title: What is the importance of the function $e^{-x^2}$ in statistics?

text: In my calculus class , we encountered the function $e^{-x^2}$, or the "bell curve", and I was told that it has frequent applications in statistics. Out of curiosity, I want to ask: Is the function $e^{-x^2}$ truly important in statistics? If so, what is it about $e^{-x^2}$ that makes it useful, and what are some of its applications? I couldn't find much info about the function on the internet, but after doing some research, I found a link between bell curves in general, and something called **normal distribution**. A Wikipedia page links these types of functions to statistics application, with highlighting by me, that states: > "The normal distribution is considered the most prominent probability > distribution in statistics. There are several reasons for this:1 First, the > normal distribution arises from the central limit theorem, which states that > **under mild conditions the sum of a large number of random variables drawn > from the same distribution is distributed approximately normally, > irrespective of the form of the original distribution**." So, if I gather a large amount of data from some kind of survey or the like, they could be distributed equally among a function like $e^{-x^2}$? The function is symmetrical, so is its symmetry i.e its usefulness to normal distribution, what makes it so useful in statistics? I'm merely speculating. In general, what does make $e^{-x^2}$ useful in statistics? If normal distribution is the only area, then what makes $e^{-x^2}$ unique or specifically useful among other gaussian type functions in normal distribution?


title: Number of events of type B given n events of type A in a Poisson process

text: I have a simple probability calculation I was working on that I came up with an answer for, but a question a colleague asked me led me to come up with a second approach - and a different answer. I put it aside at the time (a couple of years ago), but now it's niggling at me. I think my original answer is correct because of other considerations, but I failed to identify a flaw in either approach; I'd appreciate responses that throw light on why one of them must be wrong. Imagine two (independent) Poisson processes ('type A' and 'type B' say) happening along an interval; type A happens at rate $k_1 . \lambda$ and type B at rate $k_2 .\lambda$, where $k$'s are known (this could also be regarded as a partitioned Poisson process). However, we only see the type A events (we observe $n > 0$ events of type A). The idea is to try to get the distribution of the type B events given the observed type A. Putting the B events in lower case (to emphasize that we haven't seen them):                 | b  A  b b A   A   b   AA  b bb  A b |       0                                     1      Regarding type A as 'success' and B as 'failure' - recognizing that ignoring the actual times (which we don't actually observe) - and just considering the ordering of the events, it's immediately clear that the number of b's up to the $n^{th}$ A is NegBin$(n,p)$ where $p = k_2/(k_1 + k_2)$ (this being the 'count the failures' version of the Negative Binomial, rather than the 'count the trials' version). The problem comes with how to deal with the number of events between the last A and the end of the interval. One way to look at it is to now consider looking from the right end toward the left. The number of failures to the first success from the right is NegBin$(1,p)$, making the total number of type B events conditional on $n$ events of type A in the entire interval NegBin$(n+1,p)$. The other way to look at it is to regard the whole interval as being on a circle. Now, between each consecutive pair of A's, the number of B's is NegBin$(1,p)$ and with $n$ such intervals the total number of B's is NegBin$(n,p)$. (The first argument is the one the colleague's question led me to make) By the memoryless property, it shouldn't matter where the interval starts; each time, the number of events to the next A should be NB$(1,p)$ Yet the answers cannot both be correct. Which is wrong?


title: How to know whether the change in CD4 population over different time points is significant

text: I want to check whether there is a significant difference between the CD4 cells at three different time points after virus infection. I can see the increase in CD4 population, but how do I know whether the increase is statistically significant? I have done this study in mice and at each time point I had 3-6 mice.


title: Difference in statistical powers between two nested models

text: An basic question came to my mind while I was working on sample size calculation. I have some possible explanation and I would like to compare it to yours. Let us say I would like to explain some response variable $Y$ from explanatory variables $A$ and $B$. Further, let us assume that $B$ has no effect on $Y$; the corresponding regression coefficient being $0$. Two possible models are: $E(Y) = \mu + \beta_{1} A$, and $E(Y) = \mu + \beta_{1} A + \beta_{2} B$. Which of these two should I choose in order to have more power to detect some effect of $A$. What if $B$ has an effect ($\beta_{2} \neq 0$)?


title: Principal component analysis basis vectors

text: I have a question about Principal Components Analysis. I've tried to read a lot about it but have a doubt about this application. I have high dimensional data (known to be similar) which is organized into the columns of matrix Q. I perform a princomp on this data, and project the centered columns of Q onto the calculated eigenvector basis (the first 4). My question is, can I use the linearly independent vectors of the result A to form a basis for which I can test other high dim data for similarity? e.g. Solve Ax=b for an exact solution? I think the usual way of determining similarity is by Euclidean distance, but I wonder if this is possible too.


title: Exploring properties of groups in R

text: I have a data frame with the following structure:                 age gender             region homeowner.status extrovert agreeable conscientious type     1  34 Female   Northern England        Mortgaged         7         4             5    3     2  38 Female South East England        Mortgaged         4         3             6    1     3  35 Female   Northern England        Mortgaged         3         7             7    4     4  30 Female   Northern England        Mortgaged         2         6             5    3     5  60   Male           Scotland   Owned outright         4         5             6    4     6  66   Male   Northern England   Owned outright         7         6             6    3      The `type` column has been created by assigning each row to a cluster. The clusters were determined using variables that are not present in this frame. I would like to produce a general report on how the various types differ from each other in terms of the other columns. That is, I would like to generate statistics of the form:               type 1.  5% less likely to be a homeowner (p=.003)              3% more likely to be from Scotland (p=.03)              2.1 pts more extrovert than the average (p=.xxxx)      I actually have many more columns than this, so a programmatic method of tackling each column is desirable. I am using R.


title: How to estimate confidence level for SVM or Random Forest?

text: I have two classes (say 1 and 0), and want to build a classifier. It is possible to use artificial neural networks (ANN) or any "real" classifying method such as SVM or Random Forest. In case of ANN, one can easily estimates confidence level of classification. For example, if we have binary task (with outputs as 0 or 1), and ANN results for some sample is 0.92, one can suppose that ANN "sure" in classification to 1 class. Alternatively, if ANN outputs 0.52, it is considered as unsteady classification to 1 flass. But if we use Random Forest or SVM how it is possible to confidence level of classification?


title: composite scale to measure child-parent relationship

text: For my research on Mental health problems and correlates, child-parent relationship have identified as a correlate. Child-parent relationship is planned to measure with 6 questions and each question have five point scale of answers Example- question Strongly agree Agree Neither agree/disagree disagree Strongly disagree   1 2 3 4 5 Is it justifiable if I score them 5,4,3,2,1 and take the sum for each item and add up scores of 5 items together. Can I decide on cutoff points to categorized study participnts in to three groups like good relatioship, neutral relationship and poor relationship.


title: Kruskal-Wallis Test and Mann-Whitney U Test

text: I have a three samples of data where each has a non-normal distribution: ![enter image description here](http://i.stack.imgur.com/BxUwZ.png) I want to determine whether the samples differ. Therefore I apply the KW test and it gives me a significant value, i.e. the samples come from different populations? This however, doesn't tell you anything about which are different. Therefore to compare two of the samples, the MWU test can be applied to each of the samples, i.e. three MWU tests are needed. This has given me pause for thought: 1) Is it acceptable to apply the MWU test separately to each of the populations? Could this increase the false discovery rate? (Although I appreciate I don't have a large amount of populations to compare) 2) Would there be any point in using the KW test, if I am subsequently going to apply the MWU test to each of the samples?


title: Introduction to index decomposition analysis

text: Could you recommend an introductory reference to index decomposition analysis, including   * different methods (e.g. methods linked to the Laspeyre index and methods linked to the Divisa index)   * properties of decomposition methods which can be used to compare the different methods   * implementations of methods, e.g. in R ? Any hint appreciated. (could not tag as index-decomposition due to missing reputation)


title: Confused on macro-averaging and micro-averaging?

text: I'm not sure whether to use micro or macro averaging for my data or should I just present both? The data in certain classes of my confusion matrix would be a lot bigger than others, so does that mean I should use micro-averaging?? This guy seems to say so http://rushdishams.blogspot.co.uk/2011/08/micro-and- macro-average-of-precision.html Thanks!


title: Product price prediction - include important external factors

text: I need some hint over what is the general prediction solution to modelling products prices in such a case:   * I have several models (types) of the product    * I want to predict prices for each of these models separatedly, for a few weeks long time window    * I have great historical data for each of the model from last 1-2y (which makes **time series analysis** my first obvious choice, doesn't it?)   * I know there is a number of **external factors** which may have a VERY important and immediate effect on the product price (possibly more or less different for each model);  My questions are:   1. Is time series prediction a good way to follow? Maybe some kind of regression would be more approproate in this case?   2. Which approach to choose to include these external factors effects in the best way?


title: Does the dataset size influence a machine learning algorithm?

text: So, imagine having access to sufficient data (millions of datapoints for training and testing) of sufficient quality. Please ignore concept drift for now and assume the data static and does not change over time. Does it even make sense to use all of that data in terms of the quality of the model? Brain and Webb (http://www.csse.monash.edu.au/~webb/Files/BrainWebb99.pdf) have included some results on experimenting with different dataset sizes. Their tested algorithms converge to being somewhat stable after training with 16,000 or 32,000 datapoints. However, since we're living in the big data world we have access to data sets of millions of points, so the paper is somewhat relevant but hugely outdated. Is there any know more recent research on the impact of dataset sizes on learning algorithms (Naive Bayes, Decision Trees, SVM, neural networks etc). When does a learning algorithm converge to a certain stable model for which more data does not increase the quality anymore? Can it happen after 50,000 datapoints, or maybe after 200,000 or only after 1,000,000? Is there a rule of thumb? Or maybe there is no way for an algorithm to converge to a stable model, to a certain equilibrium? Why am I asking this? Imagine a system with limited storage and a huge amount of unique models (thousands of models with their own unique dataset) and no way of increasing the storage. So limiting the size of a dataset is important. Any thoughts or research on this?


title: Specify a covariance matrix

text: I'm trying to replicate the analysis presented on the bottenada.se and described in the ADA repository. Although the demo file is quite detailed, I didn't grasp where those values for the "C0" component come from. I would like to try other m0 combinations, but I couldn't understand how can I generate C0 values to fill in the model. The help file says: "You need to define your prior for the election in two steps. First give your best estimates where you believe the election will end. Then you need to specify this as a covariance matrix. In Ada we specify our priors the following way (as of august 9)."                priors2014$m0 <- c(0.227, 0.0665, 0.0645, 0.0595, 0.353, 0.08, 0.08, 0.081)         priors2014$C0 <- diag(c(0.001024, 0.000144, 0.000144, 0.000144, 0.001024, 0.000625,          0.000625, 0.000625))     colnames($)priors2014C0 <- rownames($)priors2014C0 <- names(priors2014$m0) <- c("M",          "FP", "C", "KD", "S", "V", "MP", "SD")


title: How to compare concentration ratios

text: I did a spectrometry measurement with the evaluation of the concentration of an element. For the element of interest I've two peaks (with normal distribution). $ C_1 ± \sigma_1 $ where $ \sigma_1 = \sqrt{C_1} $ and number of counts _n_ $ C_2 ± \sigma_2 $ where $ \sigma_2 = \sqrt{C_2} $ and number of counts _m_ The concentration is evaluated by the ratio between counts $ C_i $ and mass $ m ± \sigma_m $ and time $t$ (with negligible measurement error). $ A_i = C_i/(m*t) $ and the measurement uncertainty is $\sigma_{Ai} = f ( \sigma_i ;\sigma_m) $ At the end I quantify the ratio $ R=A_1/A_2 $ with $\sigma_R = f ( \sigma_{a1} ;\sigma_{a2}) $ I measured the ratio **R** for three different samples of soil (they were sampled in different places.) soil _s1_ \--> $ R_{s1} ± \sigma_{s1} $ soil _s2_ \--> $ R_{s2} ± \sigma_{s2} $ soil _s3_ \--> $ R_{s3} ± \sigma_{s3} $ **question** I have to compare these three results to evaluate if they are similar or different. How could I make this comparison? How could I make the comparison between the soil _s1_ and the average( _s2_ ; _s3_ )? one sample _t-test_?


title: How is the coefficient of determination in ANCOVA computed?

text: Given the following ANCOVA model: $Y_{ij}=\mu+\alpha_i+\beta X_{ij}+\epsilon_{ij}$, $e_{ij}\sim N(0,\sigma^2)$ i.i.d., $\alpha_1=0$, $i=1,..., m$, $j=1,..., n$ Given I have already computed $SC_{W}=\sum_i\sum_j(y_{ij}-\bar{y}_{i.})(x_{ij}-\bar{x}_{i.})$ and $SS_W(X)=\sum_i\sum_j(x_{ij}-\bar{x}_{i.})^2$ how can the coefficient of determination $R^2$ in this model be computed and what is the formula for doing so?


title: Can logistic regression estimates suffering from subsample abuse be salvaged?

text: Suppose we have some logistic regression modelling problem; $f(X) = Y$, where $Y$ is binary and $X$ is a vector of normally distributed variables. In industry it is sometimes the case that practitioners will delete rows of data where 'nothing happens' (according to an algorithmic criteria), often to reduce the dimensions of the regression (data can get quite big) or because of the perception that this 'nothing happening' is not what we want to model - we want the model only to capture when interesting events occur. Needless to say this can bias out of sample probability estimates (i.e., given some new $X$, our estimate of $P(Y=1)$ is biased). A quick `R` demonstration will show this, where the true data generating process has $Y=1$ half the time and $Y=0$ the other half but our subsampling approach has this ratio to $\frac19$:               predictions <- rep(NA,1000)          y <- c(rep(1,5000),rep(0,45000))     x <- rnorm(50000)     #y <- c(rep(1,25000),rep(0,25000))     #x <- rnorm(50000)          fit = glm(y~x,family=binomial(logit))          for(i in 1:1000) {         newX = data.frame(x=rnorm(1))         predictions[i]=predict(fit,newX,type="response")     }              mean(predictions)     plot(predictions)      *** Question *** What I would like to ask is whether our fit, $\hat{f}$, can be salvaged in the following way. For every new $X$ that we observe, we only estimate $\hat{P}(Y-1)$ if and only if it would have passed through our filter had this new $X$ been part of our in sample dataset. Intuitively I want to say yes because in-sample we have the data generating process that we artificially created, $f_\text{intrusion}(X) = Y_\text{intrusion}$, and this is the same DGP that we are drawing from out of sample to get our probability estimates.


title: Propensity score and Cox regression

text: I have a retrospective dataset of patients treated with a certain drug (treatment, $n=46$) or with placebo (control $n=96$). The stored variables are age, sex, stage of disease. I want to assess the effect of treatment on overall survival with propensity score. Here are the steps I followed:   1. I calculated propensity score with a binary logistic regression model using treatment as dependent variable and age, sex, stage as covariates.   2. I used fuzzy matching to create a 1:1 matching with 0.05 tolerance.   3. I deleted the unmatched cases and obtained a dataset of 46*2 cases (46 treated, 46 controls).   4. I used a Cox proportional regression model using propensity score and treatment as covariates. Is my procedure correct? I'm using SPSSv19.


title: In inverse theory, how do I transform the averaging kernel matrix to a new grid?

text: Rodgers and Connor (2003) describe how measurements by remote sounders can be properly compared, taking into account differences in averaging kernels and error covariances. They make the assumption that the profiles are represented on the same vertical grid. Rodgers (2000, section 3.1) describes how to transform the state vector ad the error covariance to get them on the same grid: > If we intend to compare MAP retrievals on different grids, we must ensure > that we compare like with like. Not only must the state be properly > transformed, but also the prior covariance. (...) A diagonal covariance with > elements of say 100 K^2 on a grid of 1 km spacing is not equivalent to the > same variance on a grid of 2 km spacing. (...) ...and proceeds to formulate in details how this can be done. However, neither document describes how to properly transform the averaging kernel matrix. What is a correct way to transform the averaging kernel matrix $\mathbf{A}$? The point Rodgers makes for the covariance matrix applies equally for the averaging kernel matrix; after all, the number of degrees of freedom ($tr(\mathbf{A})$), for example, must not change for the transformed matrix to remain consistent with the old one. * * * Rodgers, Clive D. Inverse methods for atmospheric sounding: theory and practice. Vol. 2. Singapore: World scientific, 2000. Rodgers, C. D. and B. J. Connor (2003), Intercomparison of remote sounding instruments, J. Geophys. Res., 108(D3), 4116, doi:10.1029/2002JD002299.


title: time-series analysis Vs statistical signal processing

text: Is there a way to identify when to use time series analysis or signal processing.   1. Time series data analysis can be divided to signal processing and normal time series analysis.   2. In signal processing the data is analysed in frequency or time domain. In the frequency domain a spectral analysis is done by using Fourier or wavelet analysis.   3. Normal time series analysis methodologies is divided to parametric and non parametric methods. For Example: ARIMA model. Why in some cases wavelet analysis is done? Its done specially from raw data signals from mechanical processes. Can't i do a ARIMA on this raw signal data? Are there any parameters in which i can i can identify when to use a wavelet analysis or a ARIMA model given a series of data?


title: How to confirm a two-way interaction in a three-way ANOVA model using SPSS

text: I have two groups of subjects, both undergoing two experimental conditions, and data are collected at two time points: pre- and post- condition. I run a 3-way ANOVA, with one between subject factor (group) and two within subject factors (condition and time). The analysis does not find a three-way interaction, but there is a group x condition interaction. My question is: How to run a two-way ANOVA for the group x condition interaction? If the interaction is condition x time, then I can pool all the subjects into one group and run a two-way ANOVA (condition, time). But for a group x condition interaction, I don't know how to do that because time is a within group factor. (I am using SPSS)


title: Comparing coefficients of two variables: one is significant, the other is not significant

text: I'm trying to test whether the coefficient for one independent variable ($X_1$) is larger than coefficient for another variable ($X_2$) in predicting the dependent variable ($Y$). For example, my hypothesis states: The effect of $X_1$ on $Y$ is larger than the effect of $X_2$ on $Y$. In my regression analysis, it turns out that $X_1$ is significant while $X_2$ is not. In this case, can I conclude that the effect of $X_1$ on $Y$ is larger that that of $X_2$? If I use the test command in Stata, the two effects are not statistically different. Is the test still meaningful when one variable is not significant? I look forward to your input.


title: Significant difference and correlations

text: Is it true that when there's no significant difference between groups then there will be correlations between groups? * * * My situation is as follows: I have a sample that was measured using two instruments $A$ and $B$. $A$ and $B$ are both questionnaires for measuring usability.   1. I compared data from $A$ with data from $B$ using a t-test to show that there's no significant difference between the means, because $A$ and $B$ are measuring the same thing.    2. I then used Pearson's correlation to see the correlation and found that they are correlated. Can I do that? Can I say that there is no significant difference between $A$ and $B$, and interpret that to mean there is a correlation between $A$ and $B$? * * * One more question, how can I use a statistical test to justify that questionnaire $A$ is better than questionnaire $B$? The difference between them is that $A$ has constructs that weren't measured by $B$ and yet they have the same usability score for a software. Any ideas?


title: Is the Durbin-Watson test appropriate for count data

text: In determining if there is any serial correlation in a time series of count data, is the Durbin-Watson statistic or similar approaches appropriate? I ask this question because the dwtest implemented in R can take an lm object as an input, but my time series comprises count data, which can be approximated by a negative binomial distribution, and lm would not be the appropriate model choice.


title: Maximizing pH values based on mean

text: It is required to maintain the pH of the water at 7.5 plus or minus .5 in a water treatment process. Two methods of treatment are available. Method 1 gives a mean pH of 7.5 and standard deviation 1.0, whereas method 2 gives a mean pH of 7.8 and standard deviation 0.5. Assume that pH follows a normal distribution. Which method is preferable in terms of maximizing the pH values within the specification? Any help is appreciated


title: Is there a way to correlate a variable with the shape of a series?

text: Given a collection of series, _e.g._ the nine depicted below, ![enter image description here](http://i.stack.imgur.com/vSFaZ.png) where each series has a rank associated with it, _e.g._ magenta is #1, orange is #2, black is #3, and so on, **I'd like to find a relationship between the _shape_ of a series and its rank. In other words, I want to know: Are there certain _characteristics_ of a series that correlates to it having a higher rank?** Of course, as stated, the problem is underspecified, since there can be as many "characteristics" as one can imagine—for examples we might describe series as being "volatile," or having certain kinds of "inflections," or "upswings toward the end," etc. But, I'm thinking there must exist a general approach—if there's not a clear technique, then I'd be grateful for terminology that may lead me to a strategy. (Currently I don't know how to word this except as "a correlation between a variable and a series.") If I had to solve it on my own, I might explore a genetic algorithm whose cost function penalizes the complexity of a characteristic ( _e.g._ number of piecewise components, or degrees of polynomials or differentials, needed to describe the characteristic), but all this is unrefined speculation. For a related example, here's the above expressed as normalized deltas (essentially a derivative): ![enter image description here](http://i.stack.imgur.com/lSFMh.png)


title: How to create a nearly orthogonal experimental design in R?

text: Does anyone know an R package for nearly orthogonal designs? I would like to create an experimental design, using 12 runs, up to 10 factors, and with mixed levels (e.g. a combination of 2 and 3 level factors). I would like to explore some nearly-orthogonal designs. There are lots of packages for orthogonal fractional designs. For example, I have looked at the documentation for AlgDesign, planor, FrF2, support.CEs, DoE.Base. In SAS, there exist a set of macros for creating orthogonal and nearly orthogonal fractional factorial experimental designs: http://support.sas.com/techsup/technote/ts723.html Does anyone know if something similar exists in R? The CRAN task view does not mention nearly orthogonal designs. http://cran.r-project.org/web/views/ExperimentalDesign.html Many thanks Tim **Update:** The Federov algorithm implemented in AlgDesign optFederov lets you create nearly orthogonal designs for mixed factors, as shown in the documentation


title: Inference of Pearson's rho from distribution perturbation

text: I would like to infer the correlation between random variables $Q$ and $R$, however, I have access only to the distribution of $Q$ and the distribution of $P=Q+R$. We can see how Pearson's $\rho$ appears here: $\sigma^2_P=\sigma^2_Q+\sigma^2_R+2\sigma_Q \sigma_R \rho$ I'm basically sure with that information alone, it's not possible to estimate $\rho$. But I thought of a cool trick. There's a treatment of $Q$ such that for an initial distribution $Q_0$, the new distribution is scaled by a constant $k_Q$ so that $k_Q Q_0=Q_k$. We can compute $k_Q$ from the ratio of $\mathrm{E}[Q_k]/\mathrm{E}[Q_0]$. The question is whether $R$ is also affected, and of course, we can't measure $R$ directly. We _can_ measure the mean effect on $R$. $\mathrm{E}[R]=\mathrm{E}[Q+R]-\mathrm{E}[Q]$ for treated and untreated. If I make one assumption: that treatment also affects $R$ by a constant $k_R$ so that $k_R R_0=R_k$, we get: $\sigma^2_P=k_Q^2\sigma^2_Q+k_R\sigma^2_R+2k_Q k_R\sigma_Q \sigma_R \rho$ We have two equations and two sets of measurements... the distribution of $Q_0$ and $Q_0+R_0$, and then $Q_k$ and $Q_k+R_k$, from which we can compute the two variances, and $k_Q$, $k_R$ from differences in the means. So we can then solve for $\sigma_R$ and $\rho$ using the two variance equations. And I've done that. and it gives a reasonable output... The question: is there any way, given the data I have to test whether my assumption about $R$ scaling is reasonable? Again, this is based on experimental observations. I _cannot_ measure the joint distribution.


title: Difference between prediction in R and SQL

text: I'm working on a prediction model for a continuous variable (amount of medicine injected) .I use R for modeling.My project flow is to multiply the prediction of a glm (logistic regression) model that is used to predict 0/1 if a medicine was injected at all with an lm (linear regression) model that is used predict amount of medicine injected - this model works rather good In R .My problem is that when I move this model to MSSQL I get different values for the prediction (i.e. for a random row the value in the R is 400 and in SQL the value for the same row is 640.The model in SQL is made by attaching the models coefficiants from the glm model to produce the glm prediction values and then multiplying it with the lm model prediction values. I don't understand why there is a difference if I use the same coefficients? Here is the code for the lm and glm models in r:               d7_lm<-lm(Ttl_Inject~UserSource+IsNewIndividual+IsCross,data=train)     d7_glm<-glm(Is_Injected~UserSource+IsNewIndividual+IsCross,data=train)      Here is a part of the r code for the prediction:               demo$d7_lm_pred<-predict(d7_lm,newdata=demo,type='response')         demo$d7_glm_pred_response<-ifelse(predict(d7_glm,newdata=demo,type='response')>0.5,1,0)     demo$glm01_lm_response<-demo$d7_lm_pred*demo$d7_glm_pred_response # this is used for a container of the prediction model's values.      Here is a part of the SQL code :               select TOP 1000*, InjectionAmount_pred= (-2.213e -1.180e+00*(case when User='IAF' then 1 else 0 end)-1.665e+00*(case when UserSource='Viral' then 1 else 0 end)     +IsNewIndividual  *  1.167e+00+IsCross )          * IIF((1 / (1 + EXP(-(-1.346e-03+1.140e-02*(case when UserSource='IAF' then 1 else 0 end) -2.975e-03*(case when UserSource='Viral' then 1 else 0 end)     -IsNewIndividual  * 1.503e-04 +IsCross ))))>0.5,1,0)


title: Regression from variable-length input vectors

text: I have a dataset that consists of variable-length categorical input vectors from which we would like to estimate the value of a numerical output variable. Is it possible to apply regression in this setting?


title: what kind of statistic test should I use for my data?

text: I am working with biological data. The project is about comparing microbiota composition in different samples who have different type of diet. Here is the experimental design of the project:               1) factor diet -> levels: a) baseline    b) white bread(wb)    c) barley bread(bb)     2) factor responded (if have high blood glucose) -> levels: a) responders   b)non-responders       sot it would be:    baseline    wb    bb        baseline    wb    bb                    |_____________________|     |_____________________|                           Responders               non-responders      I have seen big difference between amount of one bacteria in bb-responders and bb-nonResponders. Since the data is not normally districted if I use ttest I will not get any significant value. One suggested me to use Mann Whitney U-test but I am not sure what kind of test I have to use! Could you please help me with that?


title: Supervised or unsupervised learning problem?

text: currently I'm working a pattern recognition problem. I have been using supervised learning (neural network and svm with one class classification) but I think I'm doing it in a wrong way. For simplifying, the problem I'm going to describe below is just an example. In order to determine the pattern (called pattern X), I have the following training data (4 features to determine pattern X):   * 0,1,2,3,X   * 1,0,4,7,X   * 0,0,6,5,X   * 1,1,8,9,X And this is my testing data: 0,1,3,5,X ? As you can see, the first two numbers only accept binary numbers, the third number only accepts even numbers and the fourth number only accepts odd numbers.    With neural network: I think this problem is not suitable for neural network because I only have true values. And the neural network should be trained by both true and false value. Other should it be other way around ?    With svm one classification: Currently I'm using libsvm library and got accuracy at 0%, I don't know should this be problem from training data or not... So should I change to unsuperviser learing in order to find the pattern in the given training data?    Thank you very much


title: What is the problem encountered for confidence interval for single proportion when $n$ is small and $p$ is close 0 or 1?

text: My textbook says the $100(1-\alpha)\%$ confidence interval for $p$ has the form: ![enter image description here](http://i.stack.imgur.com/ZLnDB.png) It also mentions that this only works for when $n$ is large and $p$ is not too close to 0 or 1. It tried to illustrate the problem with the following exercise: ![enter image description here](http://i.stack.imgur.com/VQjzy.png) I don't understand what the exercise is trying to show. Specifically, what does 'more difficult to obtain intervals with approximately a 95% confidence' mean? Also, in the answer, it says if $p$ is between $0.1$ and $0.9$, our interval always covers $p$. I don't see how this is true from 8.10.


title: When can one say two data groups (or more) are inseparable?

text: There are plenty of measures to compute the separability between groups: the bhattacharya distance, the Jeffries-Matusita distance, the (transformed) divergence and information measures... While those measures quantify how much the data groups are separable, is there a way to say if there are not separable?


title: Probability of collisions in queues of Poisson process

text: I have a process whereby objects of width $W$ land on a gene at rate $F$ (per second, poisson process, lets assume), and then start to move along at constant speed $V$. I'm trying to work out the rate at which collisions occur - a preceding object hasn't moved out of the way sufficiently fast that a subsequent object can't fit (given it's got finite width) - does the following logic work? The 'danger zone' is the width $W$, so the existing object will be able to prevent subsequent dockings for $\tfrac{W}{V}$ seconds. And the number of expected 'arrivals' in $t$ seconds is $F\times t$, so for each existing object we'd expect it to prevent $F\times \frac{W}{V}$ subsequent objects. So for each success, we get $k=F\times \frac{W}{V}$ 'fails', so the proportion of all objects that fail is $\frac{k}{1+k}$. And I guess I can multiply by $F$ to get the failure rate per second. Seems suspiciously simplistic, and doesn't agree with a quickly mocked-up simulation, but I'm too close to the problem to see the flaw.


title: Z scores derived from a regression equation in one group applied to other groups

text: I work in the developmental psychology arena. Often papers report using regression to derive a function of age onto a score in a typical group and then applying this function to a second atypical group, saving the residuals then transforming these to z scores in respect to the typical groups mean and standard deviation. I have derived the regression function intercept and slope, for example 5.5 + (.05*age)and applied it to all children. Then computed the residual (actual - predicted) for all children. I then applied the z score equation (z = x - mean /sd) using the mean and SD of the typical groups' raw residuals. This results in a mean of zero and sd of 1 for the typical group and mean z scores much lower (as might be expected)in the atypical group. The alternative was not to use the mean and sd of the typical group and generate the z scores from all children in the group, whereby the mean z scores of the TD group are now higher than zero and the z scores reflect the distance from the mean of only my sample. My question is whether this is what others mean when they use this to standardize the sample and is there anything incorrect in either way of calculating these scores? Thanks in advance.


title: Where can I find resources about X-Means Clustering?

text: Is there resources where I can find out more about this method, it was proposed by a client and there doesn't seem to be much on this. Most of the searches seem to only turn up information about k-means.


title: Is survival analysis suitable for this comparison?

text: I'm monitoring a website where I have two kinds of users (say, A and B). These users are able to contribute to this site in a variety of ways and, for each contribution, its time is logged. What I want is to compare if the number of days that the user is active (contributing in some way) differ between the kinds of users. I'm monitoring the website since it's beginning until four months ago. Not all users joined as soon as the site was created nor users that joined at the beginning are still contributing. Is survival analysis suitable for this comparison ?


title: Checking residuals for normality in generalised linear models

text: This paper uses generalised linear models (both binomial and negative binomial error distributions) to analyse data. But then in the statistical analysis section of the methods, there is this statement: > ...and second by modelling the presence data using Logistic Regression > Models, and the foraging time data using a Generalized Linear Model (GLM). A > negative binomial distribution with a log link function was used to model > the foraging time data (Welsh et al. 1996) and model adequacy was verified > by examination of resi- duals (McCullagh & Nelder 1989). Shapiro–Wilk or > Kolmogorov–Smirnov tests were used to test for normality depending on sample > size; data were log-transformed before analyses to adhere to normality. If they assume binomial and negative binomial error distributions, then surely they shouldn't be checking for normality of residuals?


title: Is it reasonable to make inferences to the population from non-parametric tests?

text: I just had it put to me that one can not reasonably make inferences from non- parametric statistics. This strikes me as absurd as the logical rational supporting counter-factual resampling/rank order alternatives (etc), would seem to be equivalent to the rationale of frequentist approaches. Where/what is the difference in the logic? Is it reasonable to make inferences to the population from non-paramteric tests?


title: Linear Kernel in Baysian Linear Regression

text: I came up with http://mlg.eng.cam.ac.uk/duvenaud/cookbook/index.html and it is actually very useful. At some point it says > If you use just a linear kernel in a GP, you're simply doing Bayesian linear > regression, and good news! You can do this in time $O(N)$ instead of > $O(N^3)$ Why is that?


title: How to use weights in function lm in R?

text: Could anyone offer some pointers on how to use the `weights` argument in R's `lm` function? Say, for instance you were trying to fit a model on traffic data, and you had several hundred rows, each of which represented a city (with a different population). If you wanted the model to adjust the relative influence of each observation based on population size, could you simply specify `weights=[the column containing the city's population]`? Is that the sort of vector that can go into `weights`? Or would you need to use a different R function/package/approach entirely? Curious to hear how people tackle this one - didn't see it covered in any of the linear modeling tutorials I saw out there. Thanks!


title: Zero inflated correlation

text: What is the best way to correlate zero-inflated count variables with a small sample size (n=~50 and N=99)?


title: Construct probability that measurement belongs to one of two sets

text: We are dealing with a measurement apparatus that is cursed by noise and are trying to find out if a measurement was noise or an actual measurement. Assume we have a beam of light incident on an square array of photo detectors. The "counts" measured in the individual tubes follow Poisson statistics. A typical beam distributes light across 4 photo tubes, most in the tube closest to the point of incident. Noise would be distributed uncorrelated. We can measure the count distributions for a location where we know we had an actual beam hitting, and for a location where we know no beam was present. From these measurements we can construct empirical count PDFs for measurements in the individual tubes, $p_{i,\,\rm{true}}$ and $p_{i,\,\rm{noise}}$. Since for a noise measurement the counts in the individual tubes are uncorrelated I would guess that $$ P_\mathrm{noise} = \prod_i p_{i,\,\mathrm{noise}} $$ would be a good guess for the probability that this measurement was noise. Can I use a similar expression $$ P_\mathrm{true} = \prod_i p_{i,\,\mathrm{true}} $$ even though counts would be correlated for the "true" case?


title: Binary logistic regression

text: im new to binary logistic regression. I don't know whether the test is giving me real results based on my data set. My dependent variable will be the presence of STEC in fecal samples (Presence= Yes, Absence = No) . I have 9 independent variables: Season: Winter = 0, summer = 1 Geographic location: Urban = 0, rural = 1 Production systems: Backyard/extensive = 0, Intensive/semi-intensive = 1 Age: <12 months = o, >12 months = 1 lactation: dry cow = 0, lactating cow = 1 Antibiotics treatment: No= 0, Yes = 1 Parity number: no parity= 0, parity number >1 = 1 Feed: Forages = 0, others = 1 Health status: Disease = 0, healthy = 1 Is it possible to include all these predictors? should I use blocks? im a bit confused Can you help me please?


title: Compute moment and quantiles of a stream of data

text: I'd like to compute the moments and quantiles of a random variable which is the output of a sensor. I don't intend to store all the values this sensor outputs (let's say it outputs one value each 15 minutes), but rather keep batches of these values (for example a two weeks' batch) that I'll discard from time to time. I was considering estimating the probability density function of a batch of outputs with scikit's KernelDensity. I then discard the batch and when I get hold of a new batch of values I update the estimate of the probability density with these new values. I know that the probability density function that I estimate for each batch separately converges almost surely to the real density but I don't know if this result still holds if I compute the density incrementally as defined in the above paragraph. Is it the way I should do ?


title: Linear regression on grouped data

text: I have a sample composed by 200.000 projects. Each project is defined according to its size ($S$) and the presence of active users ($U$). The values for $S$ are greater than zero; on the other hand, $U$ is a binary variable (1 if the project has active users, 0 otherwise). I would like to check if $S$ and $U$ are related each other. Since I was not able to find a direct relation between $S$ and $U$, I've ordered the projects according to their sizes (ascending order). Then I've grouped the projects in 10 groups (20.000 projects per group), and for each group, I've counted the number of projects having active users. Since it seems that the number of projects having active users increases from one group to the next one, I would like to know how to proceed. It makes sense to run a linear regression analysis between the sum of project sizes per group and the number of active users per group? I should use a correlation test? ![enter image description here](http://i.stack.imgur.com/48Kxa.png)


title: Chi goodness and Fishers: some theoretical issues post-experiment

text: I have run a simple experiment under four conditions: the Control, Technique A, Technique B, and Technique C (which is a+b) The dependent variable is compliance (e.g. Yes/No) These are my results (25 participants per group): Control-- YES: 9 NO: 16 Tech A-- YES: 17 NO: 8 Tech B-- YES: 18 NO: 7 Tech C (a+b)-- YES: 24 NO: 1 I have several **hypotheses**. _First_ compares Technique A to the control, using a Chi goodness of fit, and shows significance. _Second_ compares Technique B to the control, using a Chi goodness of fit, and shows significance. _Third_ is attempting to prove that Technique C is more effective than either Technique A or Technique B in isolation; to do this I am using a Fisher's Exact test to compare A to C, and then B to C in two 2x2 contingency tables (I am using Fisher's due to the small value in Technique C). These have also shown significance. **So that is the brief!** My main (largely theoretical) questions are: 1) For a **Chi goodness of fit test, which effect size do you use**? I am being told Cramer's V, Phi, Omega and W! (There seem to be a lot more answers for a Chi association test, but not for a Chi goodness of fit test) 2) Can you use an **Odds Ratio to compliment a Chi goodness of fit test**? If yes, what would this tell us? Perhaps the odds of "Yes" happening under one of the techniques compared to the control? 3) For the **Fisher's Exact Tests** for two 2x2 contingency tables: I would love to use some **post-hoc analysis** or even a measure of effect size/odds to make my results more interesting than a simple Fisher's significance test-- but I have no idea what I can use! 4) I am worried about **correcting for multiple testing in the Fisher's test.** Do I need to do this for hypothesis 3? Maybe divide the significance level by 2 as this is the amount of tables? Or use some sort of Bonferroni test? Many many thanks statisticians!


title: Showing an estimator is consistent

text: Show that $Y_1$, the first order statistic is a consistent estimator of the parameter $\alpha $ of a uniform distribution with $\beta = \alpha+1$   Here $f_{Y(1)}(y_1)= \begin{cases} n*[1-(y_{(1)}-\alpha)]^{n-1}, & {\alpha <Y_{(1)}< \beta} \\\ 0, & \text{otherwise} \\\ \end{cases}$   By definition If $P[|Y_{(1)}-\alpha|<\epsilon]$ approaches 1 as n goes to infinity for all $\epsilon >0$ then $Y_1 $is a consistent estimator for $\alpha. $ Thus,   $P[|Y_{(1)}-\alpha|<\epsilon]$=$P[\alpha - \epsilon< Y_{(1)}<\alpha +\epsilon]$.   My question is with the limits of integration.   What is the lower and upper limits of $\int n*[1-(y_{(1)}-\alpha)]^{n-1} dy_{(1)}$ be? should it be from $\alpha$ to $\alpha + \epsilon$ Or from $\alpha $ to $\beta$.   I think it should be from because this should work for any value of $\epsilon $ and if I put limit as $\alpha$ to $\alpha + \epsilon$ then this would violate $\beta = \alpha+1$ for a particular $\epsilon$.   Please tell me what the limits should be to show that this is a consistent estimator


title: Why are Jeffreys priors considered noninformative?

text: Consider a Jeffreys prior where $p(\theta) \propto \sqrt{|i(\theta)|}$, where $i$ is the Fisher information. I keep seeing this prior being mentioned as a uninformative prior, but I never saw an argument why it is uninformative. After all, it is not a constant prior, so there has to be some other argument. I understand that it does not depends on reparametrization, which brings me to the next question. Is it that the determinant of the Fisher information does not depend on reparametrization? Because Fisher information definitely depends on the parametrization of the problem. Thanks.


title: How to pass character strings to R from IML Studio

text: If I define a function in IML:               start func(a);       submit a / R;        print(&a);       endsubmit;     finish;      and run it:               run func("character string");      I get the error message: object 'character string' does not exist. So R or IML is evaluating the character string into an object. I want R to output "character string." How do I do this?


title: Detecting a peak in an averaged profile

text: I have a very long time series (about a gigabyte in ascii format) that looks like this: 1, 0.5 2, 0.52 3, 0.3 . . . The points occur at integer time points and are predominantly 1 second apart. A small proportion are missing. The series is known to peak when a certain phenomena occurs. However, there is some underlying noise which makes it difficult to distinguish real peaks from random noise. I have an algorithm which has produced about 500 points where peaks may occur. I extracted a profile of 101 points, 50 before the position of the peak position and 50 after, to see what was happening around each predicted peak position. Inspecting by eye, a lot of the profiles didn't look very much like peaks. (I think this is because there is a lot of noise in the series.) I decided to average all the profiles together. The result is a new series where the first point is the average of all points -50 time steps away from my center point and the last point is the average of all points +50 time steps away from my central point. My hope was to see a peak close to the center point of the average profile which I did, approximately +2 time steps from the center point. I take this +2 number as a measure of accuracy of the peak locating algorithm. (If it turns out some of the points in a profile are missing, which happens rarely, I just add the ones that are and ignore the missing values, so some positions in the average profile are derived from slightly more points than others.) The average profile has a classic peak shape with the values slowly rising to a peak, as one moves from left to right, about 2 positions after the center point and declining again. If I pick 500 positions randomly, I get a relatively flat line which I interpret as almost constant. So, I think the peak locating algorithm actually does have some ability to recognizing when the series is peaking. I want a statistical test to tell me when I should judge the average profile as flat and when I should just it as a peak. Can someone please help?


title: Do I use non-normalized or normalized fft output from R to compute phase angle?

text: Is R's `fft()` output normalized or not? In other words, do I have to normalize the output first before computing the phase angle using `atan2()`? On a related point, does does anyone know what phase shift is used in R's `fft()` function?


title: Variable dimensionality samples resulting from variable length processes with fixed interval measurement

text: I have a process that takes a variable length of time from trial to trial. During a single trial, I capture snapshots of the state at fixed-time intervals. I want to model the resultant vector of snapshots (both the values and the dimensionality) for a particular trial as a sample from a multivariate distribution. Is there an established way to do this? As a simple example, consider the function $f(x) = a\cos(b x)\\!+\\!c\;\;$ where $a$,$b$, and $c$ are all drawn from their respective univariate distributions for a particular trial and remain constant during a given trial. If I find the value of $f(x)$ at discrete intervals over $x$ until I have observed a complete period and treat that vector of values as a single sample from a distribution, how might I describe that multivariate distribution? * * * It's sort of related to the idea of a Gaussian process. I like the idea, but the variable length problem throws a wrench in that particular approach for modeling my data.


title: What does "arrived at linear multiple regression models" mean?

text: I'm reading a paper on a study where a number of respondents received questions which they graded using a scale of 1-5. In the conclusion, the authors wrote that they "arrived at linear multiple regression models for the consolidated dependent variable N". What does that mean? (Disclosure: I am not a student of mathematical stats, so please use layman's terms)


title: How do you describe the influence of a set of risk factors on a response?

text: I am a clinical researcher and I have conducted a study to find factors that influence the image quality of an eye machine. The image quality has been classified as acceptable or non acceptable (2 groups) and the factors that determine it are media haze (it can be grade 1, 2, or 3), cataract (present / absent), size of eye aperture (1- 10 mm), age (20- 60 years), and gender (male / female). How should I go about describing the results and describing the influence of the these risk factors?


title: Optimizing Dynamic Regression in R

text: I am running a dynamic regression model in `R`. How can I reach at the optimal orders $p$,$q$,$r$? I tried from a few values varying from $0$ to $3$ for each of $p$ and $r$ and $0$ or $1$ for $q$. But untill what point should I check? Also if AIC is lesser for say $(3,1,3)$ as compared to $(1,1,0)$ but in $(3,1,3)$, a coefficient of the AR part is not significant. Then should I leave $(3,1,3)$ or still take that one? Basically the question is what takes precedence between significance of coefficients and AIC?


title: How to assess skewness from a boxplot?

text: How to decide skewness by looking at a boxplot built from this data: 340, 300, 520, 340, 320, 290, 260, 330 One book says, "If the lower quartile is farther from the median than the upper quartile, then the distribution is negatively skewed." Several other sources said more or less the same. I built a boxplot using R. It's like the following: ![box-plot](http://i.stack.imgur.com/1DPzv.jpg) I take that it's **negatively skewed** , because the lower quartile is farther from the median than the upper quartile. But the problem is when I use another method to determine skewness: mean (337.5) > median (325) This indicates the data is **positively skewed**. Did I miss something?


title: Question on late entry in Survial Analysis

text: In the example mentioned in the this doc for Survival Analysis on how to define time intervals, it says that the logic for creating the time variables is that, if a subject has no events, there is a single interval from 0 to last follow-up, with a status of 0. If there are events, then there is an interval for each event (0, event1], (event1, event2], etc. Each with status =1. But what if the subject had different events in same time period? Say, Part 1 and part2 both failed on an equipment id 1 during the interval 219-373? How would one represent that case? Also, if subject ID 1 was observed from time 0-414, but subject ID 2 was observed from 8-241, should I worry about the missing time interval 0-8 for subject 2?               id       tstart   tstop   status enum  treat           1         0      219      0       1  rIFN-g      1        219     373      1       2  rIFN-g      1        373     414      0       3  rIFN-g      2        0       8        1       1  placebo      2        8       26       1       2  placebo      2        26      152      1       3  placebo      2       152      241      1       4  placebo      cfit <\- coxph(Surv(tstart, tstop, status) ~ treat + sex + age + inherit + cluster(id), data=cgd)


title: Relation between stationarity, MA and AR processes?

text: So I know a time-series can be either stationary or non-stationary. I also know that non-stationary prevents us from using many of our econometric tools since our variances would be biased. Therefore, it's an unwanted characteristic. BUT, what's the relation between stationary/non-stationary processes and say, an AR process? Are all AR processes non-stationary by nature?


title: One-way Anova Meta-analysis

text: I have a series of medical studies that I would like to summarize. Each paper measures a component in the blood (call it substance X) in patients with the same disease. Furthermore, each paper has categorized patient's as having mild, moderate or severe disease (though some only categorize as moderate/severe). They do not provide measurements for each patient and only report X as a mean +/- S.D. My hypothesis is that levels of X will be increased in patients with more severe disease. What I would like to do is produce a p-value representing this. I could do a simple one-way ANOVA, but my null hypothesis would be "there is no relationship between X and disease severity", but if X was greatest in mild disease and smallest in severe disease this would still register as significant. How, do I go about taking into account the ordinal characteristic of mild/moderate/severe and only show significance if mild < moderate < severe?


title: How should I split this data when testing for heteroskedasticity

text: I have a set of time series data and am looking to split into different time periods to test for heteroskedascity or not over different time frames. Intially, I planned to do it this way: Take the daily change over the last month, last year, last 2 years etc. and test against eachother. Or would the following way be better statistically? Still using daily change, split into months and test daily change over months through the last 10 years. Then repeat with years, 2 years etc. Hope this makes sense.


title: Interpreting covariates in 2 by 2 MANCOVA

text: I am doing 2 by 2 MANCOVA with experimental data set. Followings are variables I used. DV1, DV2, IV1, IV2, and C1 where DV=dependent variable, IV=independent variable, and C=covariate. Consistent with my hypotheses, the interaction of IV1 and IV2 significantly influences both DV1 and DV2. However, here is the question that I am having a hard time to answer. MANCOVA results indicated that C1 also significantly predicts DV1 and DV2. In this case, how can I interpret C1? Several possible explanations I came up with are as follows. -The interaction of IV1 and IV2 is significant even after controlling for C1. -C1 is significant, which means the interaction can not accurately explain DVs. Which one is correct? or are both wrong? Thank you very much for your help!


title: Distribution of the ratio of dependent chi-square random variables

text: Assume that $ X = X_1 + X_2,...+ X_n $, where $\; X_i \sim N(0,\sigma^2)$ and independent. My question is, what distribution $$ Z = \frac{X^2}{X_1^2 + X_2^2 + ... + X_n^2} \;\;\; (1)$$ follows. I know from here that the ratio of two chi-squared random variables expressed as $\frac{W}{W + Y}$ follows a $beta$ distribution. I think that this assumes independence between $W$ and $Y$. In my case (1) though, the denominator of $Z$ are the components of $X$ squared. I think $Z$ must also follow a variation of the $beta$ distribution but I am not sure. And if this assumption is correct, I don't know how to prove it.


title: Problem with ets from R forecast package

text: I'm using the `ets` forecast function in R. When I fit a model to some timeseries t1:                  model<-ets(t1) [36 periods]      and the calculate forecasts from that model:                   f1 <- forecast(model,10)      so i get 10 forecasts for periods 37-48 so my question is, are these 10 point-forecast one-step-ahead forecasts wich have their seeds in $t,t+1,t+2$ with $t=37$ or are these forecasts with their seed only in $t=37$ with forecast horizon $h=1,2,3,4,...$


title: In convergence in probability or a.s. convergence w.r.t which measure is the probability?

text: I was presenting proofs of WLLN and a version of SLLN (assuming bounded 4th central moment) when somebody asked which measure is the probability with respect too and I realised that, on reflection, I wasn't quite sure. It seems that it is straightforward, since in both laws we have a sequence of $X_{i}$'s, independent RVs with identical mean and finite variance. There is only one random variable in sight, namely the $X_{i}$, so the probability must be w.r.t the distribution of the $X_{i}$, right? But then that doesn't seem quite right for the strong law since the typical proof technique is then to define a new RV $S_{n} := \sum_{i=1}^{n} X_{i}$ and work with that, and the limit is _inside_ the probability: $ Pr \left[\lim_{n \rightarrow \infty}\frac{1}{n}\sum_{i=1}^{n} X_{i} = E[X_{i}]\right]=1 $ So now it looks as if the RV is the sums over $n$ terms, so the probability is over the distribution of the sums $S_{n}$, where $n$ is no longer fixed. Is that correct? If it is, how would we go about constructing a suitable probability measure on the sequences of partial sums? Happy to receive intuitive responses as to what is going on as well as formal ones using e.g. real or complex analysis, undergrad probability/statistics, basic measure theory. I've read Convergence in probability vs. almost sure convergence and associated links, but find no help there.


title: Algorithm to calculate difference in users' tastes

text: I have data like Person $A$ like movies `['X','Y', 'Z']` and he dislikes `['V']`. Person $B$ like movies `['X','L','V']` and dislikes `['Y']`. like wise so many users. What could be a good algorithm to find mean difference of users' tastes?


title: Autoregressive model and shock events

text: I am trying to model `steel` prices using `brent` prices with following model: $steel_t=\alpha + \beta steel_{t-1}+\gamma brent_t + \epsilon_t$ I have monthly data. I fitted the parameter values with `lm` (R, is that reasonable?). Now I want to see the effect of a "shock event" in one year, e.g. what happens to steel prices if brent prices go to USD 50 in one year (it is about USD 100 nowadays). How can I do it? Does it make sense to use an autoregressive model for such analysis?


title: Decompose (deconvolve) a 2-peaked pdf into 2 elementary pdfs

text: How could I decompose a two-peaked (empirical) pdf into 2 say lognormals or other appropriate pdf in a straightforward way? I'd prefer in Matlab. ![enter image description here](http://i.stack.imgur.com/LHZ0J.jpg) to something like this: ![enter image description here](http://i.stack.imgur.com/TfPER.png) Thanks!


title: Significant differences among fit lines - ANCOVA not enough?

text: I often am in the situation of having data sets consisting of an independent variable, a dependent variable, and a factor with multiple levels - for instance, calibration curves for an instrument measured on different days. I would like to know whether the data are best described by a single fit line or by a different fit line for each factor level; i.e., whether the data are best described by a single calibration curve based on all of the data, or whether there are significant differences in the individual curves for each day. From what I understand, ANCOVA will tell me _separately_ whether the factor interacts with the slope and then whether it interacts with the intercept. What I want to know is whether the factor has a significant effect on the slope and the intercept of the line. An example in R:               require(reshape)  # for melt()     #Set up some data     set.seed(0)     x <- seq(from=0, to=2, length.out=50)     y1 <- x + rnorm(length(x)) + 0.3     y2 <- 1.2*x + rnorm(length(x))           d <- data.frame(x=x, day1=y1, day2=y2)#, day3=y3, day4=y4)          dm <- melt(d, id.vars="x")          ggplot(dm, aes(x=x, y=value, colour=variable)) + geom_point() + geom_smooth(method="lm", se=T)      ![enter image description here](http://i.stack.imgur.com/AzFEh.png)               m <- lm(value ~ x*variable, data=dm)     summary(m)      ![enter image description here](http://i.stack.imgur.com/Gl2q2.png) Here, the effect of the factor and the interaction of the factor with the independent variable are each marginally significant - at alpha = 0.05 we would fail to reject the hypothesis that the factor has an effect on slope or intercept. However, taken together, perhaps they matter. Is there a good way to assess this?


title: Unstandardized beta with an E in it

text: I have an unstandardized beta of -2.615-E5. What does the E mean? I have never seen this. Thanks!


title: Using logistic regression for a continuous dependent variable

text: I got a revision for my research paper recently and the following is the reviewer's comment on my paper: > results obtained from one model is not quite convincing especially linear > regression usually has deficiencies in dealing with outliers. I suggest the > authors also try logistic regression and compare the corresponding results > with current results. If the similar observations are obtained, the results > would be more solid. Is the reviewer's comment right? Is logistic regression better than multiple linear regression? The problem is that my dependent variable is not categorical, it's a scale variable. What can I do now? What other regression method do you recommend to evaluate my model? Score is dependent variable in the following table. Recency, frequency, tenure and last score are independent variables. ![enter image description here](http://i.stack.imgur.com/uvq3z.jpg) I've extracted these variables from a site and I hypothesize that these **independent variables** have **significant effect** on the **score**. Therefore, I represent the following models: ![enter image description here](http://i.stack.imgur.com/3LE8f.jpg) * * * By the way, the value of R squared for this linear model is 0.316! The reviewer also commented on this value too: > then the results are not convincing as there is no indicator on the quality > of learned coefficients. A small R^2 cannot indicate good performance as the > model may be over-fitted. Is 0.316 very low for R squared? In previous papers I saw the similar values a lot. ![enter image description here](http://i.stack.imgur.com/UUpAP.png)


title: Statistical tests for comparing a skewed clinical sample

text: I recently surveyed 350 low-income families -- they were randomly split into two groups: control and treatment. One of the variables I am very interested in is the amount of savings of each family. The sample is highly skewed (see below), as the vast majority of the sample has no savings ($0), and then there are few with modest savings. I'd like some input on what kind of stat test I can use to compare the two groups between each other pre-treatment to show that the savings rate and amounts are the same, and post-treatment to compare the change in savings between the groups after treatment has been applied. ![control-treatment-savings-comparison](http://i.stack.imgur.com/ifPE1.png) Thanks in advance for any input you guys may offer.


title: Why computing the variance of the extracted random effect (using ranef) is not the same as the output from lme?

text: I am applying a mixed model to predict tumor progression (y) using tumor volume (x) as the fixed effect and center ($i=1,...10$) as random intercept. The model can be written as: $y_{ij}=\alpha+\beta x_{ij}+b_{i}+\epsilon_{ij}$ I used the lme() function in R: `fit1 <- lme(PD ~ log(Volume), random = ~1 | CenterID, data=Data)` The result shows that the standard deviation of the random intercept is 0.079. Thus $b_{i}$ follows a normal distribution $N(0,0.079)$. In the meantime, I can extract the random intercept by applying ranef(fit1). This gives a list of $b_{i}$ corresponding to each center. Then I compute the standard of this vector. `sd(ranef(fit1)[[1]])` I would expect that it gives similar result as 0.079. However, it is far different. Can someone tell me why `sd(ranef(fit1)[[1]])` gives different result than the model output VarCorr(fit1)? What is exactly the relation between ranef(fit1) and VarCorr(fit1)? thanks


title: Standard error of stationary point of a 2nd order polynomial whose coefficient std errors are known

text: I have a 3-variable full quadratic polynomial $$ f(x,y,z) = c_0 + c_1 x + c_2 y + c_3 z + c_4 x y + c_5 x z + c_6 y z + c_7 x^2 + c_8 y^2 + c_9 z^2 $$ Each of the coefficients $c_i$ are known with a standard error $s_i$. Ignoring the standard errors $s_i$, I have one single stationary point, with values $(x_0, y_0, z_0)$ (thus satisfying $f'(x_0,y_0,z_0) = 0$), but I wonder: **how to deduce, from the standard errors of the coefficients, the standard errors for $x_0$, $y_0$ and $z_0$ ?** If it helps, the errors in the coefficients are (1) very small, and (2) uncorrelated. I can see that this problem can be reformulated as "transferring" the errors from the coefficients $c_i$ toward the stationary point $(x_0,y_0,z_0)$ in the system of first order equations resulting from $f'(x_0,y_0,z_0) = 0$: \begin{cases} c_1 + c_4 y_0 + c_5 z_0 + 2 c_7 x_0 = 0 \\\ c_2 + c_4 x_0 + c_6 z_0 + 2 c_8 y_0 = 0 \\\ c_3 + c_5 x_0 + c_6 y_0 + 2 c_9 z_0 = 0 \end{cases} **Edit:** I have tried to simplify Aleco's expression but started to get the jitters and then flew away after seeing the expression of $C^{-1}$: $$ \frac{1}{2 (c_9 c_4^2-c_5 c_6 c_4+c_6^2 c_7+c_8 (c_5^2-4 c_7 c_9))} %\left( \begin{array}{ccc} c_6^2-4 c_8 c_9 & 2 c_4 c_9-c_5 c_6 & 2 c_5 c_8-c_4 c_6 \\\ 2 c_4 c_9-c_5 c_6 & c_5^2-4 c_7 c_9 & 2 c_6 c_7-c_4 c_5 \\\ 2 c_5 c_8-c_4 c_6 & 2 c_6 c_7-c_4 c_5 & c_4^2-4 c_7 c_8 \end{array} %\right) $$ What I ended up doing is hypothesizing further that the coefficients were normally distributed. Then I drew a sample of 1000 coefficients for the $c_i$ from $N(c_i,s_i)$, and computed the resulting stationary point with each of the 1000 set of coefficients. These stationary points were also normally distributed, and I was finally able to deduce from their distribution their standard error.


title: Confidence interval for absolute changes

text: I have used the STDRATE procedure to get the estimates and standard errors for 5 different timeperiods between 1980-2000 in 5 years interval (1980, 1985, 1990, 1995 and 2000). This means that I have point estimate and 95% CI for $X_1, X_2, X_3, X_4$ and $X_5$ which spans between the years 1980-2000. For example i have the estimates with 95%Cl for X1 = 22 (18 to 25), X2= 30 (24 to 33) X3= 14 (12 to 16) X4= 10 (8 to 11) X5= 6 (2-7) Now if I wanted to calculate the absolute change with point estimates and 95% CI between the start of the study (i.e for X1) and the end (i.e X5), how do I do this? **A similar study have done this and this is how it looks** , their variables are a1, a2, a3, a4 and a5. This is their data. a1= 4.2 (3.8 to 4.5), a2= 3.7(3.3 to 4.0), a3= 2.4(2.3 to 2.5) a4= 1.9(1.8 to 2.0) a5= 1.5(1.4 to 1.6). Their absolute change was -2.7 (-3.0 to -2.4) I would preciate any help that I can get!


title: About kernel based estimates

text: **Kernel based operations** are common in a variety of applications, such as image processing (e.g., blurring), generating smoothed estimation maps, and so on. A common approach is to select four surrounding cells (see the following figure: **A** the green colored cells) then calculate the average value of them including the value of the central cell to be assigned as the estimate. Some questions appear to me to have a discussion: **1-** I propose two other cell selections i.e., B and C. The question is what kind of statistical justification could help to recommend one of them? **2-** Which part of statistical analysis would fit to discuss such conceptual/practical questions? ![enter image description here](http://i.stack.imgur.com/G5Vhv.png) **my understanding is:**   Because those cells (green ones in **A** ) are the closest to the central cell (the cell of interest for estimation etc). For case **B** compared to **A** it is obvious so **B** is not a better choice. In comparison between **A** and **C** however the case **A** has a disadvantage, lacking diagonal supports while for case **C** it covers full orientations. I think the case **A** is good because of less computation required however with the expense of biases diagonally.   Here is an `example` implementation of those ideas:   ![enter image description here](http://i.stack.imgur.com/WH86S.png)   Hope this helps a bit more in understanding the difference brought by applying different kernel averaging configurations. **Applications:** A sample application can be found in "After Kernel Averaging" in which type **C** has been applied.


title: What is the best method of performing a chi-squared test with only one set of data?

text: I am trying to perform a chi-squared test on the following data: Achieved: 31 Expected: 19.87179 From a total of: 182 Is it possible to perform such a test on just one item of data like this? All the examples I have encountered use multiple sets of data. Ultimately I would like to be able to quantify the % chance that the above result occurred at random.


title: Difference between confidence intervals and prediction intervals

text: For a prediction interval in linear regression you still use $\hat{E}[Y|x] = \hat{\beta_0}+\hat{\beta}_{1}x$ to generate the interval. You also use this to generate a confidence interval of $E[Y|x_0]$. What's the difference between the two?


title: How to determine the number of nearby samples for spatial estimation?

text: In many applications e.g, in mining engineering when we need to generate a map of dispersion of an element (e.g., copper) over the field of study, to depict depletion and concentration regions we have to apply estimation methods on the existing samples. Samples are dispersed everywhere: some parts appear to be more random and some other parts more regular. In estimation procedures such as IDW (Inverse Distance Weighting) and Kriging and so on we prefer to incorporate a limited number of nearby samples in the estimation of each centered point. If for the discussion we disregard the computing optimizations, **what can be the criteria to choose a right number of samples for estimation?** **Notes:**   * In mining engineering and geosciences it is a common practice to choose a number of between 10 and 15 samples for the averaging.   * Incorporating more samples generates smoother results which are in some cases the desired outputs but not for all the cases.   * The question seeks a statistical criteria instead of visualization impression which is also commonly referenced as a criteria!  **important note:**   The question appears general. Kriging or IDW are only examples. The criteria sought should be able to be generalized for any estimation method.   _Kriging:_ this method due to its basics (matrix solution) requires a small number of samples to be solvable.


title: McNemar-Bowker test for comparison of two measures

text: I am comparing two versions of a survey (an original, and a modified) and was wondering if the McNemar-Bowker test is appropriate. The survey(s) ask respondents to indicate whether they've had an experience (in this case, unwanted sexual contact) since the age of 14. Respondents have four answer choices to indicate the frequency which they've had a certain experience, coded the following way:                 "never"       "once"       "twice"       "three or more times"      My hypothesis is that participants will indicate an increased number of experiences when given the modified version, as compared to the original. In this instance I am looking at the rates which men admit to having perpetrated acts of unwanted sexual contact; with the hypothesis that the modified version will ellicit increased rates of admitted perpetration compared to the original version. My data do in fact show an increase in male admittal-rates on the modified version.                MALE ORIGINAL     (n)                      MALE MODIFIED     (n)      "Never" =         281                      "Never" =         270      "Once" =           18                      "Once" =           18      "Twice" =           7                      "Twice" =           8      "Three +" =         7                      "Three +" =        17       Total admittal =  32                       Total admittal =  43       Total n =        313                       Total n =        313      So, now i want to test for significant difference between response rates on these two surveys to see if the increase of admittal on the modified version is statistically significantly higher. **_This is a repeated measures design, so each participant took both survey versions._** I am wondering if the McNemar-Bowker test is appropriate for my data (above)? My plan is to arrange the table as such: > >                                      ORIGINAL SURVEY >   >   >                           |"Never" | "Once" | "Twice" | "Three +" | TOTAL | >                           |________|________|_________|___________|_______| >                    "Never"|  261   |   7    |    2    |     0     |  270  | >                           |________|________|_________|___________|_______| >                     "Once"|   12   |   5    |    1    |     0     |   18  | >                           |________|________|_________|___________|_______| >                    "Twice"|    3   |   4    |    1    |     0     |    8  | >                           |________|________|_________|___________|_______| >                  "Three +"|    5   |   2    |    3    |     7     |   17  | >                           |________|________|_________|___________|_______| >                     TOTAL |  281   |   18   |    7    |     7     |  313  | >                           |________|________|_________|___________|_______| >   > > MODIFIED SURVEY When I run the above analysis in SPSS crosstabs, I get a McNemar-Bowker chi- square value of 13.316, df = 6, p = 0.038. So my **questions** are:   1. Is the McNemar-Bowker appropriate for my data and hypothesis?   2. With the resulting p value of 0.038 can I now say that the results from the two surveys are significantly different -- with the modified version showing an increase in male admittal?


title: Bayesian inference of a clinical trial for clinicians

text: I am a clinician who is more adept than average at interpreting clinical trials in a frequentist manner. At this point, interpreting a trial as a frequentist has kind of become a procedure: check internal validity, check null and alternative hypotheses, check power assumptions, look at effect size and confidence intervals, look at p value, etc. The Bayesian philosophy appeals more to me intuitively, though. I understand what the philosophy of Bayesian inference is because I've taken clinical epidemiology (pre-test probability of disease is updated with test results and likelihood ratio to produce a post-test probability). What I don't know yet is whether there's a similar "procedure" for interpreting a clinical trial the way there is for a frequentist. What numbers/assumptions/figures do I need from the authors to interpret a study as a Bayesian (e.g. for a frequentist that would be p-value, confidence intervals, etc)? Is there a good "procedure" for interpreting a study as a Bayesian, much like there is for a frequentist interpretation? Are there good publications you know of that explain the process I'm asking for, with clinicians as the intended audience? I appreciate your help!


title: Correlations within Factor in a psychometric measurement instrument

text: I am working with results of a psychometric test. There are twelve factors and each have three subfactors. My problem is that one of the subfactors correlates very highly 0.3 to 0.7 with subfactors of other factors in fact with all but 10 of 36. And this is a consistent observation across three healthy samples, n1=427, n2= 353 and n3=945. Though this can be explained by looking at the meaning of these traits. But, should it question the robustness of the factor structure or the instrument? Additionally intra factor correlations are also low and insignificant at many instances. Please note that this is a very well researched and well known psychometric instrument. Need help with interpreting this information. Any ideas, what I should make of this observation? Added this for additional information: Not really, measurement model does shown some strong correlations for the subfactor outside its own factor (3 such correlations are strong), none of which are more than o.5. In my samples I consistently see two correlations at 0.7 and two others at 0.6 (I see 23 of 36 correlations above 0.3). I wish I could share the correlations here, but the information is proprietary. And the subfactor has very low correlation with another subfactor within its own factor. These others are very high as compared with the correlations given in the manual. Could it be a sampling error? What inferences can I draw for the samples if these correlations are exaggerated (more than what was seen in the norm group)?


title: How to determine which similarity measure to use when computing the robust gamma rank correlation coefﬁcient?

text: I am interested in calculating the correlation between sets of expected and observed data. I am trying to compare observed (n<12) and expected values that are small proportions ranging from 0 to 7.4 and that are non-normally distributed. I have tried a number of transformations, but none seem to transform my observed data enough to meet tests which require normally distributed data. I am currently using non-parametric tests from the `rococo` R package that need me to specify similarity measures based on fuzzy orderings. The PDF for this package outlining how to choose the appropriate similarity measure for the data is somewhat hard to translate. Can anyone point out some tips/resources on which measure to use, for example, linear, gauss(ian) or exponential? I have tried multiple similarity measures using the different measures, and they give quite different correlation coefficients. I would really appreciate understanding which measure is best for different situations.


title: Overall goodness-of-fit/p-value for multiple items IRT model in R (ltm)

text: The data set which I am trying to analyse is Student Test Data. I have a data of responses (either 1/correct response or 0/incorrect response) on some questions of a set of students. I have fitted a "3 parameter logistic model" for each question and then calculated the goodness of fit estimate and hence the respective p-values. Now my problem is I don't know how to aggregate these individual goodness of fit estimates to get a total goodness of fit for the whole model, Is there any measure (preferably in the programming language R) which can suggest about the whole model depending on the p-values of multiple tests. The package which I used is ltm inside R.


title: Statistics version of "What is..." columns in Notices

text: Each issue of the AMS monthly magazine _Notices_ contains a "What is..." column. This column is an expository article on a given topic, e.g. "What is...Noncommutative Topology". Do statisticians have something similar?


title: Neural Network Black Box Workarounds

text: I am dealing with a data set that includes rich textual data (e.g., blog entries, magazine articles, essays, book reviews, etc.) as well as a host of proprietary metrics, including numerous quantitative measures of consumer resonance. All of the textual data is in the same language and the data set sample size is approximately 400. There is some human pre-processing of the textual data designed to summarize roughly 10 key aspects of each textual work (these metrics do not change across the sample), and these key aspects can be either boolean, categorical, or discrete values. Additionally, I plan on utilizing a bag-of-words algorithm across the sample to supplement the human- generated analysis just described. I am considering utilizing a neural network to attempt to predict consumer resonance using the data described above, but it is important for me to understand relative importance of each factor. Additionally, I expect that of the roughly 10 textual metrics, some will play a much larger role in affecting consumer resonance than others, and the importance of these may change depending on the subject content of the particular article. For example, individuals reading an article on women's beauty or dating issues in large cities may value different things than those reading a book review in the New York Times. I have these questions:   * Is it possible to back out this relative importance using a neural network?   * Is this data set large enough to properly train a neural network?   * Would I be better off utilizing a Bayesian tree or SVM approach, or perhaps use these in conjunction with a neural network to supplement the analysis?   * I was considering a backpropagation network, but is there a potentially more valid type I should be looking at?


title: Can time series be used to predict series which has a changing trend?

text: Supposing I have a stock return series from 2000-2013. Looking at the data, it has pattern of long trending bull market: 2001-2008, and 2010-2013, while it also has great reversal in 2008 financial crisis. So if we denote the daily return series as $y_{t}$, there are two properties: (1) $y_{t}$ has a non-zero mean part in the return series, denoted as $r_{t}$, which is something like a "macro factor". (2) $r_{t}$ can be correlated with $r_{t-1}, r_{t-2},... $, during the good years, or negative correlated during bad years. Typically what people would do using ARMA model, is firstly estimate $r_{t}$ as $r_{t} = (y_{t-1} + ... + y_{t-k})/N $, and feed the series $y_t - r_t$ into ARMA identification toolbox to get the coefficients. However, it really turns out that picking $k$ can be quite critical : the $k$ value tells how much we believe the past trend, which is really hard to say. For example, at 2008 when the financial crisis happen, you have no reason to believe the market would continue as the previous good years; while at the good years, you would assume the global economy works well and you can use a long $k$ window. Or maybe it is best not to make the model depend on choosing $k$. So back to the question regarding time series modeling: What people usually do if the time series has a non-zero mean, which is quite different at different time point $t$ ?


title: Complete sufficient statistic

text: I've recently started studying statistical inference. I've been working through various problems and this one has me completely stumped. Let $X_1,\dots,X_n$ be a random sample from a discrete distribution which assigns with probability $\frac{1}{3}$ the values $\theta-1,\space\theta,\space\text{or}\space\theta+1$, where $\theta$ is an integer. Show that there does not exist a complete sufficient statistic. Any ideas?


title: Finding correlation between unequal group sizes in SPSS

text: I am working on correlating teacher factors with student achievement. How should I prepare the data set in SPSS for 130 teachers and 4000 student scores? I also have the same question with regression, should I input them as is or do I need to do something on my data set. I'm new with SPSS and I would appreciate any help.


title: How to determine calibration accuracy/uncertainty of a linear regression?

text: To determine organic carbon concentrations in a water solution from UV spectroscopy I made a calibration curve with known carbon concentrations and measured the UV absorbance. I used R for the data analysis and the data are found below. My question now is, how to determine the accuracy of my calibration curve. I would like to tell people that I can measure the concentrations with an accuracy of 15% or 4 mg l-1, for example. How can I go on with this? Can I just use the $R^2$ (0.91) and say that I have an uncertainty of 9% (1-0.91)? What is the correct way to do this?               concentration <- c(27.2,32.4,16.5,11.6,11.9,9.87,46.0,73.6,75.4,73.1,59.5,49.0,                 79.0,81.6,66.7,26.7)          absorbance <- c(0.764, 0.923, 0.678, 0.373, 0.287, 0.253, 1.660, 2.331, 2.255,                  2.019, 1.130, 1.858, 2.404, 2.812, 2.362, 0.636)          plot(absorbance, concentration, xlab = "absorbance (254 nm)", ylab = "concentration (mg l-1)")          fit <- lm(concentration ~ absorbance)     abline(fit)     lm_coef <- round(coef(fit), 3) # extract coefficients     myr2 <- format(summary(fit)$adj.r.squared, digits = 2)     legend("topleft", legend = as.expression(c(bquote(paste("lin. equation: ",                                                         y == .(lm_coef[2])*x + .(lm_coef[1]))),                                             bquote(paste("", R^2 == .(myr2))))), cex = 0.6)


title: Average of ongoing durations

text: Suppose I run a community center and I want to find the average length of time that someone is a rec center member. It is straightforward to do this with people who have already cancelled their accounts, but I'm not sure how to handle the data of people who still have accounts. For example, suppose the average duration of people who have cancelled is 500 days. But when I take the average of _all_ account durations (assuming today is the "end date" for current members) that number become to 1000. The obvious problem with this is it is possible for a current member to cancel tomorrow or they may cancel in 10 years. Furthermore, someone who joined yesterday will have a duration of 1 day, which is not accurate. My hunch is that this involves predicting when current members will cancel, but that seems to add several layers of complexity to this (seemingly) simple question. I apologize in advance if this has been asked. I did try searching, but it's possible I don't know the correct terminology for this concept.


title: Non-conventional form for a full conditional

text: I have a full-conditional of the form : $$ p(X|...) \propto \exp(-(aX^2 +bX +c/X)), $$ All of the other full-conditionals of my model simplify to a simpler form and for a sake of simplicity, I would like to instantiate my model using Jags or WinBugs (which means that I need to explicit a form for $p(X|...)$ and that I cannot use explicitly a Metropolis step for $p(X|)$ (I think)). Is it possible to do this? If so, how?


title: Area Under Curve (AUC) - given peak mean and standard deviation (SD)

text: I am currently into a situation that i don't really know how to solve by myself. I need to calculate the AUC of each peak and then compare these areas in relation to each other. The problem is that the peaks are not completely separated and the only information i got is the mean and the SD of each peak. Does anyone know how to do this? Any hint or guess would already be really cool. Thanks.


title: Can a function be split into sub-function to prove it is a probability mass function? And how to find variance of such function?

text: I have a question that requires to prove if the following function whether is it a PMF with poisson random variable. The function is as follows... $f(x) = \pi \frac {\lambda_1^x}{x!} e^{-\lambda_1} + (1-\pi) \frac {\lambda_2^x}{x!}e^{-\lambda_2} $ where $x\epsilon \mathrm X = \\{0,1,...\\}, \pi \epsilon (0,1), \lambda_1, \lambda_2 > 0, \lambda_1 \neq \lambda_2$ Can I split $f(x)$ into 2 function $\sum_{x=0}^{\infty}\pi \frac {\lambda_1^x}{x!} e^{-\lambda_1}$ and $\sum_{x=0}^{\infty} (1-\pi) \frac {\lambda_2^x}{x!}e^{-\lambda_2}$ ~~to prove that it is a PMF and hence continue with calculating it's expectation~~ and variance using this 2 sub- function? I've found out the expectation of $f(x)$, which is $\mathbb E[X] = \mathbb E[X_1] + \mathbb E[X_2];$ where, $\mathbb E[X_1] = \sum_{x=0}^{\infty}x\pi \frac{\lambda_1^x}{x!}e^{-\lambda_1} = \pi\lambda_1$   $\mathbb E[X_2]=\sum_{x=0}^{\infty} x(1-\pi)\frac{\lambda_2^x}{x!}e^{-\lambda_2} = (1-\pi)\lambda_2$ Now my question is how am I suppose to carry to find out what is the variance as the variance i've got is different from the expectation I got above. For a Poisson random variable, the expectation should be the same as variance. Am I right? I have also tried using moment generating function. For the 1st differential, I got the same result as my expectation. But I couldn't further differentiate the 2nd time to get my variance as all the remaining terms are constant terms. Could I have some suggestion of how should I carry on from where?


title: Minor confusion on using PC features for classification

text: I have what I believe to be a simple question. I have also browsed some of the PCA related materials here. I have an unlabeled $M$ by $N$ data matrix, consisting of $M=1000$ points, and $N=500$ features. I standardize my matrix, perform PCA, and decide to use the top two PC's. Let us assume that those turn out to be fantastic features to use, and I decide to run a (k=3)-means, get my nice decision space, etc, all is good in the world. **The question:** When I have a new data point that I want to classify as belonging to one of those three clusters:   * It seems to be that I have to always retain the first two columns of my $U$ matrix that I used in the PCA, forever, so that I can always project my new point unto them. Is this correct?    * When I did PCA, I had to standardize my data, that is, remove the mean of the features, and make their std's unity. So, isnt there something I should also do to this new data point now that I have it?  Thank you.


title: Expected value of small sample

text: I've got product ratings for a few thousand products. The number of ratings for each product varies from zero to about fifty. I want to find the expected value of product rating for each product. If there are lots of ratings for the product I'd expect the expected value to be the average of the ratings for the product, but if there are only a few I'd expect the expected value to be closer to the average of all ratings. How do I calculate the true expected value? Please be gentle: I'm no statistician or mathematician. Edit 1: Joris's answer below maintains I can't calculate expected value because by definition that means I must have the entire population. In that case please can you tell me how to calculate the quantity that is similar to expected value in spirit, does not require the entire population, and can make use of prior information. Edit 2: I would expect that if each product's ratings have low variance ratings, or if there is a very high variance between different products' ratings, then the measured ratings are more significant.


title: Permulations to test if values are different than zero?

text: I have a distance measure, which ranges between 0 and 1, between several pairs of populations. I need to figure out if each distance is statistically different from zero using permutations, any suggestions on how to achieve that? Thanks


title: Software for median polishing

text: Is there any easy-to-use software for Tukey median-polishing rows and columns with lots of missing values?


title: Where can I learn about transforming uniform, random distribution into other distribution

text: I'm caring monte-carlo simulations and I am checking my code for some faults. I have just realized (the hard way) that to generate a unit vector pointing in a random direction I cannot simply pick 3 (or any other number) random number and normalize them. I didn't understand why I can't do this apart from the claim that the generated vector doesn't have even probability to point at any direction. I can't find the math behind there and I suspect that this is mainly due to lack of proper term to search. What should I search for? Where can I find explanation? Preferably without assumption of prior knowledge.


title: How to determine if a variable is categorical?

text: I am looking at the variable vs response plot, and it seems categorical for most observations, but some are not. Here is a picture of the plot. ![enter image description here](http://i.stack.imgur.com/xGchN.png) Is there a threshold on the amount of "outliers" to determine whether a variable can be categorical? edit: In the plot, `x` is the tax rate ($\times 100$) of various regions, and `y` is the median value of homes in the region.


title: Discrete time survival analysis with varying length event windows

text: I have data that appears very well-suited to discrete-time survival analysis, in that each subject has a consecutive sequence of observations from its entry into the system until its exit, and I'm interested in determining the likelihood that a subject will leave the system in the next time period. The catch is that the time periods are not uniform in length. There are several possible ways I can imagine addressing this problem. The top two that come to mind are: 1\. increasing the granularity by further discretizing my time intervals to the minimum length 2\. leaving the individual time periods as is, but weighting each period in the analysis by its actual length I prefer the second option to the first one. My questions are: 1\. Is the second option acceptable? 2\. Which of the two options is preferable? 3\. Is there a better third option? Thank you!


title: Maximum of sample variance

text: Let $X_1, \dots, X_n$ be i.i.d from the interval $[0,1]$. I am interested in situation when the sample variance $$\frac{1}{n-1} \sum_{i=1}^n (X_i - \overline{X})^2$$ (where $\overline{X}$ is sample mean) is maximised.


title: Identity in Markov Process

text: I want to know if my reasoning here is correct, it seems simple enough but I just want clarification (I am considering the proof that if a Markov process satisfies the detailed balance condition, then it is reversible). If $X_{t}$ is a discrete state space Markov process, let $\pi_{x}^{(n)} = P(X_{n} = x)$, $\pi$ be the stationary distribution and $P(X_{n+1}=y|X_{n}=x) = P_{xy}$. Then, $P(X_{n} = x |X_{n+1}=y) = \large\frac{P(X_{n+1}=y|X_{n}=x)P(X_{n}=x)}{P(X_{n+1}=y)} = \frac{P_{xy}P(X_{n}=x)}{P(X_{n+1}=y)} = \large\frac{P_{xy}\pi_{x}}{\pi_{y}}$. It is the last equality that isn't quite so obvious to me. Can I justfiy it by saying that infact: $\large\frac{P_{xy}P(X_{n}=x)}{P(X_{n+1}=y)} = \large\frac{P_{xy}\pi_{x}^{(n)}}{\pi_{y}^{(n+1)}}$, but since $\pi^{(n)} \rightarrow \pi$ as $n\rightarrow\infty$ and since $P(X_{n} = x |X_{n+1}=y)$ is constant, I can let $n \rightarrow \infty$ in the above, so that $\large\frac{P_{xy}\pi_{x}^{(n)}}{\pi_{y}^{(n+1)}} =\large\frac{P_{xy}\pi_{x}}{\pi_{y}}$? Thanks for your insight and thoughts.


title: Calibration of probabilities in online learning (real-time learning) with Vowpal Wabbit

text: How exactly Real-Time Learning techniques deal with the problem of skewed distribution in dependent and independent variables? I am getting acquainted with Vowpal Wabbit and face this problem. My data have mixed types of variables (numeric and nominal). I use it for regression and classification. I understand that probabilities I am getting still need to be calibrated, correct? I know that imbalance in present in the domain, but as real-time learning accumulates training examples, how does that work? Imbalance (skewed distribution) in independent variable is also a problem in regression, so how is that resolved? I suppose all those sophisticated techniques are developed to handle these problems, but I need some details to begin with.


title: Variable importance randomForest negative values

text: I am asking myself if it is a good idea to remove those variables with a negative variable importance value ("%IncMSE") in a regression context. And if it gives me a better prediction? What do you think?


title: How does a GAM scale with increasing number of knots, predictors, and observations?

text: I want to fit large Generalized Additive Models, and would like to know how the estimation scales, i.e. how much longer fitting the model takes when   1. increasing the number of knots by 1,   2. when adding another smooth predictor (assuming the same support and number of knots as the existing one(s)),   3. when increasing the number of observations by 1. I suspect it does not scale linearly, but something like quadratic or exponential. Does anyone have ideas or references I could check? Thanks in advance!