title: Confidence interval and margin of error

text: Lets say I have confidence interval at 85% with 4% margin of error, and I get a value from my analysis equal to $x$. Does this mean that if I were to perform the test 100 times, then at least 85 times we would expect to see a value for $x$ ranging from $.96x$ to $1.04x$?


title: SVM: plot multi dimensional data when implementing multiclass one-vs-all or one -vs one?

text: I have split the multi class svm to several binary classifications and obtained the individual predictions. My predicted labels are saved in a matrix 'predicted_vals' with each row containing predicted binary classification result for a particular class. What I have done is using the function > mode to find out the frequently occurring value. Is it a good way? Are there any better approaches? What is the best way to handle if different classes get the same vote? Also how can I plot this decision boundary in multidimensional or 2 dimensional plane?


title: How to predict the probability of a performance rank for a test taker on a future test based on previous test scores?

text: Four students sit a series of tests, each test contains 10 questions. Students are ranked 1-4 in each test depending upon who got the most questions correct.                                    Tom    Dick    Harry   Alice        Test1 Score       9      5       6       8        Test2 Score       8      3       7       4        Test3 Score       7      6       9       8        Test4 Score       9      2       5       7      Tom was ranked 1st in 3 of the 4 tests. Test 5 is happening next week. How do I go about identifying the probability that Tom will be ranked first ? What is the probability that Alice will come first ? While this is a small sample, I am interested in the general process that could apply to millions of test scores. Thus: **How do I predict the probability of an individual achieving a given performance rank for a future test based on previous performance on a set of tests for a set of individuals?**


title: Random forest “certainty / likelihood score” - how to score records in RF mode in R?

text: My question is similar to this link Creating a "certainty score" from the votes in random forests? I am trying to build a random forest for a binary response (1 & 0). Let's say we have 10,000 different records and I am building 500 trees. Is there a way to score the records in terms of the certainty / confidence / likelihood of being categorized in category 1 (for example)? The link above suggests using the number of votes among all 500 trees, but this way can only give me up to 500 different scores, how can I differentiate further for these 10,000 records? (Like regression, the scores can be easily obtained). One solution is to average the score of each tree in the forest. the tree is the probability of 1s in the final node. Anyone know how to produce that average in R? I couldnt find this in the randomForest package. I think if I write my own codes to do that it , the run time may not be as fast as a built- in function.


title: Linear Discriminant Function

text: In linear discriminant analysis, how is the linear discriminant function determined? Assuming equal variance-covariance matrices, is the linear discriminant function determined from the training data?


title: Should I de-mean a predictor variable before a dummy interaction

text: Suppose I have the following time-series linear model where $\beta$ is misspecified: $Y(t+1) = \alpha + \beta X(t) + \sum_{i=1}^{10000}\gamma_i Z_i(T) + \varepsilon$ where all parameters are in $\mathbb{R}$ and all predictors are normally distributed and play nicely with respect to Gauss Markov. In the population ('correct') model that corresponds to the above, the slope parameter on $X(t)$ is equal to $\beta_1$ when $X(t) > \text{Q}_{0.95}(X(t))$ and is equal to $\beta_2$ when $X(t) < \text{Q}_{0.95}(X(t))$, where $Q$ is the quantile function. This break in parameters is a priori knowledge. We also have that $\beta_1 \neq \beta_2$. I want to model this as follows: $Y(t+1) = \alpha + \beta_1 X(t)I(X(t) > Q_{0.95}(X(t))) $ $+ \beta_2 X(t)I(X(t) < Q_{0.95}(X(t)))$ $+ \sum_{i=1}^{10000}\gamma_i Z_i(T) + \varepsilon$ My **question** is whether it is good practice to de-mean $X(t)$ before estimating the equation with the indicator function breaks, supposing my objective is out of sample forecast accuracy? I ask because this has an effect on collinearity between predictors (which could impact some, but not all, feature selection algorithms). Note that feature selection and regularisation is done in an automated way. I do not care about standard errors.


title: How to make inferences about a group based on test scores?

text: I have two datasets:   1. I have an exam score (pretend is the GRE) for all students that took the exam from 2000-2005, although I do not have student's private information (names, id,etc) I several variables along with the score such as socioeconomic status, race, parent's education, type of school, etc.   2. I have the average GRE score with standard deviation for all students that were accepted into a certain university for the same years 2000-2005. So I know that in med school the average GRE score was 60.05 with a standard deviation of 1.95.  Now, how do I manage to make inferences about students that were accepted into this university based on the GRE score in the first dataset? If I want to know what is the predicted probability of being accepted into Med School based on the characteristics of students in the first dataset (black, from public school) how do I calculate that? I use STATA so I divided the GRE score into categories, and ran: ologit GREscore black publicschool and then: `prvalue, x(black=1 publicschool=1)` However, I am not sure if this is what I am looking for. Do I need to fit the whole model with all my variables or not? Is there another way that I can do this?


title: P-value for multinomial distribution

text: I am testing the significance of observed data sampled ($n\approx50$) using a multinomial distribution with known probabilities (with ~20 categories). Given the probability of observing the sample $P_{sam}$, I would like to compute the p-value, which is defined as the sum of all probabilities lower than this observed $P_{sam}$ i.e. $Sum\{p \mid p < P_{sam}, p \in Multinomial- distribution\}$. One way to find this is to generate a large number of random samples from the given distribution and calculate the above sum using this set. But I am wondering if there exist any exact results and whether there are any good approximations which are less computationally intensive. Thanks!


title: Writing null hypothesis and deciding on rejection criteria

text: The number of faults in one metre of a thread is Poisson distributed.It is claimed that the average number of faults is 0.02 per metre.A random sample of 100 one metre lengths of the thread reveals a total of 6 faults.Does this information support the claim? What I want to know whether my H0 should be, H0: **lambda=0.02** or should it be **lamda=2** And my next question is in deciding the rejection criteria, Is it enough to check to reject H0,P[X>=6] **< 0.025** (Testing at 5% significance) or should I have looked for P[X<=6]<0.025 as well and checked whether either one of these is satisfied. Or calculate **2** *P[X>=6] and check if it is less than 0.05


title: Web visualization libraries

text: What are some good visualization libraries for online use? Are they easy to use and is there good documentation? Thanks.


title: What method to use to test Statistical Significance of ASR results

text: I have 2 ASR (Automatic Speech Recognition) models, providing me with text transcriptions for my testdata. The error measure I use is Word Error Rate. What methods do I have to test for statistical significance of my new results? **An example:** I have an experiment with 10 speaker, all having 100 (the same) sentences, total 900 words per speaker. Method A has an WER (word error rate) of 19.0%, Method B 18.5%. How do I test whether Method B is significantly better?


title: How to examine effect of time over four time points across two experimental conditions?

text: My experiment entails comparing a mock-infected group with a virus-infected group across 4 different time points (3,5,8,10 weeks) looking at the proportion of immune cells. I have already compared the mock and virus infected groups at each time point by using an unpaired t-test. How can I compare the mock values over time to see if there are significant differences with cell proportions and whether the virus infected groups follow similar kinetics?


title: p-value of 1.00 for a dichotomous predictor -- possible reasons for this oddity?

text: In my logistic regression analysis, my dichotomous predictor variable 'A' gave a strange p value of 1.00 under Wald's test. The analysis was done with 2 continuous covariates, a continuous predictor variable and an interaction variable consisting of the continuous predictor and the dichotomous predictor. The DV is a dichotomous outcome. Prior to the above analysis (some history on how i arrived at the logistic regression analysis), the DV was a continuous variable which was transformed into a dichotomous variable with values 0 and 1. The transformation was made because of the extremely skewed distribution. In the boxplot of the DV, 8 outliers were shown (out of a total of 98 persons) at the upper end of the graph and the rest were represented by a horizontal line at the lower end in the graph. I am wondering if the above issue of the strange p value was due to the transformation of the DV. Hierarchichal multiple regressions were conducted with other similar predictors, covariates, the same DV and 'A', and 'A' showed a consistent significant value (all p < .01). It was only in the logistic regression that gave the strange p value. If it is not due to the transformation of the DV, could it be because logistic regression was not the appropriate analytical method, or could this all be due to possibe statistical error (which seems in order though)? Thanks.


title: Why does post hoc test on pair of group means become significant when looking at subset of levels of factor in ANOVA?

text: I have something that seems to me an incongruence in ANOVA post hoc tests, and I would like to have an explanation. Basically I performed an ANOVA at the global level on my data and then I performed a post hoc test using both LSD and Tukey's HSD procedure. First I did the ANOVA on the whole data set and post hoc tests showed some pairs significantly different. Afterwards I subdivided the data set considering only some stimuli and I performed the post hoc test on them. In this second case some of the pairs that were non- significant in the "global" post hoc test become significant in the subset case. How does this happen? More in detail I show you what I did in R.               fit5<- lm(Response ~ Stimulus, data=scrd)     library(agricolae)     df<-df.residual(fit5)     MSerror<-deviance(fit5)/df          comparison <- LSD.test(scrd$Response, scrd$Stimulus, df,                       MSerror, group=FALSE, p.adj="bonferroni")          Study:          LSD t Test for scrd$Response      P value adjustment method: bonferroni           Mean Square Error:  4.292088           scrd$Stimulus,  means and individual ( 95 %) CI                                scrd.Response   std.err replication      LCL      UCL     dry_leaves_dry_leaves      6.833333 0.7768754          12 5.306018 8.360649     dry_leaves_gravel          6.750000 0.5383054          12 5.691706 7.808294     dry_leaves_metal           3.250000 0.5093817          12 2.248570 4.251430     dry_leaves_sand            6.583333 0.5701984          12 5.462339 7.704328     ...     ...          alpha: 0.05 ; Df Error: 396     Critical Value of t: 3.987986           Comparison between treatments means                                                    Difference   pvalue sig         LCL      UCL     dry_leaves_dry_leaves - dry_leaves_gravel 0.08333333 1.000000     -3.28963527 3.456302     dry_leaves_dry_leaves - dry_leaves_metal  3.58333333 0.017792   *  0.21036473 6.956302     dry_leaves_dry_leaves - dry_leaves_sand   0.25000000 1.000000     -3.12296860 3.622969     dry_leaves_dry_leaves - dry_leaves_snow   1.41666667 1.000000     -1.95630194 4.789635     dry_leaves_dry_leaves - dry_leaves_wood   1.83333333 1.000000     -1.53963527 5.206302     dry_leaves_dry_leaves - gravel_dry_leaves 0.58333333 1.000000     -2.78963527 3.956302     dry_leaves_dry_leaves - gravel_gravel     0.41666667 1.000000     -2.95630194 3.789635     dry_leaves_dry_leaves - gravel_metal      3.66666667 0.011649   *  0.29369806 7.039635     ...     ...     wood_sand - wood_snow                     1.08333333 1.000000     -2.28963527 4.456302     wood_wood - wood_sand                     3.00000000 0.274864     -0.37296860 6.372969     wood_wood - wood_snow                     4.08333333 0.001244  **  0.71036473 7.456302      Now I extract from the whole dataset a subdataset and I perform the ANOVA and post-hoc test on it:               # Row extraction     audio_wood <- subset(scrd, Audio == "wood")          #-------wood-------#          fit5_wood<- lm(Response ~ Stimulus, data=audio_wood)     anova(fit5_wood)               > anova(fit5_wood)     Analysis of Variance Table          Response: Response               Df Sum Sq Mean Sq F value    Pr(>F)         Stimulus   5 236.24  47.247  12.604 1.333e-08 ***     Residuals 66 247.42   3.749                           ---     Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1       Here I do the post hoc test               df<-df.residual(fit5_wood)     MSerror<-deviance(fit5_wood)/df     comparison <- LSD.test(audio_wood$Response, audio_wood$Stimulus, df,                      MSerror, group=FALSE, p.adj="bonferroni")          Study:          LSD t Test for audio_wood$Response      P value adjustment method: bonferroni           Mean Square Error:  3.748737           audio_wood$Stimulus,  means and individual ( 95 %) CI                          audio_wood.Response   std.err replication      LCL      UCL     wood_dry_leaves            3.916667 0.5143153          12 2.889803 4.943530     wood_gravel                2.916667 0.3361622          12 2.245497 3.587836     wood_metal                 7.333333 0.4143877          12 6.505982 8.160685     wood_sand                  4.000000 0.7487363          12 2.505100 5.494900     wood_snow                  2.916667 0.7329717          12 1.453242 4.380092     wood_wood                  7.000000 0.4767313          12 6.048175 7.951825          alpha: 0.05 ; Df Error: 66     Critical Value of t: 3.045792           Comparison between treatments means                                        Difference   pvalue sig        LCL      UCL     wood_dry_leaves - wood_gravel 1.00000000 1.000000     -1.4075050 3.407505     wood_metal - wood_dry_leaves  3.41666667 0.000798 ***  1.0091617 5.824172     wood_sand - wood_dry_leaves   0.08333333 1.000000     -2.3241716 2.490838     wood_dry_leaves - wood_snow   1.00000000 1.000000     -1.4075050 3.407505     wood_wood - wood_dry_leaves   3.08333333 0.003409  **  0.6758284 5.490838     wood_metal - wood_gravel      4.41666667 0.000007 ***  2.0091617 6.824172     wood_sand - wood_gravel       1.08333333 1.000000     -1.3241716 3.490838     wood_gravel - wood_snow       0.00000000 1.000000     -2.4075050 2.407505     wood_wood - wood_gravel       4.08333333 0.000036 ***  1.6758284 6.490838     wood_metal - wood_sand        3.33333333 0.001155  **  0.9258284 5.740838     wood_metal - wood_snow        4.41666667 0.000007 ***  2.0091617 6.824172     wood_metal - wood_wood        0.33333333 1.000000     -2.0741716 2.740838     wood_sand - wood_snow         1.08333333 1.000000     -1.3241716 3.490838     wood_wood - wood_sand         3.00000000 0.004843  **  0.5924950 5.407505     wood_wood - wood_snow         4.08333333 0.000036 ***  1.6758284 6.490838      As you can notice, the pair wood_wood - wood_sand which was not significant in the previous global pst hoc test, now is significant. Which of the two analysis I have to believe? And why?


title: How to find the Fisher Information of a function of the MLE of a Geometric (p) distribution?

text: Suppose that we have $X_1,...,X_n$ iid observations from a Geometric($p$) distribution. I found that the MLE of p is $\hat{p} = \frac{n}{n+\sum{X_i}}$. I am trying to find the MLE of $\theta = \frac{p}{1-p}, \hat{\theta}$ and the Fisher Information of a single observation $I_1(\theta)$. Now, by the functional invariance property of MLE estimators, I found that the MLE of $\hat{\theta}$ is just: $\hat{\theta}$= $\frac{\hat{p}}{1-\hat{p}}$. However, I have no idea how to find the Fisher Information $I_1(\theta)$. I know of two ways of doing this, one is to exploit the asympotic efficiency and the lower bound princple of MLE's. That is, $Var(\hat{\theta})=\frac{1}{nI_1(\theta)}$. Also, is the parametrization formula ${\mathcal I}_\eta(\eta) = {\mathcal I}_\theta(\theta(\eta)) \left( \frac{{\mathrm d} \theta}{{\mathrm d} \eta} \right)^2$ where ${\mathcal I}_\eta$ and ${\mathcal I}_\theta$ are the Fisher information measures of $η$ and $θ$, respectively. Is there another easier way to do this? I have no idea how to find the variance above. Thank you!


title: Implementation of generalized cross-validation for constrained LASSO problem

text: I have a non-negativity constrained LASSO problem like this: > min: $||Cx-b||_2^2 + \lambda||x||_1$ > > subject to: $x\geq0$ where C is a matrix, and x and b are column vectors. Now I want to use cross- validation to determine the regularization parameter $\lambda$. I want to use generalized cross-validation because it requires less computation time, but I don't know how to implement for my problem. Golub et al. has a paper on generalized cross-validation (see the paper here), but the influence matrix $A(\lambda)$ in page 2 equation 1.5 seems only designed for ridge problem. I am just wondering how to do something similar to LASSO problem and with non-negativity constraint. Any hint is greatly appreciated!


title: Reference for age-period-cohort models

text: I work in the loss forecasting area and would like to know a little bit more about the theory and implementation of age-period-cohort models. Several papers pop up in google search but I need a more comprehensive material relating to identification/estimation. Thank you.


title: Least square estimation error, calculus question

text: I want to estimate $n+m$ parameters with the equation: $$\hat\theta=\left[\frac{1}{N}\sum_{t=1}^N \varphi(t)\varphi^T(t)\right]^{-1}\left[\frac{1}{N}\sum_{t=1}^N\varphi(t)y(t)\right]$$ where $\varphi$ is a vector of regressors and $\theta$ the vector of parameters. Then my book derive an expression for the estimation error: $$\hat\theta-\theta_0 =\left[\frac{1}{N}\sum_{t=1}^N\varphi(t)\varphi^T(t)\right]^{-1} \left[\frac{1}{N}\sum_{t=1}^N\varphi(t)y(t)-\left\{\frac{1}{N}\sum_{t=1}^N\varphi(t)\varphi^T(t)\right\}\theta_0\right]$$ this makes no sense to me! Why the term in curly braces? And why inside the second factor? Please help.


title: Calculating a marginal distribution for the joint density distribution of an exponential distribution with a rate given by a Gamma distribution

text: This is a follow-up question from one I asked over at MathOverflow: http://mathoverflow.net/questions/158806/is-there-a-simple-closed-form- solution-for-the-joint-density-distribution-of-an The previous question was: I have an exponential distribution with rate $\lambda$, where $\lambda$ is drawn from a Gamma distribution with shape and scale parameters $(k,\theta)$. I'd like to calculate an exact PDF for values, $v_i$, drawn from the exponential distribution if, for each sampling event, we randomly sample a value of $\lambda$ from the aforementioned Gamma distribution. Is there a simple closed-form solution for the PDF of the $v_i$? My question here is: Is it possible to calculate a marginal distribution for the PDF of the $v_i$?


title: What general pattern in the means would produce a non-significant ANOVA result but a positive Tukey?

text: Why would the results of the ANOVA be non-significant, while a pair-wise comparison using Tukey's WSD is significant? Is their a general pattern in the means of the data that would typically produce this result?


title: What's the difference between add and subtract of two random variables, if zero mean?

text: Assume $X$, $Y$ are independent zero mean random variables. Define $Z_1=X+Y$ and $Z_2=X-Y$. Then, their mean values are the same. How does one check that $Z_1$ and $Z_2$ are not the same random variables? And how to describe their variances?


title: Visualizing sets of numbers, 0-255

text: I realize this may be a very general number, however, I'm very new to data visualization. Any links to appropriate material is greatly appreciated. Actual question/background: My team and I, for a school project, are assigned to write a program to visualize the AES encryption algorithm. Problem comes, all of us are primarily programmers, and not very creative. We were wondering if any existing visualization methods exist to display the differences between two sets of numbers. This would be the plaintext and the ciphertext (Represented as a number 0-255). Currently, we're using two methods of displaying the data and potentially a third, pixel colors in an image, sound levels in a short music sample, and a graph showing the weight of each value. Any help would be appreciated, if more data is needed, please ask. Thanks!


title: Why isn't k-means optimized using gradient descent?

text: I know **k-means** is usually optimized using **Expectation Maximization**. However we could optimize its loss function the same way we optimize any other! I found some papers that actually use stochastic **gradient descent** for large scale k-means, but I couldn't get my question answered. So, do anyone know why is that? Is it because Expectation Maximization **converges faster**? Does it has any particular guarantee? Or is it a **historical reason**? Thanks! Pablo


title: R crossvalidation cv.glm: prediction error and confidence interval

text: I am using the R package boot and the cv.glm function. The output 'delta' gives me the un-adjusted and adjusted prediction error. Here is an example on the top of page 10: http://www.unt.edu/rss/class/Jon/Benchmarks/CrossValidation1_JDS_May2011.pdf My question is are 'prediction errors' similar to standard errors? To make a 95%, confidence interval, do I simply multiply this error by 1.96 (then +/- that result)? Thanks


title: ED50s and LD50s

text: Suppose that we are given 1000 doses between 0 and 10 and various ordinal values (1 = normal, 2 = critical condition, 3 = death). So typical data points would of the form: $(5,1), (6,1), (10,3)$ etc. What is the purpose of using regression to obtain LD50? Couldn't we just look at the data and see at what dose $50 \%$ of the animals died? I guess regression is used to obtain an estimate of the LD50 if the experiment was repeated multiple times with different doses?


title: Linear regression model that best suit for data with errors

text: I'm looking for linear regression algorithm that is most suitable for a data whose independent variable (x) has a constant measurement error and the dependent variable (y) has signal dependent error. ![enter image description here](http://i.stack.imgur.com/3luKI.png) The above image illustrates my question.


title: What is the correct use of Cronbach's alpha?

text: I am trying to use Cronbach's alpha in order to see if some variables are measuring the "same", but I have some doubts about the correct use:   1. Can I compute alpha for numerical, ordinal or dichotomous variables? If it is correct to do that, then may I mix dichotomous, ordinal and numerical variables?    2. If Cronbach's alpha cannot be used for numerical, should I turn these variables into ordinal, or not?   3. If all my variables are ordinal, do they have to be on the same scale? I mean if variable 1 goes from 1 to 5, and variable 2 goes from 1 to 3, may I use Cronbach's alpha for these 2 variables?


title: How can be profile plots in EVT interpreted and what is the theoretical nature of it?

text: I have two question about profile log-likelihood. One is of theoretical nature and one is about a plot in R. I illustrate the two questions in a application of profile-likelihood in EVT. I use the package `evd`.               > data(lisbon)     > fgev(lisbon)          Call: fgev(x = lisbon)      Deviance: 241.2459           Estimates         loc    scale    shape       96.0319  12.8526  -0.1988            Standard Errors        loc   scale   shape       2.6171  1.8346  0.1284            Optimization Information       Convergence: successful        Function Evaluations: 24        Gradient Evaluations: 13           > plot(profile(fgev(lisbon)),ci=c(0.95,0.99))      The result is the following plot: ![enter image description here](http://i.stack.imgur.com/SrCjF.jpg) My two question are about this plot. **theoretical one:** For log-likelihood profile you can write one of the parameter as a function of the other, i.e. your log-likelihood function $l(\alpha,\beta)=l(\alpha,f(\alpha))$ for some function $f$.   Here I have a log-likelihood function $l(\mu,\sigma,\gamma)$ of three parameters location $\mu$, scale $\sigma$ and shape $\gamma$. In the profile log-likelihood of the location parameter I fix $\sigma$ and $\gamma$ and just vary $\mu$. But which values are used for $\sigma$ and $\gamma$? I don't think it is possible to write $\sigma=f_1(\mu)$ and $\gamma=f_2(\mu)$ for two functions $f_1$ and$ f_2$? **about the plot** : In the plot what is indicated by the dashed lines? Many thanks for your help


title: Environmental risk assessment based on sample number and concentrations of a toxic element

text: Happy New Year '14 everyone. I'm seeking a formal solution to a simple problem. I'm working with a database of a toxic element in different regions. I want to prioritize regions where immediate action should be taken based on my soil data (mean concentration of the toxic element in a region and the fraction of soil samples contaminated in that particular region). The idea is to put more weight on the fraction of soil samples contaminated. But I'm not sure if there's a nice way for quantifying the risk associated with such observations. For instance:               #region 01     #toxic element concentrations in 5 soil samples     #these observations have concentration unit (e.g. ppm, ppb, etc.)     t1<-c(0.05,0.05,0,0.05,0.05)     #mean of toxic element concentrations     m1<-mean(t1)      > m1     [1] 0.04      #fraction of contaminated soil samples in region 1 (we assume that,     #anything above 0.025 unit is dangerous for organisms     f1<-(length(which(t1 > 0.025))/length(t1)) ##no unit since fraction     > f1     [1] 0.8          #region 02     t2<-c(0.25, 0.25, 0, 0, 0.05)     m2<-mean(t2)     > m2     [1] 0.11     f2<-(length(which(t2 > 0.025))/length(t2))     > f2     [1] 0.6      The above observations in two different regions clearly demonstrate that there's no relation between fraction of soil samples contaminated and the mean concentrations; i.e. a region can have a large number of samples that are contaminated (or not contaminated) with a low (or high) mean concentration of the toxic element. A real STUPID way of prioritizing zones based on the provided data might be just adding fraction and mean:               #region 01     > m1+f1     [1] 0.84          #region 02     > m2+f2     [1] 0.71      This means that action should be taken in region 01 before region 02. But this solution is unacceptable in a global scale because I'm adding concentrations to the fraction which are unlike. If the concentrations were big numbers (say, 100 times the values shown above for another toxic element) and the safe threshold were 2.5 unit concentrations, the fraction remains same (0.8 and 0.6), but prioritized zones are flipped since region 2 ($11+0.6=11.6$) $>$ region 1 ($4+0.8=4.8$)! In other words, normalizing the toxic element concentrations to a fraction level is needed (if I want to put more weight on the fraction part), but I'm sure there's a statistically elegant way to deal with this. Is there any plausible relative probability distribution technique? Any package in R dealing with relevant problems?


title: Discrete choice prediction

text: I have a set of images of same color cars which I have users rate on a scale from 1-5 (integers only) based on how attractive they think the car design is. For each image I have a set of parameters about the cars in question, mostly various ratios of dimensions (say height at middle, width at trunk, curviness of hood, etc). I first give the user a set of training images and have them rate them. I would like to use this training set to predict future ratings by a particular user based on the ratios of the car in question. While a number prediction would be great (ie a prediction of what the user will rate the car), I am also willing to settle on a predicting of whether the rating is above or below say, 3. I don't really have a background in stats, but I think this has something to do with logistic regressions and discrete choice? I was wondering what a good reference for this would be. To add a little more info, there are no strong individual correlations amongst the ratios and the users rating. Moreover, a linear regression is out of the question because there isn't a simple relationship between changing a particular ratio and the attractiveness. Moreover, I also do not want to intoduce too many ratios because then they become correlated with each other (say, height/width and height/length can give length/width upon division).


title: Measures of effect size for chi-squared tests

text: What are the different measures of effect size used for chi-squared tests / binomial data? I'm already at least somewhat familiar with the phi coefficient and odds ratio. I've heard of Cramer's V and relative risk as well. However, I'm still not sure how to decide which effect sizes to report in different circumstances. What are the arguments for / against the different measures of effect size and in what circumstances are each most informative? Additionally, though I have resources for calculating confidence intervals for the odds ratio, I have struggled to find information regarding confidence intervals for the other effect size measures.


title: Which quality measures are available for non-binary problems?

text: I'm wondering which quality measures are available for non-binary classifiers. I've read this article https://en.wikipedia.org/wiki/Receiver_operating_characteristic I understand, that the idea of Confusion matrix can be generalized easily. However, such metrics as sensitivity,specificity or precision seem to make sense only for binary classification problems. So, my question is: are there any developed methods for evaluting the quality of non-binary classifiers? Thank you.


title: Dimensionality Reduction using PCA, with SVD of correlation matrix

text: I have computed a correlation matrix from certain data set of dimension 6 The correlation matrix is:                1.000000000000000 -0.142753907555000 -0.192138186332000 -0.523853268770000  0.124444699394000  0.002606132276000     -0.142753907555000  1.000000000000000  0.035741740609000  0.052631092693000  0.017746700663000 -0.361033166149000     -0.192138186332000  0.035741740609000  1.000000000000000  0.119582107782000 -0.346476373927000 -0.213738934536000     -0.523853268770000  0.052631092693000  0.119582107782000  1.000000000000000 -0.128989440426000  0.048428923563000      0.124444699394000  0.017746700663000 -0.346476373927000 -0.128989440426000  1.000000000000000 -0.434021957325000      0.002606132276000 -0.361033166149000 -0.213738934536000  0.048428923563000 -0.434021957325000  1.000000000000000      Then I applied SVD and got: $U$               -0.568204264304213  0.191580010050053  0.344459905939842  0.057623813300714 -0.711866787458235  0.108229201374774      0.092203508694887 -0.529503080968478  0.114920527567713  0.783048200494938 -0.053598897055946  0.286148469345589      0.399921613786302 -0.136693414712789  0.669267980471650 -0.404735519465042  0.004693646229437  0.457848350968476      0.545763234721263 -0.103897122514854 -0.450027359344472 -0.130624377930601 -0.685583680847584  0.041659858939764     -0.440777193839364 -0.406362489876042 -0.425730028151024 -0.360866606588038  0.093306539446761  0.566046038136009      0.128697656758384  0.698794255931705 -0.190919707811538  0.269103577206925  0.107794763719302  0.612075746815743      $S$               1.757508326665900 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000     0.000000000000000 1.561285064351230 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000     0.000000000000000 0.000000000000000 1.108208836182120 0.000000000000000 0.000000000000000 0.000000000000000     0.000000000000000 0.000000000000000 0.000000000000000 0.829989535414321 0.000000000000000 0.000000000000000     0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.469300557533083 0.000000000000000     0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.273707679853350      $V^T$               -0.568204264304213  0.092203508694887  0.399921613786303  0.545763234721263 -0.440777193839364  0.128697656758384      0.191580010050053 -0.529503080968478 -0.136693414712789 -0.103897122514854 -0.406362489876042  0.698794255931705      0.344459905939843  0.114920527567713  0.669267980471650 -0.450027359344472 -0.425730028151024 -0.190919707811538      0.057623813300715  0.783048200494939 -0.404735519465042 -0.130624377930600 -0.360866606588038  0.269103577206925     -0.711866787458234 -0.053598897055946  0.004693646229437 -0.685583680847584  0.093306539446760  0.107794763719302      0.108229201374774  0.286148469345589  0.457848350968475  0.041659858939764  0.566046038136010  0.612075746815743      Now what do I do with this decomposition, Maybe i am wrong but $S$ has the eigenvalues, and $U$ has the eigenvectors. So how can I reduce dimension in this example? How would you interpret these results? Can you give a little guidance?


title: Temporal abstraction in Churn analysis: Why do we need it?

text: Could you explain the need of temporal abstraction in churn analysis intuitively with a simple example? I tried Google but there are not any clear answers , especially for churn analysis.


title: MSE of filtered noisy signal - Derivation

text: I'm working on understanding the derivation of the optimal time constant for filters based on minimizing mean squared error. Unfortunately the text made a big jump between steps and lost me. Here's part of the derivation: $$ \theta_g = \theta +\sigma_g^2 +B_g $$ Where $\theta$ is the actual value, and $\theta_g$ is the input signal with variance $\sigma_g^2$ and constant bias B_g. Note that $\sigma_g^2$ is gaussian white noise. This signal is sent through a high pass filter, which results in scaling of magnitude $$ f(\omega)=\frac{\tau}{\sqrt{\tau^2\omega^2+1}} $$ $$ \hat\theta = f(\omega)*\theta_g $$ The text then goes on to calculate the mean square error for this estimator. Here's where I got lost: $$ E(e_{\sigma_g}^2) = \frac{\sigma_g^2}{2\pi}*\int_{-\infty}^{\infty} \! f(\omega)^2 \, \mathrm{d} \omega = \frac{\tau\sigma_g^2}{2} $$ $$ E(e_{B_g}^2) = \tau^2*B_g $$ $$ E(e_{total}^2) = \frac{\tau\sigma_g^2}{2} + \tau^2*B_g $$ From what I can see, the integral sums all the magnitudes squared. $2\pi$ is the range of values $\theta$ can take on. I'm just not sure why the equations are set up this way. Why is the function squared in the integral, and why is $2\pi$ in the denominator? And how was $E(e_{B_g}^2)$ calculated? Could anyone help me out here?


title: Sum of S|C Formula?

text: I have a formula like that: ${ \sum_{s} P(S=s|C=c)} $ I don't know what does it mean? **i.e.** P(S|C)+P(~S|C)?


title: What practical application is there for the Asymptotic Mean Integrated Squared Error in kernel density estimation?

text: ## Introduction For some time now I have been struggling to understand how theoretical results can be applied in practice. Fortunately in most cases the link between theory and practice is not hard to find, for instance:   * You can directly use a theoretical result for your calculations.   * You cannot actually find a solution, but you at least have an upper or lower bound that can indicate how good your real solution is.   * By observing the theoretical formulas you can make something 'similar' and hopefully with similar performance   * The general case is only theoretical, but in special cases it can be applied directly. Well, this is not everything, but I won't try to make a complete list of how statistical theory can be useful. Until now I have always succeeded in at least grasping the idea of how theory can be applied in practice. However, it now appears that I have stumbled opon something of which I cannot see how it could possibly be applied. I was studying some smoothing methods, amongst which (nonparametric) kernel smoothing, as described on Wikipedia , and found that there is a theoretical solution for the `AMISE`, as well as the bandwidth,`hAMISE` that optimizes it. However, both in the notes that I was reading, as well as the Wikipedia page I have been unable to find any practical application of this. ## The Question **Is there any practical application for the`AMISE` and its optimal bandwidth?** Discovering that there actually is an application for would be very motivating so I hope it can be achieved!


title: Plotting correlation matrices

text: I have a time series of correlation matrics (generated from a multivariate time series over time). Here I am wondering what is the best way to graphically represent the times series of corr matrices? Should I just extract individual pair and plot time series individually? What is the best practice? Thanks for your views.


title: Sparse estimated data recommender system

text: ## Premise: For a product and a user, the system has to recommend him/her other non-users related with him/her that are most likely to be interested in that same product. ## Available data includes:   1. **Non-users:**     * Basic demographics ( _e.g._ age, gender, location, date of birth).      * Simple relationship with other users.   2. **Users:**     * Non-user data     * Products the user interacted with in the past ( _e.g._ datetime, level of interaction).     * System usage data ( _e.g._ time since last login, first login datetime).   3. **Product:**     * Value.     * Datetime when the product was made first available. ## Data properties:   * Most non-users have very few relationships with other users (not uncommon to have only 1).   * Most users have few medium interactions and even less strong interactions with products.   * Demographic information is quite similar among all users (unevenly distributed).   * Total registered interactions for some products are quite limited in number. ## Initial approach:   1. Create a matrix of product interactions and users.   2. Normalise the data.   3. Assign to each non-user the average score for each product, calculated from the scores of all users that have relationships with that non-user (using the above matrix).   4. Perform a KNN for the user and the other non-users he/she is related to. Use their "estimated" product interaction scores to obtain most similar non-users. ## Question: What other approaches could be applied in order to reduce bias and/or increase recommendation quality?


title: How to measure dispersion of a single point within a group of points

text: Lets say you are given a set of $n$ points and for each one, you want to get a measure of dispersion of its closest $d$ points. So a point far away from others will have a large measure of dispersion while points in a cluster have a low measure of dispersion. Anyone know which statistic would be useful for this? I feel like this is something similar to standard deviation but standard deviation uses an average instead of just that single element. Would summing the distance differences between that single point and its $d$ closest points work?


title: Confusion related to scaling factors in HMM

text: I was reading about HMM in C.M. Bishop's book Pattern Recognition and Machine Learning. I was going through the forward and backward algorithm using $\alpha$ & $\beta$ For forward messaging passing $\alpha(z_n) = p(x_n|z_n)*\sum_{z_n-1}\alpha(z_{n-1})*P(z_n|z_{n-1})$ Since the probabilities are <=1. As we move from $z_1$ to $z_N$ the $\alpha(z_n)$ tends to get smaller and smaller and we might have floating point issues. For that the book mentioned something called scaling factors where u normalize the $\alpha$ So we have $\hat{\alpha(z_n)} = p(z_n|x_1...x_n) = \alpha(z_n)/p(x_1...x_n)$ which we expect to be well behaved numerically because it is a probability distribution over K variables for any value of n. I didn't get how it is well behaved and how it solves the floating point issue and what does it mean by probability distribution over K variables. I just didn't get this part


title: Gaussian processes benefits

text: I have this confusion related to the benefits of Gaussian processes. I mean comparing it to simple linear regression, where we have defined that the linear function models the data. However, in Gaussian processes we define the distribution of the functions means we don't specifically define that the function should be linear. We can define a prior over the function which is the Gaussian prior which defines features like how much smooth the function should be and all. So we don't have to explicitly define what the model should be. However, I have questions. We do have marginal likelihood and using it we can tune the covariance function parameters of the gaussian prior. So this is similar to defining the type of function that it should be isn't it. It boils down to the same thing defining the parameters even though in GP they are hyperparameters. For eg in this paper http://www.cs.ubc.ca/~hutter/EARG.shtml/earg/papers05/rasmussen_gps_in_ml.pdf. They have defined that the mean function of the GP is something like m(x) = ax ^2 + bx + c i.e a second order polynomial. So definitely the model/function is defined isn't it. So what's the difference in defining the function to be linear like in the LR. I just didn't get what the benefit is of using GP


title: Expected value of min X for bernoulli success?

text: I take a SRS sample of size n from a population of x values ranging from 1 to N. Each selected unit also has a probability p of success or q = 1-p of failure (i.e. the probability of success/failure is independent of the value of x). What is the expected value of the minimum x value for which success occurs? I believe that when n = N I can just use the geometric distribution, but I can't deal with the problem once only a proportion of the units are sampled. Many thanks in advance.


title: Forecasting: Different Model for 1 month, 2 month, 6 month forecasts?

text: I'm still trying to expand my statistics and forecasting technique knowledge. Right now I'm forecasting seasonal contact patterns, so the simplest model I can understand with seasonality is a Holt-Winters/ Triple exponential smoothing model. For many of our decisions, we not only need predictions for one month out, but also two months out, and even as far as 6 months out, or more. Obviously accuracy increasingly goes out the window as you try to forecast 6 months out, but, I guess we do what we can. The first thing I immediately noticed after creating the triple exponential smoothing model and initiated the seasonal/ trend values -- is that there are certainly different 'optimal' alpha (level), beta (trend) and gamma (seasonality) values, depending on whether you are forecasting one month out, or 6 months out. As you might have guessed, the 1-month forecast has far greater accuracy for a test data set with a higher alpha value (stronger recent values) -- and the 6 month is better with a lower alpha value. Not that it matters much, but the 2 month is closer to the 1 month model, and the 3-4-5 seems to be closer to the 6 month, but they all have their local 'optimal' values. So if I need a rolling forecast for the next 6 months --- am I to use one model, or am I supposed to use a different set of alpha-beta-gamma values depending on how far out the predicted value is? I have a minor background in statistics, but I know it's easy to misuse/ misread them. I don't know if using two or three different models like this can lead to mistakes.


title: Biplots: Adding supplementry points to a biplot which only use a subset of the variables

text: I have a matrix X where the rows denote the cases and the columns the variables. I use a standard row-metric preserving biplot in order to represent the cases in a subspace. If I wanted to add a supplementary point (i.e. a row) I can simply project the point it into that space. But, how can I optimally fit a point into the biplot if the supplementary point does only have values on some of the variables, i.e. if it only uses subset of the variables the biplot is constructed from? How can I add such a row point to a biplot?


title: Use of row rank and column rank in regression

text: I am struggling to understand full column rank vs. full row rank to estimate parameters of multivariate regression case. When do we use row rank, and when do we use column rank?


title: Fisher information for handwashing

text: I have two sets of data: the bacteria count on people's hands (n=1e4) without washing `x`. They then washed their hands and the count was made again `xwash`. What is the best way to estimate the reduction between these groups? Is there a formalised/sophisticated comparison test or am I reduced to hypothesis testing? What are my options? Best regards. (PS. other parameters may alter hand washing effectiveness)


title: Can deep belief networks be applied to sparse feature vectors/classification problems?

text: I am trying to beat the performance of an SVM classifier in a text classification task. Input is a bag of words model of sentences with 1 representing presence and 0 representing absence. Output is 1 of 4 possible labels. Are DBNs directly applicable to this class of problems? I get an error of 0.38 on the test set and I could not make it any better. Please refer me to papers/posts where I can find similar problems/applications.


title: Fitting a Weibull function in R

text: I am trying to model some data that follows a sigmoid curve relationship. In my field of work (psychophysics), a Weibull function is usually used to model such relationships, rather than probit. I am trying to create a model using R. My model will have a 'shape' and 'scale' value. In order to generate this model, I would have thought that R would need to have input vectors from both my X and Y axes (I am plotting size of a visual stimulus against the probability of it being seen). However, the R help page gives details of functions that only require either a vector of quantiles (size of stimulus) or a vector of probabilities (presumably, this means my 'probability of seeing' data). I don't see how R could possibly fit a model on the grounds of only having _half_ of my data (i.e. using my X axis (quantile) data, but none of my Y axis (probability of seeing) data). Am I missing something? I'm not a statistician so I fear there's something fundamental that I've overlooked. Any help would be appreciated.


title: Interpolating time series wind data using correlation

text: Is there a way to use correlation to interpolate missing data? I know the wind speed in 6 locations every hour. This shows the correlation between the separate locations. Hourly Data I know the wind speed in one of the locations every 5 minutes. 5 Min Data Is there a way to use this information to create 5 minute wind speed information for all locations? I have been using Python to analyze my data so far but can use R as well, just less familiar with R.


title: Continuous time markov chains, is this step by step example correct

text: Following my initial question (stats.stackexchange.com/questions/108874/continuous-time-markov-chains- simple-explanation) and some further reading on CTMC ([1], and [2] and also stats.stackexchange.com/questions/46389/solving-the-kolmogorov-forward- equation-for-transition-probabilities). I have some further questions... and most importantly whether the step-by-step example I provide below is overall correct. Let's assume 3 possible states $S = {1, 2, 3}$. There are the following data instances $D = {d_1, d_2, d_3, d_4}$ from which we want to model as a CTMC. In parenthesis the time spent at the state before transiting to the next. $ d_1 = 1(3min) \rightarrow 2(3min) \rightarrow 3\\ d_2 = 1(8min) \rightarrow 2(12min) \rightarrow 3\\ d_3 = 1(2min) \rightarrow 3\\ d_4 = 2(2min) \rightarrow 3 $ The main question(s) I want to ask and answer from the CTMC is:   * **Problem:** What is the probability of transiting from state $i$ to state $j$ given that $s$ time has elapsed since entering state $i$? So to form the CTMC I start with the _embedded transition matrix_ for $D$ which is: $ p_{ij} = \begin{bmatrix} 0 & 2/3 & 1/3 \\ 0 & 0 & 1 \\ 0 & 0 & 1 \end{bmatrix} $ Now the next step is to calculate the rates. From [1, p3] it says that _"Each time state i is visited, the chain spends, on average, $E(Hi) = 1/a_i$ units of time there before moving on"_. From what I understand $a_i$ is the set of rates for state $i$, also denoted in other texts as $\lambda_i$ or $v_i$, for example an operator receives 2 calls in an hour, hence $a_i = 1$ call in half an hour. However, I am not sure how to calculate this from the above data $D$. But, I can calculate the average time spent at each state and then from the above equation, solving for $a_i = 1 / E(H_i)$ I can obtain these rates... $E(H_i) = \begin{bmatrix}4.33 & 5.66 & 0 \end{bmatrix}$, denoting with 0 the terminal (consume) state. Hence, $a_i = \begin{bmatrix}0.23 & 0.18 & 0 \end{bmatrix}$, again 0 is for consume state.   * **Q1:** Is this valid?   * **Q2:** Given that in this way I assume that the probability distribution of time spent is an exponential distribution, I lose the true variance of the actual distribution as the variance of the exponential distribution is the same as the mean... Is this OK? Is there any other way?   * **Q3:** How can terminal/consume states be represented? Again from [1, p3] I can now compute the transition rate matrix (infinitesimal generator) $Q$: $Q_{ij} = -a_i$, if $i==j$ $Q_{ij} = a_i * p_{ij}$, if $i != j$ $ Q = \begin{bmatrix} -0.23 & 0.15 & 0.08\\ 0 & -0.18 & 0.18\\ 0 & 0 & 0 \end{bmatrix} $   * **Q4:** Although [1, p3] states that $Q_{ii} = -a_i$, [2, p227] says that $Q_{ii} = 0$. Which one is correct then? Now that we know $Q$, from [1, p2] I can calculate $P_{ij}(t)$ as follows: $P_{ij}(t) = e^{Qt}$ (from what I read there are a number of ways to compute that and I can use a numerical software that implements them)   * **Q5:** I assume $t = s$, ie the elapsed time since entering state $i$   * **Q6:** Why do I need to calculate the complete matrix $P_{ij}(t)$ if I am interested in one particular ${ij}$. Is there any way to calculate that.   * **Q7:** I read in [2, p.228] that if the process can be modelled as a Poisson process then the solution is: $P_{ij}(t) = \dfrac{(\lambda_i t)^k}{k!}e^{-\lambda_i t}$, where $k$ is the number of events in an interval of length t, which in my case above is $1$. Can I assume that the above process is a Poisson process?   * **Q8:** And lastly is everything I have explained in the above step-by-step example correct? Thank you the most for your time reading this and if you can provide any answers to my questions.


title: Autoregressive model with exponential lags

text: I have a very highly sampled time series that I would like to fit an autoregressive model (AM) to (~3 million samples). From knowing what they represent, I have believe there should be unique information out to ~1k sample points in the past. Thus I'm considering modifying a more standard AM. This is what I'm calling the "standard" AM (from Wikipedia): $$X_t = c + \sum_{i=1}^p \varphi_i X_{t-i} + \epsilon_t$$ The problem with this for my purposes is that to go out 1k samples require $p=1000$, which is a lot of parameters and I worry about over fitting. What I'd like to do is use exponentially spaced history points to do the fit: $$X_t = c + \sum_{i=0}^{p-1} \varphi_i X_{t-2^i} + \epsilon_t$$ So with $p=11$ I would be considering points up to 1024 samples in the past. My question is, is there something wrong with this approach? Are there frequencies that can fall in between the powers of two that I can't fit with this model? If it does work, why doesn't everyone do it like this?


title: Bootstrap approach for testing differences in interquartile range suitable?

text: I am investigating differences in two groups of forecast errors (absolute values / unsigned) that both have a strongly right-skewed distributions. Both groups of errors have the same number of observations (about 4000). As I like to test if the interquartile range and the fraction of errors that are <0.4 differ significantly from each other, I like to implement a significance test (and attach confidence limits). Actually this is what I have considered as an appropriate approach:   I like to generate samples of about 5000 observations by drawing observations randomly from the constructed sample (about 5000 obs.) with replacement. For each trial, I will compute the IQR and the fraction of errors <0.4, and then compute the difference of such measure across different valuation methods. This process shall be repeated 100 (ore more) times and a distribution for the difference of IQR (<0.4) shall be obtained. A t-statistic could be computed as the mean divided by the standard deviation of this distribution. Would you regard this approach as appropriate for a heavily right-skewed distribution considering my aims? How would I implement that using R as I will not bootstrap the differences but the errors of the two groups (e.g., A and B)?


title: Cycles in gaussian graphical models

text: I understand examples where two nodes can be dependent but conditionally independent given their common cause. For instance the common cause is high temperature, and the children nodes are high icecream sales and more number of sharks. The children are correlated through the high temperature, but independent given the high temperature. Mathematically, this is equivalent to - \begin{align*} b&=a+\epsilon_1\\ c&=a+\epsilon_2 \end{align*} $b$ is correlated with $c$, but $b\perp c|a$ I cannot imagine a scenario where cycles would occur. What is a physical or mathematical example that leads to a graph with cycles?


title: Prediction interval for my bus journey

text: I take the bus to work and I am trying to make a prediction interval for the journey time to work so that I can leave the house and be 99% sure I will get to work on time. The journey has 2 parts.   1. Waiting for the bus, the bus is meant to come every 10 minutes but because of traffic this varies a bit. I will assume that the waiting time follows an exponential distribution with a rate of 0.1 buses per minute. I call this variable X.   2. The time spent on the the bus, I will assume this follows a normal distribution. I have taken some samples and have found the mean to be 53 minutes and standard deviation 2.4 minutes. The confidence interval of the mean is pretty small compared to the overall journey time so it is ok to assume that the population mean is equal to the sample mean. I call this variable Y. So I am trying to create a prediction interval for the total time Z which is the sum of a normal and an exponential random variable. Z=X+Y. How can I go about doing this? With the information I have I know the probability density functions are: $f(x)=0.1e^{-0.1x}$ $g(y)=1.7\times 10^{-110} e^{9.471y-0.0893y^2}$ I don't mind if I have to use numerical integration but I'm not sure how to set up the integral when there are multiple variables.


title: What are the relative merits of Winsorizing vs. Trimming data?

text: Winsorizing data means to **replace** the extreme values of a data set with a certain percentile value from each end, while Trimming or Truncating involves **removing** those extreme values. I always see both methods discussed as a viable option to lessen the effect of outliers when computing statistics such as the mean or standard deviation, but I have not seen **why** one might pick one over the other. Are there any relative advantages or disadvantages to using Winsorizing or Trimming? Are there certain situations where one method would be preferable? Is one used more often in practice or are they basically interchangeable?


title: How do I determine whether traits in a subgroup of a big group are significantly different than the traits in the whole group in R

text: I have a big group of 1052 people. Out of 18 people with trait A 11 people also have trait B. Out of the whole group of 1052 people 569 have trait B. How do I determine in R whether the trait B is significantly different distributed in people with trait A compared to the overall population?


title: What is uncorrelated noise and its significance

text: In many applications such as estimation theory, when we need to estimate a parameter then we usually consider in presence of white gaussian noise of zero mean and some standard deviation. During Maximum likelihood estimation, we also use this assumption. So, my question is -   1. Do we consider noise to be uncorrelated or correlated in estimation?   2. What is the difference between correlated and uncorrelated noise and its significance   3. Why do we consider such a property of correlated or uncorrelated in estimation and when we say that measurement noise is gaussian.


title: How to forecast a Markov Switching Model

text: I have the following Markov Switching Model. Transition Matrix: $$ \left[\begin{matrix} 0.85387 & 0.91973\\0.14613 & 0.080265 \end{matrix}\right] $$ With Regime 1: Intercept: 0.00839 AR1: 0.26694 MA1: -0.26571 Var: 0.00244 Regime 2: Intercept: -0.05615 AR1: 0.70866 MA1: -0.67383 Var: 0.00244 How can I forecast using this information? I am unsure of how the transition matrix needs to be incorporated into the forecast calculation. Thanks!


title: Do I need to do log transformations for probit?

text: I am doing a (Heckman) probit. Some of my variables (e.g. N of employees, sales) are highly skewed (skew>2000). Do I need to take log-transformation to make them closer to normal? Thanks!


title: Estimate confidence in multiple choice test

text: There is multiple-choice test and for every possible answer option my algorithm gives some score of how much it is likely to be the right one and picks the one with the maximal value as the answer. If there are 4 options A B C and D and the corresponding scores are for example 3.34, 4.01, 2.78 and 3.01 then the answer should be B. On the other hand, if score are, say, 3.34, 100.01, 2.78 and 3.01 then the answer is still B but the algorithm looks more “confident” I was wondering how to numerically capture this confidence so that I can for example debug the algorithm and see if it is improving. Like for example the maximal one minus the second biggest value or something like that for one question - and the average of this value for the whole set of questions. So I wanted to ask if there is some “standard” commonly-used approach for this.


title: "Peakedness" of a skewed probability density function

text: I would like to describe the "peakedness" and tail "heaviness" of several skewed probability density functions. The features I want to describe, would they be called "kurtosis"? I've only seen the word "kurtosis" used for symmetric distributions?


title: Comparison of two different RMSEs

text: I performed interpolation on two elevation datasets. Is it possible to compare the two RMSE beyond just which one is larger and which one is smaller, to see if they are statistically different? Thank you in advance for any help!


title: Example of time series prediction using neural networks in R

text: Anyone's got a quick short educational example how to use neural networks (`nnet` in R for example) for the purpose of prediction? Here is an example, in R, of a time series               T <- seq(0,20,length=200)     Y <- 1 + 3*cos(4*T+2) +.2*T^2 + rnorm(200)     plot(T,Y,type="l")      This just an example but what I have is jumpy-seasonal data.


title: Bootstrap randomized Lasso selection for a Cox model

text: I'm interested in variable selection for a cox proportional hazards model. I've read this article which slightly favours randomized bootstrap lasso selection over bootstrap lasso selection since it removes the necessity of selecting a lambda-value. While the authors state that they used the penalized package, I'm having trouble finding the correct function for this. As far as I can tell there's no example syntax available. **How can I apply the randomized bootstrap lasso method for selecting variables for a Cox model using the penalized package? Are there alternatives?**


title: Real-time goodness of fit

text: I've to do a check of goodness of fit in real-time in my software but I don't know how to implement it: I've got a sum of weighted chi-squared variable with 1 degree of freedom. I can do the approximation with a new chi-squared distribution multiplied by a constant, so I've got a gamma distribution with given parameters. In my software I use a recursive form, so I sum the latest value to the previous one and I know only the "cumulative value", I haven't got the "history". At this point how can I test that the values came from my gamma distribution (and I know its parameters) with a given p-value? Is there a way to do a test like chi-squared test, i.e. where knowing the degree of freedom, with a given p-value, I can check if the new value is below or up the threshold using a table?


title: How do I interpret this McNemar's agreement plot generated by SAS?

text: I'm running a simple McNemar's test for agreement on the McNemar's example from Wikipedia. The test uses the following data: ![table](http://i.stack.imgur.com/w1CHJ.png) for patients of Hodgkin's disease. Running this SAS code:               data hodgkins;     input hodgkins $ sibling $ count;     datalines;     yes yes 26     yes no 15     no yes 7     no no 37     ;          proc freq data=hodgkins;         tables hodgkins * sibling /agree;         weight count;     run;      which generates this plot. How do I interpret this plot? I know that McNemar's test deals with probabilities of discordant pairs, which I understand. In this case, the simple null would be that the probability that a Hodgkin’s patient having tonsil’s removed while sibling not having them removed is equal to a Hodgkin’s patient not having tonsils removed while sibling did, but I'm not entirely sure how this applies to the plot. ![plot](http://i.stack.imgur.com/E94QJ.png)


title: What machine learning algorithms are good for estimating which features are more important?

text: I have data with a minimum number of features that don't change, and a few additional features that can change and have a big impact on the outcome. My data-set looks like this: Features are A, B, C (always present), and D, E, F, G, H (sometimes present)               A = 10, B = 10, C = 10                  outcome = 10     A = 8,  B = 7,  C = 8                   outcome = 8.5     A = 10, B = 5,  C = 11, D = 15          outcome = 178     A = 10, B = 10, C = 10, E = 10, G = 18  outcome = 19     A = 10, B = 8,  C = 9,  E = 8,  F = 4   outcome = 250     A = 10, B = 11, C = 13, E = 8,  F = 4   outcome = 320     ...      I want to predict the outcome value, and the combination of additional parameters is very important for determining the outcome. In this example, the presence of E and F leads to a big outcome, whereas the presence of E and G doesn't. What machine learning algorithms or techniques are good to capture this phenomenon ?


title: How do I calculate the p-value for each mutual information?

text: I've calculated the Mutual Information for my dataset using ARACNE, but I need to get p-values for each Mutual information. How do I calculate them? Is there any software for this calculation?


title: Hypothesis test for odds ratios

text: I have two possible exposure variables (A and B) for use in a statistical model predicting a binary health outcome. I have fitted models with each variable separately and now know that one variable is a better predictor of my outcome based on a comparison of R-squared and BIC values. However, the ORs for the two variables are quite similar with overlapping confidence intervals. The sample composition and size is the same for each model. I would like to comment on whether the odds ratios are significantly different from one another. How can I do this? Best Regards.


title: Clear description of PCA using SVD

text: After reading thousands of articles on PCA and SVD, using them in a number of programming frameworks and even implementing similar techniques (like Random Indexing) I found out that I still have doubts about some parts of PCA for dimension reduction. So let me show what I know and what I have doubts about. Let's say we have $N$ observations of $M$-dimensional data, organized as matrix $A \in R^{N * M}$. To perform PCA we should first compute MLE estimate of covariance matrix $\Sigma \in R^{M*M}$: $$\Sigma=\frac{1}{N}\sum_{i=1}^N(x_i - \bar x)(x_i - \bar x)^T$$ where $x_i \in R^M$ is $i$-th observation, $\bar x = \frac{1}{N}\sum_{k=1}^{N}x_k \in R^M$ is mean observation. Then we can decompose $\Sigma$ using SVD as follows: $$\Sigma = USV^T$$ Now here are several things I'm not sure about:   1. What are **dimensions** of $U$, $S$ and $V^T$?   2. In $USV^T$ what exactly is considered as **eigenvalues** and which of them should I use as **principal components**?    3. How can I **project** original observations $x_i$ onto new reduced space and vice versa?


title: How to sum cluster data?

text: I have a binary variable with probit model, i.e., $P(Y_{ij}=1|X_j)= \Phi(a_i+b_iX_j)$, where $X_j$ is $\mathcal N(0,1)$, and $a_i$ and $b_i$ are regression parameters. I am wondering what the expected value of $Y_j=\sum_{i=1}^n Y_{ij}$ is. Is there a known distribution for $Y_j$?


title: Number of parameters for discrete Bayes network?

text: Suppose $X_1, ..., X_n$ are $d$-ary discrete random variables which are part of a Bayes network, in which $X_i$ has $n_i$ parents. What is the number of parameters for the Bayes network?


title: What are the modeling approaches in this cartoon?

text: What are the modeling approaches depicted here? Can you name them and their prominent proponents or a landmark model? Is there an accepted superior approach? Who prefers which approach? ![alt text](http://i.stack.imgur.com/64Pe4.jpg) (From: http://www.stat.duke.edu/~mw/fineart.html)


title: What is the  multiclass loss used in kaggles for measure performance?

text: I searched on Internet but cannot find any source about this. What is multiclass loss and what kind of metric is it? If someone can enlighten me, it will be appreciated.


title: Normal approximation to the binomial distribution

text: I am having trouble getting to the bottom of this concept for two types of questions (hw is already passed, but I have a test this week and would like to do better). Hopefully someone can help me get this through my thick skull. Question 1 > In the following problem, check that it is appropriate to use the normal > approximation to the binomial. Then use the normal distribution to estimate > the requested probabilities. > > Do you try to pad an insurance claim to cover your deductible? About 39% of > all U.S. adults will try to pad their insurance claims! Suppose that you are > the director of an insurance adjustment office. Your office has just > received 134 insurance claims to be processed in the next few days. > > (a) half or more of the claims have been padded Question 2 > Based on long experience, an airline found that about 6% of the people > making reservations on a flight from Miami to Denver do not show up for the > flight. Suppose the airline overbooks this flight by selling 272 ticket > reservations for an airplane with only 255 seats. > > Use the normal approximation to the binomial distribution to answer: What is > the probability that a seat will be available for every person who shows up > holding a reservation? * * * Adding per the comments posted. My understanding of the approximation is that given np > 5 and nq > 5 the approximation can be used, and I have in my notes the definitions of                Mean = np       Sigma = Sqrt(npq)      Nowhere does it discuss how to figure out a probability, so I assume I use the probability from the binomial distribution (is this correct)                P = nCk * p^k * q^(n-k)      But my other issues is that binomial distribution is for exactly k successes out of n attempts. That doesn't seem to be the problem for question 1 (but I suppose I could sum all of the k..0 (by -1) successes. But again I am a touch confused here as well. Looking at question 2 it is more of the same, what is the probability that more than 255 people will show up . Maybe my question is more around the application of the binomial distribution. The textbook (brase and brase) is not super strong and I have been trying to augment with web and other texts. I am taking this as a leading class in a graduate degree in Econ. I also have a team of computational finance guys that work indirectly for me, but I would like to be able to have better conversations with them, and I have always wanted to now more about probability. At the end of the day, I got a 94 on this hw assignment (so its not like this is killing my grades), but these two questions (with a few more parts to them) really stumped me. It seems like a trivial place to get stumped and I would hate for this to come back and bite me later on!


title: References for consulting statisticians to offer their clients

text: This question illustrates the difficulty of a person mastering statistics and probability on their own, in the face of weakly developed resources like Wikipedia. It occurred to me that consulting statisticians, and there are a few here, may routinely face the challenge of explaining certain concepts and methods to a client. This is the flip side of the pedagogical coin. When one has mastered the concept, it may make sense to conduct a particular avenue of analyses, but one's references may either be inappropriate or difficult to share with a client. So, are there common resources that consulting statisticians like to suggest to their clients? (See update #1 regarding more advanced or specialized topics.) I can think of a few books that may be useful, but I suspect that a lot of clients will go about searching the web, as Developer did, and will come across rather inane material on Wikipedia. In my answer to Developer, I suggested the NIST Handbook as one such reference that could be used. What else? * * * Update 1: As Peter Flom has pointed out, for more advanced material or narrower pursuits, it may not be easy to offer a single point of reference. This is correct and I should have worded the question differently for those cases. In such cases, how do consultants find and share **accessible** references? I believe that many consultants will take the time to write something new in order to explain things to their client, but those aren't references that are found and shared. Some ideas:   * Tutorials written by the consultant or others   * Case studies or analyses from projects that demonstrate the same concepts   * Excerpts of books (as I'd suggested in my answer to Developer), which describe the concept What else might be a source or how else do you actually go about finding such references? I realize this is an open ended question, but my answer to Developer shows some of the ways I'd approach this problem. I don't mean to ask of _all_ the ways that one could address this, but in one's own experience, how have you typically provided such explanatory resources?


title: How can I compute a posterior density estimate from a prior and likelihood?

text: I am trying to understand how to use Bayes' theorem to calculate a posterior but am getting stuck with the computational approach, e.g., in the following case it is not clear to me how to take the product of the prior and likelihood and then calculate the posterior: For this example, I am interested in calculating the posterior probability of $\mu$ and I use a standard normal prior on $\mu$ $p(\mu)\sim N(\mu = 0, \sigma = 1)$, but I want to know how to calculate the posterior from a prior on $\mu$ that is represented by an MCMC chain, so I will use 1000 samples as my starting point.   * sample 1000 from the prior.               set.seed(0)     prior.mu      <- 0     prior.sigma   <- 1     prior.samples <- sort(rnorm(1000, prior.mu, prior.sigma))        * make some observations:              observations <- c(0.4, 0.5, 0.8, 0.1)        * and calculate the likelihood, e.g. $p(y | \mu, \sigma)$:               likelihood <- prod(dnorm(observations, mean(prior.samplse), sd(prior.samples)))      what I don't quite understand is:   1. when / how to multiply the prior by the likelihood?   2. when / how to normalize the posterior density? _please note: I am interested in the general computational solution that could be generalizable problems with no analytical solution_


title: How to turn a function into a probability density whilst maintaining the shape of the function?

text: I have a series of functions, each one supposedly representing the density of a random variable across agents. Each function also has a domain, which describes what values of the random variable are valid. Now, if I remember my stats classes correctly, if I take the integral of one of the functions across the values described by the function's domain, I should get a value of 1.0. This does not happen however. **Is there a normalization technique that can turn a function into a true probability density, yet maintains the shape of the function?** All the functions are of the form $\frac{a}{bx}+c$, where $x$ is the random variable, and $a,b,c$ are varying constants.


title: Is there a generic term for measures of correctness like "precision" and "recall"?

text: Suppose I am building some predictive models and then creating a report detailing how "good" those models were in various ways. Is there a generic (maybe even non-technical) term for the various measures of correctness (e.g. precision, recall, etc.)? A layman might use "accuracy," but accuracy actually has a very specific meaning and is a subset of possible measures to be included in such a report. In describing the contents of such a report, should I just use "accuracy" even though there will be other measures of correctness included in such a report?


title: Difference between experimental data and observational data?

text: I'm a novice to data mining and started to read about it. What's the exact difference between experimental data and observation data? Both are obviously data; and many say observation data can lead to errors. But I guess it's not possible to do an experiment for all data sets. I'm really confused, explain me what is experimental data and observation data and say when these should be used? Thanks in advance.


title: Assuming $u\sim N(0,\sigma^2)$ when y is highly skewed

text: does it make sense to assume $u\sim N(0,\sigma^2)$ when I know from a histogram that $y$ is highly skewed. Because from the assumption $u\sim N(0,\sigma^2)$ it follows that $y\sim N(x\beta,\sigma^2)$ and I'm absoluteley not sure if the assumption $u\sim N(0,\sigma^2)$ makes sense in a case where I know that the distribution of $y$ is not bell shaped. The alternative would be just to make OLS without any assumption about the error term, but in this case I can't analyze outliers and leverages (what I'd really like to do, because otherwise I can't present much more than a line which minimizes the sqaured sum of the residuals). [addendum: I can't make an outlier analysis because I can not define "outlier" in a context where I don't assume a normal distribution, because there is no outlying without a distribution] Besides your answers I'd really like to have a recommendation for a good book, where I can find some thoughts about what assumptions should we make when y is obviously not normal distributed. Thanks for your thoughts, they'll be helpful to me! Regards


title: Data Setup for Differences-in-Differences

text: Which setup is correct for a difference in difference regression model using $Y_{ist} = \alpha +\gamma_s*T + \lambda d_t + \delta*(T*d_t)+ \epsilon_{ist}$ where T is a dummy which is equal to 1 if the observation is from the treatment group and d is a dummy which is equal to 1 in the time period after the treatment occurred 1) Random samples from each group and time (i.e. 4 random samples) or 2) Panel data where the same units are tracked over both time periods? Does it matter and if not, can OLS be used with either case?


title: What is the difference between these radial basis functions?

text: Suppose there is a data set $\{(x_i; y_i)\}_{i=1}^{N}, x_i \in \mathbb{R}^3, y_i \in \mathbb{R}$. I am trying to find a regression function such as $f(x) = \omega_0 + \Sigma_{i=1}^{K} \omega_i \phi(\|x_i-c_i\|)$ using least square approximation. And my question now is whether there are any differences if I choose $\phi(x)$ to be Gaussian ($e^{-(\epsilon r)^2}$), inverse quadratic ($\frac{1}{1+(\epsilon r)^2}$), or inverse multiquadratic ($\frac{1}{\sqrt{1+(\epsilon r)^2}}$), etc. Thanks.


title: Difference categorical variables R

text: I can't figure out how to do the following: I need the following linear predictor $\eta=team_i-team_j$ where, both $team_i$ and $team_j$ are categorical variables, which have 163 categories. Normally when using categorical variables you do the following:               model<-glm(y~as.factor(team), family=, data=)      I tried to do the following:               model<-glm(y~as.factor(team1)+as.factor(team2), family, data)      But these gave coefficients for 325 categorical variables, 163 from both team1 and team2 and the intercept. How can I get R to only estimate 163 coefficients? So there are two categorical variables in each linear predicted, but they are connected in some way. I hope my question is clear. EDIT: I try to explain it differently:               y   Team1    Team2     0   A        B     1   A        C     2   B        C     2   B        A     1   C        A     0   C        B          glm(y~as.factor(Team1)+as.factor(Team2)-1, family=poisson)      This will give six estimates namely               as.factor(Team1)A, as.factor(Team1)B, as.factor(Team1)C, as.factor(Team2)A, as.factor(Team2)B and as.factor(Team2)C.      How can I get R to only estimate three coefficients. So as.factorA, as.factorB and as.factor C?


title: Shall I treat vegetation group as an ordinal or nominal variable?

text: As a follow on from here: Correlation between vegetation and erosion, I am having some trouble with the type of data that I have. I am confused as to whether my four categories are ordered. Specifically, my four groups within my categorical variable are:   * vegetation loss,   * no change non vegetated areas,   * no change in vegetated areas and    * vegetation gain.  While they are in order of an increasing amount of vegetation, I'm not sure if they would be considered nominal or ordinal. Any advice appreciated.


title: How do I deal with large data similarity computation?

text: I have lot of records like this: ![enter image description here](http://i.stack.imgur.com/EmbqW.png) `M` is about 10 million and `N` is about 100K. Now I want to apply collaborative filtering on these data, for example, A user comes in with its features(sparse data), how do I find out which existing user is most similar to him/she ? I don't think I could compute all of the records every time a request comes in, thanks ! Or is there any other algorithm could do this ?


title: Non-differentiation

text: I'm trying to assess the level of non-differentiation for a web survey I am doing. It features six pages of several Likert items on each page with a 6 point scale, followed by a seventh page of several items with a 10 point scale. I'm looking for the best formula to evaluate non-differentiation, and particularly one that is not sensitive to the number of response items. I found this paper presented to AAPOR that could be used as a model, but I'm not sure if it is sensitive to the number of response categories, i.e. could I compare the seventh page with the first six? http://papor.org/files/2006/Taylor2006.pdf


title: Is the variance of the residuals of a linear regression useful for estimating experimental sample sizes?

text: I have a data set of $y$ values that is not particularly normally distributed. However, the $y$s do partially depend on several other parameters. A linear regression model $y=c+\mathbf{ \beta x}+\epsilon$ fits the data decently and the residuals look like they're normally distributed, have uniform variance vs. fitted values, etc. [Edited to answer whuber's question from comments] The data I have is just happenstance historical data. Say I think another parameter $q$ might affect $y$, but I don't have any data where $q$ varies. I will have to collect more data, hopefully this time with an actual experiment where I do my best to control for everything that isn't $q$, including the $x$s. I could observe some $y$ values for "low" $q$ and some for "high" $q$, while holding $\mathbf{x}$ constant and in general doing whatever else is necessary to make a valid experiment. I could then test the null hypothesis that the mean of $y$ given low $q$ is the same as the mean of $y$ given high $q$. In order to decide what statistical test to use or how many observations I need to make, I need to assume some things like the distribution of my $y$ values, how much random variation I expect in $y$, how much I expect $q$ to affect $y$, etc. My question is whether or not the fact that I have historical data and some regression results helps me fill in those assumptions. Does the fact that the residuals are approximately normally distributed mean that it's reasonable to plan to use a test that assumes normal distributions, such as the t-test? (My reasoning would be that I expect the distribution of $y$ given some $\mathbf{x}$ to be normal if the model residuals appear normal.) Would it be reasonable to use the variance of the residuals when doing a power analysis for such an experiment? It seems like the variance of the model residuals a more reasonable number to use for "expected variance" that would the variance of $y$ in the raw data set, because some of that variation is explained by $\mathbf{x}$. But I don't know if there is an even more reasonable number to use. Finally, if I knew exactly which values of $\mathbf{x}$ I was going to hold constant, would it be better to use the standard error of the forecast at $\mathbf{x}$ - i.e. what I would get from Stata with `predict std_err_fcast, stdf`? (In this particular case, the forecast standard errors for the existing observations are all quite close to the standard deviation of the residuals, so the distinction might not matter much for scoping out roughly how many new observations I need when there are still other unknowns as well.)


title: Clustering with proportional threshold

text: I'm starting learning about clustering so perhaps this is a basic question. The idea is to generate clusters out of an array of floats, 1 dimension and N dimensions, get the mean value of each dimension of each cluster, and the array elements that get into a cluster should be within a range like this:               elem_val >= cluster_mean - (cluster_mean * threshold)     elem_val <= cluster_mean + (cluster_mean * threshold)      So the limits of the cluster are proportional to the mean value of the cluster. If the cluster mean values are [3.5, 5.0, 2.5], and the threshold is 0.1 , then the limits for each dimension of that cluster would be [3.15 to 3.85, 4.5 to 5.5, 2.25 to 2.75]. What clustering algorithm would be best to achieve this? (calculating the number of clusters automatically). Is BIRCH algorithm appropriate / optimal for this?


title: Increasing degrees of freedom with mixed models

text: I used the glmulti package in R to generate a linear model with interactions. The response variable has 21 observations, set against 5 input variables, one of which is a binary factor while the rest are continuous. The "best model" produced by glmulti, as judged according to aic, had 6 predictor terms (4 main effects and two interactions). The model fit beautifully in terms of fit (R2, all terms are significant p<0.01) and diagnostics, but the degrees of freedom are a problem. Each of the 21 observations is the average of about 4 observations. This is to say that there were 21 sites with four sub-samples per site. I then re-fit the same model with the sub-samples nested within sites with a GLMM with a gaussian distribution. The same results were produced as before. The coefficients still only have 13 degrees of freedom, however the number of observations for the model in its entirety has jumped up to around 75. Do I bypass the problem of having too few degrees of freedom by opting for a mixed model? I reviewed some of the other answers concerning df in GLMM but have yet to discern a clear answer to this problem. Thanks!


title: Probabilistic problem with two dice

text: Two dice are rolled. Event A is all combinations for which the sum is greater than 9. Event B is all combinations for which at least a face is 6. What are P(A|B) and P(B|A)? Maybe you will find it very easy, but I couldn't come up with an answer based on using Bayes' theorem and not just writing down the possibilities.


title: What does it mean to use a normalizing factor to "sum to unity"?

text: Would you also be able to provide an example? I have very little mathematical/statistical knowledge and have never understood normalization.


title: What to do with '0' values in Inverse Gaussian Distribution while calculating CI?

text: I am using this module in Stata to calculate the CI of a series of number that have a inverse gaussian distribution. The problem is, while calculating the CI, the zeroes are being ignored and as a result the lower CI gets higher than the mean. Can anyone kindly suggest what to do ? Thanks.


title: What is the best tool for customer segmentation?

text: I have a customer data set with the following data:   * The number of purchases that each customer made    * The Date that they made each purchase    * The Date that they signed up    * The amount they spent on each purchase  I want to segment my users into three groups:   * Great Customers   * Ok Customers   * Bad Customers Is there a tool I can use (statistical method OR software tool) that will look at all the variables and create the segmentations? I have Stata and Excel, but your answer doesn't have to be limited to these.


title: Predicting binary values with binary independent variables in logistic regression

text: I have two variables both are binary. I would like to see how well one predicts the other. Is it possible to do this in logistic regression? variables are coded as 0 and 1 but 1 does not indicate that something is 'correct'. can I still use logistic regression to see how well it is predicted?


title: How to calculate derivative of the contractive auto-encoder regularization term?

text: **Setup** I found a paper on that has a varient on normal auto-encoders (contractive) which for its gradient uses the following regularization penalty: $$\left|\left|J_f(x)\right|\right|^2_F = \sum_{ij}{\left( \frac{\partial h_j(x)}{\partial x_i} \right)}^2$$ where $\left|\left|\cdot\right|\right|_F^2$ is the Frobenius norm, $h$ is the hidden units, and $x$ is the input. The paper also gives an alternative form (when a sigmoid is used for $f$) to the equation that looks like: $$\left|\left|J_f(x)\right|\right|^2_F = \sum_{i=1}^{d_h}(h_i(1-h_i))^2\sum_{j=1}^{d_x}W^2_{ij}$$ **Question 1** As per usual, no actual derivation is given to get the second form of the equation in the paper. I attempted to derive it myself, but would _greatly_ appreciate it if someone could check my work and let me know what mistakes I might have made. $$ a(x) = W^T x + b $$ $$ h(x) = f(a(x)) $$ $$ f(x) = \frac{1}{1+e^{-x}} $$ Then, using the chain rule: $$ \frac{\partial h(x)}{\partial x} = \frac{\partial h(x)}{\partial a(x)} \frac{\partial a(x)}{\partial x} $$ Using the standard derivation of the sigmoid: $$ \frac{\partial h(x)}{\partial a(x)} = f(a(x))(1 - f(a(x))) = h(1- h) $$ and: $$ \frac{\partial a(x)}{\partial x} = W $$ Thus, finally: $$ \frac{\partial h(x)}{\partial x} = h(1-h)W $$ **Question 2** The problem I run into is that I'm not entirely sure of how to take the derivative of $\left(\frac{\partial h(x)}{\partial x}\right)^2$ with respect to $W$ in order to be able to get the gradient. If I square my previous result, I should get: $$ \frac{\partial \left[h(1-h)\right]^2W^2}{\partial W} $$ which I would _think_ would give: $$ 2\left[h(1-h)\right]^2 W $$ And I'm having difficulty in interpreting this. I am a bit confused about whether h(1-h) is then an dot product. If so, does that just give me a scalar multiplied by W? If not, and it's an element-wise multiplication, then I think the dimensionality would be all wrong. Or perhaps I did all of this incorrectly. Any help would be greatly appreciated!


title: Data tranformation in logistic regression

text: I am running a logistic regression with dichotomous dependent variable (0-1) and with equal interval scale (1-5) as independent variable. The problem is that in order to arrive to meaningful results, i.e. reconcile with theory, I need to use the following transformation to independent variable: 1/log(x) . The problem now is that I have to document why I had to transform the variable. Does anyone help me on this? For example, taking log is used for percentage change, reduce variance, change the relation from linear to non linear, etc. Many thanks


title: How to isolate impact of event in a product's lifecycle?

text: I'm trying to figure out how a single event affects sales numbers of a song. For example, see what the effect of being featured in iTunes store compared to songs with comparable previous download numbers. How should I go about modeling this question? Thanks!


title: Link between variance and pairwise distances within a variable

text: Please, prove that if we have two variables (equal sample size) $X$ and $Y$ and the _variance_ in $X$ is greater than in $Y$, then the _sum of squared differences_ (i.e., squared euclidean distances) between data points within $X$ is also greater than that within $Y$.


title: Calculating life expectancy in Stata

text: I've been looking for resources on calculation of life expectancy uising Stata. Combing Google results so far I've only been able to locate this module: LXPCT_2: Stata module to calculate multistate life expectancies Could you point me to any materials, preferably with sample code on this topic? Thanks!


title: What is most appropriate test?

text: I have conducted a survey using 5-Point Likert scale (Strong Agree; Agree etc.) in which a panel of experts indicated their Agreement with suggested statements. There are three sub-Groups within the expert panel I am interested in evaluating the significance of Agreement for each of the different groups. My challenge is that my sample size is small (n=14). Can I assume normality? Is Chi-sq test of contingency tables appropriate? Thanks!


title: Tree size in gradient tree boosting

text: Gradient tree boosting as proposed by Friedman uses decision trees with `J` terminal nodes (=leaves) as base learners. There are a number of ways to grow a tree with exactly `J` nodes for example one can grow the tree in a depth first fashion or in a breadth first fashion, ... Is there an established way how to grow trees with exactly `J` terminal nodes for gradient tree boosting? I examined the tree growing procedure of R's `gbm` package and it seems that it expands the tree in depth-first fashion and uses a heuristic based on error improvement to choose whether to expand the left or the right child node -- is that correct?


title: Meaning of "choose a random thing independently"

text: > a random walk **chooses a random bit independently** and it moves left or > right if the bit chosen is 0 or 1, respectively. I was wondering if "choose a random bit independently" means that the random bit has a uniform distribution over {0,1}, and the walk sample this uniform distribution? I remember that I often see expression such as "choose a random thing independently". Does it mean the random thing has a uniform distribution over its range? Does it have something to do with independence between events or random variables? Thanks!


title: What is the difference between R functions prcomp and princomp?

text: Yup, I compared `?prcomp` and `?princomp` and found something about Q-mode and R-mode PCA. But honestly – I don't understand it. Can anybody explain the difference and maybe even explain when to apply which?


title: How to compare two sequences of binary data?

text: I have two differing-length sequences of binary values. In each sequence, each value corresponds to an observation which is either right or wrong. I'm looking for a statistical test that addresses the question "are there significantly more right answers in one sequence?" Which test is most appropriate? I'm looking at Fisher's chi-squared test and Fisher's exact test, but I'm not exactly sure how to use each one. EDIT: More details The study consisted of a 2-alternative forced choice recognition task. Two videos were displayed, then a third. Subjects had to match the third video with one of the first two, and the binary result collected was set to 1 if they were correct. Trials were in random order, so shuffling the response vectors has no effect. Thanks, Louise


title: PCA and EFA on bond returns using Stata

text: I've got a datafile which has bond returns (30, 20, 10, 5, 1 year) and time. The bond variables are returns on the different time bonds. I need to perform PCA and EFA on the data. I tried using time as one of the factors but it returns Heywood case as the uniqueness is 0. Should I disregard time? What i don't understand is if I disregard it then my analysis will have no practical interpretation. Already I've two principal components using PCA and I'm unable to interpret them. Please help me in understanding the practical implications of the data. Thanks in advance for your time and kind consideration... P.S. I'm working in Stata 12.1. Solutions in SPSS will be helpful too but I won't be able to perform them as my SPSS 20 does not accept datafile from this version of stata.


title: Logistic Regression with weighted instances

text: I'm working on implementing a logistic regression algorithm in code. It's based this link. Unfortunately, the paper doesn't talk about weighting the individual examples $x_{i}$. I think the relevant log likelihood function will look something like this: $$ L(\vec{w}) = \sum_{i=1}^n \log{g(y_i z_i)} r_i $$ as opposed to what's in the paper: $$ L(\vec{w}) = \sum_{i=1}^n \log{g(y_i z_i)} $$ where $z_i=\sum_k w_k x_{ik}$ and $r_i$ is the instance weight for the given instance $i$. Also, $y_i \in\{-1,1\}$ in this case, and $g$ is the sigmoid function so $1-g(z)=g(-z)$. This is discussed in the link. Unfortunately, my math skills are not solid enough to be able to solve for the first and second partial derivatives, which are required to perform the optimization. _Without_ the instance weights I'd like to add, the derivatives are: $$ \frac{\partial{L}}{\partial{w_{k}}} = \sum_{i=1}^{n}y_{i}x_{ik}g(-y_{i}z_{i}) $$ $$ \frac{\partial^{2}{L}}{\partial{w_{j}}\partial{w_{k}}} = -\sum_{i=1}^{n}x_{ij}x_{ik}g(y_{i}z_{i})g(-y_{i}z_{i}) $$ How do these translate with the new $r_{i}$ instance weight involved? Thanks!


title: Basic Inter-rater reliability and type of data

text: I am looking to find out the type of data I have and the ideal test to use calculate inter-rater reliability. Research question: Functional Movement Screening: How reliable are we in our investigations?   Aim: to look into the inter-rater reliability of Physiotherapists performing a functional movement screen (12 tests) on 2 athletes. My current data collection includes:   6 raters each scoring 12 tests with a yes and no answer.   There are 2 sets (2 athletes) in total and each rater has given me y and n data for both. Example:               Rater 1   6 Yes / 6 No       Rater 2   7 Yes / 5 No            Test 1    3 yes / 3 no (from raters)     Test 2    4 yes / 2 no      I would like to know:   1. the type of data I have?   2. am I looking at the reliability of the tests or do I need to focus on the raters scores?   3. the groups I need to organise the data into to give me the results?   4. the statistical test that would give me the inter-rater reliability of the raters and the tests?  I am afraid statistics isn't one of my strong points and so apologies if this is very basic but any help anyone could give me would be much appreciated as I am having difficult at present.