title: Fixed-effects using demeaned data: why different standard errors when using -plm-?

text: I am fitting a _Fixed-Effects_ model, with intercepts at `cluster` level. One of the most direct ways is probably to use the `-plm-` package. Another well-known possibility is to apply OLS (i.e. to adopt `-lm-`) to the _demeaned data_ , where the means are taken at the clustering level. This second approach is usually referred to as the **within transformation**. It is quite convenient from a computational standpoint, becuase we are still controlling unobserved heterogeneity at clustering level, but we do not need to estimate all the time-fixed intercepts. I have tried both of these approaches, and I came to a strange result. In practice, the coefficent of the regressor of interest, `x`, is the same in both cases. However, its standard error (and actually all the other relevant quantities of the regression: R squared, F test, etc.) is different. Please, notice that I have carefully read both the _R documentation_ about `-plm-` and the related paper of the authors, where it is stated that the package apply the _within transformation_ and then apply OLS, as I did... The R script is:               # set seed, load packages, create fake sample          set.seed(999)     library(plyr)     library(plm)          dat <- expand.grid(id=factor(1:3), cluster=factor(1:6))     dat <- cbind(dat, x=runif(18), y=runif(18, 2, 5))               ############################     #   FE model using -plm-   #     ############################          # model fit       fe.1 <- plm(y ~ x, data=dat, index="cluster", model="within")          # estimated coefficient and standard error of x     b.1 <- summary(fe.1)$coefficients[,1]         se.1 <- summary(fe.1)$coefficients[,2]               ######################################     #   OLS on within-transformed data   #     ######################################          # augmenting data frame with cluster-mean centered variables      dat.2 <- ddply(dat, .(cluster), transform, dem_x=x-mean(x), dem_y=y-mean(y))          # model fit     fe.2 <- lm(dem_y ~ dem_x - 1, data=dat.2)          # estimated coefficient and standard error of x     b.2 <- summary(fe.2)$coefficients[1,1]         se.2 <- summary(fe.2)$coefficients[1,2]               #########################     #   models comparison   #     #########################          b.1; b.2     se.1; se.2          summary(fe.1)     summary(fe.2)      Notice that in the second model it is necessary to manually eliminate the intercept from the model. Thanks!


title: Interpreting categorical vble in an inefficiency equation

text: I have a categorical variable called Torture (=0, torture frequently used by government; =1 often used; 2=never used). Regression of economic INefficiency on torture yields negative coefficient sign. Does that mean that improvements in torture (i.e., going from 0 to 2) bring down and reduce economic inefficiency? Or does it mean that torture in general reduces economic inefficiency? Thanks.


title: Investigate change in mean levels of risk within a service- alternative to unbalanced repeated measures ANOVA

text: Not sure if this is as complicated as it's ended up in my head, here goes. I have a series of measurements which measure how risk-prone psychiatric patients are (hereafter referred to as HCR and it's an interval, normally distributed outcome). These measurements come from five different locations within the hospital and are measured at different times on a varying number of occasions (I have the exact date of each measurement for each patient). It's very simple to model the change in this variable for _patients_ using my old friend lme4- the model looks like this:               HCRmodel=lmer(HCR~Day+Location+(1|patient), data=mydata)      [in English, the HCR score depends on the day (counting from 1 for each patient) plus an effect for where they are in the hospital and these measurements are clustered in patients] So far, so good. The problem I'm having is that I would also like to model changes in _locations_ over time. The theory is that each location may have changed over the 5 years of data, say, service A is dealing with riskier and riskier patients, service B with less and less risky, something like that. Clearly this kind of analysis lends itself to ANOVA, except that the measures are repeated and it's completely unbalanced. I can't even post where I've got to so far because I haven't really got anywhere- I'm thinking along the lines of calculating the mean levels of risk in each service for each day over the whole study and putting this into a balanced ANOVA design. But I'm a little unsure whether I'd just be munging the data to _appear_ balanced, or whether it really would be balanced. Any help gratefully received.


title: Is it legal to publish the code of a published algorithm?

text: Let's say I want to implement an algorithm based on a paper or book and publish it under a non-proprietary but not necessarily non-commercial-friendly license (e.g. on a blog). Is it legal to do this? I know that a general all covering answer cannot be either yes or no, so I want to know additionally how to find out quickly whether this is legal or not. Some examples would be great. Here are some examples for discussion   * Apache Commons Math stated in their developer guide, that all developers should check for license issues before committing and link to Numerical Recipes as an example. This is the first time ever I saw such a warning. I do not have access to this book (neither online nor in dead-tree-format). What does the legal warning looks like ?   * Papers linked for download on the page of the author normally do not contain a legal warning. Does it mean that the algorithms are ... free?   * According to wikipedia, Random Forest is a trademarked term. Does that mean, that noone is allowed to implement this algorithm ? One can give it another name (like Good-Luck-Forests), since reproducing the exact algorithm given that not all details are published in paper (normally) is nearly impossible.    * What about papers where the access is both restricted and not. See for example the paper _"Alternatives to the Median Absolute Deviation"_ , which has been published in the Journal of the American Statistical Association, but which can be bought now via JSTOR or be downloaded from this page. Did Frank Masci, the uploader of this paper, break the law? A lot of papers with or without restricted access can be found online. **Disclaimer:** No answer should be treated as legal advice one can refer to before court.


title: How to pick coefficients for Holt Winters?

text: I'm using Holt Winters to predict sales revenue from past performance. Seasonality and changing trends exist in the data. One of the reasons chosen for Holt Winters is that it is fairly simple (implementable in excel) and explainable to non-statisticians. If other methodologies are more appropriate I’m happy to hear about them. A key implementation detail is to choose coefficients $\alpha, \beta$, and $\gamma$. Are there “standard” values for these? The other reason for this methodology is that it auto-updates. This will be useful as the series will change with market shifts. I was planning to adjust the “standard” value by eye, to try to mitigate likely changes. Is this a reasonable approach?


title: How to make waffle charts in R?

text: How can I do plot a waffle chart as an alternative to using piecharts in R?               help.search("waffle")     No help files found with alias or concept or title matching ‘waffle’     using fuzzy matching.      The closest I found googling out there are mosaicplots.


title: Forecast with predictable market events

text: I'm trying to build a predictive model to forecast the residual value of used electronic equipment. As a first step, I created a few quick plots in order to visually identify any interesting features. Here's an example: ![used prices for a used consumer electronic product](http://i.stack.imgur.com/F360n.png) A couple of things stood out:   1. the prices are 'steppy', i.e. they have a tendency to hug round numbers and drop in chunks.   2. the price drops suddenly when a new product is released. The product release dates have a somewhat predictable cadence. At first glance, it seems like this can be broken down into three parts:   1. price erosion before the new product launch   2. the impact of the new product launch   3. price erosion after the new product launch The pieces could then be stitched together. Does this seem like a sensible way forecast, taking predictable market events into account? Are there any particular statistical methods/approaches you'd recommend for this sort of problem?


title: Regression vs Multiclassification

text: I was working with SVR, and wondering, why can't I solve a natural regression problem as a multiclassification task ? Example: I have for a regression problem: targets 1, 5 and 10, trying to fit Integers numbers inside of this range. Why can't I use a multiclassification with targets 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ? What are the advantages and disadvantages of such approach ?   * One of disadvantages that I can see, is that I must have a complete target training set (1 to 10) at the multiclass problem. Any other problems ?


title: Spearman correlation of two Weibull distributions

text: Is there a way to find the Spearman correlation between two Weibull distributions? I need it as a parameter in a copula function for the joint Weibull distribution. I learned that using the Pearson correlation, which I can easily obtain from the variances and cross variances of the given spectra, is not reliable with copulas especially when the data is not normally distributed.


title: How to model the relationship between number of subjects per group, derived using standard power analysis methods, and study specific parameters

text: I am interested in estimating how many subjects should be included in a brain imaging study. Although the design is a fairly straight forward cross- sectional comparison, there are a number of tweakable image processing steps between the raw image and the processed image in which we carry out pixel-wise comparisons. I've used the power.t.test function in R to estimate the number of subjects required to reach statistical significance, using hypothetical delta values and sd estimates from a control population. I've run these analyses with varying image processing settings, resulting in a large number of "number of subjects per group" vs "some image processing parameter" plots. So many that it looks kind of messy to provide hundreds of plots. What I would like is an empirical equation that allows users to estimate the number of subjects per group as a function of effect size, alpha, and a few other image processing specific parameters. That way I only need to report the coefficients for the equation rather than supplying hundreds of graphs. Is this kind of thing possible? So far if I model N ~ k1/effect.size^2 that looks OK for some values of alpha (for example) but doesn't work so good for others. A log-log plot doesn't model the relationship well either. I can't seem to find an explicit formula that relates the number of subjects per group with the other factors in a power analysis. Does this exist? Thanks


title: How to apply a soft coefficient constraint to an OLS regression?

text: I would like to estimate an ordinary least squares regression of the form $$ y = X\beta + \varepsilon, \ $$ except that, instead of minimizing the sum of squared residuals, $$ SSR(b)=(y-Xb)'(y-Xb) $$ I want to minimize $$ (y-Xb)'(y-Xb)+\lambda(b-\tilde{\beta})'(b-\tilde{\beta}) $$ where $\lambda$ is some constant. Otherwise, notation above is as on wikipedia. All the standard assumptions are satisfied. Is there some way I can modify the regression to perform the joint minimization?


title: What is full conditional distribution in my case?

text: If $\text{P}(M|D)$ is posterior, $a$ is the proportionality constant, $\text{P}(M)$ is the prior and $\text{P}(D|M)$ is the likelihood. I have the the prior distribution, and I know the function that can give the likelihood for any input parameter sample. Then the Bayes rule is: $\text{P}(M|D)= a\times\text{P}(M)\times\text{P}(D|M)$. If I want to sample from $\text{P}(M|D)$ by Gibbs, what is my full conditional distribution? is it $\text{P}(M)\times\text{P}(D|M)$?


title: vote population

text: which is the % of population with vote rights(above 18yo for example). Can u point me to some papers which talks about voting and ages?


title: How do we pass from proportionality back to equality in a Bayesian derivation?

text: Here's an answer from one of the questions in the newsletter. I'm not trying to suggest that it's incorrect, I just don't understand why it's an acceptable practice. I know that simplifying assumptions are made all the time, but why is this useful? \begin{equation} \begin{split} p(\theta|y) &= \frac{p(y|\theta)p(\theta)}{p(y)} \\ &\propto\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}\theta^{\alpha-1}(1-\theta)^{\beta-1}*\binom{n}{y}\theta^y(1-\theta)^{n-y} \\ &\propto\theta^{\alpha-1}(1-\theta)^{\beta-1}*\theta^y(1-\theta)^{n-y} \\ &\propto\theta^{\alpha+y-1}(1-\theta)^{\beta+n-y-1} \\ &=\frac{\Gamma(\alpha+y-1)\Gamma(\beta+n-y-1)}{\Gamma(\alpha+\beta+n-1)}\theta^{\alpha+y-1}(1-\theta)^{\beta+n-y-1} \end{split} \end{equation} My issue is that we go from "proportional-to" to "equals", which is fine, but what exactly is this saying? As in, I understand how we go from each line to the line following it, but I don't understand what this tells us. Sure, we eventually get back to a Beta distribution, but only after removing a ton of values. Are these values not important? Does it mean anything profound that a Beta prior with a binomial likelihood has the property of conjugacy? It can make calculations simpler, sure, but is this useful in practice, if so why? **EDIT 1:**   So are these constants actually thrown out, or accounted for later? I understand that constants don't matter with respect to proportionality and integration, but throwing them out completely just seems counter-intuitive to me. Sure, if you're multiplying/dividing a bunch of numbers by an expression with a constant, and all you care about is the relationship of those numbers to each other, then you can throw out constants and the relationship remains the same. Is that what's going on here? **EDIT 2:**   From what I can tell, we get rid of constants, simplify, and then take the integral and normalize that integral. By virtue of the normalization, we're inherently accounting for the constants that we removed before. Is this correct? I think I understand now, for whatever reason, I kept thinking of the original equation as a scalar value rather than a probability distribution.


title: How to compare Likert ratings for multiple methods in a usability study

text: I am designing a questionnaire to compare user experience of usage of a two prototypes of a software - prototype A and B. Users will rate prototype A for a task and then prototype B for performing the same task based on a 5 point Likert scale. The questionnaire will contain 5 questions related to the various measures of user experience and contain ratings for each measure. How can i compare the Likert ratings for the two prototypes using statistical tests?


title: Can I treat SNP genotype 0 1 2 as continuous value?

text: I'm currently working on a logistic regression problem with case-control response variable and SNP genotype as regressor. Can I treat 0, 1, 2 as continuous instead of factor?


title: Confusion related to adding noise to a gaussian data

text: I have this confusion related to two methods to add gaussian white noise to data. Suppose I have a bivariate gaussian distribution with mean vector and covariance matrix $$ \mu = \begin{pmatrix} 0 \\ 0 \end{pmatrix}, \quad \Sigma = \begin{pmatrix}1 & 0.5 \\0.5 & 1 \end{pmatrix}. $$ Now suppose I want to add a gaussian white noise. I want to know what is the difference between the following methods. **Method A**   Generate samples from the bivariate Gaussian distribution with mean $\mu$ and covariance matrix $\Sigma$, and then adding noise with zero mean and unit variance like for each i data(i,1) = data(i,1) + $\epsilon_1$ data(i,2) = data(i,2) + $\epsilon_2$ where $\epsilon_1$ & $\epsilon_2$ is noise ~ N(0,1) **Method B**   Generate samples from a Gaussian distribution distribution with mean $\mu$ and covariance matrix $\Sigma + \sigma^2I$, where $I$ denotes the identity matrix and $\sigma^2$ is the noise variance set to 1 Are these methods equivalent?


title: Quantifying heterogeneity

text: I have been trying to quantify the heterogeneity of a given variable. I'd like to say that the distribution in calcite concentrations, for example, is more heterogeneous in one rock formation than another. So far I've graphed and compared the variable distributions with overlain histograms and box plots for multiple variables, and by and large this is fine, but it takes time on the part of the reader to interpret the plots. I've been looking for more direct comparisons, say bar charts comparing an appropriate measure of sample variability in a formation e.g. the standard deviation, interquartile range, coefficient of variation, mean or median absolute deviation etc. But for various reasons I'm not satisfied with these approaches. I can't help but think there must be a neater, succinct way of quantifying heterogeneity/variability _Reply To Comments:_ I'm investigating the residual error between some predicted and observed values. The residual errors for rock A are tightly distributed and for rock B they are widely non-normally distributed. I'm trying to tie the variability observed in the error, with variability observed in other rock properties. Both rocks A and B are variable in different respects, and I’m trying to highlight the variability that might matter. So I’ve been looking at ways of characterising the variability with a single value, say plotting all the IQR’s for each variable next to each other on a bar chart for example, where one might infer that rock A is generally more variable than rock B. My problems is this: variables like porosity are generally always normally and tightly distributed. Permeability by contrast is always non-normal, heavily skewed, varies over 6 or 7 orders of magnitude and typically follows a power trend. For permeability I’m not happy that an IQR really captures the variability, and secondly IQRs for different variables don’t graph well next to each other given the difference in magnitude of the parent unit. I’ve looked at normalised measures of variability, but these fudge the values with mean or medians that I don’t feel reflect the data. Perhaps there is no ideal solution for such varied distribution styles and I should just present their distributions….?


title: Independence assumption in maximum entropy models in NLP

text: I am reading Klein and Manning's notes on Maximum Entropy for Natural Language Processing. On slide 22, they have an equation saying, $P(C|D,\lambda) = \Pi _{(c,d)\in (C,D)} P(c|d,\lambda)$. I am not sure how they obtain this or what it really means. In the coursera video on this topic, Manning says that this follows because the random variables are independent and identically distributed. However, I thought, Maximum Entropy methods are prefered over Naive Bayes because they don't make the "naive" independence assumptions. If someone has a reference for this equation, that will be great.


title: adaptIntegrate in r

text: Does anyone know if adaptIntegrate accepts a vectorized integrand? for e.g. alpha<-c(1,2) f <- function(z){ (z[1]+z[2])*alpha } adaptIntegrate(f,lower=c(1, 3), upper=c(2, 4),tol=0.01) ##does not work Is what I want possible with adaptIntegrate? Thanks in advance.


title: Fitting a surface to 3D data

text: I am building a program to predict the score in a game of twenty20 cricket. I have a series of 3D datapoints, calculated from a number of games I have data for. Along the x axis we have number of wickets a team has lost. Up the y axis we have number of runs they end up scoring. And in the z axis we have number of overs into the innings we are. For example, one datapoint might say, we are 10 overs into the game, the team are 2 wickets down: They end up scoring 160 runs. From this data I want to be able to plug in how many overs into a game we are and how many wickets a team has lost, and get how many runs a team are predicted to score. I have plotted the data here (where different colour lines refer to different overs in the z axis): ![Different colour lines refer to different number of overs in](http://i.stack.imgur.com/8SOuK.jpg) As you may or may not be able to tell, in the middle of the data the lines correlate pretty well, this is because we have far more data for "average" games (e.g. where a team is 4 wickets down after 10 overs). On the edges however the data is based on very few games (e.g. where a team is no wickets down at the 19th over), where a data point seems obviously wrong to me (an outlier), I have added a dashed line to correct it. The question is, how do I build a smooth surface which fits these points? And how do I minimize the effect of the outliers? Is there a specific name for this kind of thing? EDIT: Now that I think about it, it's not really outliers i'm getting on the edges, just less reliable data.


title: Exploratory analysis: regression model with mutually correlated predictors to explain a dichotomous outcome?

text: I am attempting to explain a dichotomous outcome variable using a large set of continuous valued sensor-derived variables. Many of these variables are highly mutually correlated, some are based on solid physical concepts (and therefore easier to interpret) others are more abstract and difficult to explain. I am attempting to examine the association of these sensor variables with the outcome (i.e. which variables or combination of variables can best explain the dichotomous outcome variable?) a secondary objective is to determine if a subset of these variables can be used to explain the dichotomous outcome. I wish to avoid using a stepwise fitting procedure (due to the danger of overfitting and, a feeling that the more easily interpretable variables should be given preference in the model over highly correlated but less easily interpretable variables). In short, I am looking for the true associations rather than noisy surrogates using correlated but less globally informative variables. To avoid multicollinearity in the analyses, I reduced the number of variables using logistic regression by block analysis. Sensor derived variables were grouped by type into blocks. The dichotomized outcome variable was used as the dependent variable in each sub-group. Working with each block, I performed a logistic regression on each independent variable and only those which were significant (α < 0.05) were retained in each block. Through this procedure I excluded all non significant variables from the analyses for the final model. I then generated a final logistic regression model using the results of each sub-group analysis. I would great appreciate opinions as to whether this is a valid approach? Can the odds ratios from the final logistic regression model can be used to interpret the associations of the included sensor variables with the outcome variable in the manner of a hypothesis test?


title: How can I a "multiplier effect" in time series data?

text: I currently have data corresponding to how often a certain set of songs were downloaded. Each song has a release date, and then the number of downloads per day going forward to today. It would look like:                                               Song1    Cumulative Sum of Song1     Release Date   5/7/2013         20       20                    5/8/2013         40       60                    5/9/2013         55       115                    .................................                                                    Song2    Cumulative Sum of Song2     Release Date   7/9/2013          13       13                    7/10/2013         44       57                    7/11/2013         92       149                    .................................      I have about 500 songs total with data just like this. I am curious to see if I can describe some sort of a "multiplier" effect for each song. What I mean by this is if the customers would be somehow influenced by how many downloads there currently are. Hence, if it is a positive influence, we should see more downloads each day as the cumulative sum gets larger. Would there be a model seeing what "multiplier" effect the cumulative sum column has on the song downloads for a next day? I tried to think about something like a logit model, but it works only for a binary dependent variable and a standard linear regression seems too easy. Any ideas or advice would be GREATLY appreciated! Thanks!


title: What are the state of the art methods for person segmentation and person pose identification

text: I have tried asking this on another forum, but after 4 months there are no posted answers, so I am asking it here: What is the current state of the art approach for determining the pose of a person (including close poses like a head shot, and hand shot, etc...), and for segmenting the person from the background? The setting here is single still images.


title: Multiple t-tests against a single control condition. Is this correct?

text: I have a control condition **A** and several other conditions ( **B** , **C** , **D** , etc.) against which I want to test **A** for differences (i.e. **A** vs **B** , **A** vs **C** , **A** vs **D** , etc.). Is it correct to run: > ttest( **A** , **B** ); > > ttest( **A** , **C** ); > > ttest( **A** , **D** ); and treat them separetely? Thank you very much.


title: How do we perform a two-tailed z-test when we *do* know the population mean of the intervention sample?

text: I am in the process of working through the Udacity course Intro to Inferential Statistics. In Lesson 9: Hypothesis Testing, the following example is given. Say that we record survey data from all students of this Udacity course detailing how engaged they are by the lecture materials. We find that the population mean engagement level (on a scale of 1-10) is 7.47 with a standard deviation of 2.41. The instructor decides to sing a song in one of the lectures to try to spark engagement, and we want to know if the average engagement level changes once the song has been implemented. We collect a sample size of n=30 and find it to have $\bar{X} = 8.3$. We are asked to perform a two-tailed z-test with $\alpha = 0.05$ to find if the mean is affected by the song. Additionally, we are given the information that $\mu_{song} = 7.8$, i.e. that the population of all students who listened to the song had an average engagement level of 7.8. We are asked to find if the z-test provides a result that accurately reflects the population, and if not, whether a type I or type II error occurred. I performed a z-test on the sample and came to a z-score of 1.88, which is less than the critical value of 1.96. Since $\mu_{song} = 7.8$ which is greater than $\mu = 7.47$, I determined that we fail to reject the null hypothesis when it is false (since $\mu_{song} > \mu$) and therefore that type-II error is occurring. However, this answer was returned as incorrect. Instead, the instructor explained that a z-score needs to be calculated for $\mu_{song}$ as well, equivalent to $\frac{\mu_{song} - \mu}{\frac{\sigma}{\sqrt n}} = \frac{7.8 - 7.47}{\frac{2.41}{\sqrt 30}} = .75$. Since .75 does not fall beyond the critical value of 1.96 at $\alpha=0.05$, $\mu_{song}$ is not significantly different from $\mu$, supporting the null hypothesis. Therefore, we fail to reject the null hypothesis when it is indeed correct. I don't understand this answer. I would have believed that since $\mu_{song} \neq \mu$, that in reality we _should_ be rejecting the null hypothesis (that they are equal), and that type-II error was occurring. Additionally, we are never given a sample size for the population, so I'm not sure where n=30 came from. Could someone clarify the correct approach to a problem like this? If you are in a situation where you know the population parameters for the intervened population, does an additional z-score need to be calculated? Or is the fact that the intervention mean is different from the original population mean enough evidence that the null hypothesis ($\mu = \mu_{I}$) is false? If you are interested in trying out the question yourself for better understanding, you can find it here.


title: Logistic regression when a covariate is measured with known but heterogenous measurement error

text: Let $y$ be a binary variable and $x$ be a covariate measured with known heterogeneous measurement error: $x_{i}$ ~ $\operatorname{Normal}(x^{*}_{i},\sigma_{i}^{2})$. Now suppose the relationship with the true covariate $x^{*}$ is $\ Y$ ~ $\operatorname{Bernoulli}(p_i) $ and $\ \operatorname{logit}( p_i )=\alpha+\beta*x_{i}^{*}$. Theoretically, it seems clear to me that for forecasting purposes, you'd want a model $\ \operatorname{logit}( p_i )=\alpha+f(\sigma_{i})*x_{i}$, where $f(\sigma_{i}) \to 0 $ as $\sigma_{i}$ gets large and $f(\sigma_{i}) \to \beta $ as $\sigma_{i}$ gets small. Is there a good functional form for $f(\sigma_{i})$? It feels like a simple enough problem that there might be, but I haven't been able to come up with anything.


title: Pattern Recognition in Practical Problem in R

text: I am using R for last 4 months with basic test analysis, regression models, ANOVA etc. Now I am looking into a particular problem. In words problem can be stated like, > A particular machine got a problem 'ABC', what will be the probability (with > certain statistical significance - pvalue) of getting problem 'XYZ'. I have recreated data for reference:               ESN,BUILDDATE,FAILEDDATE,FAILCODE     XYZ123,06-01-2010,07-12-2010,CCNI     XYZ123,06-01-2010,07-12-2010,FYDM     XYZ123,06-01-2010,25-01-2011,FBDD     XYZ123,06-01-2010,25-01-2011,FRQF     XYZ123,06-01-2010,25-01-2011,YYJQ     XYZ123,06-01-2010,17-03-2011,CCIT     XYZ123,06-01-2010,03-11-2011,FRQF     XYZ123,06-01-2010,03-11-2011,ZOFC     PBC221,06-01-2010,25-05-2011,FRQF     PBC221,06-01-2010,25-05-2011,JSDM     PBC221,06-01-2010,04-08-2011,FBPS     PBC221,06-01-2010,04-08-2011,IZBW     PBC221,06-01-2010,17-10-2011,FRIE     JKL343,06-01-2010,10-06-2010,FUJC     JKL343,06-01-2010,08-10-2010,CWHS     JKL343,06-01-2010,12-11-2010,FJDN     JKL343,06-01-2010,12-11-2010,FYDO     JKL343,06-01-2010,10-09-2011,FRQF     JKL343,06-01-2010,05-10-2011,VTEU     JKL343,06-01-2010,09-01-2012,FUDO      I want to prepare either probability chart or classification chart based on this type of data. Model should give results like probability of CCIT error after CCNI should be 0.xxx. What I have tried till now is 1) Normal conditional probability of CCIT/CCNI - But that I did by dividing number of events, which gave me point probability without and significance values. 2) I used arules package to find out patterns. I am using following code for this:               train<-split(claims$FAILCODE,claims$M_No)     listData <- list()     for (i in 1:length(train)) {         listData[[i]] <- as.character(train[[i]][!duplicated(train[[i]])]) #Line-A     }     trnsData <- as(listData,"transactions")     rules<-apriori(trnsData,parameter= list(support=.001,confidence=0.1,minlen=1))      This code is giving me answer in line with my requirements, but its not taking duplicates, as in for a single machine if a particular error happen twice algorithm is not accepting it in input. (in above code line mentioned with comment 'Line-A' is removing duplicate failcode.) So with this method I have two question: a) How to check impact of a one type of fail multiple times? b) Does this algorithm consider any sequence while preparing output? I mean when I am getting following output, it should be based on faildate. i.e. FBDD & FEQF should have happen before YYJQ.                                   rules     support confidence     lift       1: {FBDD,FEQF} => {YYJQ} 0.001039655  0.2333333 1.018168       2:      {YYJO} => {FRQF} 0.001039655  0.3684211 1.058719      I may be using wrong approach or using a package with wrong settings so please let me know how can I find solution of this problem. Some important facts:   1. I have more than 800 types of failcode    2. Particular machine might have failed 30-40times with same or different failcode.


title: lagged covariate or time series for analyzing change over time in Hgb (limited observations) in the prediction of disease status

text: I want to test whether changes in Hemoglobin (Hgb) levels over time can help diagnose Myelodysplastic syndrome (MDS). I have a cohort of thousands of patients. Each patient has several Hgb measurements over a period of 3 years. Measurements are unevenly spaced and few (anything from two to 4 measurements). My questions: 1. Is a time series the correct model for me considering I have so few observation over time for each subject ? if not how do I analyze the trend ? 2. If I should use a time series, I thought of tagging each lab measurement by its year so as to make the measurements evenly spaced (this seems sensible enough from a physiological perspective). Is this valid ? Is there a better way to handle the inequality of measurement spacing ? 3. What do I use to compare trends between two groups ? I would be most obliged if you could refer me to the appropriate further reading Thank you !


title: data prediction by regression or better ways

text: I am working on data prediction. Given data of a random variable $X$ and $Y$, find out how to predict $Y$ from $X$. I know how to do it by linear regression, $\hat{Y} = kX + b$. But, here, $X$ is always non-negative and $Y$ is required to be non-negative. Sometimes, $b$ is not non-negative so that $y < 0$. How to assure that $b > 0$ and also minimize the prediction error? Are there other better ways (not regression) to do the prediction ?


title: How to improve simulation efficiency of method of simulated moments with random coefficient?

text: I have a model with two layers. There are 4 variables with random coefficients in the second layer of the model, which is a multivariate Logit. I assume the four random coefficients follow multivariate normal distribution with different variances. Some of them even have nonzero co-variances. The values generated/simulated from the second layer are going to be put in the functions in the first layer, which is a dynamic model. The moment is the difference between real data and the product of values generated from functions of both two layers. I know direct Monte Carlo simulations do not work for this problem because the efficiency is going to be low for such a problem. It may provides poor coverage and the number of draws will have to be very large to achieve a good approximation. However, I don't think other algorithms such as MCMC can work for my problem because I have a static model with random coefficients within a dynamic model. What's more, I also have to draw another set of errors to simulate households' behavior with the simulated parameters. I also look into Laplace approximation and also find hard to apply in my situation. I wonder how I simulate efficiently in such a problem. Thank you all!


title: Basic statistical equation

text: How would I go about showing that: $$\sum_{i=1}^n (X_i-\bar{X})^2=\sum_{i=1}^n X_i^2-n\bar{X}^2\,,$$ (where $\bar{X}=\frac{1}{n}\sum_{i=1}^n X_i$)? Thank you for your help!


title: Change in proportion - two timepoints

text: In the first time point - 75% (485 out of 646) of boys and 63% (350/555) of girls are in group A. In the second time point - 80% (480/600) of boys and 45% (225/500) of girls are in group A. So that is a difference of 5% for boys and 18% for girls between the two time points: ![Table][1] I was hoping someone could point me in the right direction as to what test I should use to compare this change (5% versus 18%) between two timepoints. Note that the boys and girls in the first timepoint are not the same ones in the second timepoint. Is this percent change assumed to be normally distributed?


title: How to tell if a sample comes from a distribution with infinite mean?

text: I observe a sample from a distribution that I expect to be the hitting time $$\tau = \inf\{t>0| X(t)>a\}$$ where $X(t)$ is a Lévy process with $X(0)=0$ and $a$ is some constant. $X$ is not a Brownian motion and the experimental fit to the Lévy distribution is poor. However, I do not need to know the exact formula for the law of $\tau$. For my needs I only need to know that the expectation of $\tau$ is infinite (as in the case of tau for a Brownian motion). Is it possible to formulate and test this as a statistical hypothesis?


title: Binary classification vs. continuous output with neural networks

text: Wikipedia says in binary classification: > Tests whose results are of continuous values, such as most blood values, can > artificially be made binary by defining a cutoff value, with test results > being designated as positive or negative depending on whether the resultant > value is higher or lower than the cutoff. Is there some guidance as to whether this is a desirable thing to do or not? I have data where the output value is continuous in the training set and I'm interested to know how strong the output variable is. Ideally an accurate continuous value would be the best, but I also would be satisfied with binary classification. My layman's assumption is that the binary classification task would be a little simpler. Is there any guidance as to whether to prefer continuous output vs binary classification?


title: Overfitting with pLSA

text: I'm having some trouble with overfitting in pLSA (Probabilistc Latent Semantic Analisys). I'll use the following notation:   * $t \in T$: topic, the latent variables   * $d \in D = D_{train} \cup D_{test}$: documents   * $w \in W$: terms Each document $d$ is a vector of $|W|$ elements and $d[i] = 1$ if $w_i \in W$ appears in $d$. **Training:** I apply the EM algorithm on a training set and i estimate the probabilities $P( w | t), \forall w, t$. **Testing:** Using EM algorithm, for each document $d' \in D_{test}$ i compute $P( t | d')$; then i rebuild the document with: $$d'_{predicted}[w] = \sum_{t \in T} P(w | t)P(t | d)$$ Now i can compute _False Positive Rate_ and _False Negative Rate_ and the value of an _error_ function (let's say the area under a ROC curve). I have noticed that if i increment the number of topic the error keep to decrease, so it is like if i have some overfitting but in the validation set. Since in the validation phase i estimate some parameter with EM algorithm, is it possible that if the number of latent variables (topics) is overestimated i fit also the noise?


title: Can I use Mean values of likert resposes instead of factor scores in case of skewed values?

text: I have five point likert scale responses ((1-most imp, 5-least imp) for 14 items that I want convert them into 5 factors with confirmatory factor analysis. I already have designed which item should go into what factor, so I can't use exploratory FA. Once I get the factor scores, I will use them in a logistic regression. I have the problem of skewed responses as well, with more people responding for certain facilities to be more important in their neighborhood. My question is, shall I go for CFA, or shall I just calculate the mean value for the item responses within each amenity (e.g. Public transportation-bus, metro) and use these mean values in the logistic regression? Also for CFA, Can I use SPSS adding only the items that will be in that factor by forcing it to extract only one factor? If so, which method should I use-PC, ULS, GLS, ML, PAF? Thanks, aruna


title: Detecting convergence in Random walk

text: I am trying to detect convergence of a random walk on a graph. After doing some preliminary research, the Geweke convergence diagnostic seems to be most commonly used for this. This diagnostic calls for the z-score of the chain at each iteration of the walk and declares convergence when the z-score falls within an interval of [-1, 1]. The z-score is calculated thusly: $$ z=\frac{E(X_a)-E(X_b)}{\sqrt{Var(X_a)+Var(X_b)}} $$ where $X_a$ and $X_b$ are two subsequences of the current chain, typically the first 10% and the last 50% of the chain. My dataset is a dense crawl of a social network where each node is a user. My question is: in this context, what would the expected values of the subchains $X_a$ and $X_b$ be? Would it be the lengths, or a function of the lengths and degrees of the nodes or something else? Thank you!!


title: Explain steps of LLE (local linear embedding) algorithm?

text: I understand the basic principle behind the algorithm for LLE consists of three steps.   1. Finding the neighborhood of each data point by some metric such as k-nn.   2. Find weights for each neighbor which denote the effect the neighbor has on the data point.   3. Construct the low dimensional embedding of the data based on the computed weights. But the mathematical explanation of steps 2 and 3 are confusing in all the text books and online resources that I have read. I am not able to reason why the formulas are used. How are these steps performed in practice? Is there any intuitive way of explaining the mathematical formulas used? References: http://www.cs.nyu.edu/~roweis/lle/publications.html


title: Finding the moments for the Mauchly sphericity statistic

text: Suppose we have $V \sim W_p(n-1,\sigma^2 I)$ In order to find the moments of the Mauchly sphericity statistic $\Lambda = \frac{|V|}{(tr(\frac{V}{p}))^p}$ i read one can split it defining $\Lambda_1 = \frac{|V|}{v_{11}...v_{pp}}$ and $\Lambda_2 = \frac{v_{11}...v_{pp}}{(tr(\frac{V}{p}))^p}$ and then using $E[\Lambda^h] = E[\Lambda_1^h]E[\Lambda_2^h]$ (where they are using independence of $\Lambda_1$ and $\Lambda_2$). This statistics $\Lambda_1$ and $\Lambda_2$ are, respectively, the LR criterion for equality of variances and for independence, so its moments may me already calculated. My question is: how can we see that this two statistics are independent?


title: "Importance" metric for discrete variable value

text: (I am far from an expert in the field of statistics, so I apologize beforehand if my question is irrelevant or my use of any terms is incorrect) Let's suppose that we have a _discrete_ variable $X$, and a set of $n+1$ observations $\{x_0, x_1, \dots, x_n\}$. The values of $X$ may not be numeric and all observations are considered equal with no particular order. Therefore, we can transform the observation set above to a set of $k+1$ value/frequency pairs $\{<v_0, f_0>, <v_1, f_1>,\dots,<v_k, f_k>\}$. Is there a metric that estimates the "importance" of a specific _value_ in this set? For example, in an observation set where all values exist exactly three times, none can be considered more important than the others. On the other hand, in a set where all but one values exist 3 times, I would want that one value that exists 10 times to be differentiated from the rest. Same with a value that only appears once. A simple probability calculation, for example, $P(k) = f_k / (n+1)$ would not be of help, because it does not take into account the other values. I experimented with various combinations/formulas using the various layman- known metrics (mean, standard deviation, maximum, minimum etc), but anything I came up with seemed too arbitrary for me to trust. I am currently using the standard score of the value frequency as an estimator, but it has a major issue: it's not bounded, and I am not at all sure how to normalize it without unknowingly biasing any results. I would appreciate any reference to a metric that might get me started, or to any terminology that describes what I'd like to do more accurately, so that I can focus my search. I would especially appreciate any bounded metrics that can be computed incrementally as new samples are added to the observation set :-)


title: Creating a uniform prior on the logarithmic scale

text: A uniform prior for a scale parameter (like the variance) is uniform on the logarithmic scale. What functional form does this prior have on the linear scale? And why so?


title: Is automatic linear modelling in SPSS a good or bad thing?

text: In SPSS Version 19 there seems to be a new feature called Automatic Linear Modelling. It creates a 'Model' (which is new to me) and the function seems to combine a number of the functions that is typically required for prediction model development. The functionality seems incomplete with only a subset of prediction selection techniques and most notable it's missing Backwards step wise. ### QUESTIONS   * Do people see this as good or evil?    * And if 'good' then are there ways to decompose what it is doing?    * Specifically how do I find the regression equation co-efficients when bagging or boosting?  To me it seems to hides a lot of steps and I'm not exactly sure how it's creating what it presents. So any pointers to tutorials or the like (as the SPSS documentation isn't great) is appreciated.


title: How to calculate margin of error in linear regression?

text: As a basic question in Regression Analysis, I wanted to ask how can I calculate Margin of Error when I fit a straight line to a set of data. Assume that I have variation of parameter $A$ as a function of parameter $B$, then $A=mB \pm e$, where $m$ is the tangent of the fitted straight line and $e$ is what i'm looking for! Part of my data: {{39.7678, 2320.3}, {30.8438, 1614.21}, {125.846, 3078.81}, {55.2345, \ 1947.98}, {22.0671, 972.995}, {30.1827, 701.99}, {29.5734, 837.784}, \ {24.6913, 1134.23}, {27.2493, 918.887}, {62.7684, 4535.07}, {101.449, \ 5499.83}, {125.248, 6513.04}, {187.409, 6257.72}, {174.138, 5243.63}, \ {120.747, 3768.02}, {84.178, 3453.12}, {60.2404, 3075.15}, {63.8622, \ 3517.73}, {101.9, 7240.11}, {90.6265, 5706.74}, {100.897, 7353.84}, \ {159.316, 9867.36}, {109.798, 11471.2}, {104.311, 6924.54}, {82.7057, \ 6339.06}, {140.205, 6555.52}, {173.469, 8644.27}, {138.432, 9655.86}, \ {95.2955, 5643.33}, {64.563, 3848.77}, {50.7936, 4733.24}, {34.776, 0. - 2707.89 I}, {25.3775, 6158.}}


title: How to draw an ROC curve?

text: If I have values for `sensitivity` and `specificity` for a group of studies, for example like this:               sensitivity     specificity     -----------     -----------            66.3             74.7       87.2             65.9       56.4             76.4       79.5             94.3      How can I make an `ROC` curve for such values? Do I divide them by `100` for example and just put the values in the curve straightforward? Or, how exactly does the process work out?


title: Linear/non-linear mixed effects models

text: Does anyone know how to use (nlme)? I'm having a lot of problems using it for something that should be really simple, any help would be really appreciated. I won't put up the details until I know someone knows what I'm talking about. Data was collected by a field study concerned planting pattern, genotype, we replicated this study for 3 years and in each growing season we surveyd 10 times. My question is how to interpret the results best? A professor advised using linear mixed effect model with year as a random factor, to reduce the number of separate analyses and type I error. Anybody who knows, please do me a favor. The following is my data               site    YEAR    GENOTYPE    PATTERN T   density     1   1   G   S   T1  3.0035      2   1   G   S   T1  3.0538      3   1   G   S   T1  2.9939      4   1   G   S   T1  2.8506      5   1   G   M   T1  2.7316      6   1   G   M   T1  2.7789      7   1   G   M   T1  2.8791      8   1   G   M   T1  2.7419      9   1   S   S   T1  0.6024      10  1   S   S   T1  0.6079      11  1   S   S   T1  0.6014      12  1   S   S   T1  0.5855      13  1   S   M   T1  0.5719      14  1   S   M   T1  0.5774      15  1   S   M   T1  0.5887      16  1   S   M   T1  0.5731      1   2   G   S   T1  3.3064      2   2   G   S   T1  3.2896      3   2   G   S   T1  3.3448      4   2   G   S   T1  3.4615      5   2   G   M   T1  3.4104      6   2   G   M   T1  3.3243      7   2   G   M   T1  3.3477      8   2   G   M   T1  3.4031      9   2   S   S   T1  3.3660      10  2   S   S   T1  3.3025      11  2   S   S   T1  3.3760      12  2   S   S   T1  3.3038      13  2   S   M   T1  3.4237      14  2   S   M   T1  3.5357      15  2   S   M   T1  3.5001      16  2   S   M   T1  3.4195      1   3   G   S   T1  2.2068      2   3   G   S   T1  2.1492      3   3   G   S   T1  2.0969      4   3   G   S   T1  2.3010      5   3   G   M   T1  1.7404      6   3   G   M   T1  1.8388      7   3   G   M   T1  1.5185      8   3   G   M   T1  1.6128      9   3   S   S   T1  2.3636      10  3   S   S   T1  2.5092      11  3   S   S   T1  2.0828      12  3   S   S   T1  2.4065      13  3   S   M   T1  1.9243      14  3   S   M   T1  2.0645      15  3   S   M   T1  2.0000      16  3   S   M   T1  2.0170      1   1   G   S   T2  3.6379      2   1   G   S   T2  3.5680      3   1   G   S   T2  3.6095      4   1   G   S   T2  3.6999      5   1   G   M   T2  3.4241      6   1   G   M   T2  3.5053      7   1   G   M   T2  3.4559      8   1   G   M   T2  3.4751      9   1   S   S   T2  0.6663      10  1   S   S   T2  0.6597      11  1   S   S   T2  0.6637      12  1   S   S   T2  0.6721      13  1   S   M   T2  0.6458      14  1   S   M   T2  0.6537      15  1   S   M   T2  0.6489      16  1   S   M   T2  0.6508      1   2   G   S   T2  4.5387      2   2   G   S   T2  4.5442      3   2   G   S   T2  4.5174      4   2   G   S   T2  4.5464      5   2   G   M   T2  4.3512      6   2   G   M   T2  4.4067      7   2   G   M   T2  4.4090      8   2   G   M   T2  4.1965      9   2   S   S   T2  4.5641      10  2   S   S   T2  4.5103      11  2   S   S   T2  4.6466      12  2   S   S   T2  4.5205      13  2   S   M   T2  4.4246      14  2   S   M   T2  4.3424      15  2   S   M   T2  4.4487      16  2   S   M   T2  4.3642      1   3   G   S   T2  3.0099      2   3   G   S   T2  3.0730      3   3   G   S   T2  2.9657      4   3   G   S   T2  3.1807      5   3   G   M   T2  2.6812      6   3   G   M   T2  2.3729      7   3   G   M   T2  2.8820      8   3   G   M   T2  2.4800      9   3   S   S   T2  3.1364      10  3   S   S   T2  3.0839      11  3   S   S   T2  2.9036      12  3   S   S   T2  2.9850      13  3   S   M   T2  2.7412      14  3   S   M   T2  2.8202      15  3   S   M   T2  2.3692      16  3   S   M   T2  2.5866      1   1   G   S   T3  3.3762      2   1   G   S   T3  3.3531      3   1   G   S   T3  3.5340      4   1   G   S   T3  3.4096      5   1   G   M   T3  3.4263      6   1   G   M   T3  3.1959      7   1   G   M   T3  3.2973      8   1   G   M   T3  3.1915      9   1   S   S   T3  0.6411      10  1   S   S   T3  0.6388      11  1   S   S   T3  0.6565      12  1   S   S   T3  0.6444      13  1   S   M   T3  0.6460      14  1   S   M   T3  0.6228      15  1   S   M   T3  0.6332      16  1   S   M   T3  0.6224      1   2   G   S   T3  5.4512      2   2   G   S   T3  5.3256      3   2   G   S   T3  5.5334      4   2   G   S   T3  5.3601      5   2   G   M   T3  4.9647      6   2   G   M   T3  5.0390      7   2   G   M   T3  5.0628      8   2   G   M   T3  5.0739      9   2   S   S   T3  5.3199      10  2   S   S   T3  5.2441      11  2   S   S   T3  5.2672      12  2   S   S   T3  5.3007      13  2   S   M   T3  5.1314      14  2   S   M   T3  5.0120      15  2   S   M   T3  5.1177      16  2   S   M   T3  5.0052      1   3   G   S   T3  2.8808      2   3   G   S   T3  3.0839      3   3   G   S   T3  2.7931      4   3   G   S   T3  2.8555      5   3   G   M   T3  2.6902      6   3   G   M   T3  2.7723      7   3   G   M   T3  2.8293      8   3   G   M   T3  2.9299      9   3   S   S   T3  2.5911      10  3   S   S   T3  2.5587      11  3   S   S   T3  2.7818      12  3   S   S   T3  2.7332      13  3   S   M   T3  2.8414      14  3   S   M   T3  2.7024      15  3   S   M   T3  2.6955      16  3   S   M   T3  2.9365      1   1   G   S   T4  2.9309      2   1   G   S   T4  2.9186      3   1   G   S   T4  3.0249      4   1   G   S   T4  3.0910      5   1   G   M   T4  2.9745      6   1   G   M   T4  2.9538      7   1   G   M   T4  3.1179      8   1   G   M   T4  3.0588      9   1   S   S   T4  0.5945      10  1   S   S   T4  0.5931      11  1   S   S   T4  0.6048      12  1   S   S   T4  0.6118      13  1   S   M   T4  0.5993      14  1   S   M   T4  0.5970      15  1   S   M   T4  0.6147      16  1   S   M   T4  0.6084      1   2   G   S   T4  5.3077      2   2   G   S   T4  5.4044      3   2   G   S   T4  5.4797      4   2   G   S   T4  5.4220      5   2   G   M   T4  5.1803      6   2   G   M   T4  5.2498      7   2   G   M   T4  5.3166      8   2   G   M   T4  5.2693      9   2   S   S   T4  5.7061      10  2   S   S   T4  5.4886      11  2   S   S   T4  5.5974      12  2   S   S   T4  5.6148      13  2   S   M   T4  5.3078      14  2   S   M   T4  5.1670      15  2   S   M   T4  5.1887      16  2   S   M   T4  5.2186      1   3   G   S   T4  4.1525      2   3   G   S   T4  4.1609      3   3   G   S   T4  4.1169      4   3   G   S   T4  4.1687      5   3   G   M   T4  3.9305      6   3   G   M   T4  3.9376      7   3   G   M   T4  3.9668      8   3   G   M   T4  3.9696      9   3   S   S   T4  4.1293      10  3   S   S   T4  4.1358      11  3   S   S   T4  4.1450      12  3   S   S   T4  4.1527      13  3   S   M   T4  3.8382      14  3   S   M   T4  3.8435      15  3   S   M   T4  3.8405      16  3   S   M   T4  3.8439      1   1   G   S   T5  3.2206      2   1   G   S   T5  3.0726      3   1   G   S   T5  3.0228      4   1   G   S   T5  3.1284      5   1   G   M   T5  2.6253      6   1   G   M   T5  2.7760      7   1   G   M   T5  2.7774      8   1   G   M   T5  2.7388      9   1   S   S   T5  0.6254      10  1   S   S   T5  0.6099      11  1   S   S   T5  0.6045      12  1   S   S   T5  0.6158      13  1   S   M   T5  0.5593      14  1   S   M   T5  0.5770      15  1   S   M   T5  0.5772      16  1   S   M   T5  0.5727      1   2   G   S   T5  3.6946      2   2   G   S   T5  3.7158      3   2   G   S   T5  3.6597      4   2   G   S   T5  3.6553      5   2   G   M   T5  3.5065      6   2   G   M   T5  3.3351      7   2   G   M   T5  3.4609      8   2   G   M   T5  3.3047      9   2   S   S   T5  3.5875      10  2   S   S   T5  3.4774      11  2   S   S   T5  3.5924      12  2   S   S   T5  3.5771      13  2   S   M   T5  3.2695      14  2   S   M   T5  3.3753      15  2   S   M   T5  3.2916      16  2   S   M   T5  3.2804      1   3   G   S   T5  3.6772      2   3   G   S   T5  3.6558      3   3   G   S   T5  3.6118      4   3   G   S   T5  3.6636      5   3   G   M   T5  3.3967      6   3   G   M   T5  3.4834      7   3   G   M   T5  3.3992      8   3   G   M   T5  3.6731      9   3   S   S   T5  3.6242      10  3   S   S   T5  3.6307      11  3   S   S   T5  3.6396      12  3   S   S   T5  3.6069      13  3   S   M   T5  3.4087      14  3   S   M   T5  3.4874      15  3   S   M   T5  3.3094      16  3   S   M   T5  3.6386      1   1   G   S   T6  2.7332      2   1   G   S   T6  2.6684      3   1   G   S   T6  2.8573      4   1   G   S   T6  2.6365      5   1   G   M   T6  2.2788      6   1   G   M   T6  2.4624      7   1   G   M   T6  2.4728      8   1   G   M   T6  2.3365      9   1   S   S   T6  0.5721      10  1   S   S   T6  0.5645      11  1   S   S   T6  0.5863      12  1   S   S   T6  0.5607      13  1   S   M   T6  0.5157      14  1   S   M   T6  0.5394      15  1   S   M   T6  0.5407      16  1   S   M   T6  0.5233      1   2   G   S   T6  3.0955      2   2   G   S   T6  3.0941      3   2   G   S   T6  3.1123      4   2   G   S   T6  3.0821      5   2   G   M   T6  2.8195      6   2   G   M   T6  2.8609      7   2   G   M   T6  2.9085      8   2   G   M   T6  2.9745      9   2   S   S   T6  3.0465      10  2   S   S   T6  2.9566      11  2   S   S   T6  3.0422      12  2   S   S   T6  2.9360      13  2   S   M   T6  2.9494      14  2   S   M   T6  2.9652      15  2   S   M   T6  2.9309      16  2   S   M   T6  3.0000      1   3   G   S   T6  2.9079      2   3   G   S   T6  2.8768      3   3   G   S   T6  2.7882      4   3   G   S   T6  2.9689      5   3   G   M   T6  2.7259      6   3   G   M   T6  2.6875      7   3   G   M   T6  2.5340      8   3   G   M   T6  2.6170      9   3   S   S   T6  2.0934      10  3   S   S   T6  2.0043      11  3   S   S   T6  2.0043      12  3   S   S   T6  2.2201      13  3   S   M   T6  2.3927      14  3   S   M   T6  2.4786      15  3   S   M   T6  2.5453      16  3   S   M   T6  2.4928      1   1   G   S   T7  2.7459      2   1   G   S   T7  2.6503      3   1   G   S   T7  2.6222      4   1   G   S   T7  2.7143      5   1   G   M   T7  2.7348      6   1   G   M   T7  2.6739      7   1   G   M   T7  2.7505      8   1   G   M   T7  2.6053      9   1   S   S   T7  0.5736      10  1   S   S   T7  0.5623      11  1   S   S   T7  0.5590      12  1   S   S   T7  0.5699      13  1   S   M   T7  0.5723      14  1   S   M   T7  0.5651      15  1   S   M   T7  0.5741      16  1   S   M   T7  0.5569      1   2   G   S   T7  2.6702      2   2   G   S   T7  2.5635      3   2   G   S   T7  2.6946      4   2   G   S   T7  2.7118      5   2   G   M   T7  2.1614      6   2   G   M   T7  2.1004      7   2   G   M   T7  2.0792      8   2   G   M   T7  2.3010      9   2   S   S   T7  2.2175      10  2   S   S   T7  2.3892      11  2   S   S   T7  2.3284      12  2   S   S   T7  2.3010      13  2   S   M   T7  2.3222      14  2   S   M   T7  2.1139      15  2   S   M   T7  2.1461      16  2   S   M   T7  2.0128      1   3   G   S   T7  2.5328      2   3   G   S   T7  2.5623      3   3   G   S   T7  2.4624      4   3   G   S   T7  2.6075      5   3   G   M   T7  1.7709      6   3   G   M   T7  1.5315      7   3   G   M   T7  1.6902      8   3   G   M   T7  1.6021      9   3   S   S   T7  2.4786      10  3   S   S   T7  2.3729      11  3   S   S   T7  2.5809      12  3   S   S   T7  2.6191      13  3   S   M   T7  1.8388      14  3   S   M   T7  2.0969      15  3   S   M   T7  1.9590      16  3   S   M   T7  2.1271      1   1   G   S   T8  2.6180      2   1   G   S   T8  2.6405      3   1   G   S   T8  2.6385      4   1   G   S   T8  2.6875      5   1   G   M   T8  2.5092      6   1   G   M   T8  2.6107      7   1   G   M   T8  2.7185      8   1   G   M   T8  2.6964      9   1   S   S   T8  0.5585      10  1   S   S   T8  0.5612      11  1   S   S   T8  0.5609      12  1   S   S   T8  0.5667      13  1   S   M   T8  0.5452      14  1   S   M   T8  0.5576      15  1   S   M   T8  0.5704      16  1   S   M   T8  0.5678      1   2   G   S   T8  3.1355      2   2   G   S   T8  3.2087      3   2   G   S   T8  3.2117      4   2   G   S   T8  3.3015      5   2   G   M   T8  3.0580      6   2   G   M   T8  3.2695      7   2   G   M   T8  3.2269      8   2   G   M   T8  2.9881      9   2   S   S   T8  3.0569      10  2   S   S   T8  3.0035      11  2   S   S   T8  3.2405      12  2   S   S   T8  3.0402      13  2   S   M   T8  3.1294      14  2   S   M   T8  3.1623      15  2   S   M   T8  3.2095      16  2   S   M   T8  3.0645      1   3   G   S   T8  2.1430      2   3   G   S   T8  2.2175      3   3   G   S   T8  2.2788      4   3   G   S   T8  2.3118      5   3   G   M   T8  2.0607      6   3   G   M   T8  2.3617      7   3   G   M   T8  2.1430      8   3   G   M   T8  2.4928      9   3   S   S   T8  2.0043      10  3   S   S   T8  2.1335      11  3   S   S   T8  1.9085      12  3   S   S   T8  2.0645      13  3   S   M   T8  2.0645      14  3   S   M   T8  2.3096      15  3   S   M   T8  2.3784      16  3   S   M   T8  2.3404      1   1   G   S   T9  2.4265      2   1   G   S   T9  2.4886      3   1   G   S   T9  2.4393      4   1   G   S   T9  2.4166      5   1   G   M   T9  2.0569      6   1   G   M   T9  2.1106      7   1   G   M   T9  2.2833      8   1   G   M   T9  2.1553      9   1   S   S   T9  0.5349      10  1   S   S   T9  0.5426      11  1   S   S   T9  0.5365      12  1   S   S   T9  0.5336      13  1   S   M   T9  0.4853      14  1   S   M   T9  0.4928      15  1   S   M   T9  0.5163      16  1   S   M   T9  0.4990      1   2   G   S   T9  3.1258      2   2   G   S   T9  3.2368      3   2   G   S   T9  3.1498      4   2   G   S   T9  3.3566      5   2   G   M   T9  2.9956      6   2   G   M   T9  3.0888      7   2   G   M   T9  3.0265      8   2   G   M   T9  2.9279      9   2   S   S   T9  3.0892      10  2   S   S   T9  3.2253      11  2   S   S   T9  3.0531      12  2   S   S   T9  3.0212      13  2   S   M   T9  2.8176      14  2   S   M   T9  3.0170      15  2   S   M   T9  2.8899      16  2   S   M   T9  2.8993      1   3   G   S   T9  2.4928      2   3   G   S   T9  2.3979      3   3   G   S   T9  2.3617      4   3   G   S   T9  2.2279      5   3   G   M   T9  2.3324      6   3   G   M   T9  2.5705      7   3   G   M   T9  2.3892      8   3   G   M   T9  2.7738      9   3   S   S   T9  2.7839      10  3   S   S   T9  2.8537      11  3   S   S   T9  3.0527      12  3   S   S   T9  2.8904      13  3   S   M   T9  2.2355      14  3   S   M   T9  2.3160      15  3   S   M   T9  2.3404      16  3   S   M   T9  3.1072      1   1   G   S   T10 2.5145      2   1   G   S   T10 2.4728      3   1   G   S   T10 2.4487      4   1   G   S   T10 2.4639      5   1   G   M   T10 2.4757      6   1   G   M   T10 2.5328      7   1   G   M   T10 2.5441      8   1   G   M   T10 2.3075      9   1   S   S   T10 0.5459      10  1   S   S   T10 0.5407      11  1   S   S   T10 0.5377      12  1   S   S   T10 0.5396      13  1   S   M   T10 0.5410      14  1   S   M   T10 0.5481      15  1   S   M   T10 0.5495      16  1   S   M   T10 0.5195      1   2   G   S   T10 2.0414      2   2   G   S   T10 2.0086      3   2   G   S   T10 2.0792      4   2   G   S   T10 2.1206      5   2   G   M   T10 2.2788      6   2   G   M   T10 2.2923      7   2   G   M   T10 2.3856      8   2   G   M   T10 2.1959      9   2   S   S   T10 2.1959      10  2   S   S   T10 2.1703      11  2   S   S   T10 2.4232      12  2   S   S   T10 2.2041      13  2   S   M   T10 2.3139      14  2   S   M   T10 2.2856      15  2   S   M   T10 2.2227      16  2   S   M   T10 2.1239      1   3   G   S   T10 2.3711      2   3   G   S   T10 2.5502      3   3   G   S   T10 2.4133      4   3   G   S   T10 2.6031      5   3   G   M   T10 2.7694      6   3   G   M   T10 2.8768      7   3   G   M   T10 2.7803      8   3   G   M   T10 2.7340      9   3   S   S   T10 2.8331      10  3   S   S   T10 2.9036      11  3   S   S   T10 2.8976      12  3   S   S   T10 2.7767      13  3   S   M   T10 2.7435      14  3   S   M   T10 2.8727      15  3   S   M   T10 2.7860      16  3   S   M   T10 2.8457       @mpiktas:Just as you proposed,I didn't show my goal. My interest is to: (1)compare the response of a pest density to planting pattern and genotype;(2)is there any discrepancy among 3 successive years? I have tried linear mixed effect model using year as a random factor, to reduce the number of separate analyses and type I error using R. The concerned code is followed:               Data <- read.csv("aphids.csv", header=T)     Data$T <- as.numeric(Data$T)      library(nlme)     aphids.lme <- lme(density ~ GENOTYPE*PATTERN+year+site , random=~T| year,            method=”ML”, na.action=na.omit, data=Data)     summary(aphids.lme)      is this right?


title: Is sampling from a folded normal distribution equivalent to sampling from a normal distribution truncated at 0?

text: I wish to simulate from a normal density (say mean=1, sd=1) but only want positive values. One way is to simulate from a normal and take the absolute value. I think of this as a folded normal. I see in R there are functions for truncated random variable generation. If I simulate from a truncated normal (truncation at 0) is this equivalent to the folded approach?


title: How to find  regions where points are localized in a non-random fashion?

text: ### Experiment Let say I have a black box full of numbered balls and I draw them with replacement - so the propability of drawing one specific number in N draws is described by binomial distribution? Problem is that my black box has some intristic structure so the balls are not draw with the same p. For random box (experiment) I should observed random points draw from the box. So if you put this balls id numbers on the axis you should obtain random localization of points - yes? However my experiment generates appart from random localization also points wich should be cluster in groups (localized close to eachother). ### Question   * How can I find those regions where points (which are ids of balls) are localized in a non random fashion? My box has around 10^8 balls and I draw 10^5 balls with replacement in one experiment. I would be very gratefull for any comments and sugestion, what statistical tool would be the best for my purpose. I thought about running simulation - simulate my experiment as a random one, and then look into distribution of distances between draw points. Then this random distribution somehow compare to mine to see which distances are very rare in random distribution and can be hints of my non-ranomdnes behaviour. But I am not quite sure how to proceed after simulation of random distribuiton...


title: How to test for differences in ratings for two devices where each device is rated on a multiple item scale?

text: I have two devices, and I'm trying to figure out which one of them is preferred by users. Thirty people have rated each device on a 12-item questionnaire which asks them to grade the device on 12 parameters.   * What statistical tool should I use to decide which device is better? (just comparing the grades is not enough)   * Can I test for correlation between the "device type" and some kind of weighted grade?


title: How to save individual residuals from an observed endogenous variable in a structural equation model?

text: I am estimating a structural equation model in which two latent variables (with 4 indicators each) and the interaction between the two latent variables predict a single observed variable. I would like to save the individual residuals from the observed endogenous variable to use in further analyses. As far as I can tell, Mplus, the program I generally use, does not have a way to save the individual residuals.   * Is there a stats program that will allow me to save the residuals from an observed variable in a structural equation model?


title: Is it possible to have negative values in a simple slope analysis for dichotomous predictors?

text: I am currently using the simple slope analysis to examine the significant interaction effects between my continuous moderator and a dichotomous predictor (values 1 & 2)on a continuous DV. I have 2 continuous covariates in the analysis. My simple slope graph however showed negative values in the DV (i.e: moderator values at predictor value 2 accounted for negative amounts of DV - the slopes crossed the x-axis towards the bottom right direction). Is this possible, or is there some statistical error? I have seen slopes going into the negative values of DV for continuous predictors, but not for dichotomous predictors.


title: When to cluster features for supervised learning?

text: I'm doing a project on dog adoption patterns, and I realized that there are many (100 +) different breeds of dogs. I'd like to build a predictive model using breed as covariate, but I'm not sure whether I should turn each breed into a dummy variable and run a quick and dirty supervised learning algorithm, or try to do some clustering (based on aggressiveness of breed, for example), and then try to run the algorithm. My dataset is pretty large : 14000 entries. Any advice? Thanks!


title: Why does ca.jo has a minimum lag order of 2?

text: I am trying to use urca library to do cointegration test, and its function ca.jo, which conducts the Johansen procedure on a given data set. I think a lag order of 1 is possible for a cointegrated VECM, which means it does not have short term error correction. For example, we have VAR(1) $$ X_t=\Pi_1X_{t-1} + \epsilon_t $$ and its VECM is $$ \Delta X_t = \Pi X_{t-1} + \epsilon_t $$ where $\Pi=\Pi_1-I_2$ Why does ca.jo specify the minimum lag order to be 2? Is there a reason behind it?


title: Does R square of Cox Regression tells the same thing as Linear Regression R square?

text: My R square from cox regression is about 0.02, Does it tell me that model is doing really bad? I used coxph in R.


title: Similarity Amongst Recipes Using Ingredients and Reviews/Descriptions

text: I'm still toying with things and just learning this, so please forgive any incorrect terminology. My toy data set is a collection of recipes with a fairly significant overlap in ingredients. I'm using these as my features, and using Pearson squared distance as described in _Programming Collective Intelligence_ to determine the correlation between recipes. I'm not looking to "train" anything; I currently would just like to compare the recipes in my corpus and find $X$ nearest and $Y$ farthest neighbors. I thought it'd be fun to expand this, and include people's reviews/comments on the recipes as part of the distance calculation, leading to a closer association between recipes that were both described as "sweet" or "tart" or "sour". I don't know how to do this along with the other features though; I've found examples of bags-of-words for similarity, but I can't seem to find any information about combining it with other features. Is this a valid approach, and is there some terminology that I should be looking for? Thanks.


title: Tests for consistent measurements and outliers

text: I have 3 experiments, where some quantity was measured 3 times. Thus, 3 biological replicas, 3 technical replicas in each biological replica, 9 measurements in total. I need to answer the following questions:   1. For each biological replica: are my 3 measurements consistent? To do this, I'm using Dixon's Q test.   2. For average measurements derived for each biological replica: are these 3 measurements consistent? So, how should I go about the second question? Is it a good idea to use Dixon's Q test here too? What other tests can I use to address the above questions? I guess ANOVA is not suitable because samples are too small, right?


title: Does Weka have an online API?

text: I am doing a project with Weka with clustering algorithms (simple k-means, expectation-maximization and OPTICS). I would love to be able to call those algorithms over the internet. **So does Weka has a online API?** But I suspect there is no. In that case, **what are the alternatives?** * * * **Edit:** What I am trying to achieve is a web interface to launch and control clustering algorithms. But I think I will end up running Weka directly on the server, either by invoking command line, or using my own java program that uses Weka. Another advantage would have been not to be limited by the server hardware but this is acceptable in my case. * * * I have seen that Apache Mahout has a large list of algorithms. Apache Mahout is a library of machine learning algorythms, but also without a callable online API. Google has the kind of "machine learning as a service" platform I want which is the Google Prediction API, but it has more real world algorythms like spam detection or email classification, and not the clustering algorythms I want.


title: How to perform principal components analysis (or factor analysis) on binary data (using SPSS)?

text: I have a dataset with a large number of Yes/No responses. **Can** I use principal components (PCA) or any other data reduction analyses (such as factor analysis) for this type of data? Please advise **how** I go about doing this using SPSS.


title: multiple continuous independent variables, single dependent var: which analysis?

text: I am setting up a linear analysis of a single continuous dependent variable, with multiple (up to 11!) continuous independent variables. I am interested in doing something like an ANCOVA that will show the relative contribution and co-variation among the variables. But my understanding is that ANCOVA requires having at last one categorical variable, which my data does not have. I have already done multiple regression analysis that yields a correlation grid, but does not show covariation among variables. Is there another analysis I should consider? Should I force my data to fit into an ANCOVA analysis? Here's a sample of the data for clarity.               DV      iv1     iv2     iv3     iv4     iv5     iv6     -0.34   2.05    7.77    0.00    3.91    2.23    10.80     -0.40   2.23    0.53    3.14    1.67    3.34    13.93     -0.24   6.98    2.04    2.82    9.30    2.33    13.95     -0.39   0.00    1.48    0.00    2.62    3.06    10.04     -0.32   0.79    0.00    0.00    5.51    3.94    10.24     0.24    0.00    0.00    0.00    0.00    6.15    13.85     1.58    1.34    0.00    0.00    3.41    2.20    13.05     1.07    4.05    2.99    0.00    2.89    3.47    8.09     -0.44   2.31    1.86    0.00    1.54    4.62    13.08     -0.49   2.94    2.68    0.00    10.29   1.47    11.76


title: How can I apply models to predict the output of a test dataset using SAS?

text: I'm trying to use SAS from more of a machine learning perspective than a pure stats perspective. I want to perform resampling on a dataset to measure the predictive accuracy of a LDA and Logistic Regression. However, when I perform `proc logistic` or `proc discrim` all SAS seems to output is a bunch of statistical measures. I want to use the models to predict the output of my test dataset. Is there any way to do this in SAS?


title: How to sample from a normal distribution with known mean and variance using a conventional programming language?

text: I've never had a course in statistics, so I hope I'm asking in the right place here. Suppose I have only two data describing a normal distribution: the mean $\mu$ and variance $\sigma^2$. I want to use a computer to randomly sample from this distribution such that I respect these two statistics. It's pretty obvious that I can handle the mean by simply normalizing around 0: just add $\mu$ to each sample before outputting the sample. But I don't see how to programmatically generate samples to respect $\sigma^2$. My program will be in a conventional programming language; I don't have access to any statistical packages.


title: Optimization: The root of all evil in statistics?

text: I have heard the following expression before: > "Optimization is the root of all evil in statistics". For example, the top answer in this thread makes that statement in reference to the danger of optimizing too aggressively during model selection. My first question is the following: Is this quote attributable to anyone in particular? (e.g. in the statistics literature) From what I understand, the statement refers to the risks of overfitting. Traditional wisdom would say that proper cross validation already fights against this problem, but it looks like there is more to this problem than that. Should statisticians & ML practitioners be wary of over-optimizing their models even when adhering to strict cross validation protocols (e.g. 100 nested 10-fold CV) ? If so, how do we know when to stop searching for "the best" model?


title: Why does R coerce a one dimensional data frame result to a vector? (and how can I stop it?)

text: > age <- c(1,2,3)     > weight <- c(4,5,6)     > p <- data.frame(age=age, height=height)     > p       age height     1   1      4     2   2      5     3   3      6          > p[p$age > 1,]       age height     2   2      5     3   3      6          > age     [1] 1 2 3          > age.df <- data.frame(age=age)          > age.df       age     1   1     2   2     3   3          > age.df[age.df$age > 1,]     [1] 2 3      I want the subset of the dataframe that I asked for, not a vector, in which the rownames and column names are lost. Why does R do this? Can it be prevented?


title: LexRank damping factor

text: I am looking into using LexRank to do some text summarization. I am looking at the original paper. One thing that puzzles me is whether a damping factor is used or not. The formulae are all using it, but the algorithms provided seem not to use it. I am wondering whether that should be included or not in the power step iteration?


title: Why does not ridge regression perform feature selection although it makes use of regularization?

text: Could somebody explain why ridge regression does not perform feature selection although it makes use of regularization? So, it penalizes the regression coefficients like LASSO does, but how come we end up with using all features for all the lambda (penalty) values in range? Why don't we end up getting some zero coefficients in case of high penalization? I know this is a very basic question, but I would appreciate any response. Thanks.


title: What role can permutation test play in multiple comparison?

text: In a test, permutation test is used to estimate the null distribution of the test statistic from the sample. In multiple comparison, what is the usage of permutation test? Thanks!


title: Other ways to find line of "best" fit

text: The most common methods I've seen to find a line of best fit are Least Squares regression and median-median. Are there other good ways? Is there a way to minimize the absolute value difference and find a line of best fit that way? Or to find the distance straight to a line instead of the vertical distance to the line? Thoughts?


title: Cholesky decomposition in error covariance

text: I try to implement Unscented Kalman Filter. Everything seems to be done correctly but I do receive an error about Cholesky decomposition               raise LinAlgError('Matrix is not positive definite - '     numpy.linalg.linalg.LinAlgError: Matrix is not positive definite - Cholesky decomposition     cannot be computed      The problem is that the fail occurs always after some runs of the algorithm. I keep checking determinant and it's not zero. I tried to remove dependencies from the model. I don't have any idea how to proceed from now. I don't know it it will be any help, but this is my python's implementation of Sigma point:               def SigmaPoints(xm, P, kappa):         ''' SigmaPoints calculation'''         n = len(xm)            xm = np.asmatrix(xm)         # checks how big is state vector         Xi = np.asmatrix(np.zeros((n,n*2+1)))# creates zero matrix sigmaPoints n-rows, 2n+1 col           W = np.zeros((n*2+1,1)) # weights for each sigmaPoint as a vector         Xi[:,0] = xm            # 1st column init to input xm                   W[0] = kappa*1.0/(n+kappa) # update first weight         U = np.linalg.cholesky(P*(n+kappa)) #  returns identityMat * value         for k in range(n):                   Xi[:,k] = xm + U[k-1, :].T             W[k] = 1.0/(2.0*(n+kappa))         for k in range(n):                     Xi[:, n+k] = xm - U[k-1,:].T             W[n+k]=1.0/(2.0*(n+kappa))         return (Xi, W)


title: How do I incorporate personalization to a Bayesian ranking engine?

text: I'm looking to quickly get smart on how to add personalization into a Bayesian-based recommendation system. I'm using clickstream data and Bayesian statistics to estimate probabilities of purchase to rank a list of products on. How do I incorporate user-specific history to modify my generic probabilities for a specific user? For example, a user clicked a product in a previous site visit. How do I estimate the purchase probability during a successive visit? What is the standard approach here? Looking for papers, blogs, people that can help illuminate.


title: Interpretation of insignificant predictors in logistic regression model

text: First I should explain what I did, and it might not be right. I have a variable that represents a test outcome, it might be positive or negative.   * I have a set of observations of one important variable (my point of interest) for the last 5 days before the test was undertaken. I have computed the average for the last 3 days and all the 5 days.    * Then, I have some other (not so important to me) variables, some of them are binary (yes or no), some of them are continuous.  I want to create a logit model for my DV based on these variables. As the histograms dont seem normal enough, I have used Mann-Whitney-U on all the variables (with the outcome of the test as a grouping variable) and have seen that the test is the most significant for the 5-day average (the second most significant was the day 1 before the test was undertaken). So, I have chosen the 5-day average for an univariate logit model, and it was significant. Then I put all the other variables into the model and ran a stepwise model selection in R based on AIC. Now I have a model that contains 5 variables - the 5 day average is highly significant and two other variables, still significant, but there are also two variables that are non-significant at my chosen level (0.05). I have two questions:   1. Is the process of selecting one version of the "important" variable for the model acceptable as I did it?   2. How do I interpret the two non-significant variables? They have not been shown to have a significant influence on the DV, but in a model without them, the other variables become insignificant. Can I actually use a model that contains non-significant values? Or can I say, that the model fits the available data well, and the two variables contribute to the model but are not significant at my chosen level.


title: Nonparametric equivalent to mixed and 2-ways repeated measures ANOVA

text: I have run two psychological experiments. The dependent variable is a rating provided by the participant, that is, an integer number from 0 to 100. The first experiment is a mixed 2x2 design, with one between-subject factor (treatment) and one within-subject factor (question). Each of these factors has two levels. That is, participants are asked two questions in the experiment (Q1 and Q2), while a factor varies systematically between the two groups (Treatment1 and Treatment2). The second experiment is identical, except for that the design becomes completely within-subjects. That is, it is a repeated measures 2x2 design; participants are asked four questions that encode the two manipulations in a factorial form: Q1Treatment1, Q1Treatment2, Q2Treatment1, and Q2Treatment2. I am interested in the potential interactions between the two factors (question and treatment). Normally, I would conduct repeated measures ANOVAs for these experiments. However, the problem is that my data deviate strikingly from normality (in fact, so do the residuals). Therefore, I am looking for nonparametric equivalents to ANOVA for these two designs. I can use both SPSS and R.


title: ARIMA model selection

text: I am trying to find a model for the data below. I cannot decide whether the data is stationary or not. I do not want to take differences unnecessarily . I put results of the models below. I think ARIMA(0,1,3) is best model because AIC is lowest .What do you think? ![Data](http://i.stack.imgur.com/dDBI0.png)               [1]  3.78937537  2.15208174 -0.89160064 -1.73332645 -0.83482963 -0.41333144       [7] -1.63430676 -2.20777662 -1.35331581 -1.87955944 -1.49146551  2.49103865      [13]  4.58664031  3.98372753  3.49151073  2.80024199  1.98523634  2.00031840      [19]  1.74083830 -0.61761601 -2.88803617 -0.61886754  2.57402611  1.89283706      [25] -1.07544734 -4.26161203 -4.99017725 -2.51596702 -0.86119264  0.18702439      [31]  0.71532140  0.57777386  3.35966227  5.69130104  2.56164573  0.03846258      [37]  2.58144870  5.42012156  4.91012242  3.19472611  3.25517135  2.99351748      [43]  2.04710036  3.00828776  3.97207225  2.69966884  0.34465355 -1.43062776      [49] -0.86202757  2.17878741  4.93325672  3.31060112  0.35611519  0.46807870      [55]  1.66712453  1.51650578  1.67546667  1.55933575  0.63240519  1.28472844      [61]  1.38605161  0.18651983  0.79863043  2.87366933  4.44551597  4.30870925      [67]  3.60995558  3.35830065  1.53707953  0.08623542  0.97667994 -0.42280994      [73] -3.58389281 -3.55316081 -2.55628493 -3.04211427 -4.34371036 -4.72314008      [79] -2.32487771 -0.38168241 -0.89383414 -0.58963196  0.77158269  1.07205599      [85]  0.72950780  1.88525945  3.16151643  1.72134300  0.31586665  0.34106233      [91] -1.31065867 -2.64398487 -0.47188919  1.33507435 -0.55269545 -2.74748555      [97] -3.06504226 -3.07342362 -2.60963150 -0.96491699 -0.11554423  0.07801436     [103]  1.03953592  0.63529388  0.10310775  1.67245060 -0.08355651 -4.69707317     [109] -6.83755810 -5.48216561 -2.18589446 -1.34859551 -4.25464826 -4.44676908     [115] -0.72380753  0.10925230 -2.45630323 -4.70575810 -5.66265488 -5.50397286     [121] -5.64958955 -5.48545360 -2.79058869 -1.10528812 -3.35615732 -4.07432989     [127] -2.73350349 -2.21617434 -1.52494934 -1.58548803 -2.49378602 -1.69527130     [133]  0.74873392  0.90627894  0.30555834  2.92103049  5.83058432  6.36187212     [139]  5.22905256  4.00941490  3.03523663  1.93713509  1.67471721  1.32443755     [145]  2.32354111  5.63314456  6.62253314  3.30169519 -0.53170457 -2.45563036     [151] -3.33244848 -3.77673941 -3.54203732 -2.45898416 -0.16490056  3.04228090     [157]  3.92168880  1.24849940 -1.63284318 -1.07538492  1.65080853  1.66193852     [163] -0.42635583 -0.55887445  0.41598367  0.86267051  2.37980973  3.33955626     [169]  2.06034558 -0.23641280 -0.58777692 -0.15430770 -0.71699361 -0.97577373     [175] -1.32154837 -0.63551008  2.42331965  4.81896345  3.87979306  0.95525157     [181] -1.50232828 -3.57249760 -3.03952767 -0.28938790  1.58862094  2.81297712     [187]  3.46355587  3.35284181  3.33816359  2.39696945  0.09838492 -1.57887066     [193] -0.24116984  1.61287955  0.67580375 -0.41342769 -2.12526810 -3.82541756     [199] -1.64497472 -0.73523703      Model fitting               Call:     arima(x = data9, order = c(0, 1, 3))          Coefficients:              ma1      ma2      ma3           1.0905  -0.4871  -0.6763     s.e.  0.0750   0.1390   0.0760          sigma^2 estimated as 0.8477:  log likelihood = -268.35,  aic = 542.7          Call:     arima(x = data9, order = c(1, 0, 1))          Coefficients:              ar1     ma1  intercept           0.7070  1.0000     0.2048     s.e.  0.0497  0.0213     0.4983          sigma^2 estimated as 1.097:  log likelihood = -296.55,  aic = 599.11


title: Calculating variance for microarray data?

text: I have microarray data in which there are three biological replicates for each of the conditions. I am interested in a numerical estimate of how well the replicates correspond with one another, so I thought I should look at variance. I suppose I could calculate it individually for each probe, but I thought there might be more information by looking at all probes at once (something to do with moderation). I've been poking around in limma, as a start, but I'm not sure how to go about this.


title: Is power analysis necessary in Bayesian Statistics?

text: I've been researching the Bayesian take on classical statistics lately. After reading about the Bayes factor I've been left wondering if power analysis is a necessity in this view of statistics. My main reason for wondering this is the Bayes factor really just appears to be a likelihood ratio. Once it's like 25:1 it seems like I can call it a night. Am I far off? Any other reading I can do to learn more? Currently reading this book: Introduction to Bayesian Statistics, by W.M. Bolstad (Wiley- Interscience; 2nd ed., 2007).


title: How to deal with a non-significant threshold value in ordinal regression in SPSS?

text: I am running an ordinal regression in SPSS. I have independent variables on nominal and ordinal scales. My dependent variable is ordinal, indicating a level of action, coded 0, 1, 2. Unfortunately, I currently don’t know how to deal with the non-significance of a threshold level for my dependent variable indicated by SPSS in the parameter estimates table. The SPSS output of the parameter estimates looks like this: Threshold Level of action = 0 Estimate = .6 Sig. = .143 Threshold Level of action = 1 Estimate = 2.2 Sig. = .000 Independent variables … I read that “ _non-significance of a threshold suggests that the cutting point is not truly different and therefore some levels of the dependent variable need to be combined_ ” (Garson, G. D. (2012). Ordinal Regression. Asheboro, NC: Statistical Associates Publishers). I have not come across such a statement before. Also some ordinal regressions I had a look at did not mention an issue regarding a non-significant threshold and interpreted the output nonetheless. So now I am wondering how severe that issue is. If I stick with ordinal regression, how would this non-significant threshold impact my results (given all other assumption are met)? Would they still be valid? Or does this mean that I definitely have to combine the outcome categories 0 and 1, which basically would mean I need to run binary logistic regression instead of ordinal regression?


title: Python Convex optimization

text: Hi I am looking for a robust convex optimization problem solver in python. My problem isn't very complex, its basically a simple constrained qp program. I am using Enthought free python 2.7 distribution on a windows 64 bit machine. I came across http://cvxopt.org/install/index.html It doesn't seem very easy to install this on windows. I also came across Mosek, which seems a bit of a overkill. Does anyone know of a good solution, do people use cvxopt on windows?


For example: An initial set $[1, 3, 4, 7, 2]$ sums to 17. By dividing each element by the sum, the elements sum to 1: $[1/17, 3/17,...2/17]$ sums to 1. What is it called when a set of numbers are adjusted to sum to one?


title: G-test statistic and KL divergence

text: According to Wikipedia, the G-test statistic is "proportional to the Kullback–Leibler divergence of the empirical distribution from the theoretical distribution." To get the relationship between $ G $ and the KL divergence: $$ \begin{align*} G &= 2 \sum_i O_i \ln\left(\frac{O_i}{E_i}\right) \\ &= 2 \sum_i n P(i) \ln\left(\frac{n P(i)}{n Q(i)}\right) \\ &= 2n \sum_i P(i) \ln\left(\frac{P(i)}{Q(i)}\right) \\ &= 2n \times D_{KL}(P \| Q) \end{align*} $$ This is where I'm a little confused. $ D_{KL}(P \| Q) $ represents the divergence of $ Q $ (the expected distribution) from $ P $ (the observed distribution). This is the opposite of the statement in the Wikipedia page (assuming that "theoretical" refers to expected and "empirical" refers to observed). Now from an information theoretic interpretation of KL divergence, $ P $ is represents the "true" distribution and $ Q $ is the "approximation". But from this derivation, the result seems counter-intuitive. Shouldn't the expected distribution correspond to the true distribution and the observed distribution correspond to the approximation?


title: Expected value calculation BS2

text: I have great difficulty in calculating $\mathrm{E}\left(T_{1} T_{2}^{-1}\right)$ where $T_{i}$ is a random variable distributed according to the Birnbaum-Saunders distribution. Are there any suggestions on how to calculate the expression?


title: Bias correction of logarithmic transformations

text: I read that there is a bias when we transform a series with a logarithm and then applying the inverse function, but only in forecasts of the mean values. I don't understand what it means exactly. In the book "Introductory Time Series with R", there is a section about this very issue: > The bias in the means arises as a result of applying the inverse transform > to a residual series. For example, if the time series are Gaussian white > noise $w_{t}$, with mean zero and standard deviation σ, then the > distribution of the inverse-transform (the anti-log) of the series is log- > normal with mean $e^{σ ^{2}/2}$. This can be verified theoretically, or > empirically by simulation as in the code below:               set.seed(1)     sigma <- 1     w <- rnorm(1e+06, sd = sigma)     mean(w)     [1] 4.69e-05          > mean(exp(w))     [1] 1.65     > exp(sigma^2/2)     [1] 1.65      > The code above indicates that the mean of the anti-log of the Gaussian white > noise and the expected mean from a log-normal distribution are equal. Hence, > for a Gaussian white noise residual series, a correction factor of > $e^{σ^{2}/2}$ should be applied to the forecasts of means. In the same section, it says that an adjusted forecast {$x'_{t}$} with an empirical correction factor is: $$\hat{x}'_{t} = e^{\hat{\log x_{t}}}\sum_{t=1}^{n}\frac{e^z_{t}}{n}$$ where $\hat{\log x_{t}}$ is the predicted series given by the log-regression model. Does anyone know what this means? As far as I understand, when we apply a logarithm to a series and fit a (linear) model to the resulting values, we are doing the following: $$\log x_{t} = \alpha t + \beta + z_{t}$$ where $z_{t}$ are the residues. So, I would think that the reverse transformation is given by: $$x_{t} = e^{\alpha t + \beta} e^{z_{t}}$$ However, what is the forecasting of means and how is it involved in this procedure? I'm guessing it's $\displaystyle \frac{e^z_{t}}{n}$ but I can't see a justification. **UPDATE** : Suppose we use the model described above to predict some values $p_{t}$. Since we used a logarithmic transformation, to revert it, we have to do this for all new $t$: $$\exp{p_{ŧ}}$$ However, the book says that in order to apply the correction to each new predicted value, we need to do this: $$p_{\text{t}}^{\text{corrected}} = p_{t} \frac {1}{n}\sum_{t=1}^{n} e^{\hat z_t}$$ Why? I thought that the mean of the forecast is biased but not its individual values.


title: Whether to use friedman test to test differences in categories across four topics?

text: I have collected data where respondents provide on of the following responses               1. non interested to be involved     2. only get information on the topic but not participating     3. consulting on the topic     4. taking responsibility      for each each of the following four topics               1. setting priorities     2. raising revenues     3. distributing funds      4. monitoring      I would like to test if the topic influences the preferred way for participation. **Can you use Friedman's test to examine whether responses differ across topics?**


title: Mixed linear models with exactly the same AIC?

text: I have a dataset in which I measure the response variable of a number of individual animals. I used this response variable as the dependent variable in a linear mixed model, for which I have 5 explanatory variables. Two of these explanatory variables are nested if they are entered in the same model, as they refer to the individual animal, and the research site they are situated on. So it would be site(individual). If I do not nest them but enter them separately, the program runs into calculation errors, and throws one of the two variables out. I use SPSS. I then created a whole number of different models, with all the different combinations of explanatory variables, and ranked them according to their AIC. I then found out that a number of models had exactly the same AIC. These were the models which, all else being equal, contained either just individual as an explanatory variable, or site(individual). If the model, all else being equal, only contained site, the AIC was different. Now my questions are: a) do these results make sense? I cannot think of what I am doing wrong, but I am a bit worried about the exact same value for the AIC.. b) can I compare models this way - comparing nested variables with un- nested ones?


title: What statistic test apply to a cell count time line?

text: I am a PhD student with some difficulties about what test to choose to verify an experiment. Hope you can give me some help! My experiment: I am counting the number of a subset of cells in Drosophila overtime. I have a mutant and a wild type control. I dissect the flies at different time points: just eclosed, 1 day after eclosion, 3 dae, 6 dae, 9 dae, 12 dae and 15 dae. For each time-point I have a n number of around 50 for sample. The sample number has a range of 0-9, depending on the day and if mutant or wildtype. The problem: What happen with these cells is that in the wild type population the number of cells does not decrease overtime, while in the mutants it does. Moreover the wild type population has an average of around 6 (in all the dissection days), while the mutant starts with 4 and goes down to 2. If they were the same amount initially I could have used an anova test with a posthoc test (like tukey) to show the difference between wild type and control for each day, but in this scenario an anova would tell me only that all the wild type are different from the same dae dissected mutant (even at eclosion!) because of the basic difference they have, or that for example mutant dae15 is different from mutant dae0,1,3. Moreover, what I would like to test against is not that the wild type is different from the mutant at a particular timepoint, but that the mutant numbers decrease in a different way compared to the control numbers. Possible solutions: One possibility would be to do an anova on a normalized sample: I could divide every number (and I mean the single numbers of the count, not the average) I get for the average of the eclosion day, and do an anova on that ratio (so I would have like normalised the two populations to the eclosion day and it would not matter anymore that their raw numbers are different), but something tells me I would mess up all the statistics: I am not sure if this approach is statistically correct. Another possibility I was thinking is to use some kind of correlation or regression analysis, but in the example I found I never found this approach on my kind of experiment, so i am not sure how to work this out. In the various handbook it always show this kind of approach to smaller experiments, like comparing blood pressure and coffee assumption, and with much less numbers, so I am not sure what I would end up with blotting 600 different points together. Another possibility would be to compare the slope of the two different lines, but I am not sure how I could do this considering all the different counts and not only the final average. Someone suggested to to a Keplen Maier test, but I cannot see how to apply this kind of tests (logrank) considering all the count that compose one timepoint and not only the final average. How I am going to do this if I do not find better solution is: just do an anova (yes, one single set of counts is normally distributed) on the raw numbers and show that while Wt Day0 and Day15 are not statistically different, while the Day0 and Day15 from the mutant are statistically different (or even the Day0 from Day12 etc). Maybe this would be enough and I am just over- thinking the problem, but somehow it does not seem to fully describe what I am analysing. If you have any idea how this problem could be solved, I would be grateful!