title: Calculate trend slope after STL decomposition

text: I have a time-series of daily measurements of some quantity for 1995-2011. There's about one measurement every three days. The data show a strong seasonality (annual cycle). What I ultimately want is get an estimation for the underlying trend. STL decomposition works fine and gives me seperate trend, seasonality, and noise components of my original time series. But how do I get a quantitative measure for the trend component? A simple linear regression won't do due to the autocorrelation in the noise component. Gavin Simpson suggested using `gls()` from **R** 's _nlme_ package, but I'm not sure how I should do that. Which would be the inputs for `gls()`? Or, are there other suggestions for tackling my problem? Any help is greatly appreciated :)


title: Testing implementation of Anderson-Darling test for uniform RV

text: I am trying to write unit tests for a whole mess of statistics code. Some of the unit tests take the form: generate a sample following a null hypothesis, use code to get a p-value under that null, repeat hundreds of times, then look at all the p-values: if they are reasonably uniform, then the code passes. I usually check if the ratio of p-values < $\alpha$ is near $\alpha$ for $\alpha = 0.01, 0.05, 0.1$. But I am usually also interested in whether the p-values output deviate from uniformity. I usually test this with the Anderson-Darling test. Here is where I have a circularity problem: how can I unit test my Anderson- Darling code? I can easily feed it uniformly generated variables, and get a p-value, repeat hundreds of times, but then I just have a bunch of p-values. I can q-q plot them, but I'm more interested in an automatic unit test I can run. What are some basic sanity checks I can implement automatically? there is the naive check of ratio of p-values < $\alpha$ noted above. I can also implement a Kolmogorov-Smirnov test. What else can I easily check for? (I realize this question may seem hopelessly pedantic or naive or subject to infinite regress...) **edit** some additional ideas:   1. test the code on $\frac{i}{n}$ for $i = 1,2,...,n$, for different values of $n$. Presumably I can compute, by hand, the p-value for the A-D test in this case.   2. compute $n$ p-values by feeding many uniform samples to the code $n$ times, then regress the order statistics of the p-values, $p_{(i)}$ vs $i/n$, to get $p_{(i)} = \beta_1 i/n + \beta_0$ and test the null $\beta_1 = 1, \beta_0 = 0$. Presumably I can simplify this test by hand in such a way that inspection reveals it to be correct.   3. make sure the code is invariant with respect to permutation of the input. (duh)


title: Cool article on behavioral targeting - How do you think they solved it?

text: I just read an interesting article on how Target used purchase behavior as a predictor to whether they were pregnant or not. Here's the link. Could someone explain how Target build the predictive model, or their best guess as to the process? **Update** The original article is in New York Times.


title: Bayesian inference on noncentrality parameters of t- and F-distributions

text: Suppose I observe a random variable $x$ drawn from a non-central t- or F- distribution, and would like to perform inference on the non-centrality parameter. How would a Bayesian approach this problem? Are there known conjugate priors for these? For the t-distribution I have seen some reference to LeCoutre's 'lambda-prime' distribution, but I believe this is for a non-informative prior (and I know very little about Bayesian analysis). Is there a parallel result for the F-distribution? How are these used?


title: Specify variance structure in ASreml v3

text: I used ASreml to estimate genetic parameters for feed intake using bivariate and multivariate models. However, I am very confused about how to define variance structures (even though I went through the "Users Guide" many times). Is there a quick way to check which models will be used in the R and G structures, such as when using the Unstructured, Identity, and Autoregressive models? How would I determine the initial values?


title: Probability of raining in the weekend

text: Just trying to see if my reasoning is correct.... If the probability that next saturday will rain is 0.25 and the probability that next sunday will rain is 0.25, what is the probability that during the weekend will rain? Assuming that A and B are independent events, P(A∪B)=P(A)+P(B)-P(A∩B) and P(A∩B)=P(A)*P(B) therefore the probability that during the weekend will rain is: P(A∪B)=0.25+0.25*0.25*0.25=0.265625 It this reasoning correct? Thanks, Luca


title: Expected value of $Ye^X$ where $X \sim U(0,1)$ and $Y \sim U(0,1)$

text: I am trying to find the expected value of $Z$ where $Z = Y\cdot e^X$ where $Y \sim U(0,1)$ and $X \sim U(0,1)$. My attempt so far: $$F_Z(z) = P(Ye^X \le z) = \int \int_{Ye^X \le z} f(x,y)\, dxdy$$ Where $f_{X,Y}(x,y) = f_y\cdot f_{e^x}$ $$f_Y(y) = \frac{1}{1-0}\,, \quad y \in (0,1)$$ I am stuck trying to find $f_{e^X}$ but I cannot remember how to find that pdf.


title: Proving basic probability inequalities

text: So the title is a bit vague, but I am trying to solve some basic probability inequality. Given that $P(B\cap C) > P(B)P(C)$, how can I show that $P(B|C) > P(B|W\setminus C)$? I started by showing that $P(B|C) > P(B)$, but I am not sure if this helps. Thoughts? Thanks.


title: Directed Graph to Regression Help

text: ![enter image description here](http://i.stack.imgur.com/YwgCn.jpg) So I have this directed graph (above) Each arrow represents a causal link. Is it possible to calculate the affect of X on Z, where all variables are observed except U and W. If so, what would the regression equation(s) look like. I believe it has to do with 2 step least squares, but I haven't been able to figure it out. Any help/tips would be much appreciated. Thanks.


title: Slope of a Regression

text: ![enter image description here](http://i.stack.imgur.com/UQ88F.jpg) Given this chart, how do you determine the slope of the regression Y on F?


title: Coefficients from glmer in R

text: In a mixed effect model where the intercept is random effect and the slope is fixed effect (see the code below), I understand the output of `summary(glmer(...))`. But I do not understand `coef(glmer(...))`; it will output the intercept for each sample. In the example, how are those 100 intercepts estimated?               n <- 100     x <- runif(n,2,6)     a <- -3      b <- 1.5     s <- 1     N <- 8     id <- 1:n     r  <- rnorm(length(x),0,s) # random factor     p <- function(x,a,b) exp(a+b*x)/(1+exp(a+b*x))     y <- rbinom(length(x), N, prob = p(x,a+r,b))          library(lme4)     model <- glmer(cbind(y,N-y)~x+(1|id),family=binomial)     summary(model)     coef(model) # output here is what I do not understand          # are they estimates for r?     rr <- coef(model)$id[,1]-summary(model)$coefficients[1,1]     plot(r,rr)      The likelihood of the model, I think, is: $$ L_{i}=\int_{-\infty}^{\infty}f(y_i|N,a,b,r_{i})g(r_{i}|s)dr_{i}\\ L=\prod_{i=1}^{n} L_{i} $$ where _f_ is binomial pmf and _g_ is normal pdf. So _r_ should be integrated out. The number of the parameters of this model is 3 ( _a_ , _b_ , and _s_ ). Although it seems there are 100 intercepts estimated, they are not really considered the parameters of the model, I think. `AIC(model)` is `-2*logLik(model)+3*2`. I would like to know what method will give those 100 intercepts.


title: Small Beta weight equals small structural coefficient

text: I have two predictors (X1 and X2) with large Beta's and large predictor- independent variable correlations (say it is like rs). In addition, I have another variable (X3; Beta=0.17; rs=0.18) which was not statistically significant in predicting the scores in the dependent variable. Last, I have an X4 with zero Beta and zero rs. In interpreting this, should I only mention X1 and X2 are practically important predictors, while X3 predicts the scores at a lesser extent. Or should I just skip X3. Thanks


title: ROC curve fit characteristic

text: I have a question: > Is the ROC curve enough to assess goodness of fit? Or do you need other > tests?


title: obtaining Standard Errors in maximum likelihood estimation

text: I am trying to obtain the standard errors of my maximum likelihood model, I am having trouble to fix the error message:               **Error in optim(c(0.02, 1.75, 124, 1.5, 0.85), logA, gr = NULL, method = "Nelder-Mead",  :        non-finite finite-difference value [1]**      The code I have so far is :               logA<-function(theta, data){     #theta=(lambda, rho, tau, b1,b2)     #hpa = the data with:     #First column Age,     #Second column is failure time t,      #Third column is group indicator 1 for positve results HPA and      #0 otherwise ,and      #Fourth column is delta a censoring indicator 0 if right censored      #& one otherwise     data<-hpa     n<-length(data[,1]) # sample size     t<-data[,2] # failure time     z<-data[,3] # group indicator: covariate     delta<-data[,4]     ep<-(t<=theta[3])  # indicator function of failure time           #- greater or less that the change point tau     d1 <-sum(delta*ep)     d2<-sum(delta)     d<-(d2-d1)     a<-sum(delta*ep*z)     g<-sum(delta*z*(1-ep))     w1<-sum( ((ep*t) + (theta[3])*(1 - ep))*(exp(theta[4]*z)))     w2<-sum((1-ep)*(t-theta[3])*(exp(theta[5]*z)))          ll <- (((d2)*(log(theta[1])))+((d)*(log(theta[2])))+ (theta[4]*a) + (theta[5]*g) - (theta[1]*w1) - (theta[1]*theta[2]*w2))          return(-ll)            }          optim(c(.02,1.75,124, 1.5,.85),logA, method ="Nelder-Mead" )      Then the error comes after I run this code               p<-optim(c(.02,1.75,124,1.5,.85),logA,gr = NULL, method = "Nelder-Mead", hessian=T)     #p should be like the code below to obtain the hessian matrix but I don't know what is my y and what is my x     p<-optim(c(.02,1.75,124,1.5,.85),logA,gr = NULL, method = "Nelder-Mead", hessian=T, X= , Y= ,)      For inverting the hessian matrix, i could use the codes below if only i have the above step fixed.               I<- solve(p$hessian)      to obtain the standard error               se<-sqrt(diag(I))       Thank you.


title: Assessing an interaction term

text: Suppose we have a variable $x$ with 3 levels (e.g. $x_1, x_2$ and $x_3$). We want to see if there is an interaction between $x$ and $y$ where $y$ is a continuous variable. To test this would we include the following terms: $x_{1}y, x_{2}y, x_{3}y$ in a regression model?


title: Best algorithms for ordinal classification

text: I am working on a data set of about 34K rows trying to predict an ordinal response variable using R. I have tried association rules, random forest and ordinal regression. Does anyone have experience with other algorithms which are worth a try?


title: Free tools on Linux for multivariate logistic regression?

text: What is your favorite free tool on Linux for multivariate logistic regression? Possibilities I've seen:   * R (see paper). This question says use design.   * Can you use SciPy? Other choices? Do people have experience with large data?


title: Tools for modeling financial time series

text: What modern tools (Windows-based) do you suggest for modeling financial time series?


title: Whether to assess normality in a factorial repeated measures ANOVA by looking at distributions within cells?

text: I'm a little confused about the analysis of normality in a repeated measures ANOVA I'm doing. It's a factorial ANOVA with three rm IVs each of which has only 2 levels. I've read in a few places online and in a book or two that to assess normality in a repeated measures ANOVA, I look at the distributions within each conditions -this would be 8 distributions in my case. However, most of these articles focused on rm ANOVAs in which at least one of the IVs has more than 2 levels. My confusion is coming from the fact that when carrying out a repeated measures t test (i.e. 1 IV with just 2 levels), you don't care about the distribution for responses in the separate conditions. Rather, you focus on the distribution of the difference scores. In other words, in a repeated measures t-test, we're interested in change. My questions are:   1. Should I assess normality by looking at the 8 individual distributions associated with each conditions   2. If yes, why is it different for a t-test (which is essentially calculated in the same way)   3. If I shouldn't look at the 8 distributions mentioned above, what should I run my normality analysis on?


title: Forecasting demographic census

text: What are some of the ways to forecast demographic census with some validation and calibration techniques? Some of the concerns:   * Census blocks vary in sizes as rural areas are a lot larger than condensed urban areas. Is there a need to account for the area size difference?   * if let's say I have census data dating back to 4 - 5 census periods, how far can i forecast it into the future?   * if some of the census zone change lightly in boundaries, how can i account for that change?   * What are the methods to validate census forecasts? for example, if i have data for existing 5 census periods, should I model the first 3 and test it on the latter two? or is there another way?   * what's the state of practice in forecasting census data, and what are some of the state of the art methods?


title: Finding the PDF given the CDF

text: How can I find the PDF (probability density function) of a distribution given the CDF (cumulative distribution function)?


title: What is the definition of Top-n accuracy?

text: I'm reading a Scientific paper on image classification. In the experimental results they speak of top-1 and top-5 accuracy but i've never heard of the term, nor can find it using google. Can someone give me a definition or point me somewhere? :)


title: Types of dispersion parameter for binomial data

text: For a model with a binomial proportion as response variable, which is fitted with according to a binomial distribution, a dispersion parameter $\phi$ can be calculated, which is equal to the sum of the squared Pearson residuals divided by the residual degrees of freedom. I was following along with an example I found online and found that the $\phi$ computed by hand matches the dispersion parameter that one gets by specifying `family=quasibinomial()` in the `glm` command (rather than `family=binomial()`) and asking R for `summary(fittedModelName)$dispersion` For my toy data set, $\phi = 4.904$ in both cases. When I fit the same model with a beta-binomial distribution (using `betabin` in the `aod` package), I get a dispersion parameter estimate which is very different, sc.               Overdispersion coefficients:                      Estimate Std. Error   z value   Pr(> z)     phi.(Intercept) 1.376e-01  3.145e-02 4.374e+00 6.102e-06      I would like to know how the two types of dispersion parameters differ, and whether there is any way they can be related. These are the parameters I specified for the `betabin` function:               betabin(formula = cbind(Rcnt, total - Rcnt) ~ LANG, random = ~1,          data = toyDataSet)


title: The size of my reduced data set is greater than the original

text: I have an original data set with a number of features N equal to 135 and a number of rows equal to 32000. The last column of the data set ( column 136 ) can take either -1 or 1 depending on the class of the row. Each of the features can take a binary value of 0 or 1 ( characteristic present or not). The original data set looks like this:                   V1    V1   V3   V4   V5  ............ V136         0     1    0    0     1                 -1         0     1    1    0     1                 -1         1     0    1    1     0                  1         .     .    .    .     .                  .         .     .    .    .     .                  .         .     .    .    .     .                  . ( row 32000)      I am required to use PCA to reduce the dimentionality of my data, and then apply SVM for classification. when I used PCA to cover 96% of the variance, I got a number of principal componenents equal to 54 ( which is good, since I reduced the number of features from 135 to 54), and then I computed the reduced data matrix which looks like this:                   V1                V2                V3                 V4  .................  V54    Class         1.89463604908794 -1.15458215287551 -0.577464752091365 -0.560850946997309               -1        -0.170737500644998 0.190545093723431 0.451249831953581 1.44451854658279                 -1         ......         ......      When I run SVM on the original data set, SVM finishes quickly, however, when I run it on the reduced data set, it takes forever. A part of the problem is that the original data set was binary, and thus it takes less memory when loaded. However, because the reduced data set is composed of real number, it takes more memory. For comparison purposes, I also saved the reduced data set to disque, and I compared the size of the two files in terms of KB:                   Original Data ( not reduced) : 8500 KB         Reduced Data set             : 32200 KB      It looks like PCA reduced the number of features but it increased the total size in term of KB. Any ideas about how I can overcome this issue? Doesn't this throw away the original purpose of PCA ( reducing the dimentionality, and thus the size)?


title: The equivalence of sample correlation and R statistic for simple linear regression

text: It is often stated that the square of the sample correlation $r^2$ is equivalent to the $R^2$ coefficient of determination for simple linear regression. I have been unable to demonstrate this myself and would appreciate a full proof of this fact.


title: How can I interpret the decision tree results in R?

text: Would anyone please give me a simple explanation of how can I read the results of the decision tree in R? My target attribute (label) have 12 different labels. Thanks!!               > printcp(fittree) # display the results          Classification tree:     rpart(formula = bucket ~ txamountunit, method = "class")          Variables actually used in tree construction:     [1] txamountunit          Root node error: 73114960/87368978 = 0.83685          n= 87368978                   CP nsplit rel error    xerror       xstd     1  0.171543      0 1.0000000 1.0000000 4.7237e-05     2  0.161908      1 0.8284571 0.8284571 5.8951e-05     3  0.143099      2 0.6665495 0.6665495 6.3492e-05     4  0.138778      3 0.5234509 0.5234509 6.3428e-05     5  0.098899      4 0.3846731 0.3846731 5.9729e-05     6  0.098372      5 0.2857742 0.2857742 5.4533e-05     7  0.092354      6 0.1874023 0.1874023 4.6488e-05     8  0.049631      7 0.0950484 0.0950484 3.4592e-05     9  0.020972      8 0.0454171 0.0454171 2.4445e-05     10 0.019680      9 0.0244449 0.0244449 1.8097e-05     11 0.010000     10 0.0047651 0.0047651 8.0568e-06


title: Metropolis algorithm, what is the target distrbution and how to compose it?

text: When do Metropolis sampling or MCMC, we need a target distribution $P_{target}(\theta)$, and a proposal distribution $P_{proposal}(\theta)$, then a value $\theta_i$ is generated via $P_{proposal}(\theta)$, we need to compute the probability whether to accept or reject this $\theta_i$ via $P_{target}(\theta)$, right? My problem is, we should know $P_{target}(\theta)$ before we doing this Metropolis process, right? Then what is this target distribution, does it have to do with my prior belief of $\theta$? If I've already known it, why bother doing this Metropolis sampling, can't we just use grid approximation? As I read in the book, it takes the product of the likelihood and prior of $\theta$ as the target distribution, why? Different prior belief will result into different target distribution, does it mean we don't have a fixed desired target distribution?


title: Non normal residuals in multiple regression

text: I used height, weight, gender and age to regress on BMR (basal metabolic rate) and obtained the following qq plot of residuals![enter image description here](http://i.stack.imgur.com/m8bMS.jpg) I then regressed the above variables with log BMR and obtained the following qq plot for residuals![enter image description here](http://i.stack.imgur.com/xaqK3.jpg) Shouldn't I be using log BMR here? If so, how do I interpret my results?


title: In what situations is covariance preferred to correlation?

text: If the magnitude of covariance doesn't really convey any reliable information, and correlation provides us the same information but also gives the degree of dependence, then why use covariance at all ?


title: How to test measurement invariance with PLS?

text: I have two questions about structural equations modeling:   1. To perform comparisons of model estimates for different groups (3 clusters), do I always have to analize measurement invariance first?    2. If the answer to #1 is yes, how can I do that with SmartPLS?


title: How to sample using MCMC from a posterior distribution in general?

text: Assume one has the posterior distribution of a parameter, $p(\theta|y)$ and what I mean by having it is that for each point of $\theta$, one can use Monte Carlo method+MCMC to calculate the $p(\theta|y)$. Now my question is if I want to sample from $p(\theta|y)$, them basically I have to do one Gibbs sampling(for example) to sample from distribution and at any point I have to run Monte Carlo method on the point to calculate $p(\theta|y)$'s value right? i.e. it needs two loops, one inside of the other. Is this correct? As I got an answer to this question and I thought maybe my question is vague I will try to clarify it a bit more: From what I know by reading for a week the whole time about Monte Carlo method and MCMC, I understood(correct me if I am wrong) that: $$p(\theta|y)=\frac{p(y|\theta)p(\theta)}{\int\limits{\theta}{p(y|\theta)p(\theta)}}.$$ Now if you consider that we only have a sampling algorithm for $\theta$ and we can only calculate $p(y|\theta)$ explicitly(and not the other functions!), therefore to get values from $p(\theta|y)$ one needs to numerically integrate the denominator. And for each value of this posterior one needs to apply a sampling scheme like gibs sampling to generate a sample of $p(\theta|y)$; each new transition in the parameter space should then sample from the distribution which is $p(\theta|y)$ here and to calculate that the above proportion should be computed.


title: Relating lifetime risk to prevalence of a disease

text: I am somewhat confused about lifetime risk. The National Kidney Federation released data saying that the lifetime risk of a white American male needing dialysis was around 3%. However, considerably less than 1% of the population actually are on dialysis... why the discrepancy? Why aren't 3% of American males on dialysis if the lifetime risk is 3%?


title: Selecting the statistical model for drug response study

text: I need to study the effect of certain drug (e.g. drug1) intake on blood levels (e.g. white blood cells). I have 900 observations with data on drug1 (with yes or no); and numeric measurements for the white blood levels) What statistical model would be apt for such analysis and for graphical representation. Thanks weblink for sample of data::: http://cl.ly/image/000o2v171y2A


title: Multiple construct analysis

text: I am analysing a model which has 5 constructs. Each construct has 3 Likert scale questions. For instance, I want to analyse the relationship and effect of the construct "PERCEIVED USEFULNESS" which is the independent variable upon the construct "BEHAVIOURAL INTENTION" which is the dependent variable. Both constructs have 3 questions each. How do I proceed? I wanted to use Pearson correlation and Multiple Regression analysis. Should I compute the MEAN of each construct and then RUN the tests?


title: Is effect size (Cohen's d) scale data or just ordinal?

text: I am aware that you must be careful about when you can compare effect sizes. But let's assume that I have two effect sizes from two similar studies: similar groups, same intervention, same measurement. Let's say, two classes in the same school receiving the same intervention and measured on the same outcome. If one class produces an effect size (Cohen's d) of 0.4 and the other produces an effect of 0.2, is it reasonable to describe that the former has an effect that is twice the size of the latter [scale data]? Or can I only say that the former is larger than the latter, without specifying a factor [ordinal data]?


title: Maximum risk and sparse estimation

text: On Larry Wasserman's blog, he talks about the "Steep price of sparsity" here: http://normaldeviate.wordpress.com/2013/07/27/the-steep-price-of-sparsity/ In it, he points out that a sparse estimation procedure (one that forces "small" estimates to zero) has infinite maximum risk. In the simple case of estimating a single parameter $\theta$ given a sample of size $n$: $$\sup_\theta \mathbb{E}_\theta [n(\hat \theta - \theta)^2] \to \infty $$ And more generally, for a non-negative loss function $\ell$ and a vector of parameters $\beta$, we have $$\sup_\beta \mathbb{E}_\beta [\ell(n^{1/2}(\hat \beta - \beta))] \to \sup_\beta \ell(\beta)$$ Here, the estimates $\hat \beta$ would be generated via a sparse procedure such as the lasso as opposed to ordinary least squares or maximum likelihood. My question is, how relevant is this problem? More narrowly, what is the reason for the factor $n^{1/2}$ in the second expression? If this factor didn't appear, the problem (if it is in fact a problem) would become much more innocuous.


title: Whether to assume nominal variables are ordinal when computing distance measurement for mixed?

text: **English** following dutch In mijn afstandmaat voor mengelingen van variabele mag ik alleen een combinatie van continue en ordinale variabele gebruiken. Mijn data bestaat echter uit een mengeling van ordinale, nominale en continue variabele. Mag ik aannemen dat al mijn nominale variabele ook ordinaal zijn of maak ik dan een gigantische fout in mijn waarneemingen? **English:** In my distance measurement for mixed I can only use a combination of continuous and ordinal variables. But my data has a combination of nominal, ordinal and continuous variables. Would it make sense to assume that my nominal variables are ordinal or do I make a huge mistake in that way?


title: Can I assume (log-)normality for this sample?

text: Here's a Q-Q plot for my sample (notice the logarithmic Y axis); $n = 1000$: ![enter image description here](http://i.stack.imgur.com/QlMGU.png)   As pointed out by whuber, this indicates that the underlying distribution is left-skewed (the right tail is shorter). Using `shapiro.test` (on the log-transformed data) in R, I get a test statistic of $W = 0.9718$ and a p-value of $5.172\cdot10^{-13}$, which means that we formally **reject** the null hypothesis $H_0 : \text{the sample is normal distributed}$ at the 95% confidence level. My question is: Is this good enough in practice for further analysis assuming (log-)normality? In particular, I would like to calculate confidence intervals for the means of similar samples using the approximate method by Cox and Land (described in the paper: Zou, G. Y., cindy Yan Huo and Taleban, J. (2009). Simple confidence intervals for lognormal means and their differences with environmental applications. Environmetrics 20, 172–180):               ci <- function (x) {             y <- log(x)             n <- length(y)             s2 <- var(y)             m <- mean(y) + s2 / 2             z <- qnorm(1 - 0.05 / 2) # 95%             #z <- qnorm(1 - 0.10 / 2) # 90%             d <- z * sqrt(s2 / n + s2 * s2 / (2 * (n - 1)))                  return(c(exp(m - d), exp(m + d)))     }      I've noticed that the confidence intervals tend to be centered around a point which is slightly above the actual sample mean. For example:               > mean(x)     [1] 82.3076     > y <- log(x)     > exp(mean(y) + var(y) / 2)     [1] 91.22831      I think these two values should be the same under $H_0$.


title: Semi-parametric approach: nonparametric test of residuals from linear mixed model

text: I want to know if this is a reasonable approach. I conducted the same experiment on two different species with small sample sizes (7-25 per species) in 4 years at 6 different sites. I measured damage to plants in two treatments. There is unequal sampling across years and sites. Species, site and year differ in mean damage levels. The data are highly non-normal. what I did:   1. I got the residuals of a model damage= species +site(species) as a random effect + year    2. Then I used a Wilcoxon rank sum test to test differences between treatments. Does this seem reasonable?


title: Hausman test - wrong conclusion

text: I'm performing a Hausman test in Stata 11. After the test the conclusion is to use FE panel model. In my model I have 6 variables. In FE model only one has a significant effect and in RE model 4 variables have a significant effect on the dependent variable. What do you suggest? To use FE model based on results of Hausman test or you have some other solution?


title: Looking for pointers on how/if to model a "low t and high n" longitudinal data set to evaluate outcome-predictors

text: Say you have a longitudinal data set with 5 measurement points, which begins at 200-300 participants and declines to about 50. There is an intervention between time-point 2 and 3. There's 3 continuous IVs and 1 continuous DV measured at all timepoints. One group. The key interest is the predictive value of the IVs as far into the future as possible, not the effect of the intervention. What alternative types of modeling are applicable? Do you do a bunch of regressions of the IVs on the DV between all interesting 2 timepoint- spans? Or is there some more effective type of modeling which extracts the same information in a more compact manner using less analyses? Is there any idea to do anything but a linear model given the low number of time-points? I am thankful for any answers.


title: How to prove that the radial basis function is a kernel?

text: How to prove that the radial basis function $k(x, y) = \exp(-\frac{||x-y||^2)}{2\sigma^2})$ is a kernel? As far as I understand, in order to prove this we have to prove either of the following:   1. For any set of vectors $x_1, x_2, ..., x_n$ matrix $K(x_1, x_2, ..., x_n)$ = $(k(x_i, x_j))_{n \times n}$ is positive semidefinite.   2. A mapping $\Phi$ can be presented such as $k(x, y)$ = $\langle\Phi(x), \Phi(y)\rangle$. Any help?


title: What would be the most potent free CAS for mathematical statistics?

text: I am looking for a free CAS (Computer Algebra System) that is most able to do mathematical statistics. I thought of SAGE, but it doesn't seem to be very powerful. It can't derive t-distribution : http://ask.sagemath.org/question/1485/student-t-pdf And it seems to have trouble calculating even the mean and the variance of Laplace distribution! Here's sage input/output. INPUT:               def f1(x,mu,sigma):       return 1/sqrt(2*pi*sigma^2)*exp(-1/2*((x-mu)/sigma)^2)     def f2(x,mu,sigma):       return 1/sqrt(2*sigma^2)*exp(-sqrt(2)*abs((x-mu)/sigma))          var("mu sigma")          integral(x*f2(x,mu,sigma), x, -oo,oo)      OUTPUT:               1/2*sqrt(2)*integrate(x*e^(-sqrt(2)*abs(-mu + x)/sigma), x, -Infinity,     +Infinity)/sigma      INPUT:               integral(x*f1(x,mu,sigma), x, -oo,oo)      OUTPUT:               mu      INPUT:               integral((x-mu)^2*f2(x,mu,sigma), x, -oo,oo)      OUTPUT:               1/2*sqrt(2)*integrate((mu - x)^2*e^(-sqrt(2)*abs(-mu + x)/sigma), x,     -Infinity, +Infinity)/sigma      INPUT:               integral((x-mu)^2*f1(x,mu,sigma), x, -oo,oo)      OUTPUT:               sigma^2      Does anyone know free CAS which can calculate t-distribution and the mean and the variance of Laplace distribution with parameters unkown?


title: Examples of Lurking Variable and Influential Observation

text: I have read possible explanations for Lurking Variables and Influential Observations but I can't seem to construct a good example for myself. _A well-designed experiment includes design features that allow researchers to eliminate extraneous variables as an explanation for the observed relationship between the independent variable(s) and the dependent variable. These extraneous variables are called lurking variables._ _It is possible for a single observation to have a great influence on the results of a regression analysis. Such a variable is an influential variable_ For IV I created an example as : Assume a residential building, the X axis is the number of years spent in education and Y axis is the earnings, we expect a nice rising graph (or whatever) but suddenly there is a guy who has studied very little but earns a really high amount. He is a Influential Variable. Correct?


title: Behrens–Fisher problem

text: Is there a good published expository account, with mathematical details, of the various approaches that have been taken to the Behrens–Fisher problem?


title: Collinearity problems when analyzing age-dependent variable with age-at-death as a predictor

text: Let's say I'm analyzing the incidence of pathological lesions, observed during necropsy. No information is available about when the lesion occurred before death. So, I'm using a general linear model $Y_i \sim \beta_0+\beta_{sex}+\beta_{treatment}+\beta_{age}$. $Y_i \in \{0,1\}$ is whether subject $i$ has the lesion or not. Sex and treatment are categorical variables. Age is the age at which the subject died. I already know that if I fit a Cox proportional hazards model to $age \sim \gamma_{sex}+\gamma_{treatment}$ both effects are significant. Therefore, in the model for $Y_i$ the age term must be collinear with the sex and treatment terms. The VIFs are rather large, some of them above 100. On the other hand, if I drop all age-containing terms from the model, the fit becomes significantly worse. What strategy would you recommend for somehow breaking or adjusting for this collinearity? Would fitting deviance residuals from the Cox model in place of the raw ages do the trick? It seems to work with no errors in R: `update(originalmodel,.~sex*diet*residuals(coxph(Surv(age)~sex+diet),type='deviance'))` ...and it lowers the VIFs, but I don't know if it's actually valid. * * * @Tim: I have better than box-plots-- I have survival curves, and they clearly show that females have shorter lifespans than males, and subjects on the diet have longer lifespans than untreated subjects. So the age term in the model (really, longevity) is partly determined by sex and diet. But of course there is a component that is independent of sex and diet, and I want to include just that component in the model. So that's why I'm asking if using deviance (or some other) residuals in place of age in the model would accomplish that. And the multicollinearity is severe, here are the VIFs (the model actually includes interaction terms, which I didn't mention above to keep things on- point):                     sex         diet          age     sex:diet      sex:age     diet:age sex:diet:age      82.042992   123.579264     4.270772    37.464412    72.537500   133.874481    37.464407


title: Preprocessing of non-linear and censored variables

text: I now ran into two variables that give me problems. I know that I want to try to include them in the final model, but I'm unsure how the values should be treated in order to reflect the true nature of the phenomena they are reflecting.               Case 1      The variable depicts the age/duration of a certain contract type. If the customer subscribes to this specific type, the variable shows the age of this contract since it start. However, if the customer doesn't have this type of contract, the value is zero. If I use this variable straight out of the box as a integer, you see the problem: the value "0" is quite different from any number of days (although it is in a way a correct answer). How should/could this variable be preprocessed? Durations go from days to several years.               Case 2      In my modeling I use a dataset that goes back some 3 years. I then based on this dataset calculate if and how long a go, certain changes happened. If there has been a change, I can include the change and the time elapsed since this change. But what is the best way to include these variables? Missing values indicate no change (although a cange might well have occured, but outside the collected data), while a integer indicates `n` days since a change. Both cases carry valuable information, but the data is presented in slightly different form.


title: How should you fit ANOVA and linear regression models, if the equal variance assumption is violated?

text: This is my topic for the paper I'm working on for an undergrad stats class. It's supposed to be 20 pages... and I'll be honest, I understand very little beyond the basics and am over my head.   * From what I understand, there several common tests to test if the assumption is violated... Levene's / Bartlett's, etc... and if the assumption is in fact violated, there are basically 2 paths you can take... (non-linear) transformations, or weighted least squares?   * What are some of the common things to consider when choosing whether to go with transformations / weighted least squares? What are the most commonly used transformations?


title: Facts about the mathematical difference between Student's t-distribution and normal distribution

text: I'm looking for some facts (theorems or such) concerning the properties of the Student's t-distribution compared to the normal distribution. More specifically, I understand that for the normal distribution the corresponding Student's t-distribution would be lower around the mean and heavier in the tails, and there will be two points where they cross (one on each side) and that the absolute difference will get smaller and smaller with increasing of degrees of freedom of the Student's t-distribution. What I'm interested in is what are the papers or theorems that I can use to motivate the previous sentence theoretically? Formal comparisons of the functions are extra appreciated.


title: Ranked/Ordinal data - 3 variables

text: I have the following hypothesis:   * Novice drivers perceive road environments as more difficult when there are more pedestrians. I have the following data:   * The category each driver falls in to (1. Novices, 2. Experienced, 3. Advanced)   * Number of pedestrians identified in 5 separate sections for 26 drivers    * Ranked difficulty for each section (obtained from a questionnaire from each driver) Example: Driver (1 Novice) Section (1, 2, 3, 4, 5) No. of pedestrians (5, 6, 8, 25, 20) Ranked difficulty (1, 3, 2, 5, 4) I have ranked the number of pedestrians and compared it to the ranked difficulty in SPSS using Spearmans ranked correlation. I did this separately for each driver group. The correlation co-efficients are:   * Novice 0.57   * Experienced 0.56   * Advanced 0.77 Is this the right test or is there a better test to use to analyse the data? My advanced driver data set is also much smaller than the others, will this cause problems? Thanks in advance.


title: How to measure distance for features with different scales?

text: I'm reading the book "Collective Intelligence" and in one chapter they introduce how to measure similarity between users on a movie review website with euclidean distance. Now are the movies rated all on the scale from 1-5. But what if I want to find similar users based on features like - lets say _body height_ , _body width_ , _weight_ and _ratio of eye-distance to nose-length_. This features operate on different scales, so e.g. body height would influence the distance much more than the eye-nose ratio. My question is what is the best way to approach this example. Should one use a different distance measure (which?), or normalize the data somehow and use euclidean distance?


title: Calculation of Pearson's Chi-squared statistic

text: I am having difficulty in how to calculate the Pearson's $\chi^2$ statistic for a contingency table with entries: `(3, 0)` and `(0, 3)` (row-wise). I was studying Agresti's categorical data analysis and found this table. He calculated it as 6. Through what calculation did he reach this answer?


title: Evaluating features and similarity measures

text: I am currently developing a classificator, which is supposed to classify into a number of classes. For this purpose I am designing some features and similarity measures which I might use for a later kernel for a SVM. Now I was thinking to myself, how I can roughly evaluate my features and the similarity measures without having to develop the whole classificator first. Here are my ideas, and I'd be very pleased if you could tell me, if they are valid or not (and maybe also give papers where it was done similarly..?)   1. To roughly evaluate the features, I could make a Naive Bayes assumption and try to use one feature at a time with a NB-classifier and see how it performs.   2. To evaluate the sim. measures (and at the same time features) I could calculate the inter-class mean similarity and intra-class mean similarity and see how far it differs. I was thinking, that I could maybe use an anova test to see, if the difference is significant. However, can I just assume that the results are normally distributed and have the same variance? And Anova would simply tell me, that one mean is significantly different but it doesnt tell me which one. Maybe another statistical test is more suitable? Thank you very much!


title: Regression - Dealing with Correlated, Zero-Sum Predictors

text: I'm currently working on a regression problem where a subset of the predictor variables are zero-sum. By zero-sum I don't mean they all sum to zero, I simply mean that increasing one implies a decrease in one, all or some of the others. This fact makes these variables correlated, but it also means that there is a built-in opportunity cost when one variable has a very high value since all the other predictors must be correspondingly lower. I was just curious to see if anyone has dealt with this kind of predictor variables before and, if so, have any suggestions on how to deal with these types of predictors, especially in terms of variable selection?


title: Does the normal approximation get better as a density becomes more peaked?

text: I have a sequence of densities $f_n(x_n)$, of random variables $X_n$, with means $\mu = 0$ and variances decreasing with $n$: $$ \sigma^{2}(X_n) = \frac{\sigma^2}{n}. $$ I am approximating $f_n(X)$ using a normal approximation $N(0, \frac{\sigma^2}{n})$. I would like to check if the normal approximation is getting better as $n$ increases. My intuition was that, given that $f_n(x_n)$ gets more and more peaked as the variance decreases, it becomes more "quadratic-like" so that the normal approximation should become more accurate (is some sense). I tried to verify this in several ways, by using several criterions (relative error, KL distance etc..) to check if the approximation gets better, but I have failed so far. My suspicion is that this is probably wrong, so maybe somebody can point out to me why the approximation doesn't improve. Thanks Note: $X_n$ are not sample averages, so the CLT doesn't apply.


title: Prediction evaluation metric for panel/longitudinal data

text: I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a given month, so it is zero for ~80% of the people in any month, but there's a long right tail of heavy users. The predictions I have do not seem to respect the count nature of the outcome: fractional concerts are prevalent. I don't know anything about the models. I only observe 6 different black-box predictions $\hat y_1,...,\hat y_6$ for each person per month. I do have an extra year of data that the model builders did not have for the estimation (though the concert-goers remain the same), and I would like to gauge where each performs well (in terms of accuracy and precision). For instance, does some model predict well for frequent concert-goers, but fail for the couch potatoes? Is the prediction for January better than the prediction for December? Alternatively, it would be nice to know that the predictions allow me to rank people correctly in terms of the actuals, even if the exact magnitude cannot be trusted. My first thought was to run a fixed effects regressions of actual on predicted and time dummies and look at the RMSEs or $R^2$ for each model. But that does not answer the question about where each model does well or if the differences are significant (unless I bootstrap the RMSE). The distribution of the outcome also worries me with this approach. My second idea was to bin the outcome into 0, 1-3, and 3+, and calculate the confusion matrix, but this ignores the time dimension, unless I make 12 of these. It's also pretty coarse. I am aware of Stata commands `concord` by T.J. Steichen and N.J. Cox--which has the `by()` option, but that would require collapsing the data to annual totals. This calculates Lin's Concordance Correlation Index with confidence intervals, among other useful stats. CCC ranges from -1 to 1, with perfect agreement at 1. There's also Harrell's $c$ (calculated through `somersd` by R. Newson), which has the `cluster` option, but I am not sure that would allow me to deal with the panel data. This gives you confidence intervals. Harrell's c is the generalization of the area under a ROC curve (AUC) for a continuous outcome. It's the proportion of all pairs that can be ordered such that the subject with the higher prediction actually has the higher outcome. So $c=0.5$ for random predictions $c=1$ for a perfectly discriminating model. See Harrell's book, p.493 How would you tackle this problem? Would you suggest calculating statistics like MAPE that are common in forecasting? * * * Useful things found so far:   * Slides on a repeated measure version of Lin's Concordance Correlation Coefficient


title: Basic question re PCA and dimensionality reduction

text: OK, I get that PCA can be used for dimensionality reduction but there's something basic escaping me. Let's say I have a feature set $F$ of $N$ features. After I run PCA which of the following are true about the resulting feature set. 1) It will be a subset of the original feature set $F$ with $< N$ features where each of the remaining features was in the original set $F$. 2) It will be a feature set with $<$ the number of features in the original set $F$ and where each feature is some new feature not in the original set $F$. What I'm trying to get crystal clear is does PCA simply reduce the number of features in the original set to a smaller group that are just as predictive as the complete set or does it create new hybrid features from the original set?


title: Test if the value of a function f(i, j) depends more on the choice of i than on the choice of j

text: Say I have a 24x7 matrix in which the rows represent hours in the day and the columns represent the days of the week. The values in this matrix represent the number of people present at a particular place. Mathematically speaking the values are thus the result of a function: presence(hour, day) I am looking for a way to statistically analyse this matrix to answer the following question: does the number of people present depend most on the day of the week, or most on the hour of the day? And to which extent (using some coefficient). Preferably I would like to do this with Excel. Note that the situation I describe here is just an example. In fact my data is about something also than days/hours/presence, however my problem is analogous: I have two-dimensional matrix and I want to know whether its values are more dependent on the rows than on the columns, or vice-versa. **Edit** Here is an Excel file with the example problem (hours/days matrix) and my actual data: http://brussense.be/temp/MatrixRowColDependency.xlsx **Edit 2** Please consider that I have a pretty light knowlegde of statistics and therefore may not be using the right wording to express my question. I'm not even sure whether I was right to tag this question with "correlation". **Edit 3** Maybe my hours/days example is causing more confusion than clarity. Hence, I'll explain what my actual data is about. The data I have results from a calibration experiment for a sound level meter. The meter was exposed to pure tones (sounds consisting of a single frequency) of varying frequencies and varying sound levels. In total we produced tones at 27 frequencies and 7 sound levels, resulting in 189 sounds, each of which we measured with the sound level meter being calibrated as well as with a second trusted meter which we use as a reference. For each measurement we than computed the absolute measuring error. For example:   * say I have a tone of 1 kHz played at 70 dB (as measured by the reference), but the sound level meter being calibrated indicates 68 dB. Then the absolute error for that sound is 2 dB (w.r.t. reference).   * say I have a tone of 12.5 kHz played at 85 dB (again as measured by the reference), but the sound level meter indicates 89 dB. Then the absolute error for that sound is 4 dB (w.r.t. reference). Doing this for all 189 sounds results in a 27x7 matrix, in which each value is an absolute measuring error occurring at a specific frequency and a specific sound level. I.e. the matrix rows represent frequencies, the columns represent (reference) sound levels. The question I am trying to answer is whether the errors differ more between sound levels (columns) than between frequencies (rows), or vice-versa. Or in other words (although this may in fact be another question entirely), whether the error depends more on the level than on the frequency, or vice-versa. The second sheet in the Excel file linked above holds this data.


title: What is the effect size for the interaction of a 2X2X2 contingency table?

text: I am computing the interaction in several 2X2X2 tables (i.e. in each case, I have eight frequencies), see for an example this file. I have two predictors (low vs high 'threat' and low vs high 'efficacy') and one dependent variable (whether people perform the desirable behavior ('good') or not ('bad'). My question is whether threat and efficacy interact in their associations with behavior. I've found this post about a statistical test for difference between two odds ratios, which seems to solve this problem. My question now is: can I simply take the $e^x$ of the $\delta$ (where $\delta$ = $ln(OR_1)-ln(OR_2)$) to get an odds ratio that would then represent the effect size for this interaction?


title: How does regression with and without intercept followed by test of stationarity affect cointegration test?

text: For a simple 2 variables (say X and Y) cointegration test, how does it affect our analysis, if we perform regression on X and Y with and without the intercept, and then test the spread for stationarity. I am doing this analysis for stocks.


title: Difference between VARX and ADL models

text: Could anyone please elucidate the difference between `Vector Autoregressive Models with exogenous variables, VARX(p,q`) and `autoregressive distributed lag models ADL(p,q)`? Based on the descriptions of two models, they appear to be the same so why different names are given?


title: How should I mentally deal with Borel's paradox?

text: I feel a bit uneasy with how I've mentally dealt with Borel's paradox and other associated "paradoxes" dealing with conditional probability. For those who are reading this that aren't familiar with it, see this link. My mental response up to this point has been mostly to ignore it because no one seems to talk about it, but I feel I should rectify this. We know that this paradox exists, and yet it seems like in practice (as an extreme example, Bayesian analysis) we are perfectly fine with conditioning on events of measure $0$; if $X$ is my data, we condition on $X = x$ all the time, even though this is an event of measure $0$ when $X$ is continuous. And we certainly make no effort to construct a sequence of events converging to the event we observed to resolve the paradox, at least not explicitly. I _think_ this is okay because we have essentially fixed the random variable $X$ (in principle) before the experiment, and so we are conditioning on $\sigma(X)$. That is, $\sigma(X)$ is the natural $\sigma$-algebra to condition on because the information $X = x$ is coming to use through $X$ - if it had come to us in some other fashion, we would condition on a different $\sigma$-algebra. Borel's paradox arises because (I guess) it isn't obvious what the appropriate $\sigma$-algebra to condition on, but the Bayesian has specified $\sigma(X)$. Because we are specifying a priori that the information $X = x$ came to us _by means of measuring $X$_ we are in the clear. Once we have specified the $\sigma$-algebra, everything is fine; we construct our conditional expectation using Radon-Nikodym and everything is unique up-to null sets. Is this essentially right, or am I way off? If I'm way off, what _is_ the justification for behaving as we do? [Given the Q&A nature of this site, regard this as my question.] When I took my measure-theoretic probability we, for some reason I don't understand, never even touched conditional expectation. As a result, I'm worried that my ideas are very confused.


title: What are the most popular artificial neural network algorithms for recognising the content of images?

text: What are the most used/popular artificial neural network algorithms for recognising the content of images in general? E.g.   * If the picture is of a person, dog, cat or a car.   * If the picture is a landscape, indoor or some banner or advert. etc. I've heard of backpropagation algorithm so far.


title: Constrained least squares estimation

text: I need to fit a regression model using least squares in R with the constraint that the parameters are positive. they DON'T need to sum to one. because some times parameter sum exceeds one. Can someone help me with the code pease


title: How much does quantizing continuous data affect power?

text: Background/how this came up: A student has numeric ratings on a variety of items in questionnaires and sports performance data and would like to model the performance on questionnaire scores. He wanted to do an analysis (don't ask me why) where some of the data are split into quantiles so he can compare high, medium, and low performance groups. I suggested he do a linear regression instead so that he wouldn't lose power by throwing away the continuous data - but I realized that the only papers I've seen on this being bad were about doing median splits. The question: Intuitively, it seems that quantizing your data will sacrifice power (based on arguments against median splits)- but how is this affected by the number of quantiles you choose to use? Are there circumstances where using, say, quartiles isn't much worse than doing a regression? For a given sample size, can a minimum number of quantiles be calculated to stay above a given power level?


title: Probability of getting a specific Tetris piece given previous pieces

text: I'm doing a small reinforcement learning project involving Tetris, just for fun. Considering that each piece has a constant probability of being selected, how can I calculate the probability of receiving a _specified next piece_ , given the record of the previous spawned pieces? (maybe the piece sequence is not independent as I'm using computer-generated random numbers?) My idea is simple: I would like to 'watch' the 'S', 'Z' and 'I' pieces distributions and somehow incorporate those numbers on the players policy, because are key pieces, so i think it would make a difference on the average performance. Any ideas on how I can incorporate the variables? Thanks!


title: Are two empirically estimated Markov chains statistically different?

text: I am constructing Markov chains (with 100 to 200 states) and inferring transition probabilities empirically by simply counting how many times I saw each transition in my raw data (about 20k to 60k transitions per data set). Note that the modest number of transitions and the decent number of state results in average number of measurements per edge being between 1 and 12, thus the errors on most of the inferred transition probabilities are decently high. From each data set, I construct a Markov chain. Given two Markov chains, I want a statistical test that will tell me if they are the same or not (both trying to approximate the same unknown 'true' Markov chain, or two different ones). Any advice? Any exiting code for this (preferably in Matlab, but R is fine)?


title: Is there a way to use cross validation to do variable/feature selection in R?

text: I have a data set with about 70 variables that I'd like to cut down. What I'm looking to do is use CV to find most useful variables in the following fashion. 1) Randomly select say 20 variables. 2) Use `stepwise`/`LASSO`/`lars`/etc to choose most important variables. 3) Repeat ~50x and see which variables are selected (not eliminated) most frequently. This is along the lines of what a `randomForest` would do, but the `rfVarSel` package seems to only work for factors/classification and I need to predict a continuous dependent variable. I'm using R so any suggestions would ideally be implemented there.


title: Probability of winning a tournament

text: **Edited Question:** _As I promised I've edited this question. The previous version was written with the intention of simplifying the real question, but it ended in losing the real significance. Now I'm posting the "whole story". ;)_ My purpose is to calculate $n$ players' equity in a poker tournament (their probability of ending the tournament) in every $j$ place (1st, 2nd, and so on). > I've previously solved this problem in 2 different ways you can find here: > > For the Maths: > > http://math.stackexchange.com/questions/92942/applying-a-math-formula-in-a- > more-elegant-way-maybe-a-recursive-call-would-do-t > > And for the code: > > http://stackoverflow.com/questions/8605183/how-to-translate-this-math- > formula-in-php So, when I know every players' number of chips, I can easily apply those formulas and get their equity. * * * There are 2 problems involved that hopefully can be solved with a statistical method. (I'm not a mathematician so I'm not sure it will be feasible).   1. First problem, even if I know everyone's stack, when the number of players is high, the code is too slow to be implemented;   2. Second problem, this code should work by knowing only a limited number of stacks, belonging to the players of the analyzed table. Optimistically these 2 problems can both be solved with some kind of approximations. In particular the formulas mentioned above should be applicable to scenarios with 27,45,90 players who are distributed in tables of 9. For example in the case of 27 players there would be 3 tables: when there are 18 players left they will be redistributed in 2 tables and when there are only 9 left the final table will be opened. It's not important to take into account the players' skill since it's a high variance game where its influence is reduced to the minimum, and mostly there are coin flips that eliminate players. So I'm in a situation where I know:   * My number of chips.   * The number of chips of the other 8 players of my table.   * The total number of chips.   * The average number of chips.   * The maximum and minimum number of chips. As I suggested in the previous question, this seems to me (from my humbles math skills) to be a gaussian curve, with a maximum, a minimum and an average number of chips. _I think that's all. If you need additional details please leave a comment, and I will add them as soon as possible. I wanna thank you for your interest, and for all the previous comments and answers. I hope your statistics can help me solve this. :D_ Best Regards, Giorgio. * * * * * * **Old Question:** I'd like to calculate or approximate the probability that I have to win a tournament where every player has a determined amount of chips. Let's consider a scenario where there are 9 players and I know everyone's number of chips, to calculate my probability of winning I would do: my chips/(tot chips - my chips). Now imagine those 9 players are put on 3 different tables of 3 players each, and I know the chips only of the 2 players of my table and mine obviously. I also know the total number of chips, the max and min amount chips the players have and the average stack. Is it possible to make an approximation of my probability of winning? I have only basic math skills but I think players' stack could be approximated to a gaussian curve, then use some "statistical trick" to calculate my probability. Thanks in advance for any hint! Best regards, Giorgio


title: Using distractor.analysis of R CTT library on items with missing responses

text: R and stat noob here. I have been playing with Jeromy Anglim's code in his blog post here. However, my `items$variable` has some missing values. I tried using the `distractor.analysis` function from the `CTT` package with               distractor.analysis(cases[,items$variable], items$correct)      I get the error               Error : missing values and NaN ’s not allowed if ’na .rm ’ is FALSE      Is there a work-around for using `distractor.analysis` for data with missing responses or is there a better way to perform distractor analysis in R?


title: Calculating demand and testing the effect of days of the week

text: I am running some data analysis on a site that lets users share a ride. What I am trying to determine is what separates users who are successful in offering rides and users who are less successful. This is in order to provide some tips to less successful users. The research question I'm trying to answer now is: Is there a day of the week (or set of days) that gives you a higher probability of filling your car with fellow passengers? The data I have is the ride offers from users with date and everything, and if a deal was made via our site. What's missing is how much request there is. Answering my question would be to find a mismatch between offers and potential requests. What I have done now is to calculate the amount of offers and 'deals' for the last 3 months for every day of the week. If I plot this data as three boxplots you can easily see that (offers, deals and ratio between the two):   1. The amount of deals per day has less spread than the amount of rides offered.    2. The ratio between offers and deals fluctuates. On Mondays there is a high ratio of deals to offers; on Fridays it's the other way around. Can I conclude from this that Monday would be a good day to offer a ride? How can I test the significance of this claim?


title: $t$-tests vs $z$-tests?

text: I'm trying to figure out exactly what the difference is between $t$-tests and $z$-tests. As far as I can tell, for both classes of tests one uses the same test statistic, something of the form $$\frac{\hat{b} - C}{\widehat{\operatorname{se}}(\hat{b})}$$ where $\hat{b}$ is some sample statistic, $C$ is some reference (location) constant (which depends on the particulars of the test), and $\widehat{\operatorname{se}}(\hat{b})$ is the standard error of $\hat{b}$. The only difference, then, between these two classes of tests is that in the case of $t$-tests, the test statistic above follows a $t$-distribution (for some sample-determined degrees-of-freedom $d$), whereas in the case of $z$-tests, the same test statistic follows a standard normal distribution $\mathcal{N}(0, 1)$. (This in turn suggests that the choice of a $z$-test or a $t$-test is governed by whether or not the sample is large enough.) Is this correct?


title: repeated measures regression

text: I was hoping that I could get some feedback in terms of keywords I should searching for, or concepts I need to consider with the following: I have non-experimental data that I would like to model in a regression format. Unfortunately, I only have 12 subjects and a set of repeated measures of 6 observations: spring, summer, fall for 2 years. My question is what type of model should I consider to do this? I absolutely have some reading to do, so I am hoping for any leads on where to start.. key words, or direct advice on a literature source.


title: Binary choice models and index function

text: Let's analyse the following binary choice model: $$Y^*_i = X_i^T \beta + \varepsilon_i\\ \varepsilon_i \sim F \\ Y_i = \mathbb{I} \{X_i^T \beta + \epsilon_i > 0\}$$ where $F$ is determined by a finite number of parameters $\gamma$. Define $\theta = (\beta^T, \gamma^T)^T$. $$L(\theta) = \sum_{i = 1}^{N} Y_iF(X_i^T \beta) - (1 - Y_i)(1 - F(X_1^T \beta)$$ For instance, assume that I have to show that $Y_iF(X_i^T \beta) - (1 - Y_i)(1 - F(X_1^T \beta)$ is Lipschitz in $\theta$. Should I take into account that $Y_i$ is a function of $\beta$? When we do take into account and when we don't? For instance, when we differentiate w.r.t $\theta$ in order to calculate the score, when we don't take into account or do we? I would be very grateful is someone could tell me a general rule.


title: Need help with some surprising classifications by naiveBayes

text: We are trying to do a POC on using `NaiveBayes` to classify an establishment by the category. We loaded the following training set in R.               NAME_1,NAME_2,CHANNEL_5     Vanilla,Bar,Bar &  Grill     Zen,Bar,Bar &  Grill     cafe,havana,Bar &  Grill     cafe,hollywood,Bar &  Grill     monaco,grill,Bar &  Grill     grill,grill,Bar &  Grill     hunai,asian,Bar &  Grill     Apple,Institute,School     devry,Institute,School     usu,College,School     suny,university,School     fashion,school,School     theater,study,School     Burger,King,Hamburger     Mighty,Burger,Hamburger     one,Burger,Hamburger     wendy,sandwich,Hamburger     Burger,Heaven,Hamburger     Burgler,Burger,Hamburger        Columns 1 & 2 make up the name of the establishment. Column 3 is the class.               library(e1071);          new_training_data <- read.table("C:/Users/test/08_jul_13_training.txt",     header=TRUE , sep="\t");           model1<-naiveBayes(new_training_data[,1:2],new_training_data[,3],laplace      =2);          test_data <- read.table("C:/Users/test/test1.txt", header=TRUE , sep="\t");          predict(model1, test_data[,1:2],type = c("class","raw"));      The test_set with the expected category and the actual result from NaiveBayes classifier are given below.               NAME_1,NAME_2,Expected_category,Actual_Result_from_Naive_bayes     my,Bar,Bar &  Grill, Bar &  Grill      cafe,milano,Bar &  Grill, Hamburger     Teaching,Institute,School, Bar &  Grill     devry,Institute,School, Bar &  Grill     beauty,school,School, Bar &  Grill     fashion,school,School, Bar &  Grill     theater,school,School, Bar &  Grill      Burger,Baja,Hamburger, Hamburger        Burger,Big,Hamburger, Hamburger      Pepsi,Cola,Hamburger, Bar &  Grill     Burger,Supreme,Hamburger, Hamburger      Burger,King,Hamburger, Hamburger      `my bar` is correctly classified as `Bar & grill`. However `Cafe Milano` is classified as `Hamburger` instead of `Bar & Grill`. statistically speaking the probability of `Bar` in `my Bar` is no different from `Cafe` in `Cafe Milano`. Any pointers in how `NaiveBayes` is coming up with these results will be greatly appreciated.


title: Classifier for biased dataset

text: I have a dataset with two classes and the ratio of the number of instances in two classes is 1:10. I used SVM for classification and obtained a pathetic performance as it classified all the instances in the larger class. Is there a classifier which is shown to perform well in case of biased datasets?


title: What model should I use for this research?

text: I am currently working on a project dealing with time series data but I have little experience with time series analysis so I was hoping to get some direction on what kind of exploratory data analysis techniques and models I should be looking at, so that I can read up on my own. I have two weekly time series (movie sales), of which one is a control group. The time series of interest is from a singly country A while the control group is a combination of four other countries. My research question is to examine the impact of an event that happened in country A, using the control as a counter factual to represent actual movie sales in A had the event not happened. What kind of exploratory data analysis techniques can I use, and what possible models are there for me to directly answer this research question?


title: Regression equation

text: If you have the following correlations for the following variables:               Ugrad GPA and Grad GPA  .579     GRE and Grad. GPS       .650     GRE and Ugrad GPA       .767     Interview and Grad GPA  .321     Interview and Ugrad GPA .221     Interview and GRE       .333     Gender and Grad GPA     .278     Gender adnd Ugrad GPA   .101     Gender and GRE          .011     Gender and interview    .777      Next you hacve the following means and standard deviations:               Graduate GPA   M 3.5  SD .10     Ugrad GPA      M 2.8  SD .44     GRE            M 550  SD 90     Interview      M 7.5  SD 1.5      Questions: Compute the regression equations necessary to perdict graduate GPA. Please show how you arrive at each of the predictions you make in 1a and 2b.   1. What GPa would be predicted for a person scoring 600 on the GRE?   2. What GPA would be predicted for a person earning an interview rating of 6?   3. To what extent would you suspect multicollinearity using these four predictors of graduate GPA. Identify specific variables(s) from te correlation matrix in question and provide information which supports your response. 1)


title: Nonlinear regression / Curve fitting with L-infinity norm

text: I am looking into time series data compression at the moment. The idea is to fit a curve on a time series of n points so that the maximum deviation of any of the points is not greater than a given threshold. In other words, none of the values that the curve takes at the points where the time series is defined should be "further away" than a certain threshold from the actual values. Till now I have found out how to do nonlinear regression using the least squares estimation method in R (`nls` function) and other languages, but I haven't found any packages that implement nonlinear regression with the L-infinity norm. I have found papers on "Non-linear curve fitting in the $L_1$ and $L_{\infty}$ norms", by Shrager and Hill and "A linear programming algorithm for curve fitting in the $L_{\infty}$ norm", by Armstrong and Sklar. I could try to implement this in R for instance, but I first looking to see if this hasn't already been done and that I could maybe reuse it. I have found a solution that I don't believe to be "very scientific": I use nonlinear least squares regression to find the starting values of the parameters which I subsequently use as starting points in the R `optim` function that minimizes the maximum deviation of the curve from the actual points. The idea is to be able to find out if this type of curve-fitting is possible on a given time series sequence and to determine the parameters that allow it.


title: Why is the arithmetic mean > median on a histogram skewed to the right?

text: I am fairly new to statistics. Currently I am into (histograms) medians, arithmetic mean and all the general basics. And I came across the fact/rule that the arithmetic mean is (always) larger than the median if the distribution is skewed to the right. Why is that?   (I would appreciate a rather simple or easily understandable answer).


title: Algorithm for finding the minimum in R

text: If the minimum is not unique and the MLE is equal to zero over a range of functions, how can we find the minimum? I have read about golden section search algorithm or bisection algorithm, but I don't know if they would apply in this case or not? Is there any other better algorithm which is faster than these algorithms in R?


title: Calculating induced covariances

text: I have 3 random variables, A, B and C. Given A, variables B and C are independent. I have estimates of the variance of A, B and C and the covariance between A and B and A and C. Is it possible calculate an implied covariance between B and C? My application is that I have a large number of random variables (N) and need to calculate an empirical covariance matrix from a limited number (M) of samples, where M < N. However, most of the random variables are independent of eachother given a few of the other variables, and it seems like I should be able to estimate the covariance matrix more efficiently by using the fact that many of the variables are conditionally independent.


title: Most appropriate way to make a time series stationary? (i.e. remove serial correlation?)

text: So I have this data set of 56 users with 52 weeks worth of weekly average data for blood pressure and exercise level recordings. I would like to use change point analysis (https://sites.google.com/site/changepointanalysis/) in R to understand where changes are occurring. However to make CPA usable, the observations need to be independent (at least have no strong autocorrelation). I performed ACF and the Ljung-box test for lags up to 20, and there is autocorrletion in many of the time series. So my question is, what is an appropriate way to remove this? I've looked around and have found some possibilities, but nothing has made it overtly clear why I would choose one over another, or what is a lowest risk approach. Various possibilities I have seen (some from this site):   * Low pass filter, inverse subtraction    * First differences   * Detrending   * Seasonal adjustment    * Data transformation (e.g. convert difference operator into ratio)   * Exploratory data analysis (EDA) smoothing techniques  One of the people at work said something about using a low pass filter, using inverse subtraction to remove correlation, and then finding the frequency through fast fourier transform or spectrogram analysis. I'm not sure about all that. I really appreciate any comments, I'm kind of lost at a crossroads right now. Thanks!


title: Model specification with Deflators: methodological question on forecast model

text: I am trying to build a model to predict one year ahead Earnings per share $(t+1)$ based on variables in year $t$. I’ve seen a lot of models in practice that use the following methodology:               Earnings(t+1)/#_of_shares(t) ~ x1(t) +x2(t)+x3(t)      and I’m wondering whether                Earnings(t+1)/#ofshares in (t+1) ~ x1(t) +x2(t)+x3(t)        would be the correct model. I’ve seen researchers doing that with Total Assets as well. Are they doing it because it is simpler and there is less variation when having only 1 variable t instead 2 or is my way of thinking wrong?


title: How do I conduct a simulation using a logistic model with multiple covariates using R?

text: I'm investigating the asymptotic normality of the estimators of the logistic model. I wish to do a simulation to show that the standard error decreases as sample size n increases. Assume i have explanatory variables x1, x2,...,x7 and response variable y. How do I do this?


title: Which statistical test should i use? pre-post intervention comparisions and post-post comparisons

text: In my experiment I compare the jump height of participants pre and post 2 separate interventions. All participants completed baseline measures of jump height. Participants then underwent a training intervention followed by post test measurements. On a separate occasion the same participants completed a different intervention before post test measurements were taken. I believe I should use paired t-tests to compare these groups as the same same participants were tested pre and post. However, I also want to compare the post test scores from intervention 1 with the post test scores from intervention 2 (to see if there were significant differences between the two interventions). Is there a test I should use to achieve this in one? Or do I use the paired t-tests for pre and post comparisons and a further test for the post-post comparison? Thanks in advance!


title: How good is the sailor?

text: **Which statistical test should I use** to answer the following 3 questions, using the data collected as follows: 12 sailors participated in a day long regatta, in which there were 3 races. In every race, all sailors participated and times were recorded. The course was the same for each race and all boats were identical. The only real variable, I believe, was the talent of each sailor. Lets assume that sailing talent is distributed normaly. One sailor was very strong and won every race. 1) How many sailors would have to be in the Regatta for there to be a 95% chance the strong one wouldn't win? 2) How many races would the 2nd placed sailor have to race in order to win by chance,with a 95% likelyhood. 3) How often does the winning sailors talent occur in the sailing population? For example, in terms of "He is a 1 in a million or 1 in 10,000?"


title: How big is "sufficient" sample size?

text: I am currently doing some tests with some specific parameter settings to characterize a process. I roughly remember that a widely used rule of thumb for lab tests and whatnot is a sample size of 5 or 6. Even my tutor said to do the same test 5 times. It's what were told in two of our undergraduate courses. The thing is, now that I am conducting some experiments on my own, I'd like to have a citation on that. Although I like stats, my thesis is in the field of mechanical engineering and thus this is all for a nice detail. I don't need it, but would really like to try and write down anyway. Does anyone have a legitimate source for that? Or could someone give me a few pointers on how to give a short conclusive explanation for that minimum sample size?


title: Bayesian posterior: mean vs highest probability

text: I have calculated a posterior distribution where the highest probability (peak of posterior-curve) is at 99%. But the mean probability is lower, at about 98%. This is of course because "the curve" stretches much further toward 0 than it can toward 100. I see in literature that the mean value is what matters (and the distribution around it), but my instincts say the peak-point (where the highest probability is) is more relevant. Is there a conceptual explanation why (in my case) 98% (mean) is a more relevant result than 99% (peak)?


title: Fancy Autocorrelation plot of rugarch package?

text: On page 26 in the rugarch package you can see plots of autocorrelations. I want to use this autocorrelation plots, but not for my final model checking, but for my model identification. So these are NOT the results of a fitted model, but just data (financial returns). So how can I use this autocorrelation plots for my data without fitting a model before? I don't want to use other autocorrelation function, because later I will fit a model and therefore I will have the same plots for all autocorrelation plots. Otherwise I would have differen plot designs for the same idea. How can I do this?


title: Can and should you use data from repeated responses in a linear mixed-effects model?

text: When you collect data from participants in an experiment, sometimes you can collect repeated responses for _the same condition_ , e.g., in R:               set.seed(2012) # keep the example the same each time.          data.full <- data.frame(id=gl(10, 4),                             condition=gl(2, 40),                             response=c(rnorm(40), rnorm(40, 1)))     head(data.full)          # Output:     #   id condition    response     # 1  1         1 -0.77791825     # 2  1         1 -0.57787590     # 3  1         1  0.66325605     # 4  1         1  0.08802235     # 5  2         1  1.25707865     # 6  2         1 -0.62977450      To analyse this (i.e. does condition predict response) I would normally take the mean response for each participant, for each condition. I would do this on the basis that we are supposed to be generalizing from a sample to a population, i.e. there should be one 'estimate' response from each participant for each condition, and the collection of these single responses (for each condition) is our sample, then we do an analysis which generalizes to the population. I would transform the data e.g. like this:               library(plyr)     data.means <- ddply(data.full, .(id, condition),                         summarize,                         mean.response=mean(response))     head(data.means)          # Output:     #   id condition mean.response     # 1  1         1    -0.1511289     # 2  1         2     0.8658770     # 3  2         1     0.1510842     # 4  2         2     0.0129323     # 5  3         1     0.1857577     # 6  3         2     0.9859697      And then proceed with the within-subjects analysis (note the same process would apply if there were more conditions or a 2x2 design etc.), e.g.:               aov1 <- aov(mean.response ~ condition + Error(id/condition), data=data.means)     summary(aov1) # F = 4.2, p = .07, not significant      However, I've been told that with linear mixed-effects models, you can include all the underlying data on the basis that the lme models can include correlated data. My understanding was that they could include correlated data meant they could include responses from the same participants (within-subjects effects modelled as random effects), not that you could include the underlying data that gives the participant response estimate. My question is, can you include the underlying data collected from the multiple responses of each participant in the _same condition_ , i.e. can you do this:               library(nlme)     lme1 <- lme(response ~ 1, random= ~ 1|id/condition, data=data.full, method="ML")     lme2 <- update(lme1, .~. + condition)     anova(lme1, lme2)          # X(1) = 3.19, p = .07, not significant      Or should you do this:               lme1 <- lme(mean.response ~ 1, random= ~ 1|id/condition, data=data.means, method="ML")     lme2 <- update(lme1, .~. + condition)     anova(lme1, lme2)          # X(1) = 5.25, p = .02, significant      Which is the correct approach?


title: Probability to be less the expectation

text: Let $D$ be a positive random variable with $\mathrm{E}[D] < \infty$. Assume that there exist $y>0$ and $\epsilon>0$ such that: $P(D > \mathrm{E}[D]+y) > \epsilon$. Prove that there exist other $y'>0$, $\epsilon'>0$ such that: $P(D < \mathrm{E}[D]-y') > \epsilon'$. In words, if there is some probability to be strictly above the expectation then there should be some probability to be strictly below the expectation. I do not want to assume $V(D)$ is finite.


title: Is it reasonable to delete a large number of outliers from a dataset?

text: I need advice on what is a a reasonable number of cases to be deleted as outliers. I have conducted an outliers analysis to identify univariate and multivariate outliers from my dataset. Alltogether 30% of the data was classified as outliers. If I delete all of these outliers, my results appear to improve. Also, after deleting the outliers my sample size is still good (i.e., n=300).   * Is it reasonable to delete all the cases classified as outliers?


title: Online tool for testing voting systems?

text: Let's say that I have twenty candidates in a political race and that I want to run tests on the success of each candidate based on a variety of voting systems. Is there a software that allowed me to do this, i.e. enter the candidates, pick voting system, run user tests, get results?


title: Statistical modeling of land use - 3 types of variables

text: I am struggling at the moment with how to determine links between different sets of variables. I have data on land use/cover changes in a number of regions in a period of 20 years. They are all expressed in percentages of area surface changed. I have data on so called drivers of change (population change, employment levels, education structure changes, slope, altitude etc) also expressed in percentages or indices. I have a third type of variables - a questionnaire survey. Basically what I did was multiple regression analysis on each independent variable (land cover change - forest change, grassland expansion or reduction, arable land expansion or reduction etc.) with all of the drivers of change. So I would get R for the connection between e.g. deforestation on one side and population change and altitude on the other (other drivers were eliminated through backward stepwise regression). So after I would determine that forest cover change is affected by changes in population number or altitude, I would use data from the questionnaire survey to interpret the reasons for such connection. Is there any better way to model these variables? I am dabbling into canonical correlation - could it be useful? Land cover variables on one side, and the drivers on the other? And could I include the data from the questionnaire survey in the canonical correlation in any way? Thank you very much for your help.


title: Logic behind the ANOVA F-test in simple linear regression

text: I'm trying to understand the logic behind the ANOVA F-test in Simple Linear Regression Analysis. The question I have is like follows. When the F value, i.e. `MSR/MSE` is large we accept the model as significant. What is the logic behind this?


title: Posterior derivation with prior as Normal-Gamma distribution

text: I am studying Gary Koop's _Bayesian Econometrics_ , it is bit confusing in the beginning. I will reproduce some results from the textbook here in order to smoothly move to my question. For a simple linear regression model with normal distribution assumption for the error term, its **likelihood** can be rearranged into $$p(y|\beta, h)=\frac{1}{(2\pi)^{N/2}}\bigg\{h^{\frac{1}{2}}\mathrm{exp}\bigg[-\frac{h}{2}(\beta-\hat{\beta})^2\sum_{i=1}^Nx_i^2\bigg]\bigg\}\bigg\{h^{\frac{\nu}{2}}\mathrm{exp}\bigg[-\frac{h\nu}{2s^{-2}}\bigg]\bigg\}$$ the first curly brace on the right hand side resembles Normal distribution, the second curly brace resembels Gamma distribution, where $h=\frac{1}{\sigma^2}$, $\nu$ the degree of freedom, $s^2=\frac{\sum (y_i-\hat{\beta x_i})^2}{\nu}$, $N$ the sample size. Then we have a **natural conjugate prior** , with a Normal-Gamma density $$\beta,h \sim NG(\underline{\beta},\underline{V},\underline{s}^{-2},\underline{\nu})$$, underline indicates the prior parameters. $\underline{V}$ is the variance of $\beta|h$. After this, the author provide the **posterior** directly $$\beta,h|y \sim NG(\bar{\beta},\bar{V},\bar{s}^{-2},\bar{\nu})$$ where $$\bar{V}=\frac{1}{\underline{V}^{-1}+\sum x_i^2}$$ $$\bar{\beta}=\bar{V}(\underline{V}^{-1}\underline{\beta}+\hat{\beta}\sum x_i^2)$$ $$\bar{\nu}=\underline{\nu}+N$$ and $\hat{\beta}$ is the OLS estimate. I have no idea how author get these results, I checked the reference he mentioned, but I still could not figure out the deriviation. Would anyone show me how do get posterior as author did under these model settings? Thank you!!


title: Understanding the results of an F test in R

text: I am performing the test to see if there is any differences between the preferences of chocolate for male and female students. My null hypothesis is that there is no difference. My data is not normally distributed and data transformations failed to 'squash' the data. Therefore I performed the F test on the unchanged data:               var.test(dframeA$Chocolate,dframeB$Chocolate)                   F test to compare two variances          data:  dframeA$Chocolate and dframeB$Chocolate     F = 0.9539, num df = 238, denom df = 138, p-value = 0.7447     alternative hypothesis: true ratio of variances is not equal to 1     95 percent confidence interval:      0.7041521 1.2768820     sample estimates:     ratio of variances               0.9538888       The above shows the result. I am very new to R (and statistics for the matter). Please would you help me understand what the results mean? I am assuming that this test is 2-tailed..


title: Interaction insignificant, main effects significant

text: I am using modprobe syntax on SPSS to test an interaction between narcissism and rumination on a dependent variable, aggression. I get a significant effect of rumination ( _b_ = .6450, _t_ = 2.32, _p_ = .0216) and significant effect of narcissism ( _b_ = .5646, _t_ = 2.5, _p_ = .0135). The interaction is not significant ( _b_ = -.069, _t_ = -.824, _p_ =. 4114). I'm not really sure how to interpret this. Can I still look at the plot and slopes? Or are they now meaningless as they are insignificant? (Or should I look at them and try to see why it is insignificant, e.g., because narcissism lines are too close together at +1 _SD_ of rumination?) The modprobe output has a section called: 'conditional effect of predictor at values of the moderator' So:               Rumination -1sd:  narcis=2.25.   B=  .49.    T= 4.19. P=  .000     Rumination  Mean: narcis=3.21.       .42.       4.69.     .000     Rumination  +1sd: narcis=4.17.   B=  .36.    T= 2.86. P=  .0048      Since all these are significant, doesn't this make the interaction significant? Basically, I'm not sure what the significance in the above table represents since the main effects' and interaction's significances are already shown elsewhere. Secondly, is it possible that the interaction is not significant because of the high _p_ at +1 _SD_? All I really want to know is how to report this output, but I'm trying to grasp some understanding of it while I'm at it.


title: When Should The Following Tests Be Run

text: What are the assumptions behind each of the following types of tests (i.e. what are the prerequisites to running each test).   1. Paired-t   2. Sign   3. Two-Sample t   4. Pooled Two-Sample t   5. Two Proportions   6. F I really am getting confused seeing what would warrant one over the other given criteria like (normality, variance, sample size etc.) Any help would be appreciated. (Note: This is for a class assignment)


title: Drawing a realization from a random variable

text: Say there is a random variable `X` which can take 3 values `1`,`2`,`3` with equal probability. The task is to draw realizations from this random variable. Any simple textbook will say consider a realization as `1` if you get a realization from Uniform distribution as less than, 0.33, and as `2` if you get a realization from Uniform distribution as less than, 0.66 (and more than 0.33), and otherwise `3`. This is okay, however, I could not understand the logic of it. I understand that logically it is somehow related to the fact that the CDF of all random variable has a uniform distribution, however, I could not establish the link between above drawing policy and this CDF's distributional theory. Could somebody please explain this to me in an intuitive way?


title: What tools could be used for applying clustering algorithms on MovieLens?

text: I need to analyze the 100k MovieLens dataset for clustering with two algorithms of my choice, between the likes of k-means, agnes, diana, dbscan, and several others. What tools (like Rattle, or Weka) would be best suited to help me make some simple clustering analysis over this dataset?


title: Robust estimation of Poisson distribution

text: I have a set of numbers which are assumed to be coming from a Poisson distribution. The set has some outliers also and because of that, maximum likelihood estimates are badly affected. I heard that robust estimation procedures can help in such a situation. Can any one explain how to do this? I am not a statistics student. I found that the `glmrob` function in R can be used for this. (I am quite new to R). But I could not figure out, how to use that despite reading the manual pages. In particular I am unable to understand how to get a `forumula` which is the first argument to the glmrob function. Thanks.


title: How to choose the number of hidden layers and nodes in a feedforward neural network?

text: Is there a standard and accepted method for selecting the number of layers, and the number of nodes in each layer, in a FF NN? I'm interested in automated ways of building neural networks.


title: Is there a difference between an autocorrelated time-series and serially autocorrelated errors?

text: I'm pretty sure that I'm missing something obvious here, but I'm rather confused with different terms in the time series field. If I understand it correctly, serially autocorrelated errors are a problem in regression models (see for example here). My question is now what exactly defines an autocorrelated error? I know the definition of autocorrelation and I can apply the formulas, but this is more a problem of comprehension with time series in regressions. For example, let's take the time series of daily temperatures: If it is a hot day today (summer time!), it's probably hot tomorrow as well, and vice versa. I guess I have a problem to call this phenomenon a phenomenon of "serially autocorrelated errors" because it just doesn't strike me as an error, but as something expected. More formally, let's assume a regression set-up with one dependent variable $y_t$ and one independent variable $x_t$ and the model. $$ y_t = \alpha + \beta x_t + \epsilon_t $$ Is it possible that $x_t$ is autocorrelated, while $\epsilon_t$ is i.i.d? If so, what does that mean for all that methods that adjust standard errors for autocorrelation? Do you still have to do that or do they only apply to autocorrelated errors? Or would you always model the autocorrelation in such a setting in the error term, so it basically doesn't make a difference if $x_t$ is autocorrelated or $e_t$? This is my first question here. I hope it's not too confusing and I hope I didn't miss anything obvious...I also tried to google it and found some interesting links (for instance, here on SA), but nothing really helped me.


title: Machine learning input relationships

text: After learning about a few machine-learning models (NN, SVM, decision trees), I was wondering if these models are able to find inherent relationships when learning. For example, if I feed it two inputs A and B, but it is really A - B or the percent change from A to B that determines the output (but this relationship is unknown to me), would these models be able to pick up that relationship from just the A and B inputs? What about more complicated relationships?


title: How to control for depression when sample was measured using two different tests?

text: I am looking at pre and post test depression scores in a 2x3 design with treatment and genotype as the IVs and depression score as the DV. However, I have to use one depression test for younger people and a different depression test for older people, because I would like to covary out age. I think I need a repeated measures ANCOVA but am not sure how to compare the depression data from 2 different tests. Any suggestions?


title: Support Vector Machines and Missing Values

text: Suppose a data set has the following binary values for the variable **z** :               z  x  y       1     0     1     1     0     1      The variables are **x** and **y**. The training data set consists of the first three values. The test data set contains the last three values. When I run the following R code:               library(e1071)     tuned <- tune.svm(z~., data=train, gamma=10^(-6:-1), cost=10^(-1:1))     summary(tuned)          model <- svm(z~., data=train, type="C-classification", kernel="radial", gamma=0.1, cost=10)     summary(model)          prediction <- predict(model, test[,-1])     tab <- table(pred=prediction,true = test[,1])      I get the following error:               Error in table(pred = prediction, true = test[, 1]) :        all arguments must have the same length      I am not sure why this is happening. Maybe the predict function ignores missing values? I imputed the data and this is still happening.


title: How to derive this problem with soft-thresholding method?

text: The problem is defined as $$ \min_{x} \Bigg\{ a{\|x\|}^2+\frac{b}{2}{\|x-c\|}^2 \Bigg\} $$ where $x\in R^{n \times 1}, c \in R^{n \times 1}$ and $a,b$ are scalars. Equations 2.5 to 2.8 of this paper say this could be solved by soft-thresholding, and the $\arg \min$ is $$ x ^*=\frac{c}{\|c\|_2}\max \Bigg\{ {\|c\|}_2-\frac{a}{b}, 0 \Bigg\} $$ Could anyone provide some references on how to derivative this solution, please? My derivation is as follows: $$ ax^Tx+\frac{b}{2}(x^Tx-2x^Tc+c^Tc)=\left(a+\frac{b}{2}\right)x^Tx- bx^Tc+\frac{b}{2}c^Tc $$ Then, the minimum is at the point that gradient is zero, i.e. $$ (2a+b)x-bc=0 $$ then, $x=\frac{b}{2a+b}c$. What's wrong with my derivation?


title: Plot 3 density plots in the same graph, showing also standard deviation limits in each one

text: I have obtained from a paper next results: Variable 1: Mean=165, sd=15 Variable 2: Mean=149, sd=18 Variable 3: Mean=134, sd=25 I have simulated normal distributions using rnorm. This is the way (with a n=1000): rnorm(1000,165,15). The same procedure with the others. So now I want a common graph that shows the 3 normal distributions (density plots) at the same time. And also, that each of the density plots shows standard deviations bounds (1 and 2 sigmas) and if possible, means. This may be represented as vertical lines from the X axis til the curve of the plot. Anybody knows which is the exact code in R?


title: Using Yule Walker equations for ACF and PACF

text: When using Using Yule Walker equations for getting ACF and PACF, is it essential that the time series has to be stationary? In other words, do we really need Box-Cox transformations before we use Yule Walker for ACF and PACF Thanks, Ratheen