title: What's the best way to test the uniformity of location data?

text: I'd like a test to determine whether a number of collected 2D points (say, x,y in (0,1]) are distributed uniformly across the space. The points will be one or more 'paths' through the space - collections of points resembling movement through the 2D space. I want to determine, for the entire area, whether the entire set of points within all paths looks uniformly distributed across that area or if the points have some degree of proximity (e.g., if most paths stay in one corner of the graph). I've considered these:   1. Simple regression, but I don't expect to know how the points are distributed if they aren't uniform (polynomial doesn't seem to be a good fit)   2. Taking discrete chunks of the space and computing the sum of square error between the expected density and the observed density, or using a Chi-squared on the segments.    3. Some sort of clustering algorithm/test - not sure what I'd use here, but again, I'm not sure clusters are what I'm looking for Any other suggestions/ideas?


title: How to interpret constant with different dummy interaction terms?

text: I would like to analyse the impact of Fund Size on Mutual Fund Performance by using quintiles (based on fund size). My approach is to look at the effect within each quintile to conclude uniformity of the potential effect among all size groups. My initial thought was to use a dummy regression (quintiles=dummies) with interaction terms (dummy*Lagged Fund Size). However, I would like to include a quadratic effect of the Lagged Fund Size as well and use interaction terms. In other words, analysing both the linear and quadratic effect in each quintile by using interaction terms. To avoid multicollinearity, I have to omit one of the quintiles (i.e. dummies) (since excluding the constant term instead results in the omission of one of the interaction terms in both Stata and SPSS). My questions are the following:   1. How can I interpret the constant term as it includes both the linear and quadratic effect for the omitted dummy? Is it even statistically correct?    2. Ideally I would like to include all dummies hence excluding the constant term to better visualise the effect within each quintile. Is it possible to do this (by for example omitting the general (non-interaction) effect, i.e. either the dummies or the Fund Size effect in the regression)?    3. Are there any other models/regression forms that could avoid this hindrance and analyse the linear and quadratic effect in one regression?


title: Determine if a % Change occurred

text: I'm trying to figure out if a change to a website that occurred on a specific date increased page views on that site by 5%. I have page view data for a month before the change and several weeks after. The page views fluctuate on a daily basis by +-5%, so how can I determine if a change in page views was caused by something specific or just part of the normal fluctuation?


title: Compare Models. LOCCV implementation in R

text: I want to compare three models, one linear-regression-model, one regression- tree-model (from `rpart`) and one MARS-model (from `mda` package). I want to compare the models using a _leave one out cross validation_ using the mean square error and MAPE. I have the following implementation in R:               library(data.table)     library(rpart)     library(mda)          #Load Sample-Data     data(trees)          #The following models should be compared:     # lm(Volume~Girth+Height, data=trees)     # rpart(Volume~Girth+Height, data=trees)     # mars(trees[,-3], trees[3])          LOOCV<-function(modelCall) {       unlist(sapply(seq(1,nrow(trees)), function(i) {                  training=trees[-i,]         test=trees[i,]              fit=eval(modelCall)         testValue = predict(fit, test[1:2])              test[3]-testValue       }))     }          LOOCV_MSE<-function(modelCall) {        sum(LOOCV(modelCall)^2)/nrow(trees)     }          LOOCV_RMSE<-function(modelCall) {        sqrt(LOOCV_MSE(modelCall))     }          LOOCV_MAPE<-function(modelCall) {       sum(abs(LOOCV(modelCall)/sapply(seq(1, nrow(trees)), function(i) {trees[i,3]})))/nrow(trees)*100                                         }               cat("Cross-Validation Metrics:\n")     cat("-------------------------\n")     cat("LOOCV MSE for LM:", LOOCV_MSE(quote(lm(Volume~Girth+Height, data=training))),"\n")     cat("LOOCV MSE for CART:", LOOCV_MSE(quote(rpart(Volume~Girth+Height, data=training))),"\n")     cat("LOOCV MSE for MARS:", LOOCV_MSE(quote(mars(training[,-3], training[3]))),"\n")     cat("\n")          cat("LOOCV RMSE for LM:", LOOCV_RMSE(quote(lm(Volume~Girth+Height, data=training))),"\n")     cat("LOOCV RMSE for CART:", LOOCV_RMSE(quote(rpart(Volume~Girth+Height, data=training))),"\n")     cat("LOOCV RMSE for MARS:", LOOCV_RMSE(quote(mars(training[,-3], training[3]))),"\n")     cat("\n")          cat("LOOCV MAPE for LM:", LOOCV_MAPE(quote(lm(Volume~Girth+Height, data=training))),"\n")     cat("LOOCV MAPE for CART:", LOOCV_MAPE(quote(rpart(Volume~Girth+Height, data=training))),"\n")     cat("LOOCV MAPE for MARS:", LOOCV_MAPE(quote(mars(training[,-3], training[3]))),"\n")      Outputs:               Cross-Validation Metrics:     -------------------------     LOOCV MSE for LM: 18.15783      LOOCV MSE for CART: 69.83769      LOOCV MSE for MARS: 13.72282           LOOCV RMSE for LM: 4.2612      LOOCV RMSE for CART: 8.356895      LOOCV RMSE for MARS: 3.704432           LOOCV MAPE for LM: 14.6114      LOOCV MAPE for CART: 23.51401      LOOCV MAPE for MARS: 10.00316       Does this implementation make sense? When whould using MSE on the errors make sense? When would I use MAPE/SMAPE instead? I already read "Metric to compare models?" and the conclusion there was _it depends_ , can someone explain this further. On what does it depend? My data is not a time series, it is more like the `tree` example data.


title: Bayesian statistics example

text: I am learning Bayesian statistics for a data mining module. We were given a set of practice questions but no answers and I want to make sure that I am going about Bayes in the right way. Suppose Manchester United were about to play Arsenal in the Premiership and you assess the probability of Manchester United winning to be 0.7. You also feel that if they did win, there is a probability of 0.9 that your local pub will be packed with fans celebrating their club’s victory. Alternatively, if they loose, you believe that there is still a probability of 0.6 that the pub will be packed with fans albeit for drowning their sorrows. As someone not interested in football, you enjoy a quiet day at work and on the way home, notice a big crowd in the pub. Can you assume that Manchster United won the game? If so, what is the probability of them having won the game? Attempted solution: let $p(w)$ denote prob of manchester united winning let $p(f)$ denote prob of pub being full $p(w|f)=\frac{p(f|w)*p(w)}{p(f)}$ , where $p(f)=p(f|w)*p(w)+p(f|¬w)*p(¬w)$ $p(f)= 0.9*0.7+0.6*0.3=0.81$ $\rightarrow$$p(w|f)=\frac{0.9*0.7}{0.81}=0.78$ So prob of Manchester untied having won is 0.78. I don't think we can assume they won just because there were people in the pub as there was a 0.6 prob of them drowning their sorrows. Have I got this right or have I misunderstood Bayesian statistics? Thanks very much


title: How to display multiple density or distribution functions on a single plot?

text: What would be the easiest way to plot in the same graph, the probability density function or the cumulative distribution function of a distribution, for 3-4 different parameter values. Let's take for instance the Weibull distribution W(a,b). I would want to plot the PDF for a=1/2,b=20 , a=1/3,b=20, a=1/4,b=20, a=1/5,b=20 in the same graph and in different colors. What is the easiest way to do this? You can recommend any kind of software. Also, the program should let me define custom distributions.


title: Drawing without replacement

text: An urn contains $X$ white balls and $Y$ black balls. What is expected number of balls you will draw _before_ drawing a black ball? For example, if$X=2,\ Y=1$, the possible outcomes would be:               WWB     2 balls     WWB     2 balls     WBW     1 ball     WBW     1 ball     BWW     0 balls     BWW     0 balls      And hence $E(\text{balls before first black}) = \frac{2+2+1+1+0+0}{6} = 1$ This is not homework, I have just reworded a real world problem I have encountered. It's been a while since I've done this kind of probability. I am looking to quickly compute this value for various $X$ and $Y$.


title: Dashed lines in ACF plot in R

text: I'm going through the book 'Introductory Time Series with R' by Cowpertwait and Metcalfe. On page 36 Its says the lines are at: $-1/n \pm 2/\sqrt{n}$. I've read here R forum that the lines are at $\pm 1.96/\sqrt{n}$. I ran the following code :               b = c(3,1,4,1)          acf(b)      and I see that the lines look to appear to be at $\pm 1.96/\sqrt{4}$. So, obviously the book is wrong? Or, Am I misreading what has been written? Are the authors talking about something slightly different? *Note, I'm not interested in the 1.96 vs 2 minor detail discrepancy. I assume this was just the author using the rule of thumb of 2 sd's versus the actual 1.96 sd. Edit: I ran this simulation:               acf1 = 0     acf2 = 0     acf3 = 0     for(i in 1:5000){       resids= runif(1000)       residsacf = c(acf(resids,plot= FALSE))       acf1[i] = residsacf$acf[2,,1]       acf2[i] = residsacf$acf[3,,1]       acf3[i] = residsacf$acf[4,,1]     }     meanacf1 = mean(acf1)     meanacf2 = mean(acf2)     meanacf3 = mean(acf3)     meanacf1     meanacf2     meanacf3      I always seem to get values near $1/n$ for all 3. Further edit : I'm seeing a trend of $1/n-(k-1)/n^2$


title: Is there a statistical model for modelling variables that are measured in varying amounts and in different time points per individual?

text: I have been trying to model a dataset of variables where each individual is measured a different number of times, and at a different point in time. Most of my variables are counts, but some are not (the target variable is a count). Furthermore there is a bunch of zeros over the count variable. The closest I was able to find was a Zero-Inflated Poisson model, but as far as I've seen this model assumes that I measured each individual the same amount of times and over the same period. Neither of the two assumptions is true in my case. I saw a formula (without name), where a regression can be performed such that: $Y_i$$_t$ = $S_1$$_i$*$a_i$$_j$ + $\epsilon_i$$_t$ + $I_0$$_i$ $I_0$$_i$ = $\alpha_0$ + $\sigma_0$$_i$ $S_1$$_i$ = $\alpha_1$ + $\sigma_1$$_i$ where $i$ indicates individual and $t$ time, around the internet, but the slide simplified $a$ by removing the individual. Furthermore I am not certain if this time would allow different times for each individual. Since time here is measured in seconds and the amount of measures for an individual can vary from 5 times to over 100 times across years, plotting many constant variables would most likely to a explosion of points as well. Can statistics address this? I have been having the same issue with data mining and machine learning algorithms as well due to the time constraint. The regression is multivariate and predictors are not necessarily counts. I appreciate any pointers, books, papers on any substantive field.


title: How can we predict random non-balanced events?

text: Prediction is the ability to statistically foretell the occurrence of future events by learning from historical data. In all the cases, if we have a large enough sample of data on how people behaved in the past, we can predict their behaviors with a reasonable degree of confidence. But, how can we predict events that are random and rare. I am working on study to predict earth-quack. I am doing this study with a limited resources, so I don't have enough data about geology. I want to model this problem statistically based on what happened before. As first approach, I was thinking to use poisson model, so I count earth-quack that happened in specific place, calculate the next events using this model, how can we add a trick to make sure that this event would happen. My questions are:   * Is it good approach to use counting model to tackle this problem ?   * Can you, please, suggest any other effective ways to tackle this problem ?


title: Simple statistical test of experimental data

text: I have some experimental data: pre- and post-treatment results for 24 subjects, 12 treatment and 12 control. I created the two groups by pairing subjects according to pre-test results and randomly assigning within each pair. Now I'm trying to figure out how to test the significance of my treatment effect. I was planning to calculate the difference between pre- and post- scores for each subject, then do a paired-sample t test of treatment vs. control subjects. Does this make sense? Any advice much appreciated!


title: Weibull moment generating function and Gamma function

text: I'd be grateful for any hints or help with this question: Let $X$ follow the Weibull distribution with pdf $f(x)=\beta x^{\beta-1}e^{-x^{\beta}}$ on $x>0$ with $\beta>0$. Show that $E(X^r)=\Gamma(\frac{r}{\beta}+1)$ where $\Gamma(a)=\int_{0}^{\infty}x^{a-1}e^{-x} dx$ * * * This is how far I have got....... $E(X^r)=M_X^{(r)}(0)$ $M_X(t)=E(e^{tX}) = \int_{0}^{\infty} e^{tx}\beta x^{\beta-1}e^{-x^{\beta}}$ Let $u=x^\beta$ So $E(e^{tx})=\int_{0}^{\infty} e^{tu^{1/\beta}}e^{-u}$


title: Checking Poisson distribution plot using mean and variance relationship?

text: Im new to Stack exchange, and a student of statistics. Thus I have a question for you guys.... So my questions is asking if it is possible to check the distribution of a random sample comes from a poisson distribution by plotting the mean and the variance of the a distribution. This is what the code I get for it?                   pos<-rpois(100,5)         x.mean<-rep(0,500)         x.var<-rep(0,500)         for(i in 1:500){         samp<-sample(pos, 10)         x.mean[i]<-mean(samp)         x.var[i]<-var(samp)         }      qqplot-mean and varaince                   quants <- seq(0.01,0.99,length.out=500)         x <- quantile(x.mean, probs=quants)         y <- quantile(x.var, probs=quants)         plot(x,y,xlab="mean",ylab="variance")      And then I notice that the plot(x,y) is pretty much a straight line. So I can conclude that it is a poission distribution. Somehow this just seems wrong though, please let me know what you think? Best


title: Does cross validation work with asymmetric loss functions?

text: My simple question is does cross validation work with an asymmetric loss function? I cannot find docs on google to answer.


title: What can you do when you have predictor variables that are based on group averages with different sample sizes?

text: Consider a classical data analysis problem where you have an outcome $Y_{i}$ and how it is related to a number of predictors $X_{i1}, ..., X_{ip}$. The basic type of application in mind here is that   1. $Y_{i}$ is some group-level outcome such as the crime rate in city $i$.    2. The predictors are group level characteristics such as demographic features of city $i$.  The basic goal is to fit a regression model (perhaps with random effects but forget that for now): $$ E(Y_{i} | {\bf X}_{i} ) = \beta_0 + \beta_1 X_{i1} + ... + \beta_p X_{ip} $$ **Does some technical difficulty arise when one (or more) of the predictors are the result of a survey that has different sample sizes for each unit?** For example, suppose $X_{i1}$ is a summary score for city $i$ that is the average response from a sample of individuals from city $i$ but the sample sizes these averages were based on are wildly different: \begin{array}{c|c} {\rm City} & {\rm Sample \ size} \\\ \hline 1 & 20 \\\ 2 & 100 \\\ 3 & 300 \\\ 4 & 5 \\\ 5 & 3 \\\ \vdots & \vdots \\\ \end{array} Since the predictor variables do not all have the same meaning, in some sense, for each city, I'm afraid that conditioning on these variables in a regression model as though they are all "created equal" could cause some misleading inferences. **Is there a name for this type of problem? If so, is there research on how to handle this?** My thought is to treat it as a predictor variable measured with error and do something along these lines but there is heteroskedasticity in the measurement errors, so that would be very complicated. I could be thinking of this the wrong way or may be making this more complicated than it is but any discussion here would be helpful.


title: Efficient Portfolio Optimization Through Simulation

text: Apologies in advance for the (possibly?) poor terminology as I'm a bit of a novice in the field. I was torn whether to ask this on stackoverflow or here, so hope its the right place. Anyway, my problem is I'm trying to efficiently calculate the best weights for a portfolio of assets that will maximize the sharpe ratio (or similar measure of risk adjusted returns) of the portfolio as a whole, where portfolio returns are unknown, non-normal and dependent on each other, but simulated by a seperate function. The processing chain is as follows:   1. Simulate returns from portfolio (I already have this part; assume that the simulation uses a VAR Copula Garch model incorporating all assets in the portfolio, though how it arrives at the simulation is not important for this question, just that the asset returns are not independent but can be simulated.)    2. Calculate weights for each asset that will maximize some risk/return measure (e.g. sharpe ratio) of the whole portfolio    3. Repeat above until stable points are found (e.g. each 'simulation' is very noisy by itself/far from the average over all simulations). I thought of doing this the brute force way via monte carlo simulation, but thought it best to seek expert advise first. So my question is essentially:   1. How would you recommend I go about doing the above in R.   2. Are there any packages, algorithms or resources you would recommend I use/look into for this task? Goal is to have a proof of concept up asap, and if the ideas show practical utility to delve deeper into the theory and better methods.  The easier it is to get setup the better; ideal would be recommendation on a flexible/efficient optimization framework allowing constraint specification (e.g. can have negative weights; must not have above 20%, must not be below 2% (else %0 weighted) etc... Yes, it would be nice if it also included a free lunch ;-) ) Thank you for taking the time to read this, and any thoughts or comments you have in advance!


title: Mapping a value in a normal distribution

text: I apologize in advance for misuse of terminology. If given a number $x$ from a normal distribution with mean 0 and standard deviation 1, how can I map that onto another distribution with a different mean $m$ and standard deviation $s$? Is it simply $m + s\,x$ ? Clarification - the real-world problem I'm trying to solve: I would like to generate random numbers in a normal distribution for which I know the mean and standard deviation, however, I can only generate random floating point numbers in a distribution with mean 0.0 and stddev 1.0. How can I translate or "map" this value onto a different distribution?


title: GEE, quasi-likelihood and what it generalizes

text: Wikipedia formulates Generalized Estimating Equations (GEE) as > Given a mean model, $\mu_{ij}$, and variance structure, $V_{i}$, the > estimating equation is formed via: $$ U(\beta) = \sum_{i=1}^N \frac{\partial > \mu_{ij}}{\partial \beta_k} V_i^{-1} \\{ Y_i - \mu_i(\beta)\\} \,\\! $$ The > parameter estimates solve U(β)=0 and are typically obtained via the Newton- > Raphson algorithm.   1. Does GEE belong to maximum quasi-likelihood method (is the maximum quasi-likelihood method same as quasi-likelihood estimation?) If yes, what is its quasi-likeilhood function, or does GEE maximizes some quasi-likelihood function?   2. What does GEE "generalize"? Is it estimating equation method for estimation?  In what sense is GEE "generalized"? Is it similar to the way in which the generalized linear model generalizes the linear model? Thanks and regards!


title: how to Normalize data(with noise) into 0-1 range good scale in mean and variance?

text: i have a matrix data. Perhaps some data in one cluster and another in some cluster. data scale is between [0-1000](just example). and i want to normalize into [0-1] and good in mean and variance. it means, i want to have a Good visibility in mean and variance. for more description,mean and variance Be chosen in perfect place that Consider all data, and not ignore noisy data. Foe example one way is zi=xi−min(x)max(x)−min(x), but this way if Most of the data in one cluster and few data in another cluster, mean and variance be chosen between large cluster and not attend to noisy data.   Please introduce me a paper(if newest is better) or a way to solve this problem. Thank u.


title: When to use B'' or B''D as a measure of response bias?

text: I hope this is not a stupid question, but here it goes: Based on information from Macmillan and Creelman's _Detection Theory_ (2005) and Pallier's R-code that I found here, Computing discriminability and bias with the R software, I came up with code to calculate $B^{''}$ (used as a measure of response bias in psychological "yes/no"-experiments). Pallier suggests a diferent measure of bias than Macmillan & Creelman (pp. 100-104, 371). In the code below, "fa" stands for false alarm rate and "hit" for hit rate.               > bpp <- function(hit, fa) {         a <- ((hit*(1-hit)-fa*(1-fa)) / (hit*(1-hit)+fa*(1-fa)))         b <- ((fa*(1-fa)-hit*(1-hit)) / (hit*(1-hit)+fa*(1-fa)))         a[fa>hit] <- b[fa>hit]         a[fa==hit] <- 0         a       }      Here is the original $B^{''}_D$ code as provided by Pallier:               > bppd <- function(hit,fa) {         ((1-hit)*(1-fa)-hit*fa) / ((1-hit)*(1-fa)+hit*fa)       }      What is the difference between these two measures of response bias? When is it appropriate to use which?


title: When the selection depends on a prior experience, what statistical test is appropriate?

text: We have raised a batch of frogs on one species A fly. We then present the frogs with species A and a second species B fly. Will the frogs pick species A or B? Lets say 40% choose A and 60% choose B. is this selection random? If not random, then how do you show that it depends on prior food exposure? or it does not depend on prior food exposure? or that one fly species is preferred over another? So, we know that the frogs were raised on species A flies. The question is when a frog raised on species A flies is presented with A flies and B flies. When it makes the choice is it a random or not random choice? So, if 50% choose A and 50% choose B. Then can one say it is random selection? But if 60% choose A and 40% choose B, can we say that it is not random? Can you further state that the choice was influenced by prior exposure? When the selection might depend on a prior experience, what statistical test is appropriate? Is it possible to set this up as a chi square test? or McNemar's? How do you set it up?


title: Consistency of the learning process

text: I have two questions related to the concept of "learning consistency" for those who are familiar with statistical learning theory a la Vapnik. Question 1.   The learning process is called consistent (for the class of functions $\mathcal{F}$ and probability distribution $P$) if $$ R_{emp}(f^*_l) \buildrel P \over \to \inf_{f \in \mathcal{F}} R(f),\;l \to \infty $$ and $$ R(f^*_l) \buildrel P \over \to \inf_{f \in \mathcal{F}} R(f),\;l \to \infty $$ These two conditions are independent. On p. 83 of Vapnik's "Statistical Learning Theory" there is an example of a set of classifiers $\mathcal{F}$ such that the second convergence takes place but the first one doesn't. I was thinking about an example of a set of classifiers such that _the first_ convergence takes place but _the second_ one doesn't, and couldn't come up with anything. Can anyone help me here? Question 2.   The learning process is called nontrivially consistent (or stictly consistent) (for the class of functions $\mathcal{F}$ and probability distribution $P$) if for any real number $c \in R$ such that set $\Lambda(c) = \\{ f | R(f) \geq c \\}$ is not empty we have: $$ \inf_{f_l \in \Lambda(c)}R_{emp}(f_l) = R_{emp}(f^*_l) \buildrel P \over \to \inf_{f \in \Lambda(c)} R(f),\;l \to \infty $$ P. 81 of Vapnik's "Statistical Learning Theory" provides an illustration of why we want to consider strict consistency instead of consistency defined in Question 1, i.e., why we want to introduce $\Lambda(c)$ and consider $\inf_{f \in \Lambda(c)}$ for any $c$. All other texts that consider strict consistency essentially duplicate Vapnik's illustration when they want to explain the rationale behind the concept of strict consistency. However, I'm not really happy with Vapnik's illustration due to 2 reasons: first, it's done in terms of loss functions $Q(z, \alpha)$ and not the classifiers, and, second, Fig. 3.2. from the book doesn't really make sense when we consider the common loss function for classification problems, i.e., the function that is equal to 0 when predicted class label is equal to the true class label and to 1 otherwise. So, is it possible to give another, more sensible, illustration of the rationale behind the concept of strict consistency? Essentially, we need an example of a set of classifiers such that these classifiers are not consistent (in terms of the definition from Question 1) and some new classifier performing better that any of the classifiers from the set, so that when we add this classifiers to the set we end up with the case of "trivial consistency". Any ideas?


title: Maximization of overall hit rate

text: I'm looking for some pointers with an optimization problem, specifically how to have my data formatted If I want to use R/Excel/Octave for optimization (those 3 are the only applications I have access to). I have 4 groups A, B, C, D. Each of these groups has 5 integer values, 1 to 5. For each integer I have volume (No. cases) and No. positive cases. I would like to pick 1 integer for each group such that my overall hit rate is maximized, where hit rate is defined as No. of positive cases/total number of cases. Sure, we can enumerate all possible solutions and pick the one. In the above case, potential possibilities is $5^4$. However, what I've posted is a smaller representation of bigger problem I'm working on. I have 5 Groups and each integer value goes from 0 - 999 where the potential no. of solutions would be $1000^5$. Constraints I thought of:   * Ensure values from each group A, B, C, D are strictly integer   * From domain knowledge: > A must be between 4 & 5   >  B must be between 3 & 5   >  C must be between 2 & 5   >  D must be between 1 & 5 Constraints that must be met: The total volume across A, B, C, D should be within 1.25% of the total volume across A, B, C, D. The link points to an Excel file with the data format I have. https://docs.google.com/open?id=0B40ZgxUasZIPMjA4ZmQzNzMtYjgzZS00YjMzLWIyMGMtZjY5NTk3OTY3M2Y5


title: Test statistic for weighted average of partially dependent populations

text: I am looking for a test statistic in the case of two (or more) populations that share certain values, viz. the samples overlap partially. I've looked in many books but most discuss only the standard cases of independent populations, means, variances, medians, etc. I'm also interested to know to find out how accurate these values are by comparing the two populations. In my particular case I'm looking at a manufacturing plant, where I want to look at the two-day and fourteen-day weighted average of the cycle times for individual production steps. The weights are the quantities of the products being produced. Basically, the two-day weighted average is for the past two days and the fourteen-day weighted average for the last fortnight. Thus the two-day values are part of the fourteen-day population. What I'm interested in is to know whether these are from the same (empirical) distribution, whatever the distribution may be, i.e. whether the weighted averages are the same or if there is a drift in the cycle times. Is there a test statistic that you can recommend for my case of partially dependent populations and a weighted average? Thanks and cheers!


title: range of coefficient of determination R^2 (can be negative?)

text: According to wikipedia, the coefficient of determination is $$ R^2 = 1 - \sum_i{\frac{(y_i - f_i)^2}{(y_i - \bar{y})^2}} $$ where $y_i$ is the $i^{th}$ sample value, $f_i$ is the model predicted value and $\bar{y}$ is the mean value. I can imagine a simple case with two sample points where the coefficient of determination will be negative. Basically, the problem comes when the regression line is really bad, predicting the regression line as negative of the correlation. But in the places I've read about it, they say that it's range is [0,1]. Can anyone explain? ![enter image description here](http://i.stack.imgur.com/64Whm.jpg)


title: Examples of "Textbook" Retrospective Studies

text: I am a physician and have never had any formal training in statistics/biostatistics. However every time I come across a journal article, specifically on retrospective studies, I cannot help but feel a little suspicious that (a) the authors are trying to pull a fast one on me and/or (b) they are like myself and don't really understand what they are doing as well. So I was wondering if there are any retrospective studies, hopefully in medicine, that are considered to be "classics" or are stellar examples of logical thinking and sound applications of statistical principles. I know that the knee-jerk response might be to tell me to read textbook A or B, but unfortunately I don't have that kind of luxury at this point. I also do not have the luxury of quick access to a biostatistician to ask these kinds of questions at my current institution. \-- EDIT -- In response to some of the answers below, all of which are much appreciated, let me add the following: I am a surgeon, so I am interested in diseases pertaining to the field of surgery. AdamO suggested that I look at the NEJM, JAMA, or the Lancet, which I did. One of the studies that I found interesting but am somewhat skeptical of is: "Risk Factors for Retained Instruments and Sponges after Surgery" by Atul A. Gawande et al, published in the NEJM, January 2003. Now Dr. Gawande is something of a rock-star in general surgery, and this is one of the reasons why I chose to look at his paper after getting the suggestion from AdamO. His paper is published in the NEJM, so he must be doing something good when designing his study. He is trying to see what are risk factors for surgeons accidentally leaving things inside patients. This is a (thankfully) very rare occurrence: scouring through a Massachusetts malpractice insurance company's records from 1985 to 2001, he found only 54 cases. However, his design leaves me somewhat confused and unhappy. For instance he needs to compare 54 cases with a risk-adjusted cohort that did not have retained objects, which he will treat as control. He writes "Given an estimated 60 cases available for review, we determined that four controls for each case would give the study sufficient power to detect a risk factor present in 30 percent of patients that produced a doubling of the likelihood that a foreign body would be left behind." Most likely some biostatistician told him that would be good enough, but unfortunate statistics-ignorant lay- folk have no way of knowing whether this is indeed good practice. In any case he now has 54 patients to compare with a randomly chosen risk- adjusted cohort of 235 patients acting as control (which magically allows him to compare risks in the way he describes). When doing the chart reviews he told his residents to go through the charts looking for a predefined set of variables and to see how often they appear in all the charts. These variables include age, sex, body-mass index, whether counts of sponges and instruments were performed, duration of operation, whether the operation was performed on an emergency basis, etc. He then performs logistic regression on each of these variables to see whether they were associated with an increased likelihood of retention of a foreign body. However, he sets the level of statistical significance to be P <0.2 as the inclusion criterion for further analysis. This is something that I find somewhat disturbing. I have haunted these forums and looked up methods for variable selection and almost always the answer is that using P-values for variable selection is improper or erroneous. Again I could be wrong and Dr. Gawande might be using it properly in this case for all I know, but I am skeptical. The only people who seem to support its use are those in medicine or epidemiology. So he then uses 8 variables that have P <0.2, for multiple logistic regression to find that only 3 variables are now significant (operation performed on an emergency basis, unexpected change in operation, and high BMI). As a clinician, I would love to think that Dr. Gawande's approach to a retrospective analysis is indeed a model of the format. Except for the first part where he decides that approximately 240 subjects are needed as control to give the study sufficient power, everything else is pretty straightforward. However, is this true? Is this study really a model example of the retrospective analysis form as AdamO implies? Or has Dr. Gawande's status within the medical community perhaps allowed a top name journal to give their stamp of approval on what would otherwise be considered a subpar statistical experiment by the statistics community?


title: Computation of likelihood when $n$ is very large, so likelihood gets very small?

text: I am trying to compute this posterior distribution: $$ (\theta|-)=\frac{\prod_{i=1}^{n}p_i^{y_i}(1-p_i)^{1-y_i}}{\sum_{\text{all}\,\theta,p_i|\theta}\prod_{i=1}^{n}p_i^{y_i}(1-p_i)^{1-y_i}} $$ The problem is that the numerator, which is the product of a bunch of $\text{Bernoulli}(p_i,y_i)$ probabilities is too small. (My $n$ is large, about 1500). Hence, the posterior values for all $\theta$ all get calculated to be 0 (I am doing calculations in R). To clarify, each $y_i$ has its own $p_i$, together these $p_i$'s make a vector of $n$ elements for $n$ $y$'s. Each $\theta$ has its own $n$-element vector of $p_i$. EDIT: Adding a reproducing example (for the numerator)               p <- sample(seq(0,1,by=0.01), 1500, replace=T)     y <- sample(c(0,1), 1500, replace=T)     dbern(y, p) # 1500-element vector, each element is < 1     prod(dbern(y, p)) # produce 0     exp(sum(log(dbern(y, p)))) # produce 0 since the sum is very negative


title: Where do I put the control variables in 2SLS?

text: When I am running a 2 stage least squares, where do I put the control variables? Should I put the control variables in the first stage? The second stage? Both? Can someone explain why?


title: Proper way to match a reference population for survival analysis

text: I am performing survival analysis on a specific group under study containing thousands of individuals. I am looking for the best way to obtain a reference population from the total population (millions of individuals). The reference population must be matched on age, e.g. have the same age distribution as the study population. Since the total population is much larger than the study population, the matched reference population can also be much larger. The easiest way to proceed would be to simply draw a subsample from the total population of the same size as the study population (matched for age). Doing so would mean that the vast majority of the total population is never used, which may introduce bias and surely widens confidence intervals on the reference population. To avoid this issue, I am currently considering two resampling approaches. I use the Kaplan-Meier estimator for the survivor function $\hat{S}(t)$. **First approach**   1. bootstrap the total population to draw a large number of (matched) subsamples (lets say $n_{boot}$ samples of $n_{study}$ individuals, matched for age),   2. aggregate the subsamples into a large population $\mathcal{P}_{aggr}$ which contains duplicates ($| \mathcal{P}_{aggr}| = n_{boot} \times n_{study}$ non-unique individuals),   3. compute reference survivor function based on $\mathcal{P}_{aggr}$,   4. use exponential Greenwood confidence intervals ( _log-log_ transform). **Second approach**   1. bootstrap the total population to draw a large number of (matched) subsamples (lets say $n_{boot}$ samples of $n_{study}$ individuals, matched for age),   2. compute survivor function per subsample: $\hat{S}_i(t), i=1..n_{boot}$,   3. aggregate survivor function estimates, global survivor estimate being $\hat{S}(t)=\frac{\sum_{i=1}^{n_{boot}} \hat{S}_i(t)}{n_{boot}}$,   4. confidence intervals can be obtained directly from the set of survivor function estimates. Are there any strong statistical factors I must consider in favor of/against either approach? Is there a better way to obtain a matched reference population? I currently lean towards using the first approach as it is somewhat easier to implement and has a commonly used closed form for the confidence intervals.


title: Using R in Java Project

text: I hope this question is not off topic on CV: I am currently fitting some different models (naive bayes, logistic regression...) in R which I up to now thought of as prototypes for a later Java implementation of the same models. Now I came across multiple ways of executing R code from Java. I am sure people were in a similar situation before, and I was wondering if you stick to Java libraries (or any other programming language) for solving statistical problems, or whether you execute R code from your programming environment? Since R is such a rich language for statistical problems, I do not see much motivation for using Java libraries for solving such tasks.


title: Methods For Estimating Trend in R

text: Do you know how i can get a code for estimating trend by Mitcherlich and logistic method in R?


title: Expected value of sums of squares of a blocking factor in a split plot design

text: How can we show that in a split plot design, the expected value of the sum-of- squares of the blocking factor has expected value equal to $k\sigma^2_{\text{block}} + \sigma^2_{\text{error}}$? I tried to identify the within and between factors and used the definition of covariance, within and between block factors. **What do I need to do for this question to get attention? Can someone please help out on it? Thanks**


title: Need help calculating a Bayes estimation for a Poisson

text: My study group and I are stuck on this Bayes' estimator problem. The question is: Let X~Pois($\lambda$) Find the Bayes estimator for $\lambda$ with respect to: (i) The prior distribution: $\lambda$ ~ exp(1) (ii) The squared error loss function So for our posterior distribution we ended up with: $g(\lambda|x_1...x_n)= \dfrac{\frac{\lambda^{\Sigma x_i}}{\Pi x_i!}e^{-\lambda(n+1)}}{\int_0^\infty \frac{\lambda^{\Sigma x_i}}{\Pi x_i!}e^{-\lambda(n+1)}, d\lambda} $ which is a [Gamma distribution][1] with parameters $\Sigma x_i + 1$ and $n+1$. Then our Bayes' estimator with respect to the Prior and the Squared Error Loss Function is the expected value which we get: $\frac{\Sigma x_i + 1}{n+1} $ Which we are just not sure if this is right or not. Could someone please let us know if our thinking is correct on this one? Thanks!


title: Comparing coefficients between two linear regressions: justifying insignificant difference when the predictor is significant only in one group

text: I have the following question, any hint would be really welcome: I am trying to conduct a two-country comparison by running two separate regressions, one for each country, and testing $H_0:\,b_1=b_2$ (using Wald tests), where $b_1$ is the coefficient of an explanatory variable in regression group 1 and $b_2$ is the coefficient of the same explanatory variable in regression group 2 (the two groups have unequal size). However, I have came up with the following outcome: $b_1$ is statistically significant, $b_2$ is statistically insignicant, and $H_0$ cannot be rejected, i.e. the difference between the two coefficients is statistically insignificant. I know that this outcome is possible in terms of statistics, but it sounds a bit "problematic" in terms of logic. How could someone sensibly justify it?


title: Calculate that at least n number of x values occur from multinomial distribution

text: Let's say `X` can take on 5 values               X<-1:5      each of the 5 values occur with some probability: `p<-c(0.2,0.3,0.1,0.3,0.1)`, where `sum(p)=1` I would like to get the probability that number 5 occurs at least 2 times out of 3 trials. For the binomial case I would use:               sum(dbinom(2:3,3,0.1))      What is the equivalent of this in the multinomial case? I know that the equivalent command is `dmultinom()`, but I don't understand what the equivalent arguments are.               dmultinom(x,size,prob)      I don't understand what would correspond to 'at least 2 out of 3'


title: Model selection - is trying out all possible models a bad thing?

text: Suppose we have a DIY market and for all wrenches we are selling, we collect a various number of more or less important attributes (size, weight, price, hardness of material, etc., all of them quantitative). Also we know how many items we sold last year, so we can get the revenue. I'd like to model the revenue in a linear way depending on the attributes. Choosing the right model is obviously an important question. I've thought of generating all possible linear models (in R `revenue ~ size, revenue ~ size + weight, weight, revenue ~ size + weight + price, ...`) and compare the models. The method of comparison would be leave-one-out cross validation, comparing the average squared error of each model.   1. Is this approach of model comparison a good idea? If not, why? I'm aware of the possible memory issues if our data has too many different attributes (ie rows in the data frame). In a stackoverflow answer on how to generate all possible models in R I've found the following warning: > Make very sure that is what you want, in general this kind of model > comparison is strongly advised against. Forget about any kind of inference > as well when you do this.   2. Is the squared error from leave-one-out cross validation a good measure to pick the best model, especially when comparing models with a different number of independent variables? Are there other or better ways to compare the models?


title: MEMM label bias problem

text: I study MEMM model with application to POS tagging. The question I found that the MEMM model has one drawback - "label bias problem". More formally, > The transition leaving a given state compete only against each other, rather > than against all other transitions in the model...Conditional Random Fields: > Probabilistic Models for Segmenting and Labeling Sequence Data. The question I cannot understand this drawback in application POS tagging. I would appreciate if you could describe me this drawback with example of POS tagging.


title: Kolmogorov-Smirnov Test Alternatives

text: I'm comparing 2 data sets from nearby locations to see if they differ significantly in air quality. I cannot perform any paired tests because the data at one location does not necessarily correspond to the data at the other location in time (i.e. one sample at location A was taken from Monday to Friday, another sample at location B was taken at from Tuesday to Saturday). In addition, I have some duplicate samples (2 samples taken over the same few days at the same location, but separate physical samples, not duplicate analyses of one sample) I have been advised to compare the CDF of the two locations. Is this good advice? If so, I can use the k-s test here. Are there any other tests I could use?


title: Is there an analytical function that approximates number of clusters?

text: I have a dataset {x_i} which form 3-10 clusters. Is there any analytical function of {x_i} that I may use to estimate the number of clusters in the dataset? The fact that there might be 3-10 clusters is actually the domain knowledge I have for my problem.


title: How to interpret two-way interactions in Linear Mixed Effects modeling?

text: I've fit a Linear Mixed Effects model to some "accuracy" scores for a study with rats. The fixed effects are `TrialNumber` and `Age`, and the random effect is the individual rat (`Rat`). `Age` is included because the trials span a number of months (and `Age` is measured in days; not all rats were the same age at the start of the study, for a number of reasons). The resulting model in R was:               library(lme)     library(car) #for the logit() function, since scores are percentages     lmer(logit(Score) ~ Age*TrialNumber + (1|Rat))      I get negative values for `Trial` and `Age` (the study is designed such that we predicted scores to actually go down with more experience, which indeed turned out to be the case). However, the interaction term is also significant, but with a positive estimate. Results:               Fixed effects:                      Estimate   Std. Error  t value     (Intercept)      1.182e+00  1.689e-02   70.00     Age             -2.788e-03  2.849e-04   -9.78     TrialNumber     -1.872e-06  9.344e-08  -20.04     Age:TrialNumber  2.123e-08  1.741e-09   12.20       How do I interpret this interaction? Does it mean that for repeated trials on the same day, `Score` went up, or does it mean that older rats did better on the same TrialNumbers?


title: Logistic Regression and Linktest in Stata

text: When checking the fit of my logistic regression model through the linktest command in Stata, the results indicate that hatsq has been omitted due to collinearity. Does this result indicate that the variable which would be recommended by linktest is collinear with one of the predictor variables? Do I interpret this result as an indication that no additional predictors are recommended?


title: How do you assess for mode bias and scale effects when designing survey questionnaires?

text: We typically educate clients on the potential bias of one mode over another (online versus paper surveys for example) and the bias a scale can have when designing questionnaires. A client recently asks me how we could test for this bias and I am looking for ideas?


title: Not sure if standard error of p-values makes sense in Fisher Exact Test

text: I am working on implementing a Fisher Exact Test for some unemployment and wage data. The idea is to describe two populations (one receiving an assistance program (the "treatment") and one not receiving it) via a summary statistic. The first statistic that I am using is the difference in average wage-level post-treatment between the two populations. To calculate a p-value for such a set-up, the idea is to make a large number $N$ of random permutations of the assignment vector (the list of 0's and 1's that indicates whether a given observation belongs to the control population or not). I calculate the summary statistic for each of these random assignments, to get a distribution of summary statistics in counterfactual experiments where the control group had been different. Since I also have the summary statistic for the observed assignment vector (the actual vector of 0's and 1's for the experiment that was truly observed), I can then count the number of simulated summary statistics which are more extreme than the observed summary statistic. This proportion out of all of my simulated trials serves as the estimated p-value for my summary statistic. My question is as follows: is there a standard way to get a standard error for such an estimated p-value? Obviously, I can calculate the Monte Carlo standard error for the summary statistics in my simulations, but since all of the simulations are used for just one single p-value calculation, it''s not clear how to get the standard error for that. I had the following thought about what the standard error might be in this case. In usual Monte Carlo, we have some function $f$ that we compute at each of the simulated draws $x_{i}$ (where here, the $x_{i}$ are understood to be the assignment vectors). If I define: $$ f(x_{i}) = \mathbb{1}_{ |stat(x_{i})| > |stat(x_{obs})| } $$ then it seems that the p-value I calculate is given by $$ \hat{p} = \frac{1}{N}\sum_{i=1}^{N}f(x_{i}) = \frac{\textrm{# more extreme}}{\textrm{total samples}}$$ And following the usual Monte Carlo formulas, would it then make sense to write the variance of the estimate as: $$ Var(\hat{p}) = \frac{1}{N}\sum_{i=1}^{N}[ f(x_{i}) - \hat{p} ]^{2} $$ and then take the square root to obtain the standard error? The reason this confuses me is that for each $i$, $f(x_{i})$ will be binary, either the computed statistic was more extreme in that iteration or it wasn't. It seems like it would be error prone to sum up a bunch of binary things like that to estimate the variance in a p-value, but that could just be my unfamiliarity with this method. Can anyone confirm that this is right? Also, if I exposed any other ignorance about what I am doing here, corrective comments are appreciated.


title: Proper contingency table analysis for old diabetes study

text: I was chasing down a citation about the incidence of postoperative wound infection in diabetic patients, and I found this 1998 abstract by Pomposelli with the following statement: "In patients with hyperglycemia (> 220 mg/dL) on POD 1, the infection rate was 2.7 times that observed (31.3% vs 11.5%) in diabetic patients with all serum glucose values < 220 mg/dL. When minor infection of the urinary tract was excluded, the relative risk for "serious" postoperative infection increased to 5.7 when any POD 1 blood glucose level was > 220 mg/dL." I pulled the article and the contingency table data is as follows (excerpted):               Table III: Blood glucose and infection rate contingency tables     POD 1    Highest glucose    Infected     Uninfected    p value              <=220 mg/dL        3            23            .05               >220 mg/dL        21           46                      Table IV: Postoperative day 1 blood glucose and infection rate                contingency table excluding urinary tract infections              Highest glucose    Infected     Uninfected    p value              <=220 mg/dL        1            23            .03               >220 mg/dL        15           46                  The caption for both says "Actual p values of chi-squared shown." When I run this in R I get:               > pod1AllInfections <- data.frame(infected=c(3,21),uninfected=c(23,46))     > row.names(pod1AllInfections) <- c("above220","below220")     > chisq.test(pod1AllInfections,simulate.p.value=TRUE)          Pearson's Chi-squared test with simulated p-value (based on 2000 replicates)          data:  pod1AllInfections      X-squared = 3.8372, df = NA, p-value = 0.06997      and for the other table:               > pod1ExcludeUTIs <- data.frame(infected=c(1,15),uninfected=c(23,46))     > row.names(pod1ExcludeUTIs) <- c("above220","below220")     > chisq.test(pod1ExcludeUTIs,simulate.p.value=TRUE)          Pearson's Chi-squared test with simulated p-value (based on 2000 replicates)           data:  pod1ExcludeUTIs       X-squared = 4.7017, df = NA, p-value = 0.03598      My question is, am I doing the chi-square calculation correctly in R? I don't understand the simulate.p.value=TRUE flag that well despite reading what I can on Google. My second question is, for this study what's the correct contingency table analysis to do? My last question is about Table IV -- is it biostatistically kosher to just exclude a subgroup of patients (in this case, those with UTIs) from the table?


title: Is there a principled approach to dimensionality reduction?

text: **Background** I have walked around my office environment with a video camera. From these image sequences, I wish to determine whether or not there is a 2D embedding that approximates the camera's trajectory or the geometry of the floor plan or its semantics (i.e. offices vs corridors vs kitchen, etc). The images are converted into feature vectors of ~500D each. I have tested a variety of nonlinear dimensionality reduction techniques to reduce the dataset to 2D for visualisation, however the approach of selecting and testing such techniques does not seem very methodical. The resources that I have on dimensionality reduction simply provide a set of techniques with very little discussion on a general overarching approach. **Question** Is there a principled approach to dimensionality reduction or does one generally need to approach it heuristically (more or less what I am currently doing)? In either case, could some suggestions be provided as to how one should approach dimensionality reduction with particular reference to the problem that I am exploring?


title: how to rank items measured with a likert scale?

text: I have a 70 or so items all measured with a 5 point likert scale. What I'd like to do is take their mean value and rank the items according to their means but with respect to statistical significance in terms of differences in means. So items that do not have statistically significant means get the same rank.               Item - Likert Mean - Rank     A - 5 - 1     B - 4 - 2     C - 3.1 - 3     D - 3.099 - 3     E - 2 - 4     F - 1 - 5      How would I go about doing this?


title: What kind of distribution is this via t-distribution

text: Suppose we repeat an experiment identically and independently 100 times. Each time we construct a 99% conﬁdence interval for $\mu$ via the t-distribution. Let X = the number of times the conﬁdence interval fails to contain the true value of $\mu$. The distribution of X is: Normal with $\mu=0$ and $\sigma^2 = 1$, is this correct? Since in Normal, the mean should be zero and the variance is always one. Since this is a t-distribution, it should be Normal. I thought another answer might suit it being Normal with $\mu = 99$ and $\sigma^2 = 0.99$ Could someone explain which one could be correct, if correct at all?


title: Data arrangement ANOVA

text: I am using Stata to perform an ANOVA analysis. I would like to ask for help on how to arrange data. This is a texture analysis and I have four data sets (four different time points) ![enter image description here](http://i.stack.imgur.com/LB0NB.png)


title: likelihood problem with exponential distribution using R

text: How can I solve this problem using likelihood in R? The data below are the survival time (in years) for 10 patients over 65 years after the diagnosis of a particular type of cancer: 10.5 0.2 7.4 0.8 5.3 3.2 11.9 3.7 2.6 0.6 Get the confidence interval of 95% for λ: i) Wald 95% ; ii) Bayesian, considering a priori uninformative Gamma. I know that `ll <- (lambda^n)*exp(-lambda*sum(n))` Thank you


title: R - ROCR Library - Understanding predict and prediction method

text: My name is Abhi and I am trying to understand the difference between predict and prediction. I am using the r language and my ide is rstudio. I have created a random forest model (r package randomForest)               myModel <- randomForest(Survived ~ .,data = modelData[,-1],importance = T)     modelResponses = predict(model,type = "prob") # I am guessing this gives probability of survival for each passenger      temp1 = modelResponses[,2]     pred = prediction(temp1,trainData$Survived) #Not Sure whats is the pred object       Now here are my questions   1. What is the pred object?   2. I have seen some code which uses the pred object to plot the auc curve. I know temp1 is the probability of survival for each record. Say the probability of survival for a particular record is 0.55. How does the prediction function know to classify this as survived or not-survived?   3. How do I use this model to classify new data. Until now I was using `modelResponses = predict(model,type = "prob")` , but now I am not so sure. Again the same confusion as item 2, how does the system determine the best cut off point for probabilities.  Thanks a lot guys. Any help would be much appreciated. Regards,


title: Parental Markov Condition

text: I asked this on math.stackexchange.com with no luck.. any thoughts? I'm currently reading a text on Bayesian networks and the text is giving some very crude interpretations of what appear to be some of the most important foundations of the subject. It states the following: > Theorem 1.2.7 (Parental Markov Condition): A necessary and sufficient > condition for a probability distribution P to be Markov relative a > [directional acyclic graph] G is that every variable be independent of all > it's nondescendants (in G), conditional on it's parents. I understand the premise of what is happening here. Given some point X in the graph, if you condition over all parent nodes, it should be independent of all non-descendants. I guess the confusion for me arrises around how this makes the process Markov relative. Maybe that is the weak part of my understanding. Could someone please provide a example of graph which fails this condition and hence fails being Markovian relative to this graph? I've done some study on Markov Chains and I feel like maybe their definitions differ in some way that's been lost on me. Thanks


title: Is it advisable to use t test for comparing two modules in SAP?

text: I have two independent samples from the maintenance and materials departments regarding the effectiveness of SAP modules. I have a likert scale data on a scale of five. Shall I use independent sample t-test to compare the two modules?


title: How to select kernel for Gaussian Process?

text: In Gaussian Process (GP), the kernel (co-variance function) is used to measure the similarity between one point and a given point. There are so many kernel functions for GP, and I wonder how to select a suitable kernel. For instance, if my time-series data are not periodic, should I choose the Squared Exponential (SE) kernel? In addition, could anyone explain why the SE kernel is so popular as well? what is the feature of this kernel? Thank you for your help in advance.


title: How to find and describe regularities in a distribution of interarrival times of a recurring event?

text: I want to see if there are regularities in interarrival times of a recurring event in discrete time. I know that I can fit distributions to the interarrival time distribution, but are there other possibilities to detect and describe regularities for the interarrival time distribution? I would like to find out how the distribution of different lengths of the interarrival times are correlated along the time axis.


title: Decision criteria for logit and regression analysis

text: When we use either logit or linear regression, will be the deciding factors same (P, F, R sq, and R adj)?


title: Is it "rows by columns" or "columns by rows"?

text: I've got a data table like:               ID  Low Color    Med Color     High Color      1        234          123            324      2          4          432           3423      The rows are widgets, the columns are color levels. Would you call this table "widgets by color level" or "color levels by widget"?


title: How can I find out how shifts in a country's fiscal policies affect its economic health?

text: I have the values of certain variables for 20 years for different countries... I am unable to understand how to use the values of a particular variable for 20 years. Could anyone suggest how I should go about this? Predictor variables:   * Imports of Goods and Services    * Industry value added   * Trade in services   * Revenue excluding grants   * GNI per capita   * Gross saving    * CPIA fiscal policy rating Outcome variables:   * GDP growth (annual change as %)   * GDP per capita   * Inflation of customer prices


title: Taking the log of variables

text: Just before I start the question I would like you all to know that I have checked the other threads on taking the log of variables but I still think I have a question that hasn't been touched on yet. I would also like to thank whuber for his lengthy answer to another log question here. This question specifically relates to one of the reasons of why we take logs, namely transforming the distribution of the data. When we take the log of a variable it is usually because the distribution of the variable is skewed and we want to give it a normal distribution. A common example of this in OLS regressions in economics is a variable denoting wages, income, GDP etc. However, no one ever seems to mention the central limit theorem (CLT). The CLT says that sum of many random variables will be normally distributed even if their underlying distributions are not normally distributed. If the error is the sum of the random variables $X$ and $Y$, $\epsilon = Y - X\beta$, then surely the error will be normally distributed regardless of the distribution of $X$ and $Y$. If this holds (and the CLT seems to hold under pretty weak conditions) then why would we need to transform the variable?


title: How to calculate cumulative Poisson probabilities without adding each one if no. of outcomes is large and hence, adding (without excel) is difficult?

text: One can calculate the probability of a correct score of a football match by Poisson(Number of Team1 goals, Mean average Team1 goals) x Poisson(Number of Team2 goals, Mean average Team2 goals) So for 0:0 scoreline, Poisson(0, 1.05) x Poisson(0, 2.5) if mean Team1 is 1.05 and mean Team2 is 2.5. But how does one calculate the probability/odds of "home team to win by exactly one goal"? Adding double Poissons for each 2:1, 1:0, 3:2, etc scorelines might be one of the methods but it will be a tedious task and I am trying to avoid Excel for the time being to find the simplest method to calculate this. How do you involve the other team in your calculations? Was hoping an easier, simpler way was there (something like calculating P=0 and then 1-P(0) to find odds of P>=1).


title: Growing hierarchical self-organizing maps

text: For learning GHSOM I figured out I should study SOM as a first step. Now, I know the basics of SOM, about weighted vectors and euclidean distances: when the human brain cannot easily process more than two dimensions of data, SOM will cluster, or in other words, map the data from high dimensions into lower ones (from this awesome link). I found out what vertices of weights and similarity will do, but I still couldn't understand exactly how GHSOM works. I read in an essay that it works with layers of SOMs. With some examples I hope to understand it better. Would you please tell me a proper reference for a beginner to study about it? Or can anyone explain how the algorithm works? Thanks in advance.


title: Suggested R packages for frontier estimation or segmentation of hyperspectral images

text: An **hyperspectral image is** a multidimensional image with more than 200 spectral bands i.e. an image for which each pixel is a vector of dimension 200 (most often it is a sampled spectral curve that is encoutered in satellite imagery or medical imagery). What are the **implemented package** (I am especially interested in R packages but if other free algorithms exist, I will try them) for **frontier detection** and (unsupervised) **segmentation** of this type of images?


title: Cutoff on ROC curve

text: @cbeleites mentions in response to the question How to determine best cutoff point and its confidence interval using ROC curve in R? that if one could specify the relationship between a false negative and false positive (e.g. a false negative is 10 x as bad as a false positive) that would give you a modification of the closest point to the ideal corner. Is there a mathematical way to find that value? If so can you point me in the direction of finding it?


title: Probability that a sample will contain a defect

text: I just want to check my math on calculating the probability that a random sample will include a defective record. I have a population of 20,000,000 records. If there are 7,000 records of the 20,000,000 that have a defect, what is the probability that a 10% sample will contain 1 of the defects? A 10% sample would include 2,000,000 records. The probability of 1 record being one of the defective records is:               7,000/20,000,000 = .00035.        The probability of one of the samples Not being defective is:               (1 - .00035) = .99965.        The probability of 2 samples not being defective is:               .99965 * .99965 = .9993.        The probability of 2,000,000 samples Not being defective is:               .99965^(2,000,000) = 8.73 x 10^(-305).        The probability that at least one of the 2,000,000 samples is one of the 7,000 defects is:                1 - .99965^(2,000,000)      which is very, very close to 100%. What sample size would have a .999999 probability of having at least 1 of the 7,000 samples? $$.999999 = 1 - .99965^{(SampleSize)}$$ A sample size approximately 39,465 would have a .999999 probability of including 1 of the 7,000 defects.


title: Neural networks: $tanh$ activativation function in output layer

text: In multi-class classification problem, if the output neurons are activated by $tanh$ function, how do we determine the class assignment for the pattern? In case of sigmoid all output neurons would have value in $[0,1]$ and, using 1-of-C encoding of the response variable, the error is well-defined and the class determined by the neuron with maximum value. In case of using hyperbolic tangent, the neurons values would be in $[-1,1]$ and if we use 1-of-C encoding, the errors would be huge and it's not clear how do we classify the pattern based on the network's output? Maybe, we should encode the response with $\\{-1,+1\\}$ with $+1$ for the correct class and $-1$ for the rest?


title: 3 parameter Exponential Family and sufficient statistics

text: This is a homework problem. I've derived the following distribution from an earlier part in the problem $$ f_{X_1,X_2}(x_1,x_2) = \dfrac{\Gamma(x_1+x_2+r)\alpha_1^{x_1}\alpha_2^{x_2}\theta^r}{\Gamma(r)x_1!x_2!(\alpha_1+\alpha_2+\theta)^{x_1+x_2+r}}$$ where $\alpha_1, \alpha_2, r, \theta>0$ and $x_1, x_2 = 0,1,...$. I have to put this into an exponential family form and find what is the sufficient statistic for $(\alpha_1,\alpha_2,\theta)$, where $r$ is known and $r=1$. Here is my attempt: $\begin{align*} f_{X_1,X_2}(x_1,x_2) &= \dfrac{\Gamma(x_1+x_2+1)\alpha_1^{x_1}\alpha_2^{x_2}\theta}{x_1!x_2!(\alpha_1+\alpha_2+\theta)^{x_1+x_2+1}}\\\ &= \dfrac{\Gamma(x_1+x_2+1)}{x_1!x_2!}e^{(\log{\alpha_1})x_1+(\log{\alpha_2})x_2-(\log{\alpha_1+\alpha_2+\theta})(x_1+x_2+1)+\log{\theta}}\\\ &= \dfrac{\Gamma(x_1+x_2+1)}{x_1!x_2!}e^{\left(\log{\tfrac{\alpha_1}{\alpha_1+\alpha_2+\theta}}\right)x_1+\left(\log{\tfrac{\alpha_2}{\alpha_1+\alpha_2+\theta}}\right)x_2-\log{\tfrac{\alpha_1+\alpha_2+\theta}{\theta}}}\\\ &= h(x_1,x_2)e^{\sum_{j=1}^3\eta_1(\alpha_1,\alpha_2,\theta)T_j(x_1,x_2)-B(\alpha_1,\alpha_2,\theta)} \end{align*}$ So my sufficient statistic is $T(X_1,X_2)=(X_1,X_2,1)$??? I feel like I did something wrong.


title: Rescaling AIC values to select the model

text: I have a set of 492 observations that I fit between a exponential or a logarithimic curve. Both curves share the same amount of explanatory variables. My question lies on, to use the rule of thumb that DeltaAIC < 2, we have to rescale the data to vary between 1 to 10? Here is the result table, AIC for exponential and Log curves, and their sum(residues^2).               AIC expon  AIC log    Res^2 expo  Res^2 log      -798.6752  -346.68    0.9005307   4.5773081     -711.1705  -950.659   1.2336707   0.5212761     -562.4169  -685.1748  2.1066076   1.3545968     -800.1217  -390.1025  0.8958573   3.9153901     -688.1213  -886.8839  1.3403152   0.655689      For example line 3: -685-(-562) = 123 that's the Delta AIC, should I rescale _Dividing it by 100_? resulting in Delta AIC - 1,23 thus both models are equally explicative?


title: Adjusting p-values for a single predictor

text: I have 42 DV's that I have individually modelled versus the same single IV - using GAMs via the mgcv package in R. This has produced a total of 42 individual models for each of my DV's and the associated p-vlaues. My question is: should I be adjusting p-values under the above circumstances? I understand that if one has more than 1 IV then the p-value is simply divided by the number of IV's (i.e. 0.05/20 - in the case of 20 IV's). However, I'm struggling to find any information to support this approach when there is only a single IV being considered.


title: Why do we say that the variance of the error terms is constant?

text: I always think about the error term in a linear regression model as a random variable, with some distribution and a variance. So if the error terms come from this random variable, why do we say that they have a constant variance?


title: Problems running a Factor Analysis

text: I'm given a data set to run factor analysis on. The number of variables was greater than the number of cases so I had to decide which variables to remove. I did it looking at the correlation matrix, removing those variables who were too highly (almost totally) correlated. So here are a couple of questions: **1) What's the best method to get rid of variables when dealing with such problem?** The number of my cases is 48 while I have reduced the number of variables to 24. I know the ratio of number of observations to number of variables must be at least 5 but I'm lost as to decide what other variables I need to remove. I ran the factor analysis on this data set and I got this error: > No local minimum was found, extraction was terminated. Which must have to do with my data set not being suitable for a factor analysis, I assume. So, what should I do now? **2) Do I get the error because I still have lots of variables for which I don't have sufficient number of observations?**


title: lm() function in R

text: This is kind of a follow-up question from this post: Gradient descent vs lm() function in R? Is there any literature available for the QR decomposition concept involved in the `lm()` function in R?


title: Parameter estimates for skew normal distribution

text: What are the formulaic parameter estimates for the skew-normal? If you can, the derivation via MLE or Mom would be great too. Thanks **Edit**. I have a set of data for which I can tell visually by plots is slightly skewed to the left. I want to estimate the mean and variance and then do a goodness- of-fit test (which is why I need the parameter estimates). Am I right in thinking I just have to guess the skew(alpha) (maybe do several skews and test for which is best?). I would like the MLE derivation for my own understanding - would prefer MLE over MoM as I am more familiar with it.   I was unsure that there was more than one generic skew normal - I just mean a neg skewed mean! If possible, the skew exponential power param estimates would be helpful too!


title: Is there Bayesian search theory about search through the space of Bayesian models?

text: Consider we start with a specific Bayesian model, say an infinite mixture model with a Dirichlet Process as a prior. I know there are wildly many variants on this theme, from the Hierarchical Dirichlet Process, the Nested Dirichlet Process, the Transformed Dirichlet Process and other dependent versions that try to incorporate spatial or temporal dependencies. And then we have the "mirror world" with the Beta Process and its extensions. Continuously in this field a scientists rewrites one of the priors into a process and corresponding hyperprior(s), preferably using some dining metaphor. And voila, another paper. Did someone attempt to search through this space? I admit it is huge. It is likely however that if this search is done properly we do not need to recompute everything all the time. If I have a vat with marbles that turn out all to be blue, and now I pick a marble from another vat, which turns out to be red, it is reasonable to have some prior corresponding to this event. It can be governed by the prior on the "color level", or on the more abstract level of "uniformity". The search through the "space of priors", seems to me the most interesting one in the world of Bayesians... Some pointers to work in this direction would be great!


title: correlation statistics

text: I want to do a proposed study on the frequency of a sleep disorder, namely sleep paralysis,and test to see if a trait for mindfulness is correlated with it. A questionnaire will be given to people that may have this disorder to confirm that they do and a scale to measure mindfulness will also be given. So the scale measures for mindfulness are on a 6 point Likert scale so that would be an interval scale I suppose. I guess after I receive the results from the sleep paralysis survey I can categorize the frequencies of the experience accordingly to show the correlation..? So the idea is to simply test the alternate hypothesis that people who experience sleep paralysis more frequently have less trait for mindfulness...So I know this is correlational by design but exactly what statistics to use I am not sure of....


title: Evaluation methods for personalized recommendation

text: I am currently trying to verify and evaluate a personalized recommender system I am working on, which seems like a huge task. Evaluating a static recommender system is rather easy and can be done with a simple bayesian framework the fast way. But my problem lies in how to find a good way to measure the degree and quality of personalization without too much user feedback. User interaction would be OK, but the whole thing should work without the user giving much explicit feedback. Are there any good meta-strategies to tackle this problem? What are good values I should measure? At the moment I think about values that do not directly depend on a conscious user feedback (i.e. "satisfaction"), but on more statistically inferable factors like convergence factor of the learning algorithm, etc. Hints anyone? Thanks guys. I don't need a full strategy, I am more hoping that anyone already found good resources how to deal with a variety of these problems, then I may just look what suits best for my specific case. Best Martin


title: Centering constraints for regression - specifically GAM

text: I asked the following question and (don't think) really received a thorough answer. I qualify my impression about the suitability of the answer, because perhaps I am lacking pre-requisite knowledge. I. In ordinary linear regression without nominal covariates, i.e. $y= b_o + b_1 \times x_1 + b_2 \times x_2 + e$ where $x_1$ and $x_2$ are numeric covariates, is there any constraints on $b_1$ and $b_2$? (I know that the X design matrix will not be invertible when there are "dummy variables" and more than 2 levels of a given nominal predictors - hence I mention only numeric covariates here). If there is no constraints on the coefficients $b_1$ and $b_2$ then why are there constraints on $f_1(x)$ and $f_2(x)$ in a GAM? From Simon Wood's text: ![enter image description here](http://i.stack.imgur.com/Ve6gR.jpg) II. Are these constraints required when considering a generalized linear | additive model?


title: R vs SAS, why is SAS preferred by private companies?

text: I learned R but it seems that companies are much more interested in SAS experience. What are the advantages of SAS over R?


title: Comparable salary in a different city calculations

text: I am doing a salary comparison depending on 2 cities. Very similar to this website So you select a source city, then a destination city, your current salary in the source city and it will output the "comparable" salary in the destination city. I want to confirm that what I am doing is correct, just bare with me for a minute. I will give you an example. All the data is fictional.               Source city: Cleveland, OH      Destination city: Seattle, WA      Cleveland Salary: $120,000      The way I compare is I take the housing prices, transportation prices and see where it's higher/lower. So, let's assume I will take into consideration only these two pieces of spending: housing and transportation. On average housing represents 30% of spending, while transportation is 20%. Let's assume that housing is 10% more expensive in Seattle, while transportation is 5% cheaper. Now, here is where I want your help. Am I doing this right?   1. 30% + 20% = 50% in salary will fluctuate (since I take into consideration only these two spendures)   2. Total amount that will fluctuate from source salary is: $120,000 - 50% = 60K   3. Housing is 30% (=$36,000) from 60K   4. Transportation is 20% (=$24,000) from 60K   5. Housing is 10% higher, so 36K + 10% (3,6K) = $39,600   6. Transportation is 5% lower, so 24K - 5% (1,2K) = $22,800   7. Comparable Salary in Seattle would be (the untouched) 60K + 39,600 + 22,800 = $122,400 Do these calculations seem right to you?


title: Bayesian rank of game characters

text: I have a set of game characters of a single shape (A) but a different size (small (1) to big (5)) and so the set of characters is (A1,A2,A3,A4,A5). They interact with eachother in a game grid in a pair-wise fashion, an interaction during which one will win and one will lose a battle. This goes on for a finite amount of time. Is there a way I can use Bayesian inference to:   1. create a ranking of these characters in terms of who has won the most, and    2. create a variable that represents the gradual acquisition of knowledge about the difference in size between the characters?  I currently have a list of Beta variables to characterise the rank, and can see how one could possibly use some kind of inference to rank them, but I am unsure as to make the second part work. The size difference needs to be used in some way to inform a variable, but I am unsure beyond that. So in sum, help greatly appreciated on points 1 and 2.


title: Significance of R Squared in Random Forest / GBM and GBM Tuning Parameters

text: I often get different level of responses when I discuss about R-Squared and its relevance to measuring the performance of a Random Forest or GBM model. In general, RMSE is a better and more appropriate measure, but I sometimes encounter cases where the RMSE is acceptable but the R-Squared is way low ~ 0.18 for example in a recent test. How should one interpret this ? Also, re: tuning parameters in GBM - I use caret in R and as usual I set a range of values for tree (e.g. 1:100), for interaction depth and so on. I need to check the relative influence of the variables once the GBM is run and in different levels of tuning (eg., shrinkage = 0.01 vs shrinkage 0.001) the rel. influence of the variables can become "very" different. Is this indicative of something fundamentally concerning the data or conversely, how does everyone generally address such an issue (i.e, at what point would you stop tuning further given each run takes a long time (large datasets of x 100k rows) and consider a model as with a given set of tuning parameters as final). Although GBM selects the optimal n trees and depth, finer tuning can change that selection, and consequently alter the results of the variable importance. Same question also for Random Forests. Thanks in advance for your thoughts on this.


title: Test if people drop out or decrease bets after repeated losses

text: I have data on a series of winning and losing bets over 5 rounds of betting with attrition after each round. I am using a decision tree like the following to display the data. ![enter image description here](http://i.stack.imgur.com/qvQAQ.jpg) The nodes towards the top of the tree are those that are having winning bets, and those towards the bottom of the tree are having runs of losing bets. I want to look at (a) attrition at each node (b) changes in mean bet sizes at each node. I'm looking at the rate of attrition at each node from the previous node, and the survival rate (using the expected amount of people at each node if the probability is 50%). For example, if the probability is 50% at each node, out of the 1000 that started, roughly 500 people should be in each of the second nodes, W and L. The hypothesis is (a) the rate of attrition is higher after losing bets (b) mean bet size is reduced after losers and raised after winners. I just want to do this in a very simple univariate setting first. How can I perform a t-test to show the the change in mean bet size from node WW to node WWW is statistically significant if 50 people have dropped out? I'm not sure this is the right approach: each subsequent bet is independent, but people are dropping out after losers, so the sample is not matched. If it was just a case of the same class taking a series of exams one after the other with no-one dropping out, I'd understand how to perform the appropriate t-test, but I think this is a bit different. How can I do this? Also, if the results are being skewed by a small number of customers, how could I take out the top 5% and bottom 5%? Just remove the customers with the highest cumulative stake size from bet 1 - 3? I have the data from which the figure was generated, so I have the mean, std, std error etc at each node.


title: Estimating number of RNA-seq reads needed to detect differential expression

text: So I've spoken with two professors about this and I'm still confused. Maybe I need to see it in writing (I'm a visual learner). This is an experiment with Drosophila (fruit flies). Currently, our samples are going to get at least 30 million reads each. Given that we have 3 biological replicates per condition, that gives us 90 million reads. Let's say we only get 50% of those aligned and only count 100bp segments (even though Illumina has paired-end, I think that only improves accuracy of mapping). That gives us 45e6 x 100 = 45e8 bp of reads. The Drosophila exome (including non-coding genes) is 17e3 genes x Avg 6e3 bp per gene = 1e8 bp. 45e8 bp of reads / 1e8 bp coding and non-coding exome = 45x coverage. Please let me know if I've done the calculations right. It's a lot of guesstimating. My numbers for the drosophila exome are from here: http://flybase.org/static_pages/docs/release_notes.html One additional complication: Due to the nature of the experiment, only about 1/3 of the animal is affected by my experimental condition. My worry is that the other 2/3 will "cover-up" any gene expression changes that I might see in the affected tissue. The question is how many genes will I actually be able to detect as differentially expressed at this 45x coverage? Am I only going to be able to detect genes with high abundance? Will increasing it to 75x help or just be a waste of money? I was told by one professor to not worry about it and another that it is a hypergeometric test in some way. Thanks much. p.s. I've found this tool to do power analysis with RNA-seq: http://euler.bc.edu/marthlab/scotty/scotty.php but I don't have pilot data (and don't quite know how to make simulated data) to use it with.


title: Transform continuous variables for logistic regression

text: I have large survey data, a binary outcome variable and many explanatory variables including binary and continuous. I am building model sets (experimenting with both GLM and mixed GLM) and using information theoretic approaches to select the top model. I carefully examined the explanatories (both continuous and categorical) for correlations and I am only using those in the same model that have a Pearson or Phicorr coeff less than 0.3. I would like to give all of my continuous variables a fair chance in competing for the top model. In my experience, transforming those that need it based on skew improves the model they participate in (lower AIC). My first question is: is this improvement because transformation improves the linearity with the logit? Or is correcting skew improves the balance of the explanatory variables somehow by making the data more symmetric? I wish I understood the mathematical reasons behind this but for now, if someone could explain this in easy terms, that would be great. If you have any references I could use, I would really appreciate it. Many internet sites say that because normality is not an assumption in binary logistic regression, do not transform the variables. But I feel that by not transforming my variables I leave some at disadvantage compared to others and it might affect what the top model is and changes the inference (well, it usually does not, but in some datasets it does). Some of my variables perform better when log transformed, some when squared (different direction of skew) and some untransformed. Would someone be able to give me a guideline what to be careful about when transforming explanatory variables for logistic regression and if not to do it, why not?


title: Meaning of p-values for an interaction between a categorical variable (w/ >2 cats) & continuous variable

text: I have a question about the meaning of the p-values when testing an interaction between a continuous and a categorical variable (with more than 2 categories). When I do that in a glm model using R, I obtain a p-value for each class of the categorical variable vs. continuous one. However, I would like to get the p-value for the interaction itself using R, and to understand what the meaning of this p-value is. An example of the code and results:               model_glm3=glm(cog~lg_hag+race+pdg+sex+as.factor(educa)+(lg_hag:as.factor(educa)),                     data=base_708)          Coefficients:                              Estimate Std. Error t value Pr(>|t|)         (Intercept)               21.4836     2.0698  10.380  < 2e-16 ***     lg_hag                     8.5691     3.7688   2.274  0.02334 *       raceblack                 -8.4715     1.7482  -4.846 1.61e-06 ***     racemexican               -3.0483     1.7073  -1.785  0.07469 .       racemulti/other           -4.6002     2.3098  -1.992  0.04687 *       pdg                        2.8038     0.4268   6.570 1.10e-10 ***     sexfemale                  4.5691     1.1203   4.078 5.15e-05 ***     as.factor(educa)2         13.8266     2.6362   5.245 2.17e-07 ***     as.factor(educa)3         21.7913     2.4424   8.922  < 2e-16 ***     as.factor(educa)4         19.0179     2.5219   7.541 1.74e-13 ***     as.factor(educa)5         23.7470     2.7406   8.665  < 2e-16 ***     lg_hag:as.factor(educa)2 -21.2224     6.5904  -3.220  0.00135 **      lg_hag:as.factor(educa)3 -19.8083     6.1255  -3.234  0.00129 **      lg_hag:as.factor(educa)4  -8.5502     6.6018  -1.295  0.19577         lg_hag:as.factor(educa)5 -17.2230     6.3711  -2.703  0.00706 **       **I would like to have the p-value of the interaction and its meaning (i.e., just one p-value for the whole interaction, not one for each category)**


title: Correlation & Regression Prediction

text: I have a homework question. I have solved most of it already, but am unsure how to proceed with one specific part that involves prediction (Parts B & C). I am not looking for anyone to just give me an answer, but was hoping someone might point me in the right direction of how I might solve this part of the question. If this isn't the appropriate place to seek this type of assistance, I apologize. This is what I have done so far: > A sales manager wishes to examine the relationship between the numbers of > miles her sales representatives travel and the amount of their monthly > sales. She wishes to use this information to formulate an equation that she > will use to predict Sales (y- or dependent variable, dollars) based on > `MilesTraveled` (x- or independent variable). The following information was > collected: >   >   >     Miles Traveled:   50,   120,   200,   250,   300   >     Sales:          2500, 10000, 15000, 17500, 21000 >   > > a) Write the equation of the regression line: > > $\quad$ _y = 101.695 + 71.186x_ > > **b) Predict Sales for a person traveling 150 miles: > > c) Predict Sales for a person traveling 205 miles:** > > d) What is value of the Pearson correlation coefficient? > > $\quad$ _r= 0.9903, p = .001_ > > e) Briefly define regression: > > $\quad$ _Regression attempts to describe the dependence of a variable on one > (or more) explanatory variables._ > > f) Explain what the slope of 71.186 represents: > > $\quad$ _71.186 represents the average change that occurred in the dependent > variable (sales)._ > > g) Explain what the y-intercept of 101.695 represents: > > $\quad$ _101.695 is the point where the regression line intersects the > y-axis._


title: Applying kernel function to input data before giving it to algorithm

text: I have gene expression data, I do dimensionality reduction and clustering with self organizing maps, but self organizing maps do not perform well with my data.   I want to map my data to feature space using different kernel functions and then give this data as input to self organizing maps to compare their performance. My input data is of dimension $m\times n$, and upon applying a given kernel it becomes a square matrix, which I don't want to. How can I map my data to feature space without changing dimension of my input matrix?


title: R's stats::aov() and nlme::lme() functions for repeated measurement anova

text: I am running an experiment testing reaction times under different conditions. I have a data sample located here and I have added a graphical plot of my data below in order to ease your understanding: ![enter image description here](http://i.stack.imgur.com/owWXn.png) I would like to check whether weak emotion recognition has significantly higher error rates than all other conditions. I was told that the proper way to do this was a repeated measurement ANOVA. I have found out that this can be done via R's `stats::aov()` function. I am interfacing with R via RPy and you may see my exact code under this notebook. I am getting the following resulting summary:               Error: ID               Df Sum Sq  Mean Sq F value Pr(>F)     Residuals  6  0.022 0.003666                         Error: Within               Df  Sum Sq  Mean Sq F value Pr(>F)        COI        6 0.02628 0.004379   3.468 0.0083 **     Residuals 36 0.04547 0.001263                       ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1      How does this help me address my issue? Additionally, in a discussion resulting from this other question I have been told that while anova is acceptable in some cases (such as this one) linear models such as the one generated by `nlme::lme()` are preferable. I have used that function in this other notebook, and the output reads as follows:               Fixed effects: ER ~ COI                       Value  Std.Error DF  t-value p-value     (Intercept) 0.01928571 0.01514819 36 1.273137  0.2111     COIem-hard  0.07028571 0.01899579 36 3.700069  0.0007     COIsc-11    0.00403687 0.01899579 36 0.212514  0.8329     COIsc-15    0.01700000 0.01899579 36 0.894935  0.3768     COIsc-19    0.00417857 0.01899579 36 0.219974  0.8271     COIsc-23    0.00432488 0.01899579 36 0.227676  0.8212     COIsc-27    0.00417857 0.01899579 36 0.219974  0.8271      how am I to interpret those p-values in the context of the point I'm trying to make? Also, why is my first COI (COIem-easy) absent from the list? As a general point I would also be very happy to hear which of these 2 approaches you advise I should use.


title: Conflicting Answers from Marketing Test: Both Answers are Statistically Significant

text: I'm running a marketing test comparing banner A against banner B against banner C. Each banner received >10million exposures and based on response rate we concluded early in the test that banners B and C were both beating banner A. However, after the test ran longer we analyzed the results and it now appears that banner A is our #2 performing banner and banner B has fallen to the bottom. These latest results are also statistically significant. My first instinct is that something must have changed structurally about the test for this to occur. For example, I thought perhaps the demographics of the exposed audience shifted causing a different outcome. But so far there appears to be no structural change. With all of that as context, can someone please explain how you would approach interpreting these results? Would you trust the early findings or the recent findings? Am I correct to believe that the fact that the late findings contradict the early findings is indicative of something changing structurally about the test? After all, from a statistical standpoint isn't it highly unlikely for the early results to change with continued exposure of the ads all things being equal. That's why we run the test for statistical significance. I greatly appreciate any explanation or advice someone might be able to offer.


title: Bootstrap confidence interval programming

text: Suppose I have the following data set called form the file `results`. I want to write a bootstrap program using both SAS and R to determine the confidence interval for the mean proportion. If a student get > 50 marks then it is a pass or else it is a fail. The number of replication must be 1000. exam 23 56 48 87 94 56 98 62 63 45 85 75 49 79 68 58 36 82 81 73 36 69 76 88 96 50 40 49 66 48 70 61 58 63 78 85 78 30


title: What to call this graph showing icons for artists on a horizontal axis indicating number of unique words used?

text: What type of graph is this developer using? ![graph](http://i.stack.imgur.com/NHsZL.png) Copied from here.


title: Why would a statistical model overfit if given a huge data set?

text: My current project may require me to build a model to predict the behavior of a certain group of people. the training data set contains only 6 variables (id is only for identification purposes):               id, age, income, gender, job category, monthly spend      in which `monthly spend` is the response variable. But the training dataset contains approximately 3 million rows, and the dataset (which contains `id, age, income, gender, job category` but no response variable) to be predicted contains 1 million rows. My question is: is there any potential problems if I throw too many rows (3 million in this case) into a statistical model?I understand the computational expenses is one of the concern, are there any other concerns? Are there any books / papers that fully explain the data set size issue?


title: How can partial eta squared be 5% and yet SPSS says 0.045?

text: The partial eta squared indicates that gender explains 5% of the variance in drugs and ethnicity explains 6% of the variance in drugs. These are small effect sizes. I am not sure I understand this because when I look at the ANOVA table produced by SPSS it says that the Partial ETA squared for gender is .045 and for ethnicity is .060. Are these figures just rounded up and moved over a few decimal places?