title: Formula for chi square test for heterogeneity

text: Assuming to have $k$ 2x2 tables which count successes and failures for two different treatments, which is the exact formula to give the chi square test for heterogeneity?


title: Adding labels to points using mds and scatter3d package with R

text: I have a dataset forwhich i have performed an mds and visualized the results using scatterplot3d library. However i would like to see the names of the points on the 3d plot. How do i accomplish that? Each column belongs to a certain group i would like to see which points belong to which groups on the 3dplot.               #generate a distance matrix of the data     d <- dist(data)          #perform the MDS on  3 dimensions and include a Goodness-of-fit (GOF)          fit.mds <- cmdscale(d,eig=TRUE, k=3) # k is the number of dimensions; 3 in this case          #Assign names x,y,z to the result vectors (dimension numbers)     x <- fit.mds$points[,1]     y <- fit.mds$points[,2]     z <- fit.mds$points[,3]          plot3d <- scatterplot3d(x,y,z,highlight.3d=TRUE,xlab="",ylab="",pch=16,main="Multidimensional Scaling 3-D Plot",col.axis="blue")


title: R - ARIMA model with long seasonal periods - Error: "length of x and xreg does not match"

text: i want to use an ARIMA model in R for predicting an electrical load on a minutely basis. By examining the ACF I figured out which model could suit. The ACF has shown that the value one day ahead has a periodic autocorrelation. Therefore I'd like to implement a seasonal difference with a lag of 1440 (min/day). Thus, I found this page (http://robjhyndman.com/hyndsight/longseasonality/) describing how to deal with long seasonal periods in R. However, by applying that method, I experienced the following problem in R:               >Arima(x,order=c(2,0,2),xreg=fourier1(1:length(x),4,1440))          Error in stats::arima(x = x, order = order, seasonal = seasonal, xreg = xreg,  :      lengths of 'x' and 'xreg' do not match      x is the dataset as a zoo-Object with the following structure (it's just an example, I do not have access to the real structure at the moment; main difference: much more data!):               > str(x)     ‘zoo’ series from 2010-01-01 00:00:00 to 2010-01-01 00:06:00     Data: num [1:7] 1 2 3 4 5 6 7     Index:  chr [1:7] "2010-01-01 00:00:00" "2010-01-01 00:01:00" ...      Since the number of rows in xreg should be exactly the same as in x, they are apparently not. Does anyone has any suggestions about or experiecend this? I'll appreciate any hints! Marc


title: Implementation of online classification algorithms?

text: I'm looking for implementations of online learners. I guess that is possible with AdaBoost. Where you train the model and then you modify it by adding later more training data. However you don't have to re-train the entire model. Are you aware of this online implementation of AdaBoost in java?


title: "Constrained" Classification Problem

text: I need to classify objects based on a set of features - typical classification settings. The additional information is that I know the number of individual which fall in a certain class. In the sample I need to classify, I have N individuals that I need to classify in k classes. I do not know which individual falls in class C1, C2, ..., Ck but I know the number of observations falling in each class. Can somebody please suggest a way to take this information into account? Thanks, Paolo


title: What's the order of correlation?

text: What is expressed by the terms zeroth-, first-, second-, third-, etc. order of correlation? Thanks!


title: How to project high dimensional space into a two-dimensional plane?

text: I have a set of data points in a N-dimensional space. In addition, I also have a centroid in this same N-dimensional space. Are there any approaches that can allow me to project these data points into a two-dimensional space while keeping their relative distance information in the original space. Is PCA the correct one?


title: Coding the regresssion model with several binary variables

text: Say I have 5 binary variables and 2 normal variables. I want to get the probability of success, say one of the variable 1 or 0, 1 for success. How can I do that? I tried `glm(A~B+C+D+E, binomial, data)` in R. $A$ is a binary variable for which I want to know the probability of success. $B$, $C$, $D$, $E$ are all binary variables. Am I doing right?


title: Binomial random variables versus chi-square for A/B testing?

text: I've been reading on how to calculate confidence levels from A/B tests, as I'm running some ads on Facebook and want to better understand the results. The clearest explanations I've found seem to offer two alternatives: Chi-Square and Binomial Random Variables. I'm hoping someone can explain which is more appropriate. I ran two ads on Facebook:               Ad    Impressions    Clicks    CTR            1     58,028         3         0.005%         2     189,724        24        0.013%          According to this Chi-Square calculator: http://www.usereffect.com/split-test- calculator I have not yet achieved statistical significance, and will need over 40k more views to achieve a 90% confidence level. This site explains Binomial Random Variables: http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to- know-about-mathematics-of-ab-split-testing/ According to that math, both ads have a very, very small standard error, and so I can trust that the Click- through Rate is pretty accurate. Any help would be appreciated.


title: For lognormal data, what are necessary sample sizes for a difference in means between treatment / control

text: Given a population to be divided between treatment / control, if I believe a variable is distributed lognormal in the population, how do I know what the necessary control size is in order to guarantee a difference in means is significant at some arbitrary level for a specific effect size? (Reference to R or SAS example appreciated)


title: Cox proportional hazards with time-dependent covariates: predict in R

text: I would like to use the predict function (or something similar) in R to generate expected values from a Cox proportional hazards model with time- dependent covariates. The model takes the form               model <- coxph(Surv(start,stop,event.time)~covars..., data=data)      Normally, it's pretty straightforward to predict expected values by generating a new data frame where the covariates are all held at some value, except for one of interest (in this case, age). But when I use               newdata <- with(data, data.frame(acovar<-values, bcovar<-values...)     y <- predict(model, newdata, type="expected")      I get an error that says that the "Time" variable is not numeric. I have seen workarounds to this (i.e., be explicit about data source) for ordinary Cox PH, but don't know what to do for Cox PH with time-dependent covariates. The data look something like this                start     stop     event.time     acovar     bcovar      0         1        1              m          1      1         2        0              f          1      2         3        0              f          1


title: Does $\sum_{i\neq j} \text{Cov}(X_i, X_j) = 0$ imply $\text{Cov}(X_i, X_j) = 0, \,\forall\,i \neq j$

text: Let $X_1, \dots, X_n$ be random variables, for some integer $n > 2$. Then, $$\text{Var}\left(\sum_{i=1}^nX_i\right) = \sum_{i=1}^n \text{Var}(X_i) + \sum_{i \neq j} \text{Cov}(X_i, X_j) $$ There is a natural interest in the special case when the $X_i$ are such that $$\sum_{i \neq j} \text{Cov}(X_i, X_j) = 0 \quad \Leftrightarrow \quad \text{Var}\left(\sum_{i=1}^nX_i\right) = \sum_{i=1}^n \text{Var}(X_i)$$ Now, the condition $$\text{Cov}(X_i, X_j) = 0, \,\forall\,i \neq j\tag{1}$$ clearly implies the condition $$\sum_{i\neq j} \text{Cov}(X_i, X_j) = 0 \;.\tag{2}$$ Is the converse true? Alternatively, can someone give me an example where $(2)$ holds but $(1)$ fails? (My guess is that $(2)$ does _not_ imply $(1)$, since I realize that, in general, $\sum_1^{n>1} a_i = 0$ does not imply that all the $a_i$ are zero, but I can't readily come up with an example of this general fact when the summands are the terms $\text{Cov}(X_i, X_j),\;(i\neq j)$.) (My remaining questions are moot if, contrary to what I suspect, $(2)$ _does_ imply, and is therefore equivalent to, $(1)$.) When the random variables $X_1, \dots, X_n$ are described as being _"uncorrelated"_ , does this mean that they satisfy $(1)$, or just that they satisfy $(2)$? Lastly, I've often seen the term "pairwise uncorrelated", which clearly means that $(1)$ holds, is there a name for the condition in $(2)$ (when $(1)$ is _not_ assumed)?


title: Thinning chains in BUGS/JAGS

text: Hi I have a quick question about the details of running a model in JAGS and BUGS. Say I run a model with n.burnin=5000, n.iter=5000 and thin=2. Does this mean that the program will   * run 5,000 iterations, and discard results,  and then   * run another 10,000 iterations, only keeping every second result? If I save these simulations as a CODA object, are all 10,000 saved, or only the thinned 5,000? I'm just trying to understand which set of iterations are used to make the ACF plot? Thank you


title: A test for a binomial r.v. vs. sum of two binomial r.v.'s?

text: Suppose I draw a sample from one of the two possible random variables $X$ or $Y$ where $X\sim\text{Binomial}(p,N)$ and $Y=A+B$ with $A\sim\text{Binomial}(p,M)$ and $B\sim\text{Binomial}(q,N-M)$. Assume unknown $q\geq p$, with $p$, $M$ and $N$ known. My objective is to distinguish which random variable my sample came from, that is test hypotheses $q=p$ vs. $q> p$. It seems like a simple threshold test is appropriate, but I can't figure out how to justify it mathematically. Neyman-Pearson lemma isn't applicable since the alternate hypothesis ($q>p$) is not simple. Is there an UMP test for this problem? Some other measure of optimality? Any suggestions?


title: Correlation influence on two variable coefficients

text: Consider the linear model $Y = B_{0} + B_{1}X_{1} + B_{2}X_{2} + e$, where the columns $X_{1}$ and $X_{2}$ of the design matrix have mean $0$ and length $1$. That is $X_{i}'X_{i} = 1$ and $X_{i}'J = 0$ where $J$ is a column entirely of ones. Let $\rho$ be the correlation between $X_{1}$ and $X_{2}$. **Q:** Determine what values of $\rho$ will make the variance of $\hat{B}_1$ and $\hat{B}_2$ larger than $5\sigma^2$. **A:** My thought is that this is a question related to the variance inflation factor. Therefore, I could plug in $1 / (1 - z^2) = 5$. Here, $z$ is the correlation I am trying to solve for. After performing a few algebra steps, I get $z = 0.8$ (Correlation). Please let me know if I am on the right track with the variance inflation factor or if I am supposed to use another methodology (covariance matrix, etc.) Thank you.


title: Nested/SplitModel - RepeatedMeasures/MixedModel ANOVA: levels of nesting & scripting in R

text: My data set has the following variables:   * Treatment (4 types- fixed)    * Location (8 locations- fixed)   * Position in Location (3 positions per location- fixed)   * Samples are taken in each position (3 samples per position-random)      * Time (two sampling times - fixed)     * Mineralisation rate (as result of analysis of samples taken) Two locations are used to test each treatment (ie 4 treatments, 2 locations per treatment, 8 locations total). I want to do a split-plot (/nested?) repeated measures (/mixed model?) ANOVA in `R` using the above variables. Q.1. Does this sound suitable? My goal is to see if there is an affect of 1) position, 2) treatment, 3) time and 4) interaction of all (ie pos*treat*, pos*time, treat*time, pos*treat*time) on mineralization rates. Q 2. Is location nested in treatment? Is sample nested in position? Q 3. What are the between- and within- factors? Q 4. What is the subject/plot? \- Is it the location or position or sample or rate? Q 5. How can I put time as repeated measures in my R formula? Q 6. Would you use aov, lme, or ezANOVA? Q 7. How do I code the seperate independent variables, and their interactions into a proper R formula? I have literally been trying to figure this out for days and I cannot seem to find an answer that makes sense...


title: Confusion related to condition number

text: I have this linear equation                 [A] = [C]x       [B]   [D]      where A=Cx and B=Dx. I have combined these two to form the above equation. I was reading a paper where it was mentioned that if I place a weight w on D i.e, multiply D by a constant w, then depending on the value of w, the solution of x will vary. If it is small, it will not tend to fit B=Dx properly(I mean will fit but with errors), if it is large it will tend to fit B=Dx properly with the expense of not fitting A=Cx properly. I didn't get how this all fits in. Is it about condition numbers or anything? I didn't understand how adding a small variable w and multiplying D can decide if we want to fit B=Dx properly or A=Cx properly. Can anyone please elaborate this? Suggestions?


title: Time series prediction where each datapoint has a sequence

text: I am Computer Science major, and new to stats, so please bear with me and point me to the right direction if what I'm asking is pretty obvious. I have a dataset, where each data point consists of `M` features (e.g. `name`, `age` etc. both continuous and categorical - I might bin them or something and organize them properly) and a time series with data for times `T1, T2, ..., Tn`. The event dates are in months. The time series can be a sequence of dates of joining new jobs, which means that the whole dataset describes job - changing characteristics for a few persons, each datapoint representing a single person. I want to use machine learning to train a model with this data, and then given an exactly similar test dataset, I need to predict the time `Tn` if I'm given the sequence from `T1, T2, ..., Tn-1`. Something like that. So this isn't exactly the common time series prediction. I'm confused about what tool I should use and how this thing can be used to build a model. Linear regression? If yes, how? How does my data fit into it? HMMs? SVMs? ANNs? The problem seems very similar to future retweet time series prediction if we're given historical retweet data for a few tweets (in hour scale), where each datapoint represents a time series of retweets. But I can't seem to grasp my head around this. Any help will be really appreciated.


title: How to compute y caret

text: I have been given a data points $x_i,y_i,1\leq i\leq n$. I was asked to compute the standard error of regression coefficients estimates. As in the formula, I have to compute $\sum_{i=1}^n (y_i-\hat{y}_i)^2.$ But how one defines $\hat{y}_i$?


title: Post-hoc analysis when variable effect is non-significant

text: In a dataset with 3 related measurements for each case, I ran a Friedman test, and found no significant effect. Is it appropriate to now run (and report) pairwise tests, as I had planned initially? On one hand, the existence of these pairwise effects may convey important information (e.g. there's a difference between having one day of learning or having none, but no difference between having one day or one week.) But, more fundamentally, if the main effect isn't there, shouldn't that be the end of the story as far as reporting this particular study?


title: Simulation of remission and after remission times under specific conditions

text: This is basically a data generation problem. Say, $t$ is an exponential lifetime with mean two years. $tr$ is the remission time and $ts$ is the after remission time. So, $t=tr+ts$. I need to simulate these quantities for $100$ patients. Now in real life, usually when $tr$ is smaller, that is when the patients remit quickly, their after remission time $ts$ are longer (so that the lifetimes $t$ are also longer). When $tr$ are longer, that is the patients remit slowly, their after remission times $ts$ are shorter. One thing is known that the distribution of the lifetimes, $t$ is exponential with mean two years. I don't have any information about their exact relationship but I know, this is the scenario in real life. I need to simulate data under this scenario because I want to see the performance of an estimator under these conditions. So any feasible assumption regarding how they are correlated is absolutely okay for me. But I cannot understand how to generate data from the above mentioned scenario. If some sort of randomness can be put along with maintaining the conditions, that will be great! Your suggestion will be a great help.


title: Relationship between sample mean and sample survival probability

text: I am not sure if this question is very naive, but I was wondering if there is a relationship between sample mean and sample survival probability. For an exponential distribution with mean $\mu$, $$S(t)=1-\exp(-\dfrac{t}{\mu})$$ Now if I generate data from an exponential distribution, can the sample mean $\bar{x}$ be used to find $\hat{S}(t)$ as an estimate of $S(t)$? How about if I generate lifetimes from exponential distribution with different means (say, for males and females). Can the sample mean be used to find out the sample survival probability as an estimate of population survival probability then too?


title: Treatment of interactions in multiple regression

text: I'm confused about how interactions are treated in multiple regressions. When doing a factorial ANOVA, I'd just look at the graphs, say they're not parallel so there's an interaction, run my ANOVA with the interaction term, and determine if it's significant. However, in multiple regression, I add in my interaction term, see it's not significant, and none of my factors are either as my p-values are all screwed, so I then centre all my factors, do the analysis against those and the "centred" interaction term, then discover my interaction isn't significant... But I'm apparently meant to keep this interaction term in the model. I don't think I fully understand why interactions are treated differently in factorial ANOVA compared to multiple regressions - is there something I'm missing here?


title: What is a good proposal distribution for this density (in rejection sampling)? Is mine correct?

text: I have this density, I want to find a density "larger" than this $f(x)=\frac{1}{c}g(x)(1-\sin (20x)/4))$ , where $g(x)$ is N(0,1). I am looking for a proposal distribution for f(x), basically find a density that is "larger" everywhere than the one above? I was told to use g(x), but g(x) is not always "larger" because of the sine function. I was hinted to use g(x), possibly the answer. But it just doesn't make sense to me. Can someone give me a hint? Thanks!


title: What is the difference between standardized and unstandardized estimates in SEM (thinking of AMOS in particular)?

text: AMOS, like other SEM packages offers standardized and unstandardized estimates of the parameters. What is the difference? Are unstandardized estimates based on the covariance matrix and standarized on the correlation matrix? Or is it something else?


title: Excluding false values with association rule mining in Weka

text: I am using Weka 3.6 to do Association Rule mining. In our data set, each transaction is a word, and each letter in the word is an item. The rules that we are mining would be in the format of `{a set of letters} -> {another set of letters}`. So far, I have formatted six transactions by representing the existence of a letter with `'1'` and the absence of a letter with `'0'`, but this is giving me some unwanted rules. Specifically, I am getting rules where the absence of a letter implies the absence of another letter, e.g. `C='0' ==> U='0'`. How do I filter out rules that represent the absence of items? * * * For what it's worth, here is my `.arff` file:               @relation transactions     @attribute A {'1','0'}     @attribute C {'1','0'}     @attribute D {'1','0'}     @attribute E {'1','0'}     @attribute I {'1','0'}     @attribute K {'1','0'}     @attribute M {'1','0'}     @attribute N {'1','0'}     @attribute O {'1','0'}     @attribute U {'1','0'}     @attribute Y {'1','0'}     @data     '0','0','0','1','0','1','1','1','1','0','1' %monkey     '0','0','1','1','0','1','0','1','1','0','1' %donkey     '1','0','0','1','0','1','1','0','0','0','0' %make     '0','1','0','0','0','1','1','0','0','1','1' %mucky     '0','1','0','1','1','1','0','1','1','0','0' %conkie     '1','0','0','0','0','1','1','1','1','0','1' %mankoy


title: How to deal with gaps/NaNs in time series data when using Matlab for autocorrelation and neural networks?

text: I have a time series of measurements (heights-one dimensional series). In the observation period, the measurement process went down for some time points. So the resulting data is a vector with NaNs where there were gaps in the data. Using MATLAB, this is causing me a problem when computing the autocorrelation (`autocorr`) and applying neural networks (`nnstart`). How should these Gaps/NaNs be dealt with? Should I just remove these from the vector? Or replace their entry with an interpolated value? (if so how in MATLAB)


title: Rationale behind using ANOVA to compare means

text: Let's say we have 7 samples of data and we wish to test whether there is a significant difference between the means of each sample. We could use t-tests, which would require 21 different t-tests. However, I've read that this isn't advisable as the risk of drawing the wrong conclusion (1 in 20) is increased. Instead, an ANOVA is advised. But if 21 t-tests were performed using these data and because there are only 7 groups, some of the same data would be used in each test. So although we are calculating 21 different t-tests, we aren't using different data on 21 different occasions. So how does this situation increase the chance of drawing the wrong conclusion?


title: Divergence interpretation

text: I'am getting familiar with the statistical notion of Divergence. The word "divergence" is also used in physics (or vector analysis, see here http://en.wikipedia.org/wiki/Divergence). As I was more familiar with the second use, I wonder If there is a link between the two uses. There seems to be none because the first one is a local volume variation in a flow and the second seems to reflect the difference between two distribution. But I cannot convince me there is no link between all these integrals. Can you help me ? Should I understand "divergence" in probability only as "difference" ?


title: Sharing a model trained on confidential data

text: I have a regularized logistic regression model using scikit-learn and would like to share it with others, however the data it is trained on is confidential and must remain protected. The model uses bag-of-words style features to automatically classify texts describing injuries, and would be useful for a broad variety of injury surveillance tasks. Is it possible to share a fully trained model like this without revealing potentially protected information, such as the words that occur in the texts it is trained on? If so, how would I do this, and how strong would the confidentiality protection be?


title: What does seasonality of 1 mean?

text: What does having a seasonality of 1 mean? Suppose I have hourly data and I define the seasonality to be 1, does that mean the data will be dealt with as if there is no seasonality?


title: trouble in prediction in neural network classifier

text: I am training a 4-class neural network classifier. The details of my data are: featurelength = 280                             training      testing     ---------------------------------------------     no. samples      438           250     samples class1   203           135     samples class2   19            10     samples class3   16             5     samples class4   200           100            After I train the neural network. The confusion matrices are as follows:               203 0 0 0      19  0 0 0     16  0 0 0     200 0 0 0      Similarly for test data too. All the data is predicted to be of class 1. How can I debug this issue? What is the reason behind such performance? In addition to this, no. of samples for class 2 and class 3 are very less. How can I make sure that they are not dominated?


title: How does boosting work?

text: What is the easiest way to understand boosting? Why doesn't it boost very weak classifiers "to infinity" (perfection)?


title: How to perform Normalization on Call Details Record to perform k-Mean Clustering

text: I'm new to data mining and currently doing mining project on telecom customer segmentation (based on profile and call details record). I have gender, age, call time and call duration and have to perform k-mean clustering. My question is:- How to normalize data to perform clustering. Any reference or suggestion are welcomed.


title: Balancing gender in an experiment

text: Say a study aims to see if two variables A and B are correlated in the general population. It isn't clear to me how extraneous variables like gender should be treated when the sample is selected. Each of the two methods of "controlling" for gender that I am aware of (see below) seems non-ideal, so is it a case of selecting the "least unideal" one for the study at-hand? 1) You only use subjects of one gender, because you don't want gender to even exist as a variable. Surely then, any effect you find is vulnerable to the doubt that perhaps it only exists in that gender but not in the other? 2) You use 50% males and 50% females and thus claim that the effect exists in the general population because it exists in your gender-matched sample. Surely though, what you have proved then is an effect that exists when averaging across genders, and thus may even be an effect that perhas doesn't exist when looking within each gender separately. It seems to me that even if gender effects are not of interest, if an interaction between gender and the independent variable exists, it still needs to be decomposed (broken up), such that potentially, the effect remains in only one gender but not in the other. Method 1) doesn't know what might be happening in the case of the other gender while method 2) averages together what could well be two very different effects, thus possibly either masking two individually-existing effects or creating an artificial effect that does not exist in each gender individually. Confusing! Help please?


title: Combining repeated experiments into one dataset

text: I hope you all won't mind a basic question. We are examining the effects of a compound at various concentration on the behaviour of an organism. The compound is administered once at the beginning of the time course. Observations are made every minute for a period of 2 hours. Each concentration is applied to 12 individuals. If we repeat this experiment with different individuals, what is the best way to combine the 2 experiments? The issue that we're having is that although the distribution and shape of the data from each experiment is very similar, the plot of the medians by time is usually significantly shifted up or down between exp1 and exp2. It will no doubt be very evident that I'm unskilled in statistics. If this question needs more information to be answerable, let me know. Thanks, Jen


title: Programming a new random effects structure in lme

text: I am seeking advice on programming a new random effects variance-covariance matrix/structure (pdmat) in R for use in lme()? I've checked out the lme source code (as suggested if you want to program your own pdmat), but remain confused. Does anyone have a good reference? Thanks in advance!


title: In multinomial logistic regression, can I compare the categories that are not the reference category with each other?

text: I am conducting multinomial log. regression, my DV has three categories, therefore my output gives me the comparison between the reference category and the other two. I would also like to compare those two which are not reference with each other. I know I can do this by changing the reference category in a second analysis, but are there problems associated with this? Thanks


title: Fitting conditional functions in nls

text: I'm trying to fit two equations with nls() function in R. The two functions are: $f(x) = c_{1} \exp\left(-\left(\frac{x-\mu}{\sigma_{(x)}}\right)^2\right)$ where $\sigma_{(x)} = \sigma_{11}$ if $x \le \mu$ and $\sigma_{(x)} = \sigma_{12}$ if $x > \mu$ and $f(x) = a K \exp\left(- \frac{a}{b} \exp\left(-b x\right) - bx\right)$ Below is my attempt with factitious data:               x <- seq(from = 17, to = 47, by = 5)     y <- c(26.2, 173.6, 233.9, 185.9, 115.4, 62.0, 21.7)     Data <- data.frame(y, x)     Fit1 <- nls(formula =  y ~ if (x <= Mu) Mean <- c1*exp(-((x-Mu)/Sigma11)^2) else Mean     <- c1*exp(-((x-Mu)/Sigma12)^2),                      data = Data, start = list(c1 = 240, Mu = 25, Sigma11 = 5, Sigma12 = 14), trace = TRUE)                    Fit2 <- nls(formula =  y~K*a*exp(-(a/b)*exp(-b*x)-b*x), data = Data,                     start = list(K=4250, a=10, b=0.1), trace = TRUE)      Both codes produce Error and Warning messages. Any help to figure out these problems will be highly appreciated. Thanks


title: Testing significance of factors and covariates along with modeling causality among responses

text: I'm wondering how to test the significance of factor(s) and/or covariate(s) along with modeling the causal relationship among responses. Let me explain this with a concrete example. ### Example: Suppose a researcher observed four responses **Y1** , **Y2** , **Y3** , and **Y4** along with three covariates **X1** , **X2** , and **X3** from an experiment involving _ab_ treatment combinations from a fixed factor **_A_** with _a_ levels and a random factor **_B_** with _b_ levels. Based on past experience, it is assumed four responses are correlated and **Y1** is also influenced by the other three ( **Y2** , **Y3** , and **Y4** ). ### Question:   * How can I test the significance of Factors ( ** _A_** and **_B_** ) and covariates ( **X1** , **X2** , and **X3** ) and causality among responses ( **Y1** , **Y2** , **Y3** , and **Y4** ) using a single model?


title: Significant difference between two small populations sampled without replacement

text: Suppose I sample and perform tests from two **small** populations, **without replacement** , and I want to test for a significant difference in the proportions of positives in each population. How can I perform this test? I know how to do this if the populations were large, or I was sampling with replacement, but how does the t-test method generalise when you need to use the hypergeometric distribution. Would appreciate a link to some relevant reading if possible.


title: How are final model coefficients estimated when using k-fold cross validation?

text: It is i'm sure quite a simple question but i was not really able to find an explanation of this online and does not seem to be explicitly called out in the typical expositions i've seen. I get how 10 fold cross validation works to recursively partition the dataset into 9 train and 1 test set 10 times to get an estimate of out of sample performance. What i'm not sure is how the 10 models are then aggregated into one single model that can be used to predict against a new dataset. e.g. say i do 10 fold linear regression. my understanding is that i then have 10 individual models (with individual coefficients, se, p-values etc.) built on 10 subsamples of 9/10's of the data - is the 'final model' just an average of each of the individual models e.g. just average to coefficients of each model? This is the bit i was never quite sure on...


title: Poisson regression for change in count following an event

text: I have a count of sickness absences before and after an accident, and I want to find out whether an accident increases the sickness absences differently in different groups. I'm trying to formulate a Poisson model for this, but I'm not sure if I'm doing it correctly, or if I should be doing something completely different. Some of my subjects have an accident (once), and I have split the data to two rows for such persons, before and after the accident. Some never face an accident, and so they always count as healthy, and therefore they only have one row in the data. I have a variable ("state") which indicates whether the row concerns time before or after accident. The model I've come up with:               fit <- glm(count~state*group+age, family="poisson", data=d)      Is this a correct approach? To take this further, I would also like to take into account person years the subjects are in the study before and after accident. Would adding `+offset(log(person_years))` to the dependent variables achieve this?


title: How do I model the volatility of an arithmetic mean?

text: Suppose I have a model of stock prices developed using Brownian motion. I have a second time series derived from the first. At each price point in the first time series, I take the arithmetic mean up to that point and that is the data point for my second time series. The volatility of the second time series is lower because the arithmetic mean has a dampening effect. Are there any statistical tools to model the volatility of the second time series?


title: Help in fitting linear mixed model in R (lme4 function)

text: In my design I have 48 permanent plots in 3 summits (16 plots for each summit). Plots are grouped in clusters arranged in the cardinal direction (4 plots for cardinal direction = 16 plots) and grouped into summits (16 plots for summit x 3 summits = 48 plots). I have two observations of the response variable for each plot: 2001 and 2012. The response variable is a vegetation indicator S. Plots are nested into cardinal direction and summits (plots are nested within the combinations of summit and direction?) I 'd like to test if the vegetation indicator S is changed over time. To do this, I'm trying to buid a linear mixed model in R (`lmer` function of `lme4` package). I assume that time is a fixed factor as both summit and direction (3 summits and 4 directions are few to treat them as a random factor). The random factor is related to repeated measure on plot over time. I fit this model:               model1<- lmer(S~time + Direction + Summit + time:Direction+time:Summit+(time|plot       What do you think?


title: Why NB and Poisson performs superior than ZIP, ZINB and Hurdle in presence of lots of zeros?

text: I am working on a data which contain nearly 80% of zeros and positive counts as large as 7. The dataset is very large, nearly 16,000 cases. It is a health related data. I have fitted ZIP, ZINB and Hurdle models on it with five covariates, same for zero and positive counts. Later I have fitted negative binomial and Poisson on the data for same covariates. All the covariates were significant on each of the models. As I expected, AIC's are much better (I mean lower) for zero inflated models. So, I was happy. Vuong test also confirms the superiority of ZIP, ZINB and Hurdle models over Poisson and negative binomial models. So, I was happier. Later I estimated the frequency of counts for each models. I used the following codes: > round(colSums(predict(zip, type=""prob))) #zip is the ZIP model and so on > for the latter.   >  round(colSums(predict(zinb, type="prob")))   >  round(colSums(predict(hurdle, type="prob")))   >  round(colSums(predprob(poisson)))   >  round(colSums(predprob(nb)))   > and   > table(data$TRUE) # this is for the true counts   > Poisson and NB has as good estimate as ZIP, ZINB and Hurdle. Even, NB has somewhat better than ZIP. All were very close to the TRUE counts. Can anyone please tell me what might be the reason? I really liked the ZI models. I simulated data and explored them in various ways. But now I am kind of shocked! By the way, I have used **pscl** package for ZI models and **glm** for Poisson and **glm.nb** for negative binomial model in **R**.


title: Modeling using neuralnet package in R - lots of issues

text: I am using neural net package in R. While I understand the basic neural network concepts, the details and back end is still a tough nut for me. Currently all I can do is use brute force to change the parameters one at a time(changing neuron layers, adjusting hidden neurons, threshold,etc) I tried testing the package on a manually created data set. I had a few questions regarding the same.   1. What are the types of output functions for which this package can be used.    2. In cases when the error is high, what is the workaround and how can we bring down the error.   3. How to decide on the number of hidden layers and the number of neurons in each   4. Which algorithm and error function to choose   5. I also find the results are very unpredictable. Many times, even for simple straight forward linear functions the algorithm doesn't converge.    6. Do we need to treat the data and perform checks like whether the data is normally distributed or not before we can proceed to model using neural net?   7. Finally is neural net package worth it? In other words can we do full fledged modeling on the same with reasonable accuracy using this? I have to soon start modelling on the same and hence any help will be greatly appreciated


title: Regression on means of log-transformed variables

text: Suppose that I want to study the relationship between two variables $Y$ and $X$ using the linear model $Y \sim X$. Unfortunately, both $Y$ and $X$ are not normally distributed, say they are both skewed and have long right tails. So I'd do the trick of log-transforming both of them to make them look more like bell-shaped and do the regression. Now I consider doing something quite different. Say I have $n$ independent observations of $(X, Y)$. Suppose that $n$ is a very large number. Now I'd do $k << n$ iterations of the following process to get $k$ independent observations whose both dependent and independent variables are guaranteed to be normally distributed: For each iteration $i$, I randomly sample $m$ pairs of $(y_i, x_i)$, I then do the log-transformation on those $m$ pairs and take the average, call them $(\bar{y}_i, \bar{x}_i)$. I finally do the linear regression on those means of samples $(\bar{y}_i, \bar{x}_i)$ -- there are $k$ of them. My question is: Will the two methods described above produce the same results? That is, are they equivalent? Or are they completely different from each other? Which one would you choose if you want to look at the relationship between $Y$ and $X$ using a linear model?


title: Word entropy / frequency in human speech

text: I am wondering how to best approximate the **information value** of a word $x$ in **general human speech**. By information value I literally mean its entropy: $H(X) = \mathbb{E}_{X} [I(x)] = -\sum_{x \in \mathbb{X}} p(x) \log p(x).$ with $p(x)$ the probability of the word $x$ occurring in general human speech (assuming nothing of the participants or their context). But, what is the state-of-the-art in the study of $p(x)$ and its properties? Are there any known "good estimates" of $p(x)$? (at least in standard English) I would imagine that the number of websites returned by Google or statistics of word frequency in Wikipedia can help approximate $p(x)$. But **has the statistics or computational linguistics community ever tried to estimate $p(x)$**? If so, how? From what I understand, the security community resorts to approximations of $p(x)$ for estimating password strength (e.g. see this Cryptography thread XKCD #936: Short complex password, or long dictionary passphrase?). So I imagine that good sampling distributions of $p(x)$ exist and are used at least by some communities. **Other notes:**   * According to Wikipedia, the Oxford English Dictionary had 301,100 main entries as of 30 November, 2005. So $p(x)$ can be understood as a vector with ~300K values between 0 and 1.


title: Analysis of survival data using binomial GLM with offset

text: We are interested in determining whether there's an association between frequency of screening visits and cancer outcomes and whether that differs by race. We have Medicare data to analyze this. Typically, the knee jerk reflex for modeling survival times is that of Cox proportional hazards models, but the problem is that we're directly interested in modeling the effect due to time, since this influences the number of screening visits that a person would be eligible for. The outcome is binary: whether or not a person died at the end of a study interval. Study intervals technical differ from subject to subject. At most, we have 5 years worth of screening data, where eligible in subjects, though some subjects are censored because they enroll in other health care plans and we can't determine what their further COAs are for screening, diagnosis, or treatment). In my mind, these data are cross sectional, not prospective, since we use visit data from the future to infer about proportions that are homogeneous in the past. Yet, subjects differ in weight between one another due to the length of available follow-up. I'm struggling to write a correct linear model for these data and propose a valid analysis plan for them. How does one write a linear model with an offset?


title: ZIP converges but ZINB does not. Should I drop this model?

text: **Background:** I am building a count data model with an abundancy of zeros. More precesely, I am trying to estimate the number of competitors that will enter a certain market. 70% of my data consists of zeros (no entries). So, I am trying to model using a Zero-Inflated Poisson (or Neg. Binomial) model instead of a Poisson (or Neg. Binomial) model. **Problem:** I got a nice ZIP model (smallest AIC, BIC, good p-values), and I did a Vuong test and got that it is better suited than a Poisson model. I want to compare this to the ZINB model (Stata can do that. I guess by testing if alpha = 0). But, when I run the same model as ZINB, the model fails to converge. What does this mean? Does this mean that this model should be discarded? Everytime I get a convergent ZINB model, it points out that it is similar to a ZIP model. My data has 112 entries (30 of which being non-zero entries). The models are very sensitive, adding a new variable to the model may lead to a non-convergent model. All in all, it was hard to find models that converge. **Sample of correlation matrix** This is to give an example of the corr. matrix of one of the models I used.


title: Standard error of sample standard deviation of proportions

text: I recently started reading Gelman and Hill's, "Data Analysis Using Regression and Multilevel/Hierarchical Models" and the question is based on that: The sample contains 6 observations on proportions: $p_{1}, p_{2}, \dots, p_{6}$ Each $p_{i}$ has mean $\pi_{i}$ and variance $\frac{\pi_{i}(1-\pi_{i})}{n_i}$, where $n_{i}$ is the number of observations used to compute proportion $p_{i}$. The test statistic is $T_{i} = $ sample standard deviation of these proportions. The book says that Expected value of the sample variance of the six proportions, $p_{1}, p_{2}, \dots, p_{6}$, is $(1/6)\sum_{i=1}^{6} \pi_{i}(1-\pi_{i})/n_{i}$. I understand all this. What I want to know is the distribution of $T_{i}$ and its variance? Would appreciate if someone could let me know what it is, or guide me to a book or article that contains this information. Thanks a ton.


title: Sidak correction

text: For variables A (product research) and B (product purchase), I am proposing to do three levels of analysis as follows:   * **Test 1: Overall:** The goal here is to see if there is a relationship between A and B. My data is at the interval level and the scatter plot shows a linear relationship, so I am doing bivariate correlation using Pearson's r.   * **Test 2 (Demographic difference in the relationship between A and B):** The goal here is to see if gender, age and income level affect the relationship between A and B. I am finding this out using GLM (ANCOVA), so I am running 3 separate tests (i.e. on gender, age and income).   * **Test 3 (Equality of the correlation coefficients):** I know this is controversial and perhaps meaningless but I am curious! My goal here (if I do this test) is to see how A and B correlate for different demographic characteristics (i.e. it could be higher for males, or for people under the age of 18 years). There are eight controls - two for gender; three for age (e.g. less than 18 years, 18 to 45 years and Over 45 years) and five for income level. I am doing three equality of correlation tests again. Given the above, I am performing seven tests on the same data. **Question: Should I do do Sidak correction based on all seven tests or do separate corrections (i.e. one for Test 2 and one for Test 3 - unsure how to handle the correction for Test 1)?**


title: Determine whether a number "fits" with a group of numbers

text: I need to determine if a single number "fits" with a group of numbers. All the numbers will be percentages, so we'll use decimals. For example: The group is: `0.1, 0.25, 0.3, 0.4, 0.9` and we'll test against two single numbers: $y_1$ = 0.2 and $y_2$ = 0.8 The group is all low numbers, except for the one outlier, 0.9. So I need to know that $y_1$ fits in with the group, but $y_2$ doesn't. I'm thinking I need to do confidence interval. So I take the median of the group, in this case 0.3, then check to see if the single number is within a tolerance of the median. So in this example, if the single number is "good" it should be within the range `0.3 +/- 0.2`. But how do you find that tolerance? Note: this is not a math homework problem, so I don't get mean or variance or anything just handed to me. All I have to work with are the group of numbers and the single number.


title: Writing a Monte Carlo simulation in R

text: I am trying to write a Monte Carlo simulation in R and I am really stuck! I want to know the probability distribution of a random person in the UK becoming ill from eating a cooked 100g piece of chicken. I have the following information: out of 1000 pieces of chicken tested 20 had bacteria in question and I have data for the $\log_{10}$ counts of these 20 pieces, I also have min and max $\log_{10}$ counts (0.1 and 3.0). I also know the average person in UK eats 2 x 100g portions of chicken a week. The model for risk of illness _given an ingested number_ of the bacteria is predicted by $R=1-\exp(-aD)$ where $D$ is the ingested number of organisms and I have a value for $a$. I can write basic Monte Carlo simulations but I am struggling with the start of this one as I can't get my head around the model being ingested bacteria and the question being risk from eating a 100g portion.   * Is my first step here to obtain the CDF?    * And what is the distribution I should use?


Can the percentages in a pie chart add up to more than $100\%$ and the data or analyses will still make sense (or be logical)? Let's say the percentage in a pie chart adds up to $100\%$ . Does this make statistical sense?


title: Panel data descriptives, plots and 'feel for the data'

text: Allright, so I have a dataset for around 40k firms over fiscal years 1950-2011 with about 430k firm-years. If I'm not mistaken I have panel data. I created a unique identifier `ticn` for each firm. Years are indicated by `fyear`. For now my variables of interest are yearly sales `sale`, yearly advertising `xad`, and yearly R&D expenses `xrd`. _So my question, I want to get a feel for the data, give descriptives and maybe make a few plots, can you give me ideas for some analyses?_ Also because of the size of the data it's not really easy to just scroll through it to get a good feel for it. Due to the nature of the data (alot of observations) I decided to use STATA, also because I have a little experience with it (I don't have any experience with R, SAS or Matlab). If I scroll through the data I see gaps and missings so we may conclude that I have unbalanced panel data. * * * Here I have some descriptives but I can't think of more relevant descriptives, statistics, plots, or analyses to get a feel for the data.               xtsum ticn fyear xad xrd sale              Variable         |      Mean   Std. Dev.       Min        Max |    Observations     -----------------+--------------------------------------------+----------------     ticn     overall |  16839.99   9333.823          1      32930 |     N =  429426              between |              9506.36          1      32930 |     n =   32929              within  |                    0   16839.99   16839.99 | T-bar =  13.041                      |                                            |     fyear    overall |  1990.307   13.88944       1950       2011 |     N =  429426              between |             11.81111       1959       2011 |     n =   32929              within  |             7.754303   1959.807   2020.807 | T-bar =  13.041                      |                                            |     xad      overall |  35.16947   224.9083          0       9315 |     N =  110114              between |              137.133          0   7171.027 |     n =   14310              within  |             119.1316  -3952.802   6351.562 | T-bar =  7.6949                      |                                            |     xrd      overall |  47.81559   340.4286          0      12183 |     N =  153958              between |             197.5934          0     8502.3 |     n =   15455              within  |             201.0934  -4044.806   10102.42 | T-bar =  9.9617                      |                                            |     sale     overall |  1174.005   7084.566          0     470171 |     N =  379747              between |             4244.681          0   172849.2 |     n =   29918              within  |             4546.078  -115788.6   335794.2 | T-bar = 12.6929      What I did to get these. First, I dropped missing cases from my panel variables.               drop if missing(fyear)     drop if missing(ticn)       Then, I reported, listed, tagged and removed duplicates in the panel variables.               duplicates report ticn fyear     duplicates list ticn fyear     duplicates tag ticn fyear, gen(isdup)      drop if isdup == 1     drop isdup      Then, I removed negative values.               drop if xad < 0     drop if xrd < 0     drop if sale < 0      Finally I am able to declare panel data and enable panel data analyses!               xtset ticn fyear      Thanks in advance for your suggestions!


title: Visualize sets and their connections

text: I collected a data-sets which tested for many users (>100.000) which out of ten features in a software-product they use. They can use multiple features but for each feature there is only "use" or "don't use". So in terms of software-development, I have many sets of features in a length of zero to ten. I want to visualize the data in such a way, that it answers two questions:   1. Which features of the software are the most popular ones; How many percent of the users use FeatureX?   2. Which features are most often used together? So which sets of features are the most popular. For example, if most people tend to use Feature1 together with Feature2 (no matter what other features they use or don't use), I want to see this somehow in my visualization. To clarify: The data is already collected, I'm searching for a good way to show this data. I'm not sure whether you can (and should) visualize this in one graph or whether you should choose to display multiple graphs. I came up with a simple solution for 1.: Creating a bar-graph which shows for each feature how many people selected it solves this problem. But it doesn't help for 2.


title: Prediction error and covariance penalty

text: I am studying the paper by Efron: The Estimation of Prediction Error: Covariance Penalties and Cross-Validation. To estimate prediction error in the Gaussian case by Mallows' $C_p$ or other methods studied in the paper, one needs to estimate the variance. The usual recommendation is to use a model with more degrees of freedom and take residual variance from there. This recommendation is a bit vague, unfortunately. When one needs to compare models, the value of variance, and thus the choice of the "bigger" model used for its estimation, may be critical. Doesn't it render the whole approach virtually useless in many cases?


title: True and false discovery rate in variable selection

text: I have a question about how I can calculate true and false positive rate in a simulation study? I have seen some articles and thesis by different definitions. One of them is the following one: > TPR=the ratio of the truly selected covariates relative to the number of > active covariates and > FPR= the ratio of the falsely selected covariates relative to the number of > inactive covariates. and other definition: > >     True positives (TP): >     Non-null variables correctly identified as non-null variables. >      False positives (FP): >     Null variables incorrectly labeled as non-null variables. >      True negatives (TN): >     Null variables correctly identified as null variables. >      False negatives (FN): >     Non-null variables incorrectly labeled as null variables. >     TPR=TP/(TP+FN) >   Can anyone tell me which formula is used in variable selection in simulation studies? And what are the definitions for active and inactive covariates?


title: Confused about independence and prediction power of data

text: What is the correct way (if there is one) to think about when authors claim that stocks have produced some percentage annual return X over every 20 year period of time? They might calculate this by using the monthly starting price of the S&P500 and compare it to the monthly ending price of the S&P500 20 years later to the month. They might measure the return from Jan 1970 - Dec 1989 but they will also use the return from Feb 1970 - Jan 1990 as another point of data and so on. My questions are threefold:   1. Since the time periods used for the analysis overlap each other except for the 1 month on each end as we move forward in time, is this really an accurate way to measure returns? Isn't the next 20 year period of time's rate of return highly correlated to the previous rate of return?   2. Does it make more sense to use non-overlapping 20 year periods to measure this? i.e. Jan 1970 - Dec 1989 being one data point and the next one being Jan 1990 - Dec 2009?   3. Do these measurements have any predictive capability or do they just tell us what stock returns have been in the past?


title: Analysis of compounds using PCA - selecting the right PCA "type" for the data...?

text: I'm completing scientific analysis of chemical compounds in consumer products. As a non-statistician, I would really appreciate any thoughts from the experts here. My data is non-normal so I've used non-parametric tests like MW and KW for hypothesis testing between samples so far. However I now have to conduct a principal component analysis (PCA) of the different compounds measured in the different products (measured in different units). The stats add-in I was using asks that the type of data format be specified (eg: observation/variable table, versus a correlation or covariance matrix). I'm working with straight data so used the observation/variable table set-up. But it _also_ asks me to specify the PCA type from the following options (Pearson(n), Pearson (n-1), Spearman, Kendall, Covariance...). I tested the same data set with the Pearson (n) option and the Spearman option and got very different eigenalues and eigenvectors. The final biplot is naturally quite different. Any help someone can provide regarding what the difference is, and what PCA type should be used would be greatly appreciated. **UPDATE** : I was using XLSTAT (an Excel add-in). Is it okay to use Pearson as the "PCA type" when the correlations between the variables are non-linear? For example this "PCA type" option does not appear in other stats programs (eg: SPSS). So for example if using SPSS, the novice user would by default use Pearson "pca type".


title: Categorical fixed effect w/ 3 levels in LMER

text: I have a categorical fixed effect with 3 levels that I'm trying to enter into an LME.               a = lmer(Reaction ~ ContinuousEffect * CategoricalEffect + (1|Subject) + (1|StrengthOfStimulus))      `StrengthOfStimulus` is a numerical variable that's being used as a random effect, along with `Subject`. Now, for some reason, when I run this model I only get information on two levels of the CategoricalEffect, when there are really three levels:               Fixed effects:                   Estimate Std. Error t value     (Intercept)   6.853550   0.035681  192.08     ContinuousEf -0.005463   0.002046   -2.67     TypeB        -0.213765   0.010707  -19.97     TypeC        -0.198775   0.010595  -18.76      Where A, B, and C are the levels of the `CategoricalEffect`. What happened to level A?


title: Cross-validation with multiple validation samples

text: I'm wondering if I can call the following procedure “cross validation”. I extracted _k_ independent sets of data with comparable size from the same population. One of them was used for model development / training and all other sets were used to validate this particular model. So there is no “rotation” where each of the _k_ data sets take turns to act as the training set. So is this a “cross-validation”?


title: generalized linear models, explanatory variables completely determine response

text: I am very new to stats, so I apologize for this naive question. Say I have the following data                Yi      X1 X2 X3     (5,17)    A  C  E       (8,10)    A  C  F     (1,2)     A  D  E     (2,4)     A  D  F     (6,18)    B  C  E     (9,11)    B  C  F     (2,3)     B  D  E     (5,9)     B  D  F      where the $Y_i$'s are binomial response variables ($y_1=(5,17)$ means 5 heads during 17 coin tosses) and $X_1$, $X_2$, and $X_3$ are binary explanatory variables. I am thinking about modelling this data with a GLM utilizing the logit link function. The goal is, of course, to understand the effect of $X_1$, $X_2$, and $X_3$ on $Y$. What I would like to know is whether the fact that each value of `y` is completely specified by its associated factors (`x1`, `x2`, `x3`) would make the fit & subsequent inference problematic? Based on what I've read, to fit such model, one needs to create one dummy variable for each level of each explanatory variable, effectively making the number of variables the same as the number of observed responses $y_i$.


title: Pros and cons of controlling for age when examining effect of cognition on driving performance

text: I am reviewing some studies on the relationship between cognition and driving. Some studies have controlled for age and some have not (age is correlated with cognition. Can anyone please provide any good references on statistical control I can use to explain the positives and negatives of controlling for age?


title: Using Poisson family for log+1 counts in GLM

text: I'm running a GLM using counts as my response variable and would normally use `family=poisson`. But I'm also running SAR models, which couldn't easily handle my counts as there was such a small amount of variation between them (high number of counts at very low numbers) so my supervisor suggested I use the hack of `log(counts)+1` as my new response variable. Should I be using the family Poisson? The use of the family Poisson will give more conservative results (significance wise) when compared to the same model without Poisson family specified, but will obviously give me warnings as I have non integers. Because of this (I assume) it won't give me an AIC either.


title: How can I estimate the time at which 50% of a binomial variable will have transitioned?

text: I have the following data, representing the binary state of four subjects at four times, note that it is only possible for each subject to transition $0\to 1$ but not $1\to 0$:               testdata <- data.frame(id = c(1,2,3,4,1,2,3,4,1,2,3,4,1,2,3,4,1,2,3,4),                            day = c(1,1,1,1,8,8,8,8,16,16,16,16,24,24,24,24,32,32,32,32),                            obs = c(0,0,0,0,0,1,0,0,0,1,1,0,0,1,1,1,1,1,1,1))      I can model it with a logistic regression:               testmodel <- glm(formula(obs~day, family=binomial), data=testdata)          > summary(testmodel)               Coefficients:                   Estimate Std. Error t value Pr(>|t|)         (Intercept) -0.018890   0.148077  -0.128 0.899907         day          0.032030   0.007555   4.240 0.000493 ***      First, how can I account for repeated measures on the same individual within the model? Second, how can I estimate, with uncertainty, the day on which 1/2 of the subjects will have made the transition from $0\to 1$?


title: Random Effect Model

text: One factor random effect model: $$y_{ij}=\mu+\tau_{i}+\epsilon_{ij}\quad i=1,2,\ldots,a; j=1,2,\ldots,n$$ where, $y_{ij}$ is the $j$th observation of $i$th treatment effect $\mu$ is the overall mean $\tau_{i}$ is the $i$th treatment effect and $\tau_{i}\sim NID(0,\sigma^2_{\tau})$ $\epsilon_{ij}$ random error term and $\epsilon_{ij}\sim NID(0,\sigma^2)$ and that $\tau_{i}$ and $\epsilon_{ij}$ are independent.   * Why do we need the independence assumption of $\tau_{i}$? Is that $\tau_{i}$ are dependent for fixed effect model? And for random effect model it is written that : "Testing hypotheses about individual treatment effect is meaningless,so instead we test hypotheses about the variance components $\sigma^2_{\tau})$, $$H_0:\sigma^2_{\tau}=0 $$ $$H_1:\sigma^2_{\tau}\ne 0 $$ "   * Why is testing hypotheses about individual treatment effect meaningless?   * Why do we assume $\tau_{i}\sim NID(0,\sigma^2_{\tau})$ ? Why not other distributional pattern except the normal distribution?


title: Time series factor model with one series more frequent

text: Let's say I have two time series, one of which updates more frequently than the other: $x_0,x_1,x_2,\dots,x_t,\dots$ $y_0,y_{10},y_{20},\dots,y_{10t},\dots$ I want to fit a model to this that predicts $y$ from $x$ (and possibly from previous values of $y$) at each of the values $1,2,3,\dots$, i.e. it gives a prediction even for values of $y$ for which we won't make an observation (equivalently, assume that there are true values for $y$ at every value of $t$, but we only observe it at $t=0,10,20,\dots$) Is there a canonical way to do this?


title: Omitted variable bias in logistic regression vs. omitted variable bias in ordinary least squares regression

text: I have a question about omitted variable bias in logistic and linear regression. Say I omit some variables from a linear regression model. Pretend that those omitted variables are uncorrelated with the variables I included in my model. Those omitted variables do not bias the coefficients in my model. But in logistic regression, I just learned that this isn't true. Omitted variables will bias the coefficients on included variables--even if the omitted variables are uncorrelated with the included variables. I happened to find a paper on this topic, but I can't make heads or tails of it. Here's the paper: http://esr.oxfordjournals.org/content/26/1/67.full.pdf?keytype=ref&ijkey=R3RzO51agvA7eKQ You can also find some powerpoint slides about this here: http://www.cannectin.ca/workfiles/Nov%2013.09-Slides.pdf The bias, apparently, is always towards zero. Can anyone explain how this works? Any insight would be much appreciated. Please let me know if I can clarify anything or elaborate.


title: How to interpret a positive beta coefficient in linear regression?

text: Could someone provide a sample interpretation for the following:                                               dependent variable     Categorial variable      category 1                    Beta estimate of 1.78 (-3.39,6.69).       cat 2                         2.45 (-287, 9.34)      cat 3                         4.49 (-3.65, 7.08)


title: Significant first-step coefficient becomes insignificant in second step of hierarchical multiple regression

text: I've tried to find the answer to this on this website but haven't been able to, so apologies if this has already been resolved. I am carrying out a hierarchical multiple regression. In the first step, the model is significant, and the predictors $X_1$, $X_2$ and $X_3$ have significant coefficients. When I add $X_4$ in the second step, the model is significant and the $R^2$ change is significant. The coefficient for $X_4$ is also significant. However, the previously significant coefficient of $X_1$ becomes insignificant. Why would this be happening? There do not appear to be any problems with multicollinearity. I don't know if the following is relevant, but $X_1$ and $X_4$ are moderately positively correlated and are equally correlated with the dependent variable. $X_2$ and $X_3$ are dummy variables of a categorical variable with 3 levels. All other variables are continuous. Thanks!


title: Best statistical notation for expected probability density

text: Assume that we have two multivariate normal distributions $\mathcal{N}_1 = \mathcal{N}(\mu_1, \Sigma_1)$ and $\mathcal{N}_2 = \mathcal{N}(\mu_2, \Sigma_2)$. We do these two steps:   1. Pick a point, say $x$, from $\mathcal{N}_1$.    2. compute $\mathcal{N}_2(x)$ (probability density of x in $\mathcal{N}_2$). What is the **best correct statistical notation** to show **expected value for $\mathcal{N}_2(x)$**? The only notations that come to my mind are $E_{x\sim\mathcal{N}_1} \left( \mathcal{N}_2(x) \right)$ and $E_{x\sim\mathcal{N}(\mu_1, \Sigma_1)} \left( p_{\mathcal{N}(\mu_2, \Sigma_2)}(x) \right)$ **NOTE 1:** Instead of $\mathcal{N}_1, \mathcal{N}_2$ (numerical subscripts), I have to show the actual parameters $ \left( \mu_1, \Sigma_1, \mu_2, \Sigma_2 \right)$ in the formula. **NOTE 2:** I already know the solution (It is solved here). I am just thinking about the correct notation.


title: How do I store regression results from loops in Stata?

text: I have built a model which basically does the following:               run regressions on single time period     print results      The regressions output two numbers that I am interested in storing. However, I intend to extend the single time period to rolling through a large timeframe, in essence:               for i in timeperiod{         run regressions         store output     }      I am coming from an R background, where storing a variable in a vector is very simple, but I'm not quite sure how to go about this in Stata. In the end I want a 2xn matrix (a separate dataset) of numbers, each pair being results of one run of a rolling regression. Sorry for the very vague description, but it's better than explaining what my model is about. Any pointers (even if it's to the right manual entry) will be much appreciated. Thank you.


title: 2^k factorial design and blocking

text: If we have a standard model in the setting of a $2^3$ design with factors $A$, $B$, and $C$: $$ y_{ijkl} = \mu + \alpha_j + \beta_k + \delta_l + \alpha\beta_{jk} + \alpha\delta_{jl} + \beta\delta_{kl} + \alpha\beta\delta_{jkl} + e_{ijkl} $$ where $j = 1, 2;~k = 1, 2;$ and $l = 1, 2$. Assuming that $j = 1$ corresponds to the high level and $j = 2$ the low level, and so forth for $k$ and $l$.   1. The constraint on the parameters in the model imply that $\alpha_1 = \alpha_2$, so that: we consider just a single parameter $\alpha = \alpha_1$; How to show that the interactions can each be similarly written in terms of a single parameter $\alpha\delta,~\alpha\beta,~\beta\delta$, and $\alpha\beta\delta$? Also, what is the relationship between these 7 parameters and the original set of parameters indexed by $j$, $k$, $l$? And how can we show this?   2. Using these 7 parameters, write the expected value of $y_a$, where $y_a$ is the mean of the observations getting treatment $a$, using Yates notation. And how to do this for the 8 treatment combinations?   3. If we Let $e_A$ be the contrast effect that we defined for the special case of the $2^3$ experiment to measure the main effect of $A$, with $$ e_A = \frac{1}{4r}[y_a + y_{ab} + y_{abc} + y_{ac} - y_1 - y_b - y_c - y_{bc}] $$ then how can we show that the expected value of $e_A$ is $2\alpha$? Also, what is the expected value of $e_{AB}$?


title: ARMA(p,q) model interpreting PACF and ACF

text: ![enter image description here](http://i.stack.imgur.com/WxrrY.png) Here I have two time series (ACF, PACF), one on the left and one on the right. I have difficulties interpreting the results. Both PACF/ACF couples look the same and I can't distinguish any geometric decay or cutoff points. Would this be a AR/MA or ARMA model and how many lags would you specify for p and q??


title: How does random Forest work for regression?

text: I am an absolute beginner in field of machine learning, I started doing titanic assignment in Kaggle and found(read some where) Random Forest is the best fit. I started reading about random forest and found the Explanation by Edwin Chen in this question intuitive. This made me "understand" how I can solve the Titanic assignment which predicts if one survives or not(classification). But I cannot understand How random Forest will work for regression which is continuous. Please don't mind to point out any mistakes in my assumptions or the way I started things. Any advice would be helpful, This looks very vast and Don't even know where to begin.


title: Categorical variable with very few observations

text: I'm intending to do a bayesian zero-inflated regression, where some of my regressors are categorical variables, like this one, the country regions:               >table(db$region)     REGIAO1  REGIAO2  REGIAO3  REGIAO4  REGIAO5  REGIAO6  REGIAO7  REGIAO8          4851     2293     5471      410     1157      626     3683     1712      REGIAO9 REGIAO10 REGIAO11 REGIAO12 REGIAO13 REGIAO14 REGIAO15 REGIAO16          3550     1719    48208     3647     5293      716      545     2115      REGIAO17 REGIAO18 REGIAO19 REGIAO20 REGIAO21 REGIAO22 REGIAO23 REGIAO24           318    13884     1759     1048     3518      162     2899      559      REGIAO25 REGIAO26 REGIAO27 REGIAO28 REGIAO29 REGIAO30 REGIAO31 REGIAO32           574      407      326      110      194      367       85       36      REGIAO33 REGIAO34 REGIAO35 REGIAO36 REGIAO37 REGIAO38 REGIAO39 REGIAO40            70       11       33      478      388     1758     1185      160      REGIAO41 REGIAO00           116        4       What makes me worried is that we have few observation for some of these variables. So, far I considered three solutions:   1. Ignore the problem and use all the observations anyway.   2. Remove the categories where we have very few observations like `REGIAO00`.   3. Do my regression considering only the observations where region=11. The questions are:   * **(1)** If I choose option 2, what is theoretical motivation I'd have to explain it in my article?   * **(2)** If choose number 1, would this cause problems in the estimation of the other categories and/or other regressor? **(3)** I've heard that bayesian inference has less problems with unbalanced designs, so would this be the case? One might say that the number of observations there are not that small, but what would you say if I have a category with only one observation (this is also truly present in my dataset):               ANO_MODELO1994 ANO_MODELO1995 ANO_MODELO1996 ANO_MODELO1997 ANO_MODELO1998                 420           1676           9327          12843          17984      ANO_MODELO1999 ANO_MODELO2000 ANO_MODELO2001 ANO_MODELO2002 ANO_MODELO2003               20812          17055          24397          10036           1894      ANO_MODELO1992                   1       **(4)** In this situations, should I remove this category ANO_MODELO1992?


title: How can I make a predict plot varying two variables in R?

text: I want to plot the results of a regression model, but allowing two variable to vary simultaneously. I guess I could do that using predict() function in R, but I am running a model that does not have such function developed yet. Let's suppose we have a data set               x1 <- rnorm(100)     x2 <- rnorm(100)     y <- 1 + x1*5 + x2*3 + rnorm(100)      And we run a simple model and calculate yhat.               mod <- lm(y ~ x1 + x2)     yhat <- predict(mod)     dt <- as.data.frame(cbind(yhat,x1,x2))      How can I plot the expected value of y for different values of x1 and x2? I tried this, but it didn't work:               contour(dt$x1, dt$x2, dt$yhat)          Error in contour.default(dt$x1, dt$x2, dt$yhat) :        increasing 'x' and 'y' values expected


title: Relationship between poisson and gamma

text: Supposing that I have an interval n units long with m arrivals, and I model arrival using a poisson distribution with lambda = m/n, it's pretty easy to show that the distance between successive arrivals has an exponential distribution lambda * exp(-lambda * x). However, I'm having a hard time generalizing this to a variation of the gamma distribution when I want to look more than 2 arrivals (say I have 3 successive or four successive arrivals). Can someone explain the logic there? I believe the density should be $\frac{\lambda^{k-1}} {(k-1)!} x^{k-2}e^{-\lambda x}$ but I am not sure how!


title: Logical extensions after logistic regression

text: I have a number of predictors to use for a binary (Classes 0 and 1) classification task. Let us call them $x_1, x_2, x_3, ... x_n$. The way these are calculated, my naive heuristic assumption is that $S = \sum_{k=1}^n x_k$ should predict Class 1. That is higher $S$ means it is more likely that it is Class 1. Since this did not perform as well as I had hoped, I want to show that my scores $x_k$ can collectively achieve a better performance, than, say some other baseline methods and even my own single scores such as $x_1$. Due to the way I justify my calculations of $x_k$, I do not want to use a black-box technicque such as SVM or Random Forest. More concretely, if I use a very complex model, there is a greater burden on me to prove that I am not over- fitting or just getting as lucky as any other combination of various predictors used in the field. I first tried to use logistic regression and its performance was not very spectacular. I then proceeded to use MARS models, and I got much better classification performance (using Precision-Recall and ROC curves). But scientifically, I am not sure I can simply justify using MARS models and comparing it to logistic regression. So I want to know which models are the logical extensions to use after logistic regression to add a little more complexity. I have thought of splines, MARS, additive models, adding binary interactions $x_i x_j$ as inputs to the logistic regression. I have planned to use PR and ROC curves on cross-validation data sets to do the comparisons. For logistic regression, one can use model likelihood as a measure but are there any other measures to use across models such as these?


title: On error probability bounds in Bayesian hypothesis testing

text: In the Bayesian version of (binary) hypothesis testing one has to decide which of two hypotheses $A$ and $B$ holds true. The two hypotheses are given prior probability $p(A)$ and $p(B)$, summing up to 1. $A$ and $B$ induce two probability distributions on a set of possible _observations_ $X$, say $p(x|A)$ and $p(x|B)$. One has two decide between $A$ and $B$ after looking at one observation $x$. It is known that the best strategy, that is the one that minimizes the probability of uncorrect guess, is to choose the hypothesis $H\in\\{A,B\\}$ that maximizes the $p(H|x)$, where $x$ is the observation. The probability of error (averaged on all $x$ and $H$) can be expressed $$P_e = 1-\sum_x \max( p(x|A)p(A), p(x|B)p(B))\tag{1}$$ The expression (1) is often regarded as 'intractable' due to the presence of the max operator. Hence tractable bounds are seeked. An example is the harmonic lower-bound $$P_e \geq E_x[P(A|x)P(B|x)]\tag{2}$$ ($E_x$ is expectation over $x$; see e.g. : Routtenberg, Tabrikian, "A General Class of Lower Bounds on the Probability of Error in Multiple Hypothesis Testing", http://arxiv.org/abs/1005.2880, May 2010, and references therein). My questions: 1) In what sense are expressions like the rhs of (2) "more tractable" than (1)? Computationally, they still require integration over (functions of the) PMF's $p(x|A)$ and $p(x|B)$. Maybe they are more convenient from an analytical point of view? 2) An exact and simple expressions for $P_e$ is: $$P_e = 1-\frac{||p(\cdot|A)p(A) - p(\cdot|B)p(B)||_1 + 1}2\tag{3}$$ (Here $p(\cdot|H)$ is viewed as a vector in $R^{|X|}$, and $||\cdot||_1$ denotes the norm-1). This is conceptually interesting because it relates probability of error to a distance between PMF's. Is this expression regarded as "intractable" in the same sense as (1)? M.


title: Is it technically "valid" to fit a logistic regression with a dependent variable that is a proportion?

text: Several posts (here and here) suggest that beta regression is more appropriate when the dependent variable is naturally bounded between 0 and 1. My question is, leaving appropriateness aside, is it technically incorrect to fit a logistic regression to proportional response variable? R will throw a warning but still produce a result. It seems to me that the likelihood function will not be a valid likelihood when the response variable is proportional instead of binary, but mathematically speaking, it can still be minimized to give a solution. I wonder what violation/mistake, if any, is made when fitting a logistic regression to proportional data.


title: Why are the residuals computed manually different from those computed by R?

text: I have done a simple test using R, take a look at the code below: First of all I have created three samples:               > a = rnorm(10)     > b = rnorm(10)     > c = rnorm(10)     > a      [1] -0.2485833 -1.3077108  0.4019243  0.4453618  0.3024991 -0.4228684      [7] -0.3817301  0.2195161  1.3693408 -1.1030199     > b      [1]  0.1795048 -0.8764778  0.6097460 -1.0405654  0.8688749 -1.5191619      [7]  0.8955941  0.7544877 -1.0576488  1.7317130     > c      [1] -0.29437630 -0.96914835  0.16349019  0.08477029 -0.52696295  0.28241544      [7] -1.49860175 -0.26208560 -0.60898891 -1.67154576      Then, I've done a linear regression on those three samples:               > mod = lm(a ~ b + c + 0)     > mod          Call:     lm(formula = a ~ b + c + 0)          Coefficients:            b         c       -0.02748   0.37456        Perfect, now I have the residuals of the linear regression. _( **note:** I set the intercept to zero)_ Now, I print the residuals:               > mod$residuals              1          2          3          4          5          6          7      -0.1333896 -0.9687917  0.3574424  0.3850173  0.5237530 -0.5703937  0.2041941               8          9         10       0.3384146  1.5683807 -0.4293429       If I'm not wrong in this case(without intercept) the residuals are calculated doing:               a - (b*coef_b + c*coef_c)      If I do it manually I have:               > a - ((b*-0.02748) + (c*0.37456))      [1] -0.1333890 -0.9687922  0.3574432  0.3850155  0.5237550 -0.5703965      [7]  0.2041971  0.3384162  1.5683795 -0.4293383      if you look at the numbers closely you will notice a slight difference. Could someone explain me the reason? Thank you!


title: Quantify significance between measured value and prediction (with error)?

text: I have a question about the proper way to describe the results I get to a prediction (both of which have statistical errors). I get a result with 1-sigma, let's say:               -1 +/- 2       There is a published prediction (and 1-sigma error) for this measured value, let's say:               4 +/- 1.5.      If there is a precise way of describing my result? I mean, should I say, "this is consistent within 2-sigma?" or "it's inconsistent... sorta"? **I would like to be able to quantify the significance of the disagreement (or agreement) between the measurement and the prediction.** If it's easier to use variables instead of numbers, let's use my result is a +/- b; predicted value: x +/- y.


title: Bounding the difference between square roots

text: I want to compute the value of $\frac{1}{\sqrt{a + b + c}}$. Say I can observe a and b, but not c. Instead, I can observe d which is a good approximation for c in the sense that $P( |c-d| \leq 0.001 )$ is large (say 95%), and both c and d are known to have $|c| \leq 1, |d| \leq 1$ so a difference of 0.001 is actually small. I want to argue that $\frac{1}{\sqrt{a + b + d}}$ is a good approximation because d is a good approximation for c. Is there anything I can say about the difference $\frac{1}{\sqrt{a + b + c}} - \frac{1}{\sqrt{a + b + d}}$? Maybe I could say something like $P(\frac{1}{\sqrt{a + b + c}} - \frac{1}{\sqrt{a + b + d}} \leq 0.001) = x?$ I'm worried that being under a square root might mess things up. Would I need to find out the exact distribution of $c - d$, or anything else before I can make such claims? Thanks for all your help!


title: R code for zero inflated Poisson

text: I have used `glm()` to model some data I have. The code looks like the following:               for(ddm_idx in 1:90) {         for(ppm_idx in 1:90) {             mdfit <- glm(cuse[[4]] ~ cuse_ddm[[3 + ddm_idx]] + cuse_ddm[[3 + ddm_idx]]^2 +                                       cuse_ppm[[3 + ppm_idx]] + cuse_ppm[[3 + ppm_idx]]^2 +                                       cuse_ddm[[3 + ddm_idx]]*cuse_ppm[[3 + ppm_idx]],                           family=poisson(link=log))             mdfit_dev[ddm_idx, ppm_idx] <- deviance(mdfit)         }     }      It turns out that for each "case", I have about 90 different data points for `ddm` and `ppm` and so that's why I have the `for` loop run twice. I know this is correct because a post-doc in stats also ran the same in MATLAB and got the same results. However, my next task to to use **zero inflated Poisson distribution** as I have a lot of zeros in my dataset. Some of these zeros are "true" zeros and some of them false. How can I modify my code to use `glm()` for this distribution?


title: Covariance v. correlation based PCA, theoretical view

text: In principal component analysis (PCA), one can choose either the covariance matrix or the correlation matrix to find the components (from their respective eigenvectors). These give different results (PC loadings and scores), because the eigenvectors between both matrices are not equal. My understanding is that this is caused by the fact that a raw data vector X and its standardization Z cannot be related via an orthogonal transform. (Mathematically) similar matrices have the same eigenvalues, but not necessarily the same eigenvectors. This raises some difficulties in my mind: (1) **Does PCA actually make sense, if you can get two different answers for the same starting data set, both trying to achieve the same thing** (=finding directions of maximum variance)? (2) When using the correlation matrix approach, each variable is being standardized (scaled) by its own individual standard deviation, before calculating the PCs. **How, then, does it still make sense to find the directions of maximum variance if the data have already been scaled/compressed differently beforehand?** I know that that correlation based PCA is very convenient (standardized variables are dimensionless, so their linear combinations can be added; other advantages are also based on pragmatism), but is it correct? **It seems to me that covariance based PCA is the only truly correct one (even when the variances of the variables differ greatly) and that whenever this version cannot be used, correlation based PCA should not be used either.** I'm still puzzled, hence these questions, open for answers and comments. [I know there is thread on a similar topic, but it seems to focus only on finding a pragmatic solution, which may or may not also be a algebraically correct one.]


title: How many studies are required to assess heterogeneity of effect size in meta-analysis?

text: I have read that inferences on heterogeneity should be applied with caution if there are few studies. I have 6 studies for meta analysis.   * Is 6 studies sufficient to assess heterogeneity of effect sizes?   * How many studies are required to assess heterogeneity?   * What is the consequence of having an insufficient number of studies to assess heterogeneity?


title: How to 'normalize' data to account for level at baseline?

text: In the beginning of an experiment, subjects are required to report on their initial emotions / mood using Questionnaire#1. After the experiment is conducted, the subjects fill another mood questionnaire#2. How do we interpret these two sets of data (from the initial questionnaire and the final questionnaire)? Since it is not possible for every participant to have the same initial emotions or level on the Likert scale (which goes from _strongly disagree_ to _strongly agree_ ) at the start of the experiment, how do we analyze the data to "normalize their emotions" from questionnaires.


title: Interpreting of hypothesis prediction after reversed relationship

text: I am currently facing a big issue with regard to my master thesis. I developed following hypotheses: Hypothesis 1: Managers’ investment in innovation increases when the organization is judged in decline state _( **positive** relationship between organizational decline and investments in innovation)_ Hypothesis 2: The relationship between organizational decline and investments in innovation is **positively** moderated by managerial ownership: Managers’ with higher company ownership will invest more in innovation when the organization is judged in a decline state Hypothesis 3: The relationship between organizational decline and investments in innovation is **positively** moderated by institutional ownership: Firms with higher institutional ownership will invest more in innovation when the organization is judged in a decline state Here you can find my conceptual model: http://imageshack.us/photo/my- images/29/kvji.png/ This is the outcome of the regression: $log(Investments in Innovation) = 6.984 - 0.852Decline - 4.703Managerialown + 3.030ManagerialownDecline-1.870Institionalown + 1.14 Institionalown x Decline $ My issue is now at the beginning I predicted a positive moderation in Hypothesis 2 and 3 based on a positive relationship in Hypothesis 1. Now the return shows that there is a negative relationship in Hypothesis 1 (so managers invest less), yet both moderators Managerialown and Institutionalown mitigate this relationship (so managers invest less at a slower rater with increasing ownership) Do I confirm my Hypothesis 2 and 3 or reject them? As I understand, a positive moderation intensifies the main effect. However, now the moderator lessens the main effect of Hypothesis 1, yet my second description of the hypotheses is still somehow valid ( _Managers’ with higher company ownership will invest more in innovation when the organization is judged in a decline state_ ) Thank you very much for your help. Tom


title: How to measure association of nominal values for few observations?

text: I have a sample with 181 observations. I measured two nominal variables for each observation and now want to calculate the association between the two variables. The problem is that one variable has 15 possible values, the other one 16. So when I tried to compute Cramér's V, R complained that "Chi-squared approximation may be incorrect", which the Internet tells me comes from many cells in the contingency table having too few observations in them. ## Is there any way to correct for this? The two ways I can think of won't work:   1. Increase the sample. I don't have access to more observable items. With lots of effort, I may get 20 more or so, but nowhere near enough for 5 values per table cell.    2. Combine variable similar variable values into categories. My variables are truly nominal, with no real similarity between the values. I can't think of any combination which will make sense.  Is there some possible treatment of the data which will reduce the error of calculating chi squared? Is there an association measure other than Cramér's V which is better suited to small sample sizes (or large variable domains)? Is there anything else I can do? ## If I cannot correct, what are the implications for my analysis? If there is nothing I can do to correct this, what will happen if I do the analysis anyway? Will the results be considered so bad as to be unusable? Will I be able to use the results, but keep in mind that they may be a bit off? If they are a bit off, how are they off? Is the resulting Cramér's V likely to be too high, or too low, or is there no way to tell?


title: Stationarity and homoscedasticity in break tests

text: Let me start by saying that I'm really not an expert in statistics/econometrics. Now to my questions: I have a data set containing weekly prices of a stock. I want to check whether a structural break has occurred at a specific date using a Chow test. I know how to conduct it (using EViews 7), but I'm not sure exactly which preliminary tests I need to do in order to get accurate results. I know that the Chow test requires homoscedasticity, but what about stationarity? I further want to verify my result using a Quandt-Andrews breakpoint test. I basically ask myself the same thing; does it require homoscedasticity and stationarity?


title: How to find an optimal threshold

text: Is there a way to find an optimal threshold? I am using MATLAB for this project. I have 2 groups of triangles (distances and angles). I need to find the similarities between two triangles in each set. I order the distances from largest to smallest. I need to compare distances and angles. If they are below a certain threshold, I consider them a match. My problem is how to find the threshold for largest distance, mid distance, and small distance, and the same thing for angles.


title: Exercise of Hypothesis Testing

text: An insurance company is reviewing its current policy rates. When originally setting the rates they believed that the average claim amount was $1,800$. They are concerned that the true mean is actually higher than this, because they could potentially lose a lot of money. They randomly select 40 claims, and calculate a sample mean of $1,950$. Assuming that the standard deviation of claims is $500$, and set $\alpha = 0.05$, test to see if the insurance company should be concerned. My attempt:   * Null hypothesis: $H_0:\mu\leq1800$   * Alternative hypothesis: $H_1:\mu>1800$ Since our sample size is large, we will do $Z$-test. The test statistic is $$Z=\frac{\bar x-\mu}{\frac{s}{\sqrt n}}=\frac{1950-1800}{\frac{500}{\sqrt 40}}=1.897,$$ and the rejection region is $Z>1.96$. Conclusion: We fail to reject null hypothesis. But here, they have considered a $t$-test. But the sample size is large enough to do a $Z$-test. Again, the link doesn't consider 'upper tail'. So their conclusion is in contradiction with mine. Which one is correct?


title: LDA - Why differents formulas to calculate covariance and pooled covariance matrix

text: Reading materials from differents sites some questions have risen about covariance and the pooled covariance matrix calculation to implement LDA:   **Definitions**   Ci - covariance matrix of group i (C1 and C2)   C - pooled covariance matrix   u - global mean (dataset)   ui - mean fo group i (u1 and u2)   N - number of samples   Ni - number of group i's samples (N1 and N2)   Xi - feature i (X1 and X2) **1 – Some sites use this formula**               C1 = ((x1 – u)T(x1 – u) ) / N1         C2 = ((x2 – u)T(x2 – u) ) / N2         C =  (N1/N)C1 + (N2/N)C2            **2 – Others use this formula**               C1 = ((x1 – u1)T(x1 – u1) ) / N         (can be N or N – K, where K is the number of groups)         C2 = ((x2 – u2)T(x2 – u2) ) / N         (can be N or N – K)         C =  (1/N)(C1 + C2 )                    (N or N – K)       Why the difference   in Ci calculus, using global mean or group mean?   in C calculus, using (1/N) or (Ni/N)? **Observation:**   In my 4 datasets,   LDA and LDA with quadratic kernel   If I use `((xi – ui)T(xi – ui) ) / N or ((xi – u)T(xi – u) ) / Ni` the result is the same But in QDA   If I use `((xi – u)T(xi – u) ) / Ni` the result (classification) is poorer   than `((xi – ui)T(xi – ui) ) / N`


title: How do I analyse the data from my spaced repetition system

text: As part of my MSc, I have recently developed a web application designed to assist toddlers with learning difficulties in the comprehension of spoken instruction. The system shows a series of animations to convey an object and then tasks the child to pick out the correct object from a series of objects. I then collect the data as to whether the child selected the correct object. Then, through utilisation of a spaced repetition algorithm the cards are re- ordered and reshown at cutom intervals. So I have have a load of data that basically looks like this: Card Name - Correct/Incorrect(binary) I have produced a line chart in Excel that plots the total number of questions posed vs the number of questions answered correctly over time. The correct answer line looks quite straight, so I was hoping that I could show the children are learning the slope of the two lines are equal? Would this signify the children are answering all the questions correctly all the time? Are there any other deductions I could look to be making? I'm not a stats person and quite new to all this.


title: How to think while comparing methods

text: I'm a CS master student focusing on data mining. Now I'm doing my master thesis and the contribution of my thesis is to compare different approaches/methods of one topic (e.g. clustering of text documents). What I did so far is looking at the state of the art on the topic and read the papers. However now I need to think of what to compare. Of course there is the obvious comparison question: which method give best results? But that is so obvious. My supervisor once mentioned to see how those methods compare their results and look that maybe there is a better way to compare the results of the methods. That was helpful to me to think about questions like this. But still I'm so stuck and I can only think of obvious stuff. This is the first time I do something like that. I'm sure some of you went through something like this, so I was wondering if you can even tell me some /basis/ stuff and questions that people address when they compare methods in computer science. My main questions to you are: 1- When I read the papers of the methods to compare, in which way I should read it? Critical? Questionable? What to look exactly in them? 2- Is there a general scheme for comparing methods in academia? 3- As for a master thesis, what which stuff are a must-do for comparisions? 4- Any great references/papers related to comparisions that could help me?


title: Questions about nominal data

text: Can nominal data have uncountably many values? Are categorical data the same as nominal data? Are discrete data the same as nominal data? Thanks!


title: Using AIC to test the direction of causality

text: I would like to ask the opinion of this community in regard to the following discussion between me and my colleague. The case is this: we have two variables, let's call them Y and X. The AIC of a simple linear regression of Y on X `(Y = b1*X + e)` is A, and the AIC of the reversed model, regression of X on Y `(X = b2*Y + e)` is B. If B is much larger than A, can we say that the first model is preferable? I have never encountered such use of AIC, but I also couldn't find any references that would mention a restriction of the comparison based on AIC to models with same directionality. In other words, how close do the models in comparison have to be for an AIC-based comparison to be valid? Any references are welcome (with preferences for less technical ones, but anything is better than nothing). Thanks!


title: Exponential smoothing models backcasting and determining initial values python

text: I have made python code for exponential smoothing (ES) that takes in about 15 different cases including:   * Simple Exponential Smoothing (SES)   * Simple Seasonal models (both multiplicative and additive)   * Brown's Linear Exponential Smoothing   * Holt's Double Exponential Smoothing   * Exponential trend method   * Damped-Trend Linear Exponential Smoothing   * Multiplicative damped trend (Taylor, 2003)   * Holt-Winters Exponential Smoothing: multiplicative trend, additive trend, multiplicative season, additive season, and damped models for all four variations 14 of these cases can be found in page 8 of Exponential smoothing: The state of the art Part II. What values are commonly used for the initial values for the different ES models? What methods are used to determine points at time 0 and 1? (I am using `Nan` (not a number) to substitute in the code right now).


title: Multidimensional irt data generation

text: My aim is to generate multidimensional binary data with specified correlation between the dimensions. Let's assume that there are 100 items grouped under 2 dimension that are correlated at specified level


title: How can I calculate F critical value in ANOVA method knowing df1, df2 and alpha?

text: I'm very new to the statistic methods but at the beginning of my wonderful math-journey I found some problems. Two days ago I got to know the ANOVA method. I'm kind of familiar with it today, but there's one issue that I can not understand. I know how to calculate F, and df1 so as df2. But how about F critical value? How can I calculate this? (I calculated F, df1, df2, have n, k and alpha given.)