title: Transformation optimizing stationarity of the residual of a regression

text: I am trying to define an objective function or a method to find the transformation and coefficients optimizing the stationarity of regression's residuals. For instance, if I want to regress $X_1$ vs $X_2$ and I use a Box Cox transform. I want to find $\lambda$ and $\beta$ s.t. $\varepsilon = \frac{X_1^\lambda-1}{\lambda} - \beta*\frac{X_2^\lambda-1}{\lambda}$ is stationary. Any ideas?


title: Project in Bayesian biometrics

text: I'm working on a project about specific bird species on an island in the Northern Atlantic sea. The data seem reliable and pretty detailed, and range from the 18th century, however their structure and reliability is precise and detailed enough since 1970 for Bayesian biometrics inference. I want to model puffin natural mortality rate with Bayesian methods. I have stumbled on some articles about biometrics and I was wondering if someone could point out articles or material similar to what I have, since I would like to see multiple ways to approach my problem.


title: How to control for industry effects in regression?

text: Right now I'm working on an analysis of influence of cultural aspects on investment mode preference. However I have to control for many other factors, for example industry, since some industries, for example, are more direct investment intensive, whereas others are more appropriate for licensing. But I have no clue how to manage this issue. Which kind of information do I need in order to be able to control for industry influence and calculate variables expressing the influence of each industry on dependent variable, which is a ratio of royalties to FDI income. I guess I need to include industry dummies, however, I do not really understand how to estimate them and based on which kind of information.


title: How to analyse RCT where significant baseline differences exist despite randomisation?

text: I'm analyzing a double-blind, placebo controlled RCT using an ANCOVA in R, where the predicted "treatment effect" shows up before the treatment has occurred! The goal of the study is to determine if treating a disease reduces a particular behavior. Folks with both the disease and the behavior were randomized 50-50 into treatment and placebo control arms. As predicted there was a significant interaction between the intervention and baseline disease state (all variables are interval, except for "intervention", which is a two-level factor variable identifying treatment vs. control group):               MODEL 1:          Call:     lm(formula = follow_up_behavior ~ baseline_behavior + baseline_disease *      intervention, data = d)                                             Estimate Std. Error t value Pr(>|t|)         (Intercept)                            11.28464    0.51177  22.050   <2e-16 ***     baseline_behavior                       0.77872    0.05112  15.234   <2e-16 ***     baseline_disease                        0.36726    0.23110   1.589   0.1146         interventiontreatment                   0.74738    0.70254   1.064   0.2895         baseline_disease:interventiontreatment -0.64681    0.31374  -2.062   0.0414 *        The interaction was predicted because the intervention is a very effective treatment of the disease, but baseline disease state varies along a continuum from near 0 to very high. Hence, folks with high baseline disease got the biggest benefit from the treatment, and should therefore have had the biggest reduction in the behavioral response. So far so good. On a lark, I ran a very similar model of **baseline** behavior:               MODEL 2:          Call:     lm(formula = baseline_behavior ~ baseline_disease * intervention,      data = d)                                             Estimate Std. Error t value Pr(>|t|)       (Intercept)                             -0.6350     0.7620  -0.833   0.4062       baseline_disease                         0.7422     0.5016   1.480   0.1415       interventiontreatment                    1.1941     1.0611   1.125   0.2626       baseline_disease:interventiontreatment  -1.3320     0.6510  -2.046   0.0428 *      As you can see, there is a very similar significant interaction between disease state and intervention group, even though though the intervention has not yet occurred. This would seem to be a failure of randomization. My primary concern is that because follow_up_behavior is very highly correlated with baseline_behavior, the significant interaction in the first model is due to the pre-existing interaction seen in the second model, and is therefore not a consequence of the intervention. My questions are:   1. Do I actually have a problem?   2. If so, does including baseline_behavior as a control variable in model 1 fix the problem, i.e., guarantee that the significant interaction in this model is not a consequence of the pre-existing interaction seen in model 2 but is instead due to the intervention?   3. If including baseline_behavior as a control is insufficient, is there anything I can do salvage the study? Many thanks in advance for any help or insights.


title: learning an hierarchical linear model - overfitting / identifiability issues?

text: I'm currently looking at a paper where they've done this using two layer support vector regression and I'm trying to figure out whether they have biased the performance of their classifier and whether there is a better way of doing this. I have a sequence of bins and within each of these bins I have counts of specific features (all bins have the same set of features) and I have a number $Y$ associated with that sequence which I am trying to predict. So following the paper - for each bin I can learn a model which predicts Y_{i} (predicting of Y using information from bin i) ie $Y_{i} = \beta_{i} X{i} + \epsilon$. Now given the prediction of each of these classifiers, in the paper they then use these to learn another linear model to predict $Y$. They use the same data to learn both the $\beta$s for the individual bins and the coefficients for the second model. My question is this....is there an overfitting problem here - essentially using the same training data twice? Can the parameters of the model be learned reliably - ie can we actually learn the coefficients of both "levels" of the model at once......? Or is there a better way of formulating such a model? If this way of explaining doesnt make sense, let me know and I'll try to make a diagram/explain it more. Thanks


title: What test to use to determine if someone is actually answering on a questionnaire vs. responding randomly

text: I am new at using statistical tests and here is a very easy question but I cannot find the answer online. I am going to send out a survey and my professor told me that I should ask the same question three times but then rewrite it somewhat differently in order to test whether people filled in the survey with attention instead of just clicking the dots as fast as possible. There is a test which you can use in order to see if the three answers regarding the same variable are somewhat the same. What is the name of this test?


title: Does the average distance in K-means have to be monotone decreasing?

text: I'm implementing the k-means algorithm myself. I don't see any obvious mistake in my code and it seems to work well. However, there's something I don't understand. My algorithm, working on vectors $x_i \in X$, cetroids $c_j \in C$ and labels $l_i \in L$ looks like:   1. choose centers $C=\\{c_1,c_2,...,c_k\\}$ from $X_i$ randomly (no repetitions)   2. label each vector with the index of nearest centroid $$l_i = \text{arg}\min_{j \in \\{1..k\\}}(||x_i-c_j||, c_j \in C)$$   3. find new means of each cluster $$c_j = \text{mean}\\{x \in X_i \;\text{and}\; l_i=j\\}\;\text{for}\;j=1..k $$   4. if the centroids have moved signifficantly in the previous step, goto 2 A plain vanilla k-means, I guess. However, when I make a plot of the average of point-to-respective-centroid distances after each iteration, $$\text{mean}\\{||x_i- c_j||\;\text{and}\;j=l_i\;\text{for each}\;x_i \in X \\}, $$ I can see that the mean distance is not monotone decreasing (although in general it is decreasing and converging nicely): http://i.stack.imgur.com/ySgWf.png (sorry it won't allow me embed the image) I remember to be told it should be always decreasing, and my intuition is the same. What's wrong? My algorithm (implementation) or my assumption about the mean of distances?


title: SE of fit versus SE of prediction

text: I would like to get the standard error on a prediction. Using R `glm`, I can get the SE of the fit for a specific prediction:               mod <- glm(y~wa_WSI, data=mydata, family=gaussian(link="identity"))     predict.glm(mod,newdata=newdata, type="response", se.fit=T)      But when I compare the predictions with the actual values, this number seems way too small. I found a formula for "standard error of the estimate" which is $\sqrt{s/(n-p)}$ where $s$ is the sum of the squared residuals, $n$ is the number of data points, and $p$ is the number of terms in the regression. This gives me a much larger result, but is not for a single prediction. My question is, is the SE formula above the formula I should use and is there some way to get it from the value R gives me for `se.fit` so that it is specific for a particular prediction?


title: Characteristic function and Fourier transform

text: I understand the definition of characteristic functions used in probability theory: For a random Variable $X$ with probability density function $f_X$ the characteristic function is defined as: $$\phi_X(t) = E(\exp(itX)) = \int_{\mathbf{R}} e^{\mathrm{i}tx}f_X(x)\, dx.$$ I read that > [...] the characteristic function in nonprobabilistic contexts is called the > fourier transform (Page 342, of Probability and Measure. P.Billingsley 3rd > editon). But I still can't see this from their definitions, because the fourier transform $f^*$ of a function $f$ is defined as follows: $$ f^*(t) = \int_{\mathbf{R}^n} f(x)\,e^{-\mathrm{i} t \cdot x} \,\mathrm{d} x$$ If the function $f$ is a density, for e.g. $f=f_X$, then the fourier transform can be written as: $$ f^*_X = \int_{\mathbf{R}^n} f_X(x)\,e^{-\mathrm{i} t \cdot x} \,\mathrm{d} x \neq \int_{\mathbf{R}} e^{\mathrm{i}tx}f_X(x)\, dx = \phi_X(t) $$ The definitions differ with a minus, why can they than be used both for the same thing (e.g. for deconvolution)?


title: How do I calculate the ranking of some galleries based on the rankings of the artists represented by them?

text: The mean is not good in this case, because there are galleries that have an artist with a high rank and several other artists with way lower ranks. I'm thinking about doing a weighted mean, but I don't know how to split the weights to the ranking ranges. Any ideas? Thank you.


title: General distributions for which binomial and Poisson are special cases

text: I have an unknown discrete distribution $f(x; a,b,c,d)$. Of the 4 parameters one is redundant. I have found some limiting cases: $$X\sim\text{Binomial}(g(a,b,c,d),c/(b+c)))\;\;\; b,c \gg a,b$$ $$X\sim\text{Poisson}(a/d)\;\;\; b \gg a > d \gg c$$ I shall leave $g$ undefined for now. I am wondering which distributions would be good candidates for $f$.


title: How to do I find a joint density?

text: Can someone please help me with finding the joint density of $Y_1$ and $Y_2$ when $Y_1=X_1+X_2$ and $Y_2=X_1-X_2$?


title: inverse logistic regression with binary covariates

text: I am currently using a hierarchical Bayesian framework to investigate a problem with both a single binary response variable and binary covariates, $P(Y=1 | X_i=1), i=1,\ldots,n$. Using R/JAGS I can characterize the coefficients, find the odds-ratios, etc. However, I would like to make statements regarding the inverse, $P(X_i=1 | Y=1)$. Any suggestions for an approach or pointers to literature would be appreciated.


title: Estimating the Marginal Effect of a Misclassified Binary Regressor Without Instruments

text: In ordinary least squares, we can use an errors-in-variables model when one or more of the regressors are measured with additive noise to ameliorate the effects of attenuation bias. We just need to know the reliability $r = \left( 1-\frac{\text{noise variance}}{\text{total variance}} \right)$.   1. Is there an analogous result to estimate the effect of a misclassified binary regressor on a continuous or a binary outcome that does not rely on using an instrumental variable?   2. If I am willing to assume something about the misclassification errors, can I just divide the estimated coefficient by $(1-p-q)$, where those subtracted terms represent the Type I and II error rates to see how sensitive my estimates are?  * * * Here's one possible nonparametric regression approach by Chen, Hu, and Lewbel. Has anyone ever used this on actual data that I can try to replicate to make sure I am doing this right?


title: When and why does the "brittleness" of mutual information cause overfitting?

text: I have frequently heard concern over "brittleness" of entropy and mutual information as performance metrics for a statistical fitting and the fact that it leads to overfitting. You can see an example of such concern in this blog post. However I have trouble understanding what exactly "brittleness" means in this context, and in which cases it would be a basis for overfitting.   * In which cases entropy and mutual information should not be used?    * If they are used, how can you ensure that no overfitting occurs?


title: Interpretation of n-way ANOVA results using different models in MATLAB

text: I have been trying to understand the function `anovan` in MATLAB to perform n-way ANOVA to test the effects of multiple factors on my data. What caught my eyes when I read the help page for this function is that, in their example, the p-value for factor `X1` changes from being insignificant (p>0.05) to being significant (p<0.05) when the model is changed from `default('linear')` to `'interaction'`. How should I interpret this result? Also, what if I don't know whether the two factors I'm testing have any interaction or not, which model should I use?


title: Learning multiple output

text: Can you suggest me an algorithm and probably a real code for multiple output learning, where input of the model is vector of around 10 000 values and output is, for each input vector, an output vector of 1500 dimensions (so, it's kind of a big dataset)? In my opinion neural network can handle this number of values, but what else? Supervised SOM? Multi-class multikernel learning? Any examples of the real world data model with the same type of learning (large multiple output?) Any suggestions and opinions will be very valuable for me. Update: Data example can be found here - http://pastebin.com/Z4E87d3d


title: What method to detect structural breaks on time series?

text: I have two lists of price (stock) and I need a method to detect if there are structural breaks inside those series. What method can I use? For each serie I have around 600 prices (daily). Thank you in advance


title: Calculating confidence interval from binomial distribution

text: I have two variables:               a: 12,13,15,10,9,8     b: 15,15,15,15,15,15      variable 'a' is aggregate numbers of choosing a particular answer (0 or 1) for each case (6 cases in this example). So for example for the 1st case 12 out of 15 participants chose 1, 13 out of 15 in the second case, 15 out of 15 in the third etc. variable 'b' predicts behavioral responses for each case. Here it predicts that for each case the majority of participants should select option 1. To compare 'a' to 'b' I calculated the difference between each aggregate response:               diff: 3,2,0,5,6,7 which is a total of: 23      I would like to compute a confidence interval for this score.


title: Training a Tic Tac Toe brain - am I on the right track?

text: My only experience with Machine Learning is Andrew Ng's Coursera course, but I did work through that just fine and passed with 100%. I decided to practice by making up some problems and solving them. My idea was to create a simple tic tac toe game, and have it just play millions of random games, and use the data to learn to predict the next move based on sequences that won in the past. **Is it reasonable to expect to learn to predict a good move from 1 million samples of games that player X won?** I have a function `getWinSequences('X', 1000000)` that returns of list of sequences that win for X. A sequence represents series of choices, 1 to 9, on the game board. The board would look like this:               1|2|3     -----     4|5|6     -----     7|8|9      So `WinsX` looks like this:               [         [4,2,5,3,6],         [1,4,2,8,3],         [1,2,4,3,5,8,9]     ]      So in the first sequence, that means that player X chose 4, O chose 2, X chose 5, O chose 3, X chose 6 So X won with 3 in a row, 4,5,6 ([ **4** ,2, **5** ,3, **6** ]) Now, I'm not doing machine learning on the winning sequences, because I want the program to get a "current board position" not a sequence. In other words, **given a board with none or some X's and O's, assuming it's player X's turn, predict a good move (1-9) based on past observations a boards, and knowledge of the next move that was chosen** So I have a function that converts a winning sequence into a series of board positions, and a label saying which move is next. So the actual data that I train on is like this               X[0] = [0,2,2,             1,1,0,             0,0,0]     y[0] = 6      **0 means empty, 1 means an X is there, 2 means a Y is there. So based on my winning sequences, I am able to automatically label that position as "6", because the board is a recreation of what the board looked like just before we "randomly" won by choosing 6.** Because I want to be able to predict a good move even if there is no winning move, then whenever I have a winning sequence, **I don't just create a board as it was just before the win, I create a board for every step along the way of a winning sequence, and label it with the next move from the winning sequence.** So a winning sequence will usually result in 3 to 5 examples of a board position, and a label (1-9) saying what the move for that board should be. I figured that with a million examples of winning sequences, and the board positions that led to the win, I would be able to predict the next move. **Is that reasonable?** If so, what is a good way to train on the data? When I trained on 1 million examples, I got a score of .29 from GradientBoostingClassifier. the relevant code looks like this:               winsX = getWinSequences('X', 1000000)     samplesX = samplesFromWins(winsX)          X = [row[:9] for row in samplesX]     y = [row[9] for row in samplesX]          X_train, X_test = X[:700000], X[700000:]     y_train, y_test = y[:700000], y[700000:]          print("got data, ready to train")          clf = GradientBoostingClassifier(n_estimators=100, learning_rate=1.0,         max_depth=1, random_state=0).fit(X_train, y_train)          print(clf.score(X_test, y_test))      **I just want to know if I'm on the right track, or if there is a fundamental flaw in what I'm doing.**


title: Measuring representativeness of a (non-random) selection

text: I am looking for a numerical value that expresses how representative a (non- random in this case) sample from a population is regarding the distribution of a certain attribute. Take for example, the age distribution in a country and in its elected body of representatives. I'll want to compare this value across many dimensions, so the value needs to be normalized in some way and not depend on the nature of the dimension. Some of the dimensions will be numerical and continuous (age, income), others discrete with many (county of residence, trained profession) or with few classes (gender). The goal is to figure out in which of these dimensions the sample is most (or least) representative, i.e. in which ways the non-random selection (the electoral process) is biased. Notes: There is a question of almost the same title here already, but I think it talks about something else, and this other question seems to ask for a very similar thing too, but there is no good answer for me there, either.


title: Results Difference: Frequentist vs. Bayesian

text: I fit a lognormal model on some data points using both frequentist and Bayesian (using a non-informative prior) approaches. However, I got different results. Here are my codes and outputs: Frequentist:               > data1 = c(0.32618457, 0.29166954, 0.27427996, 0.23844847, 0.25148180)     > n=length(data1)     >      > lln1 = function(par){ if(par[2]>0) return( -     > sum(log(dlnorm(exp(data1),par[1],par[2]))) ) else return(Inf) }     > optim(c(0,0.1),lln1)     >      >   mu              sigma      [1] 0.27641155     0.03091169      Bayesian with 20,000 MCMC and 4000 burn:               model      {             for( i in 1 : N )              {                 x[i] ~ dlnorm(mu, tau)              }                  mu ~ dunif(0, 1)             tau ~ dunif(0, 1)                  sigma <- 1/tau     }             list(N = 5, x = c(0.32618457, 0.29166954, 0.27427996, 0.23844847, 0.25148180))                    Node     mean    sd  MC error   2.5%    median  97.5%   start   sample     mu      0.2417  0.2182  0.001976    0.006612    0.1759  0.8226  4000    16001     sigma   2.625   2.22    0.01899 1.049   2.015   7.755   4000    16001      Since I'm using a non-informative prior, I was wondering why the estimates of mu and sigma are different.


title: Bayesian batting average prior

text: I wanted to ask a question inspired by an excellent answer to the query about the intuition for the beta distribution. I wanted to get a better understanding of the derivation for the prior distribution for the batting average. It looks like David is backing out the parameters from the mean and the range. Under the assumption that the mean is $0.27$ and the standard deviation is $0.18$, can you back out $\alpha$ and $\beta$ by solving these two equations: \begin{equation} \frac{\alpha}{\alpha+\beta}=0.27 \\\ \frac{\alpha\cdot\beta}{(\alpha+\beta)^2\cdot(\alpha+\beta+1)}=0.18^2 \end{equation}


title: Pandas / Statsmodel / Scikits-learn

text: 1. Are Pandas, Statsmodels and scikits-learn different implementations of machine learning/statistical operations, or are these complementary to one another?   2. Which of these has the most comprehensive functionality?   3. Which one is actively developed and/or supported?   4. I have to implement logistic regression. Any suggestions as to which of these I should use?


title: Logistic functions in ANN. Biologically inspired?

text: Was the introduction of logistic/sigmoid activation functions in artificial networks justified by biological evidence, practical convenience or both? What are some good references on this topic?


title: Durbin Watson test statistic

text: I applied the DW test to my regression model in R and I got a DW test statistic of 1.78 and a p-value of 2.2e-16 = 0. Does this mean there is no autocorrelation between the residuals because the stat is close to 2 with a small p-value or does it mean although the stat is close to 2 the p-value is small and thus we reject the null hypothesis of there existing no autocorrelation?


title: Excel's CHIDIST function in MATLAB for large values

text: This question is related to Excel's CHIDIST function in MATLAB The minimum answer larger than zero I can get in Excel is about 2.2e-308 that correspond to `realmin` in MATLAB. However, in MATLAB the minimum answer is 2.2e-16, or `eps(1.0)`. I understand this is because I first calculate `chi2cdf` that is close to 1 for large values, and the precision in this case is `eps(1.0)`. Is there any other way to reproduce Excel's CHIDIST and not to loose the precision? Do you know exactly how CHIDIST works? **EDIT:** I have to do this test multiple times, and in many cases the resulting p-values are close to zero. I need to rank the p-values, this is why I need high precision in p-value calculation.


title: Assumptions and pitfalls in competing risks model

text: Question: **What are the major pitfalls and assumptions for competing risk analysis?** _Background_ I'm trying to learn competing risks and I use the `cmprsk` package in R based upon Gray's algorithm. I've read Melania Pintilie's book, Competing Risks: A practical perspective, but although it's a very nice introduction to the field I lack a list of the common pitfalls. My mentor has warned me about the competing risk analysis still being a developing field and I would therefore really appreciate if I could get your opinions on this. In the answer to this question it's mentioned: > just make sure that you understand what assumptions you are making As I understand it the major assumption is the proportional hazards, similar to the Cox model, but other than that assumption I'm not sure that I know what other assumptions and pitfalls I'm to look out for.


title: How to determine if two time series are significantly related to each other

text: Based on our knowledge of other characteristics of these two variables, we have reason to believe that changes in admits to a ward has an impact on a certain bad outcome on that ward (these are counts collected monthly):               > dput(admits)     structure(c(3L, 4L, 3L, 22L, 54L, 74L, 35L, 58L, 59L, 45L, 38L,      52L, 37L, 29L, 39L, 27L, 14L, 4L, 6L, 15L, 31L, 10L, 12L, 14L,      11L, 18L, 36L, 33L, 42L, 35L, 20L, 28L, 22L, 54L, 26L, 41L, 26L,      41L, 40L, 34L, 31L, 23L, 34L, 22L, 21L, 11L, 29L, 13L, 27L, 40L,      41L), .Tsp = c(2010, 2014.16666666667, 12), class = "ts")          > dput(badOutcome)     structure(c(12L, 14L, 13L, 12L, 42L, 55L, 47L, 29L, 25L, 28L,      17L, 22L, 54L, 30L, 31L, 25L, 26L, 9L, 12L, 7L, 14L, 17L, 13L,      13L, 14L, 12L, 15L, 20L, 17L, 30L, 35L, 41L, 18L, 19L, 26L, 15L,      12L, 5L, 15L, 12L, 21L, 13L, 18L, 22L, 19L, 21L, 12L, 8L, 7L,      15L, 12L), .Tsp = c(2010, 2014.16666666667, 12), class = "ts")      Since the data are serially correlated, my understanding is that the assumption of independence for regular regression techniques is violated. What are the steps then, and techniques, to determine if "admits" is significantly related to "badOutcome" ?


title: Can we compare correlations between groups by comparing regression slopes?

text: In this question they ask how to compare Pearson r for two independent groups (such as males vs females). Reply and comments suggested two ways:   1. Use Fisher's well-known formula using "z-tranformation" of r;    2. Use comparison of slopes (regression coefficients).  The latter could be easily performed just via a saturated linear model: $Y = a + bX + cG + dXG$, where $X$ and $Y$ are the correlated variables and $G$ is a dummy (0 vs 1) variable indicating the two groups. The magnitude of $d$ (the interaction term coefficient) is exactly the difference in coefficient $b$ after model $Y = a + bX$ conducted in two groups individually, and its ($d$'s) significance is thus the test of difference in slope between the groups. Now, slope or regression coef. isn't yet a correlation coef. _But_ if we standardize $X$ and $Y$ - _separately_ in two groups - then $d$ will be equal to the difference _r in group 1 minus r in group 0_ and therefore its significance will be testing the difference between the two correlations: we're testing slopes but it appeares [as if - ?] we're testing correlations. Is that I've written correct? If yes, there's left the question which is a better test of correlations - this one described or Fisher's one? For they will yield not identical results. What do you think? **Later Edit:** Thanking **@Wolfgang** for his reply I nevertheless feel I miss understanding _why_ Fisher's test is more correct a test for r than comparison-of-slope-under-standardization approach described above. So, more answers are welcome. Thank you.


title: Labels in the Digit1 dataset for semi-supervised learning

text: I am working with the "Digit1" dataset introduced by the book "Semi-Supervised Learning" by chapalle et. al, as one of the benchmark datasets in the field. In the dataset description located at: http://olivier.chapelle.cc/ssl- book/benchmarks.pdf comes: > The class label was set according to the tilt angle, with the boundary > corresponding to an upright digit. Somewhere else in the document comes that there are only 2 classes in this dataset. From these two explanations I still can not understand how exactly the labels are defined. I would be thankful If someone could help me with this.


title: Is there an R implementation to some mixed models quantile regression statistical procedure?

text: I would like to find some solution for performing a mixed effect model of quantile regression. From my google searching, I was not able to find an R implementation for such a procedure (only warnings that "this is not for the faint hearted"). I would like to solve a simple situation where we have one x one y, and one "subject" variable. Any suggestions on what to do with this?


title: Average effect across multiple linear models?

text: I have several OLS models with robust s.e.'s that predict an outcome variable Y For instance: Model 1 $Y=B_0 +B_1X_1$ Model 2 $Y=B_0 + B_1X_1 + B_2X_2$ Model 3 $Y=B_0 +B_1X_1 + B_2X_2 +B_3X_3$ I am interested in giving an average effect for $B_1$ across Models 1-3 with an accompanied 95% CI. Can I just take the average of $B_1$'s across Models 1-3 and the average of standard errors to construct my confidence interval? What is this called?


title: Latent semantic classification

text: How can I create a training data set for document classification using LSA? I have created a term-to-document matrix and have class labels also. I don't know whether to add these class labels in a term-document matrix or to create anther matrix. I don't know the exact steps to follow to create training data.


title: Meaning of 0-weighted linear weights in SVM

text: I am working with linear SVM (Using SVMlight) and I'm assisting to a weird phenomenon. The training algorithm weighted some features 0. Does that means such features are irrelevant for the classification? Looking at the dataset I found that the vectors containing such features belongs only to a category. And that to me sound like the features are very relevant to classify a new observation. Moreover I have examples of other features belonging to a single category that have a non zero weight and that confuses me even more. Is that behavior correct or I am missing something?


title: What are the main differences between classical and Gibbs sampling Latent Dirichlet Allocations?

text: In these weeks I have been studying the classical Latent Dirichlet Allocation (LDA) algorithm by David Blei and colleagues (2003), and the LDA variant based on Gibbs sampling introduced by Tom Griffiths. What are the main differences between the two methods?


title: Using confidence interval from full sample to test statistics from a small sub-sample?

text: This may be a rather stupid question. However, I have already spoken to a statistician and he did not seem certain about the answer, so perhaps it may be of use to others. The study is not typical, so please bear with me while I try to explain it. I have about 75,000 sentences of English conversations for a sample. I'm interested in whether sentences with a certain grammatical structure (specifically, missing verb phrases) tend to clump together. First, I calculated the relative frequency (empirical probability) of these sentences and got 910 hits/75,000 total sentences ≈ 0.012. To get confidence intervals for this statistic, I used bootstrapping with 10,000 resamples of the entire data set with replacement, and found that the 99% confidence interval is about 0.0109 - 0.0134. To see whether and to what extent these sentences tend to be near each other in the data, I made a new sub-sample. For each sentence with a missing verb phrase, I counted how many sentences also had missing verb phrases in the five previous sentences. So, I have five "slots", each with a relative frequency of how often sentences in that slot lack verb phrases (i.e., the immediately preceding sentence is a slot, then the next preceding sentence, and so on). This gives me results like this: Slot 1: 56 hits/910 total sentences = 0.061 Slot 2: 32 hits/902 total sentences = 0.035 etc... My question is this. Can I compare these relative frequencies for each slot to the confidence interval I calculated from the whole data set? In other words, if slot 1 has a relative frequency of .061, that is beyond the 99% confidence interval from my bootstrap of the whole data set, so is it therefore valid to claim that this is higher than could be expected in that slot? And if it isn't, is there a relatively straightforward way to make it work?


title: Implementation of CRF in python

text: Is there a popular implementation of **Conditional Random Fields** in **Python**? I can't seem to find any that is widely used and popular!


title: Cronbach's alpha continuous and dichotomous data in SPSS

text: I have one question about using SPSS to calculate Cronbach's alpha of a measure that draws on two types of items: one yields ordinal data and the other dichotomous. The ordinal data are produced through 4-point rating items and the dichotomous data through true-and-false questions. Is it still OK for me to use the reliability analysis option to calculate the Cronbach's alpha for the measure?


title: Is it possible to determine the relationship between X and Y, with only X and Z, where Z = X + Y?

text: I have a set of data composed of two variables $X$ and $Z$, where $Z = X + Y$. I want to make a statement about the relationship between $X$ and $Y$. For instance, I'd like to claim that $X$ and $Y$ vary together or correlate. Is there a way to do this with only measurements from $X$ and $Z$? As an additional wrinkle, consider if X and Z are corrupted by some noise $X$-noise and $Z$-noise. $X$ is actually $X + X$-noise and $Z$ is $X + Y + Z$-noise. In addition, $X$ is scaled by some coefficient $a_X$ and $Z$ is scaled by some other coefficient $a_Z$. Therefore, the data I collected, which are paired measurements, are $[a_X*(X + X-\text{noise}), a_Z*(X + Y + Z-\text{noise})]$. Can I still estimate $Y$ from my measurements? If I do not know $a_X$ or $a_Z$ is this problem unsolvable?


title: Independent event proof

text: I'm doing a stats past paper for my first year exam, and I'm having some trouble answering the following question. Could someone please help me? Thanks 'Show that if 3 events $\\{A, B, C\\}$ are independent, then $(A \cup B)$ and $C$ are independent as well'


title: Low classification accuracy, what to do next?

text: So, I'm a newbie in ML field and I try to do some classification. My goal is to predict the outcome of a sport event. I've gathered some historical data and now try to train a classifier. I got around 1200 samples, 0.2 of them I split off for test purposes, others I put into grid search (cross-validation included) with different classifiers. I've tried SVM with linear, rbf and polynominal kernels and Random Forests to the moment. Unfortunately, I can not get accuracy significantly larger than 0.5 (the same as random choice of class). Does it mean I just can't predict outcome of such a complex event? Or I can get at least 0.7-0.8 accuracy? If it's feasible, then what should I look into next?   * Get more data? (I can enlarge dataset up to 5 times)   * Try different classifiers? (Logistic regression, kNN, etc)   * Reevaluate my feature set? Are there any ML-tools to analyze, which features make sense and which don't? Maybe, I should reduce my feature set (currently I have 12 features)?


title: Logistic Regression and how to judge model fit and parameter influence

text: I want to show statistical significant impact of about 15 different independent variable on a binary dependent variable (I am not a statistician). Some of my independent variables are term counts in text so they are likely correlated with the length of the text which is also one of the independent variables. I selected this set of variables because they seemed to impact the outcome at least individually. I performed matching on the length variable to try to control for this effect (length alone explains a lot of variance in my outcome/dependent variable). Now I want to figure out which variables still matter in the presence of the other variables and what their relative importance is. Question 1: What is the best way to deal with the possible collinearity of my independent variables? Variant 1: Replace term_counts with residuals after regressing them on length. Variant 2: First, do logistic regression model on all variables. All factors where the parameter is significantly different from zero are significant factors (this is a claim I would like to make). And then models where I remove a single variable and show that the fit gets significantly worse to show that every single one matters, and by how much (how to do this exactly leads to Question 2). Question 2: What is a reasonable and standard way of assessing superior fit of the different logistic regression models? (also see Assessing logistic regression models) What measure should I use to compare fit? Likelihood doesn't really work since it will always get better for more complex models. Classification seems stupid since our empirical estimates for p(Y=1) are only 20-40%. We were also thinking about plotting p_empirical versus p_model after binning multiple observations to get a clearer picture. I am sure people have done this before and I would be very happy if you could point me to how to visualize the fit/performance. Question 3: What is the best way to normalize the features so I would be able to compare the model parameters (i.e. how much more important is one feature over another?)? Is it reasonable to scale everything between zero and one. Should it be "minus mean, divide by std". I have some ordinal, some binary, and a few continues dependent variables. Thanks!


title: Mathematical technique to find activity windows in a moving average plot

text: I have some raw data that comes from a simulation. I take the sample points, and produce a moving average variable, and plot it against time. So imagine a signal that has peaks and valleys. The peaks represent periods of high activity, and the valleys represent periods of low activity. What filter can I apply to this data to report the windows of high activity? We can assume that the windows do not overlap. EDIT: The words "what filter can I apply" are not used correctly. I am not looking for a "filter". I am looking for a technique to **find the windows**. I start with the data, and all I want to do is find where the interesting windows of high activity are, so I can analyze them. You could say "just look at them", but I'm dealing with many data sets so that doesn't scale.


title: Conditional probability questions (homework)

text: I copied the exact question. Can anyone check my answer? I am so confused between the choices, all of them seems to be workable. > An company divides their customers according to how long they take to pay > their bills into three groups: fast (< 1 month), slow (1 to 6 months), > incredibly slow (> 6 months). From their previous records they know that the > proportion of customers that are fast is 0.3, the proportion of customers > paying slow is 0.65 and the proportion of customers who are incredibly slow > is 0.05. Among those that are incredibly slow, half will still pay within > two years, the other will never pay. > > Which of the following statements are true? > >   1. One out of 40 people never pay their bills >   2. A customer who has not payed his bill for 1 month has probability of > 0.0357 of never paying. >   3. It's more likely that a customer pays within 6 months, than taking > longer than 1 month >   4. Given a person is fast or very slow, it is more likely that the person > is very slow > My answer is 2 3 4. Reasons:   1. One out of 40 people never pay their bills. - False In probability terms, we usually add a #Taking the average#   2. A customer ... probability of 0.0357 of never paying. - True Consider reduced sample space of 0.65+0.05=0.7 Consider chance of never paying = 0.025 0.025/0.7=0.0357...   3. It's more likely ... taking longer than 1 month. - True Probability (Within 6 months) = 0.3+0.65=0.95 Probability (Greater then 1 month) = 0.65+0.025=0.675 (Assumption: People who does not pay)   4. Given a person .. is very slow. - True By 2 cases P(fast)=0.3 P(slow)=0.7


title: Is there a simplistic model to disaggregate census data based on years and smaller zones?

text: In a nutshell, here's what I have:   * Annual population estimates for the State   * Periodical (5 years) age, population, and basic census data per zones Here's what I want to do:   * Create a simplistic model to generate the data for the missing years between the period for each zone, and have the total sums add up to where the yearly state population estimate is. All in all, I'm looking for a non complicated statistical model that is able to generate values with an acceptable (doesn't have to be super high) precision.


title: How to analyse a study looking at relationship between one set of five items (predictors) and a second set of five items (outcomes)

text: My study is about customers’ perception of the effectiveness for a Malaysia corporate Weblog. In my study, customers’ perception of the effectiveness defined as perceived ease of use, perceived interactivity, and perceived trustworthiness; whereas the corporate Weblog defined as Weblog publishing software, Weblog comment system, and Weblog blogroll and hyperlink. In my study, I want to determine whether there is a relationships between: a) Customers’ perception of ease of use and Weblog publishing software; b) Customers’ perception of interactivity and Weblog comment system; and c) Customers’ perception of trustworthiness and Weblog blogroll and hyperlink. In the questionnaire, each section (perceived easy of use; perceived interactivity; perceived trustworthiness; Weblog publishing software; Weblog comment system; and Weblog blogroll and hyperlink) consist of 5 questions. As the questionnaire design in 7-likert scale (ordinal measurement), therefore I’m using non-parametric correlation (Spearman’s correlation) to analyzed the data. Besides that, as in my dissertation chapter 1, there have some research hypothesis; therefore I’m selected one-tailed significance. ### Questions:   1. Is Spearman Correlation the appropriate method to analyse my data?   2. Because each section has 5 questions (eg: Perceived ease of use and Weblog publishing software), the statistical output has 10 correlation coefficients, but how should I interpret these 10 correlation coefficient into one conclusion as to whether there is a relationship between customers’ perception of ease of use and Weblog publishing software?   3. If I wanted to do a regression for customers’ perceived ease of use (5 questions) and Weblog publishing software (5 questions), which regression should I use?


title: What is the pull distribution?

text: In principle a simple question: What is the pull distribution? (All I could find out is that it is the error-weighted distribution of estimators around the true value.) I'd be interested in the precise mathematical definition, how, why, and when to use it, what is it expected to look like, and if both estimator values and "true" value have errors associated to them, do these add up in quadrature. PS: Feel free to add / change tags, I didn't find any good ones.


title: Is a Bayesian Classifier a good approach for text with numerical meta-data?

text: I'm trying to come up with an approach for detecting scam adverts on my website. I think the problem has a lot in common with detecting spam email (for which a naive Bayesian classifier is a common solution) since many of the signals that indicate a scam will be found within the text of the advert. However, there are certain other pieces of information which can be good scam indicators, but I'm not sure if/how a Bayes classifier could use them, because they involve numeric values (with values at the extremes of the range being suspicious) rather than simple binary values corresponding to the presence or absence of a word in the text. For example, many scam adverts have the price of the item set very low (to attract lots of views), so I would like a lower than normal price to be a strong indicator that the advert may be a scam. Is Bayes still a good fit for my requirement, if not then could you recommend a different approach?


title: Graphical representation of weekly spending

text: What is the correct graphical representation for these data of weekly spending (in dollars) on soft drinks for 20 people:               12,13,17,21,24,24,26,27,27,30,32,35,37,38,41,43,44,46,53,58      I want to separate the data into 5 bins:               10-20(f=3),    20-30(f=6),    30-40(f=5),    40-50(f=4),    50-60(f=2)      and I title axis: "Weekly spending (dollar amount)". Which one would be more appropriate for y axis? (For which I put `f=3,6,5,4,2`.)   1. Frequency per \$1 spent   2. Frequency per \$10 spent


title: Distribution of standard deviation

text: This question addressed the normal distribution, but I am wondering what is known about the distribution of the standard deviation of a sample of size _n_ drawn from an arbitrary distribution. In particular, what is the standard deviation of the standard deviation? For a normal distribution, the s.d. of the s.d. is $\sigma \over{\sqrt{2n}}$. Is this approximately true for an arbitrary distribution as $n \rightarrow \infty$ ?


title: What are modern, easily used alternatives to stepwise regression?

text: I have a dataset with around 30 independent variables and would like to construct a GLM to explore the relationship between them and the dependent variable. I am aware that the method I was taught for this situation, stepwise regression, is now considered a statistical sin. What modern methods of model selection should be used in this situation?


title: How to create a composite variable to use as a response variable?

text: I am a student doing my master's thesis and I have a question regarding my study. I am working with country data for 25 countries and I am looking into cultural values, attitudes and sociodemographics as predictors of environmental behavior. For the dependent variable environmental behavior I would like to combine several aspects (Generation of waste, waste treatment, GHG emissions, energy consumption, public transportation expenditure, car usage, environmental protection expenditures) Could you please give me some advice on how to combine these into one composite variable? If I transform each variable into quartiles (1 being environment protection and 4 environment degradation) can I then add the "rankings" and form a composite? or is this not a valid way to create a composite? Also, is it a good idea to combine the variables into one composite dependent variable, or should I focus on making regressions for each aspect of environmental behavior separately? I use the software Eviews and SPSS. Please if I am not clear enough let me know to give further details.


title: References regarding rules for multidimensional histograms

text: Can someone give me references on the theory of histograms applied to multivariate data? Are there any rules like the one dimensional Freedman- Diaconis available for the high dimensional case? What are the most common drawbacks? I'm planning to use the histogram to do density estimation.


title: Calculating  sample mean and sample variance on all samples vs distinct subsets

text: Consider 1000 samples drawn from an unknown distribution. Whats the difference between the following two ways of calculating sample mean and sample variance?   1. find sample mean and sample variance over all 1000 samples.   2. find sample mean over first 500 samples and sample variance over remaining 500 samples. What is the difference and which method is preferred?


title: ANCOVA with repeated measures in R

text: I am a novice to program R and have been trying to perform a repeated measures ANCOVA with Temperature as the dependent variable, Site as the independent variable, Date as the covariate and Year as the repeated measures. My dataset consists of temperatures from 4 sites, over 20 days, during 2 different years. There appears to be a significant effect of site, date and year on temperature, but no significant interaction between site and year or site and date. I originally performed two separate ANCOVAs, but my coauthor wants them in a single model. As far as I can tell, there is no simple way to do a repeated measures ANCOVA as there is for ANOVA. I have tried using: `lm(formula = Temp~Site*Date + Year)` , but wouldn't this just put year as an interaction that doesn't affect the model? Any help you can provide would be GREATLY appreciated!


title: Calculating point estimates from model-averaged parameters

text: I'm using an IT-approach and multi-model inference with some count data. I have used model averaging to obtain parameter estimates from several GLMMs with Poisson-lognormal errors (Poisson family plus observation-level random effect). Model-averaged parameter estimates are as follows: (Intercept) 1.9522   landuse -2.0318   treatment -0.8861   hedge -0.6943   sun -0.5191   landuse:treatment 1.0529   bound.area -0.2389 I am now trying to work out point estimates from these. Landuse is a two level factor (a and p). Treatment is a two level factor (AF and MC). Hedge, sun and bound.area are all continuous variables. So the intercept is treatment = AF and landuse = a. I am interested in working out point estimates for:   1. treatment = AF when landuse = a    2. treatment = AF when landuse = p   3. treatment = MC when landuse = a   4. treatment = MC when landuse = p I have Grueber et al’s 2011 paper on MMI but I am not sure whether I have correctly managed to work out how to apply their methods (detailed in the paper's appendix) to my parameter estimates. My predictors were standardised to a mean of 0 and SD of 0.5, so I understand that I have to substitute standardized predictors and then solve the model for the required estimate at the mean (i.e. 0) of all the other predictors. So for example for a point estimate of treatment=AF when landuse=a, can I simply do exp(intercept)? [Which is the same as exp(1.9522 – 2.0318*0 - 0.8861*0 - 0.6943*0 - 0.5191*0 + 1.0529*0*0 – 0.2389*0).] But how do I work it out for any of the other combinations? Would treatment=AF and landuse=p be worked out like this? : exp(1.9522 – 2.0318 - 0.8861*0 - 0.6943*0 - 0.5191*0 + 1.0529 – 0.2389*0) I have a feeling I’m missing something / have misunderstood... Finally, am I right in thinking that I can’t work out a treatment effect by itself (as it’s included in an interaction)? Any help much appreciated.


title: Independent vs. Dependent Compound Poisson Distributions

text: I have an issue with a section of some Actuarial lecture notes that I am reading. Here are the snippets from the notes: "Consider a portfolio consisting of $n$ independent policies. The aggregate claims from the $i$-th policy are denoted by the random variable $S_{i}$, where $S_{i}$ has a compound Poisson distribution with parameters $\lambda_{i}$ (all i.i.d), $\textbf{not known}$, and the CDF of the individual claim amounts distribution is $F(x)$, known." The notes then go on to say that this implies that all of the $S_{i}$s are i.i.d. This makes intuitive sense. However, the next section has the same set-up, but now the Poisson distribution parameters are all $\lambda$. The notes state "If the value of $\lambda$ were known, then the $S_{i}$ are i.i.d". I.e. the $S_{i}|\lambda$ are i.i.d. Implying that the $S_{i}$ themselves (i.e. with $\lambda$ $\textbf{not known}$) are dependent. This seems to contradict the first section. If you can help me get my head around this, intuitively, that would be a great help. Thanks very much.


title: Does random effects allow me to do repeated measures?

text: I am analysing some behaviour data from several participants in Matlab. I used to do "repeated measures" in SPSS. But is this the same as adding "subject" as a factor, and choosing it as a random effect? i.e. my normal repeated measures data matrix x looks like this                                condition A      condition B     subject 1          x(1,1)           x(1,2)     subject 2          x(2,1)           x(2,2)     ...                 ...              ...     subject 7          x(7,1)           x(7,2)      so for Matlab, I tried               COND = [1 2                   SUBJ = [1 1             1 2        and                2 2                ...                           ...             1 2]                          7 7]      then ran `anovan(x(:), {COND(:), SUBJ(:)}, 'random',2 )` Now, this does not seem to give me the same results as SPSS, for repeated measures... Perhaps I have misunderstood the 'random' parameter - I thought you could construct a "repeated measures" test from it?


title: Goodness of fit of sorted data

text: Say I have a list of 10000 sorted counts $\ge 0$ that sum to 20000 and I want to determine that they came from placing 20000 balls randomly, independently and uniformly into 10000 cells. A straight Chi-squared test would be reasonable if I assume independence, but 10000 counts of two would be blessed by chi squared, but I'd be somewhat skeptical of independence (randomness as well). I could compare the counts with a Binomial(20000,1/10000) ~ Poisson(2), but the counts are no longer independent, so a Chi-squared test is probably no longer reasonable even if I compute the covariance matrix for the counts. Surely this has been dealt with before. (I'm interested in the case where the probabilities are non uniform, but want to understand the simple case first) **Added:** Calculation of variances. (All the cleverness in what follows is due to Robert Israel and André Nicolas.) If we draw $n$ samples from a $\mathrm{Poisson}(\lambda)$ distribution then we expect to draw a count of $r$ $$E(X_r) = n \frac{\lambda^r}{r!}e^{-\lambda}$$ times. As the process of drawing a count of $r$ can be considered as $n$ independent Bernoulli trials, the variance would be $$\mathrm{Var}(X_r) = n \frac{\lambda^r}{r!}e^{-\lambda}\left(1-\frac{\lambda^r}{r!}e^{-\lambda}\right)$$. Now, if we place $\lambda n$ balls in $n$ cells we will (as $n\to\infty$) get the same expected number of counts. The variance is somewhat more problematic. Let $Y_{r,i}$ be an indicator random variable representing the event that cell $i$ receives exactly $r$ balls. Then $X_r = \sum_i Y_{r,i}$. We have that $\mathrm{Var}(X_r) = E(X_r^2)-E(X_r)^2$. Splitting $X_r^2$ into a sum of $Y_{r,i}^2=Y_{r,i}$ (as $Y_{r,i}$ is either $0$ or $1$) and a sum of $Y_{r,i}Y_{r,j}$ over $i\ne j$. For $i\ne j$ we have $$E(Y_{r,i}Y_{r,j}) = \binom{\lambda n}{2r}\binom{2r}{r}\left(\frac{1}{2}\right)^{2r}\left(\frac{2}{n}\right)^{2r}\left(1-\frac{2}{n}\right)^{\lambda n - 2r}$$ So we have $$ \text{Var}(X_r) = n E(X_r) + n(n-1)E(Y_{r,i}Y_{r,j}) - (nE(X_r))^2$$ Taking the limit as $n\to\infty$ we get $$\text{Var}(X_r)=n\left(\frac{\lambda^r}{r!}e^{-\lambda}-\frac{\left(r^2\lambda^{2r-1}-(2r-1)\lambda^{2r}+\lambda^{2r+1}\right)}{(r!)^2}e^{-2\lambda}\right)$$ (taking the limit is fiddly, see Robert Israel's post for the simplest case). This can be extended to compute the covariances (I'm working on it in my copious free time) and from there we can compute a better Chi-squared statistic. I am still not convinced that we cannot do better. I am also surprised that this is not a well known problem. _Update_ This is the final update (I have now gotten the algebra to cover the handwaving in the initial formulation.) The covariance of the counts is $$\Sigma = n(D - p'p - q'q)$$ where $p = p_0,p_1,\dots$, and $q = q_0,q_1,\dots$ where $$p_i = \frac{\lambda^i}{i!}e^{-\lambda},$$ and $$q_i = \frac{i-\lambda}{\sqrt{\lambda}}\frac{\lambda^i}{i!}e^{-\lambda},$$ and $D$ is the diagonal matrix made of the elements of $p$.   Note that the covariances are far from 0, making the assumption of independence underlying the use of a straight $\chi^2$ test somewhat suspect. This result is kind of cute as we can use the Sherman-Morrison-Woodbury method to find the inverse (of a truncation of the infinite matrix as the inverse diverges.) We can use the fact that if an $n$ dimensional random variable $X$ is distributed as $N(\mu,\Sigma)$ then $(x-\mu)\Sigma^{-1}(x-\mu)$ is a $\chi^2_n$ random variable. As the numbers of each count is essentially a binomial R.V. we can pretend that at least for the small counts they are normally distributed. This sugests using $(x-\mu)\Sigma^{-1}(x-\mu)$ as a test statistic, where $x$ is the number of small counts. Ignoring the larger counts has the nice side effect of making the problem full dimensional. If we do this in practice we see that if the number of bins we use is small enough, the result is nearly $\chi^2_n$ distributed.               p = sapply(0:7,function(i){2^i*exp(-2)/factorial(i)})     q = sapply(0:7,function(i){2^i*exp(-2)*(i-2)/(factorial(i)*sqrt(2))})     poiscov = diag(p) - (p %o% p + q %o% q)     poiscovinv = solve(10000 * poiscov)     gendat = function(){tabulate(tabulate(sample(1:10000,20000,replace=TRUE),                                   nbins=10000) + 1,nbins=8)}     dat = t(replicate(10000, gendat()))     mn = 10000 * p     ch = apply(dat,1,function(v){(v-mn) %*% poiscovinv %*% (v-mn)})      The mean and variance look about correct               > mean(ch)     [1] 8.092341     > var(ch)     [1] 17.10295      It passes the eyeball goodness of fit test.               > library(MASS)     > truehist(ch)     > x = seq(0,40,length=100)     > lines(x,dchisq(x,8))      ![Distribution of purported chi-squared statistics](http://i.stack.imgur.com/8h8TP.png) It turns out, unsurprisingly, that the distribution looks incrasingly unlike a $\chi^2_n$ distribution as $n$ increases, the tail becomes much heavier, and I am not certain what to do about that. I have tried adding all of the counts in the tail to the final bin, but that does not seem to help. I cannot say that this test is definitely more powerful, than the straight Chi squared test, as I can find points that are accepted by one and not the other, but somehow this seems to take into account more of the structure of the problem. Intuitively this method takes into account the shape of the contours of the probability distribution. It not surprisingly places a fair amount of weight on the condition on the sum of the bins, and actually weights the higher numbered bins less than the smaller bins (the opposite of the chi- squared test).


title: Whether to report untransformed data when performing ANOVA on transformed data?

text: I'm trying to understand how to analyse transformed data using ANOVA. If my data does not meet the assumption of normality, and I have transformed it using a log transformation to fix that, would I then run my analysis on the transformed scores and cite the transformed values? **Edit:** More specifically, when you have transformed data so it meets the assumption of normality and then run a 1 way ANOVA on the transformed scores, if you want to graph the data, should you use the transformed scores as well, or the original means? Since graphing illustrates relationship between the variables I would guess use the transformed values.


title: Hypothesis Testing and specifing the null hypothesis

text: I am having difficulty with the null hypothsis. Suppose I have a set of data from a polulation and you want to test if this sample has a lifetime of greater than 2000. Would i set the null hypothesis to be <=2000


title: Confused about the statistical test to use

text: I am a little bit confused about the statistical test I should use. The following is my case: I am measuring participants' personality traits (agreeableness, extraversion, conscientiousenss, openess, neuroticism) and I asked them how likely it would be that they performed five different actions on a 7-point scale (let's call them A, B, C, D, E). So the data I have are the 5 personality traits that I measure and the 5 response actions on a 7-point scale. What I want to know is whether the likelihood that they would perform the actions can be explained by the score of each of the personality traits. For example; someone who scores high on neuroticism is more likely to involve in action A than one of the other actions. I was thinking of a repeated measures anova as I am measuring 5 times the response action for each participant, but how do I handle the 5 personality traits as they are also with-in subjects


title: Interpret Variable Importance (varImp) for Factor Variables

text: When I run variable importance on a random forest (or any other model), the factor/categorical variable names have the factor name as the suffix. For example,               SALARY ~ STATE + CITY + AGE + …, the result of varImp(model) could look like,          > varImp(model)     rf variable importance          only 20 most important variables shown (out of 1050)                                Importance     AGE                       100.00     STATECA                    91.84     STATEAZ                    86.24     CITYSTANFORD               74.15     STATEVT                    71.27      In terms of relative importance, would it be right to interpret this as AGE is the most important predictor, followed by STATE followed by CITY? The importance values also do not say anything about the relationship between the predictor and the outcome--for example, does higher age equate to higher salary, does STATE CA mean higher salary, etc. Any suggestions on how such measures can be obtained for "black box" models such as random forest, gbm, etc. would be very helpful.


title: Regression with TBATS error?

text: I'm working on a time series model which includes multiple seasonal components (daily and weekly). I believe the best way to approach this would be BATS/TBATS model, however I have a concern if I can incorporate external factors in a proper way. I'm thinking about a regression with BATS/TBATS error, in analogy to a regression with ARIMA error:   1. Fit a linear regression to forecast the outcome based on available external predictors    2. Fit BATS/TBATS model to forecast residuals; Do you see any fundamental problems with this approach and is there a better way to build a model with multiple seasonal components and external predictors?


title: Statistical analysis for non-human experiments

text: I am running an experiment where no humans are involved. My mobile robot performs a tracking task (following a line in the ground) under 5 different conditions (different control variables). I evaluate the tracking error **at each instant** and then I average it over time, so that I have, for each condition, a MEAN ERROR $\pm$ S.D. How can I tell if one condition is better than another one? I can see if the MEAN ERROR is lower, but if both have high S.D.s they may have been much closer than I think. Can I treat each instant as a subject and apply a standard stat analysis (e.g. repeated-measure ANOVA across the 5 conditions)? Thank you.


title: Adding additional terms to a piecewise regression

text: I am exploring the probability of flight in a seabird (1=flight, 0=no flight) using binomial logistic regression. My predictors are distance to a disturbance (continuous), hour of the day (continuous), site (factor), season (factor), sea state (dichotomous), and group size (dichotomous). I have explored the use of piecewise regression in relation to the distance to a disturbance as this variable spans a large range (out to 74 km) and there is no way that this is affecting flight at the largest distance. When the model was fit with just reference to distance to a disturbance within the R program 'segmented' it points to a break in the data at 3.9 km. The slope up to this distance is negative and statistically significant while the slope estimate for distances further than 3.9 km is estimated to be 0 and non- significant. I would like to now sequentially add in additional terms to the model to see if there is any reduction in the deviance when the additional terms are added. Can a term be added just to the section before or after the break? I cannot seem to find any information in the literature regarding this My questions is can I do this? Or do I need to split the data into two chunks, before and after the breakpoint and explore additional terms this way. Also the motivation to do this analysis is more to find and identify the breakpoint. Instead of adding in terms after I assess the breakpoint should I explore the breakpoint within a the model including all the terms? Would this find the break in the data in relation to the other terms or does the algorithm completely ignore the other terms in the model when searching for a break in the distance to disturbance variable. Thanks,


title: Modelling the zero-inflation parameter in a ZIP regression model

text: I have data from two agricultural experiments, one on 101 sites, the other on 97 sites. In both experiments, the response variable, $Y$, is a count, with an excess of zeros and there is a binary covariate, $I$, which indicates whether the site received a particular treatment. In the first experiment these data were successfully modelled with a zero inflated Poisson (ZIP) model, providing estmates for the probability of zero, $\phi$, such that $\phi=0$ would correspond to an ordinary Poisson model, and for the treatment effect. However, in the second experiment there is also a continuous covariate, $I$, which is associated with $\phi$. How can I model these data so that I can estimate the association between $\phi$ and $X$, and also treatment effect ?


title: Leave one out cross validation for rare events logistic regression in SPSS

text: I am writing thesis about predicting which corporate takeover targets attract a competing bidder. I first estimated a model using a logistic regression. The sample consists of 1350 single-bidder targets and 51 multiple-bidder targets. I want to test the predictive ability with a holdout sample and use this prediction to construct a portfolio that generates more returns than the market (CAR); however this holdout sample has 250 single-bidder targets and only 13 multiple-bidder targets.   1. Does it matter for the holdout sample that the number of multiple-bidder target is so small?   2. Would a "leave-one-out-cross-validation" (LOOCV) be superior to the holdout sample to test the predictive ability? i.e. not model accuracy   3. How I can do a "leave one out cross validation" (LOOCV) using spss?


title: Cox proportional hazard lift gain charts

text: I have built a Cox PH model. I obtained the predicted survival probabilities. My questions are:   1. How to get the predicted survival time, or time to event, using survival probabilities?   2. I need validation using lift charts, as in how many events is the model capturing in the first three deciles.


title: find the point at which the curve significantly shoots up

text: so this is getting a little complex for me and hope someone can help me out. I do not have a mathematical background. I have a time series of daily rainfall for 50 years for a particular location. That location is marked by a very defined rainy season (normally starting around first week of June every year). Therefore you would expect a sudden surge of rainfall around June which lasts for approx 2-3 months and then subsides down. I was interested to know what sort of statistical or mathematical techniques can be used to mark the dates for each year when the monsoon begins. Basically I want to compute the rough onset dates for each year. Sorry if this question is a little vague. Thanks


title: Duration weighted Optimal Matching Algorithm using R

text: Besides the ado provided by Brendan Halpin for Stata, is there a way to pursue an Optimal Matching using the R-package `TraMineR` which weights OM’s elementary operations inversely with episode length [1]? [1]: See Halpin, B. (2010) 'Optimal Matching Analysis and Life-Course Data: The Importance of Duration', Sociological Methods & Research 38(3): 365–388.


title: Is it possible to use Hellinger distance for environmental variables?

text: Here is the problem, Euclidean distance is not recommended for datasets with many zeroes (like matrices of species/site), as there is the risk of the abundance paradox (Orloci, 1978). Whereas to calculate environmental distance (i.e., using Temperature and Precipitation variables) the Euclidean distance is widely used. The problem is these are not easily comparable. Is it correct to use Hellinger distance on environmental variables (normal distribution)?


title: How can there be a standard deviation of true scores?

text: Dudek (1979) describes the following formula as "the standard deviation of true scores when the observed score is held constant" (p. 337): $SE_{est} = SD \sqrt{(reliability × (1 - reliability))}$ As I understand it, my true score is a hypothesized 'real' score for me on some test, for example an IQ test. Due to measurement error it won't necessarily be my actual score on any given test, but if I could repeat the IQ test infinitely many times with no practice effects then my true score would be the ~~mode~~ mean. If I just have a single underlying true score, how can we talk about there being a standard deviation of true scores? Dudek FJ. 1979. The continuing misinterpretation of the standard error of measurement. Psychol Bull 86:335–337.


title: Meaning of "a test task is nonparametric or parametric"?

text: How is a test task being nonparametric or parametric defined? My understanding is that the test task is parametric, if and only if it assumes a parametric model on the distribution of the sample, regardless of whether its null is true or not. For example, the test task that t tests solve assumes the sample is normally distributed, regardless of whether their nulls are true. But what if the distribution of the sample isn't assumed to be parametric, but the null specifies the distribution of the sample being parametric? For example,   * when the null specifies that the distribution of the sample is normal, i.e. testing normality of the data. Is the test task parametric or nonparametric?   * when the null specifies that the sample has a specific distribution, i.e. goodness-of-fit test, is the testing task parametric or nonparametric? Such as the test tasks that the chi-square test solves, and the test task that the Kolmogorov-Smirnov one-sample test solves? Thanks and regards!


title: Reconstructing joint distribution from marginals

text: I think this is a rather open question. Suppose I have bi-dimensional data $(x_i, y_i)$. I have some reasonable model for the marginals, say distributions $F_X$ and $F_Y$ (parametric). How to reasonably construct a bi-dimensional model for the joint distribution of $(X,Y)$ that respects say, the correlation between $X$ and $Y$ or some other higher moment? Of course I'm not considering normal variates. Thanks in advance!


title: Insignificant VAR coefficients

text: I am not quite familiar with vector autoregression (VAR). I am thinking of using VAR/IRF (impulse response functions) to illustrate the relations between some time series variables. However, most of the coefficients are not significant. One note I found says it is rare to report the coefficients. I am kind of confused here. To my purpose, is it a big problem if the coefficients are not significant? Thanks.


title: Another p-value fallacy

text: Wikipedia examples say that a coin is unfair if it generates a sequence 1111111. A high alternation rate, such as 1010101010, would be similarly unfair. What is a fallacy to think that a coin is unfair on the grounds that it is equally improbable to see any sequence? I mean that it is normally "resolved" by stating the fact that P(11111111) = P(00000000) = P(01010101) = P(10101010) = P (010101000) = P(any other sequence). This explains that the coin is fair. But, I interpret it as there cannot be any fair coin just because of identity of all probabilities and because P(all ones) = P(any sequence) etc, all being highly improbably. Now, we have that fair coin cannot be fair. Where is the fallacy and how to apply the p-test properly? **edit** The hypothesis is that the coin is fair and statistic is the probability of occurring the sequence. I can compute the probability of occurred sequence from the hypothesis. For any sufficiently long sequence, the theoretical probability is too low and, therefore, fairness must be rejected. Where is the fallacy? **edit2** Why nobody can simply say that the pitfall is pointed out in the first Wikipedia example: **p-criteria does not take the sample size into account**? I can even trivialize the problem. Forget the series. Let's evaluate the probability of picking a single item38 under assumption of uniform 0-100 distribution. Obviously, it is 1%, which is sufficiently low to be picked by chance. But, statistics shows that item appears in 100% of the cases (1 time per 1 experiment). This obviously cannot be by chance, according to p-level test, yet, sample size is also insufficient. So, p-test must be complemented by sample size analysis. It is a fallacy to forget this. Right? A related question: which distribution will have the probability of picking item38 if I draw multiple samples? How do I take the integral of "extreme cases"?


title: Inverse covariance matrix, off-diagonal entries

text: Let $\Sigma$ be a covariance matrix. According to the material in this link, > If the elements of $\Sigma$ are all positive, most of the off-diagonal > elements in $\Sigma^{-1}$ will be negative this is actually written about the correlation matrix, but the principle should be the same. What does "most" here mean? Is there a common condition that would make _all_ the off-diagonal elements negative?


title: Probability that a sample mean is between two values using Central Limit Theorem

text: Mean is $2.707$, standard deviation is $.049$, sample of $35$ is drawn from the population.   What is the probability that the mean price for the sample was between $2.683$ and $2.716$? It was suggested that the Central Limit theorem be used.


title: Steps to follow for correspondence analysis when each brand is not shown to every respondent

text: I want to understand the steps followed for **correspondence analysis when each brand is not shown to every respondent**. Till now I used to assign a number (proportion) to each brand for each attribute and this **proportion is calculated as follows =** **(No. of time respondents have selected particular brand for particular attribute / Total no. of respondents to whom particular brand is shown)**. I used this technique to give weight to a brand according to no. of times it is shown to respondents. I have doubt whether this methodology is correct as correspondence can be applied on counts only not on proportion and if this method is wrong then what should I do to deal with this issue.


title: How do I find if a dependent variable is affected by another dependent variable?

text: I have one experimental group with language disorders (independent variable) and I have administered them 4 tests (dependent variables). The "language" test which comprises of the sum and two subparts (vocabulary and grammar), the "IQ" test, the "questions" test, and the "word" test. I want to see:   1. If the scores of "IQ" test _affect_ the scores of "language" test.   2. If the "questions" test _affects_ the "language" test   3. If the "word" test _affects_ the "language" test Is this a simple Pearson's r coefficient analysis? Or is it multiple regression analysis? Which method do I have to use in SPSS? Should I run an RM ANOVA? Thank you!


title: Interpretation of posterior variance in gaussian regression

text: I'm trying to understand how the posterior variance of a Gaussian regression depends on the datapoints. Suppose $y=X\beta + e$, with Gaussian priors over $\beta$ and $e$ and known variance of $e$. Then the Bayesian posteriors over $\beta$ will have variance $\sigma^2 (X'X+\Sigma_0)^{-1}$. I want to know if there's a nice interpretation of how a shift in the vectors in $X$ affect elements of $Var(\hat{\beta})$: in particular, suppose we have just two observations, is there a geometric interpretation of how a shift in the relative position of these vectors changes the informativeness, i.e. gives us more/less information about each element of $Var(\beta_i)$? I can see it can be written as ratios of determinants, but doesn't help me with intuition. Put another way, what are the comparative statics between some $x_{i,j}$ and $Var(\beta_k)$? I've tried this myself, and searched a lot, but can't get a nice answer (except for the 2x2 case).


title: How to create an "alignment" variable from mutual rating variables

text: I want to create an "alignment" variable out of two other variables. Meaning, that if two raters (say manager and employee) rate each other similarly, alignment is high, but if they rate each other differently, alignment is low. I tried looking into IRR measures such as ICC and Cohen's Kappa, but they seem to generate only an overall alignment score. I'm looking to generate one alignment value for each pair of raters rating each other - effectively creating a new alignment variable. How can I accomplish this? Technically I'm using SPSS, but the most important is for me to understand what generic technique to use.


title: Estimating overlap from subsamples (a mark-recapture problem?)

text: I have a population at two time points, $N_t$ and $N_{t+1}$. Some members persist from one time point to the next, but many die out, and many are added. **I want to know the fraction of $N_t$ that persists to $N_{t+1}$ given two subsamples of these populations.** In other words, I have a sample $P_t$ (with $P_t <<N_t$) and a sample $P_{t+1}$ (with $P_{t+1} <<N_{t+1}$), and some members are shared. I want to know, for the whole population ($N_t$ + $N_{t+1}$), the fraction of all individuals that are found in both years, and the fraction of individuals in one year persisting to the next. I don't know $N_t$ or $N_{t+1}$. I'd like to calculate confidence intervals of estimates. My stats in this area are quite rusty, so I'd love pointers and suggestions of things to read if explicit solutions are not straightforward. _Edit:_ It looks like I need to be using a Jolly-Seber type model, which assumes open populations. I'm digging into the (vast) literature here. An additional caveat is that I have these values $P_t$ and $P_{t+1}$ for many completely independent populations--I'm uncertain how to handle that replication.


title: Which is the right method of analysis for data over one time-period?

text: I am in the process of designing a survey that will collect data from users of services provided by my organization. We will collect data from all those users (other smaller organizations) working with us; therefore, we will have data about the whole population. We will design the survey in such a way that we get only numeric observations: continuous, discrete and categorical variables. At first I thought I would use panel data analysis, but the problem is that I will -at first- have data over just one time-period. The question then is, can I still use panel data analysis? Which other method would it fit best this kind of dataset?


title: R-squared for user-defined prediction algorithms

text: I've been working on a machine learning project for a while and I've come up with an algorithm that does what I want it to do (predict some values). I wondered if it is possible to calculate R-squared for my results. My results are like this (test):               Predictions  Observations:     4.25         4.30     4.15         4.25     3.70         3.75     3.25         3.15     ...      How can I calculate R-squared for this?


title: How to properly color clusters for visualization

text: I've some data that I can express as an image. Within this data I found some clusters (shown in the image below). I can just assign different colors to different clusters and visualize the image but if I do so randomly I lose the spatial relation that can be observed within the image. ![Projection of my clusters into a 2D to visualize distances between them](http://i.stack.imgur.com/8FVee.jpg) To solve that in order to let close clusters have close colors I thought that a good solution might be to map this space into a color map (or the other way around) Just to say I guess that I should use the color map such that in the center is black rather than white just for displaying purposes. (white marker in white canvas does not show up much ;) ) ![Color Space](http://i.stack.imgur.com/8gUnz.png) A possible solution is to directly map the PCA of my centroids to 3D coordinate system and use this one to map HSV coloration. ![enter image description here](http://i.stack.imgur.com/yJEu6.jpg) Centroids nearby have similar coloration but differences in saturation and brightness does that nearby differ and are close to other centroids far away in the first two coordinates but close in the third one. A solution is to use the same coordinate for saturation and brightness leading to this coloration (or even just use hue) ![enter image description here](http://i.stack.imgur.com/DJJCA.jpg) However clusters on the top can be easily differentiated meanwhile the clusters on the bottom they all look similar (dark). On the other hand using only hue it would lead to a reduced subset of colors. Saturation can be mapped as distance to the center but still, the color space is not properly used to have the best coloration. The final goal is to re-project back into an image the coloration of the clusters to obtain a colored visualization of its texture, as follows: ![enter image description here](http://i.stack.imgur.com/7uZoE.jpg) Thanks.


title: Distribution of number of values less than cutoff within symmetric Dirichlet

text: Assume have a symmetric Dirichlet distribution with $a_1= \dots =a_k = a$ $ (X_1, \ldots, X_K)\sim\operatorname{Dir}(a) $ I am trying to determine the distribution of the number of values less than some cut-off. That is, for $\Pr(N=n|c,a,k)$ $n$ values were less than $c$ while $k-n$ were greater than $c$ Example for two values: $\Pr(N=2|c,a,3) = \Pr(X_1<c, X_2<c, X_3>c) \+ \Pr(X_1<c, X_2>c, X_3<c) \+ \Pr(X_1>c, X_2<c, X_3<c) $ _(which being symmetric Dirichlet should simplify, but wanted to show individually all combinations as an example)_ Did 1E7 simulations for $(c=.1, a=.5, k=3)$ and got:                      n     count            0     2,492,498            1     5,527,905            2     1,979,597            3           0      So out 1E7 simulation, around 25% of the cases had zero values that were less than .10 Is there a general formula for the distribution of $N$ as a function $c$? (given $a$ and $k$) Or, an approximation for small c? ($c\ll\frac{1}{k}$)


title: Factoring two matrices into shared and unique variance

text: I have two square symmetric matrices, A and B, each of which contains the pairwise cosine distances between two different sets of vectors. The sets of vectors are different for A and B, but they are correlated. The original vectors from which these distance matrices were constructed are no longer available. These matrices are not positive definite. I would like to factor A and B into AB, A1 and B1. AB would contain the shared variance between A and B, whereas A1 and B1 would contain the unique and error variance of A and B respectively. If I project the singular values of A and B into the space of orthogonal bases of A and B and then create a pairwise distance matrix for all of said bases, only the first few bases are correlated. In other words, only the first few principal components of A and B appear to have significant shared variance, if that helps.


title: When fitting a model in Amos with ordinal data do I use Bollen stine adjustment or Baysian techniques?

text: I am trying to run a model validation/fit in AMOS using 18 items and three factors. The data is ordinal (1-4 scale) for each item. My data is non-normal both on a univariate and multivariate level. I am unsure whether I should estimate using Bayesian techniques or whether I should use a ML method and adjust using the Bollen Stine bootstrap method. When I try and do the Bayesian approach it doesn't run well (I have to make the tuning paramater so small) and I also get some negative values (which when I try and specify '0' as the least value it won't run at all). When I do a ML method and the bootstrapping the model it works better. In this case I have a ns chi square (using the adjusted measure)and mostly good measures of fit. I guess I would prefer it if I could use the ML method with the Bollen Stine, but want to make sure I am doing what is considered appropriate in the literature for ordinal data .


title: When to use the Wilcoxon rank-sum test instead of the unpaired t-test?

text: This is a followup question to what Frank Harrell wrote here: > In my experience the required sample size for the t distribution to be > accurate is often larger than the sample size at hand. The Wilcoxon signed- > rank test is extremely efficient as you said, and it is robust, so I almost > always prefer it over the t test If I understand it correctly - when comparing the location of two unmatched samples, we would prefer to use the Wilcoxon rank-sum test over the unpaired t-test, if our sample sizes are small. **Is there a theoretical situation where we would prefer the Wilcoxon rank-sum test over the unpaired t-test, even that the sample sizes of our two groups are relatively large?** My motivation for this question stems from the observation that for a single sample t-test, using it for a not-so-small sample of a skewed distribution will yield a wrong type I error:               n1 <- 100     mean1 <- 50     R <- 100000     P_y1 <- numeric(R)     for(i in seq_len(R))     {         y1 <- rexp(n1, 1/mean1)         P_y1[i] <- t.test(y1 , mu = mean1)$p.value     }     sum(P_y1<.05) / R # for n1=n2=100 -> 0.0572  # "wrong" type I error


title: Are survivor functions meaningful with proportional hazards models?

text: Does the survivor function estimated after running a Proportional Hazards model give valid predicted probabilities of an event happening after a number of time periods? The only source I've found on this question says that Proportional Hazards estimates can only be used to discern the factor that two hazards rates differ by when comparing two different levels of an independent variable. They argue that the survivor function is invalid because proportional hazards models (like Cox) do not estimate an underlying hazard function, and so it is impossible to say what the actual predicted hazard is at any point. According to this powerpoint, the answer is no. I haven't been able to find any other sources that agree or disagree with their claim that proportional hazards models are only useful for comparisons among hazard rates.


title: Regression model with non-constant variance

text: I have a dataset to analysis and the following information is known: $y_i \sim N(\mu_i, \theta(\mu_i)^2)$ The link function is => $ln(\mu_i) = (\beta)^T X$ $y_i$s are cound data. The model parameter is beta and theta. I need to find an estimation method and fit a model. I have looked at the over-dispersion model and negative binomial models. But they don't seem quite right... Can anybody point me to the right direction ..? :) Update: After some more research - I am looking for if there is any R-package will model data with the above properties?


title: relation between confidence interval and likelihood function

text: I once meet the following question,which is also listed by book written by Cosma Rohilla Shalizi               what is the relation between between confidence intervals and the likelihood function.      I am not very clear how to connect these two concepts?


title: Modeling techniques for dichotomous data

text: I have dichotomous data where some of my independent variables are categorical, some are continuous and some are binary (0/1). My dependent is a binary response (Fail/NoFail, 0/1). The data is some readings collected every day over a period of time. The goal is to use this data and see if we can figure out cause of failure, the end response. Example data format Date, Type, Mileage, S1, S2, S3, ... , Response 03/02/2013,A,32000,1,0,1,..., 1 03/03/2013,B,32400,0,0,0,...,0 03/04/2013,C,45000,0,1,1,...,1 Can we do time series modeling? Any other modeling techniques that can be used?Any suggestions on what type of other exploratory analysis can be used to figure out patterns in data?


title: Dimension of multivariate skew normal

text: Please, in this multivariate skew-normal loglikelihhod equation, \begin{equation} l=- \frac{1}{2}nlog|\Omega|- \frac{1}{2}n \,tr(\Omega^{-1}V)+\sum_{i}\zeta_{0} { [\alpha^{'} \omega^{-1}(y_{i}- \epsilon_{i})]} \end{equation} What is the dimension of \begin{equation}\sum_{i}\zeta_{0} { [\alpha^{'} \omega^{-1}(y_{i}-\epsilon_{i})]}\end{equation} Is it 1*1? Thanks in advance.


title: How can I compare regression coefficients in the same multiple regression model

text: I think this question has been answered in bits and pieces here and there, but I am still a bit unsure about what the best approach for this is: how to compare two coefficients from a multiple linear regression to see if the effect strengths are significantly different. For example, I am interested in relating attitude (IV1) and behavioral control (IV2) to the medication adherence (DV). I found that standardized betas were 0.30 and 0.19 for attitude and behavioral control, respectively. Is it reasonable to say the attitude is the strongest predictor of medication adherence? If so, how can I test whether its effect is significantly different from those of the other predictors? I am using SPSS v19.


title: Confusion related to calculation of variance

text: This might be a very simple question. But I was going through this lecture where it is mentioned that If I have n-1 neighbors and there is a probability p of having an edge with a neighbor. Then the average degree of any node is given by (n-1)*p. This is fine I get it But how come the variance of degree is (n-1) _p_ (1-p). I didn't get how this variance was derived.


title: Simulation power analysis

text: How do you perform a power analysis for linear mixed model using R? I measure area of five groups of plants across time. Each group of plants grow under two treatments. I have 12 replicates for each plant. I have fitted a lme with plant group and treatment as fix effects and the replicates as random effect. How can I go from that to the actual calculation? I have found tools for ANOVAs and MANOVAs but not for lme. Thanks.


title: Predict sales levels with decision trees

text: I need to build a model using climate variables (temperature, rainfall) to predict monthly sales (horizon of 6 months) for certain product. The data has strong seasonality and a standard regression model would works fine, the problem is that the historic data will not be updated, meaning that the observed data points will not be incorporated into the model. Whats a good way to solve this? What if i split the sales data into levels (say 'WEAK', 'NORMAL', 'HIGH', VERY HIGH') and then use a regression tree? Is there any 'danger' in doing this? For a standard regression model, how i deal with the seasonality if the new points will not be incorporated? I'm using R, thanks!


title: Why proc mixed give different DF than proc glimmix?

text: I am trying to move data analysis from proc mixed to glimmix for the easy of the ilink transformation. However I am finding a big difference on the denominator DF when using either of them. As background. My data comes from a group of 95 calves that were fed 4 diets (about 24 calves per diet), my measure is neutrophil number in blood which was repeated 4 times (7, 14, 28, 42 d of age). Thus I am using sp (pow) as cov structure. When using proc mixed with next statements: I got about 85 DF for the diff contrasts               CLASS calf diet parity sex age;     MODEL  RBCx=  diet sex age diet*sex diet*age  sex*age diet*sex*age / DDFM=KR;;     random calf (diet*sex);     repeated age /type=sp(pow)(age)  subject=calf (diet*sex) ;      However when using proc glimmix, for same contrast I got 336 DF when using next:               proc glimmix data = CBC_2011 IC= PQ;     CLASS calf diet parity sex age;     MODEL  RBCx=  diet sex age diet*sex diet*age  sex*age diet*sex*age / dist=Gaussian DDFM=KR;;     random age /type=sp(pow)(age)  subject=calf (diet*sex) residual;      Thanks in advance. Miriam


title: Chi square statistic and contingency coefficients

text: I need some advice on explaining $\chi^2$ as a contingency measure from a pedagogical perspective. I usually introduce the individual terms in the $\chi^2$ sum as scaled deviations from independence; my argument claims that $d_{ij} = (n_{ij} - E_{ij})^2$ is an _absolute_ measure but we want relative measures that are independent of the units so we should divide by $E_{ij}$. However, the natural question raised is why we do not divide by $E_{ij}^2$ which would truly normalize the counts in the numerator. In addition, it is difficult to find good, intuitive explanations for the final form of the contingency coefficient which is $\sqrt{\chi^2/(\chi^2 + N)}$


title: Hypothesis testing and total variation distance vs. Kullback-Leibler divergence

text: In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples are independently and identically distributed from one of these two distributions (though the distributions may be related: for example, $Q$ may be a mixture of $P$ and some other distribution.) The null hypothesis is that samples come from $P$, alternate hypothesis is that samples come from $Q$. I am trying to characterize the Type I and Type II errors in testing the sample, knowing the distributions $P$ and $Q$. Particularly, I am interested in bounding one error given the other, in addition to the knowledge of $P$ and $Q$. I have asked a question on math.SE regarding the relationship of Total Variation distance between $P$ and $Q$ to hypothesis testing, and received an answer that I accepted. That answer makes sense, but I still have not been able to wrap my mind around the deeper meaning behind the relationship of Total Variation distance and hypothesis testing as it relates to my problem. Thus, I decided to turn to this forum. My first question is: is total variation bound on the sum of the probabilities of Type I and Type II errors independent of the hypothesis testing method that one employs? In essence, as long as there is a non-zero probability that the sample could have been generated by either one of the distributions, the probability of at least one of the errors must be non-zero. Basically, you can not escape the possibility that your hypothesis tester will make a mistake, no matter how much signal processing you do. And Total Variation bounds that exact possibility. Is my understanding correct? There is also another relationship between Type I and II errors and the underlying probability distributions $P$ and $Q$: the KL divergence. Thus, my second question is: is KL-divergence bound only applicable to one specific hypothesis testing method (it seems to come up around the log-likelihood ratio method a lot) or can one apply it generally across all hypothesis testing methods? If it's applicable across all hypothesis testing methods, than why does it seem to be so very different from the Total Variation bound? Does it behave differently? And my underlying question is: is there a prescribed set of circumstances when I should use either bound, or is it purely a matter of convenience? When should the result derived using one bound hold using the other? I apologize if these questions are trivial. I am a computer scientist (so this seems like a fancy pattern matching problem to me :) .) I know information theory reasonably well, and have graduate background in probability theory as well. However, I am just starting to learn all of this hypothesis testing stuff. If needed, I will do my best to clarify my questions.


title: Picking random elements from a list – still random if they are reinserted at the end?

text: Say I have a list with $n$ elements (say number $1$'s) and I want to do 1000 random changes to them, such as a $+1$. If I (1) picked a random element, (2) changed it, and then (3) did these two steps (1)+(2) another 999 times, this would probably be completely random, right? Now, what if, whenever I change an element, I take it from the list and reinsert the changed element at the end of the list (i.e. this is the new step (2)) – would the changes also be completely random then? I have the feeling that it might decrease the probability for already changed elements to get changed again, but can't quite put my finger on the cause. Also, consider this variant: instead of inserting them in the end, I insert them in a random new location after the change. How would that influence the probability for picking any element? (PS. If someone knows good tags for this, please add them and delete this paragraph...)


title: Doubts about the standardization process that I should perform with a set of variables

text: I have a set of categorical variables that I need to encode between $[0,1]$ or $[-1,1]$ to solve a problem in which I have to compare their values with metrics like Euclidean Distance or the Dot product. These metrics require my variable to be in the mentioned ranges. As a first approach I represented each category $X_i$ of a variable as an integer $(1,2,\ldots)$, and then used the following formula to normalize them between $[-1,1]$: $$X_i(\mbox{normalized}) = \frac{X_i - X_{min}}{X_{max}-X_{min}}$$ But this formula supposes that variables follow a uniform distribution which in my case is not true, as there is a great variance in some variables and a little variance in others. This means that in some variables a few categories contain the most part of the data, while the other categories are empty, or with a few outliers. For this reason I think my set follows a normal distribution which I have to standardize with mean 0 and variance 1 and after dividing each variable by its modulus, get a range of $[0,1]$ for them. But the problem is how should I code those variables, as they are categorical? The idea of represent them as integers (for example: red=1, yellow=2, green=3) would lead to strange results without sense for the mean (yellow in the example) and therefore also for the variance. Which is the best way to code categorical variables?