title: How to define a posterior probability of y given x when the model is not probabilistic?

text: Suppose we have a very simple online k-means where each new data-point is assigned to its nearest center (the mean is updated incrementally). Each center (cluster) is labelled with the most common label of data-points assigned to that cluster. In this special configuration: is it possible to compute a sort of "posterior probability"? I.e., can the posterior probability of a class label $y$ given a data-point $x$ ($P(y|x)$) just be $1/\text{distance}(x, m_y)$, where $m_y$ is a center labelled with $y$ which is nearest to $x$?


title: Independence over time

text: I have some categorical data from a clinic taken over a year long time period. The plan is to perform a Pearson's Chi Squared test (of independence) on this data. I am extracting data from a clinic's database and my aim is to see if male patients are more likely to cancel their appointment than female patients. My query is this: Chi squared requires independent samples but there is a chance that patients may have had multiple appointments within that time period. Whilst I could identify these multi-attenders, how would I then assign them to any one category (presuming some would appear in multiple different categories ). For instance... John Doe may have:   * Attended a cardiology appointment in January   * Attended a cardiology appointment in February   * Cancelled a cardiology appointment in March To push this to it's limit:   * John Doe then goes through a (very fast) gender reassignment procedure and becomes Jane Doe and then cancels their cardiology appointment in April. So from one person we have:   * Male/Cancelled: 1   * Male/Attended: 2   * Female/Cancelled: 1 Is Chi Squared the correct test to use in this instance? If so, is there some data processing I should be performing on my data to account for patients who appear multiple times in different categories?


title: Why does the y-intercept of a linear model disappear when I standardize variables?

text: I was fiddling with my independent variables in a linear model and I note how the y-intercept becomes 0 when all the variables become standardized. Intuitively I understand that I'm shifting the line by changing with the mean of the normal distributed variables, but I was wondering if there is some more rigorous/theoretical way of explaining/deriving this? Assuming we start with a model of the form y = Xb + e where y is a vector of variables, X is the design matrix and b is the coefficient matrix, and we subtract from every random variable its respective sample mean and divide by the respective sample standard deviation. How do we see that the design matrix gets all zeroes in its first column?


title: Why don't we use significant digits?

text: Any idea why we don't use significant digits in statistics? Something along the line of we are using estimates so rules about precision don't apply ;) ?


title: How do I extract difficulty and discrimination from a non-parametric IRT model

text: I've been using the `KernSmoothIRT` module in R to do non-parametric analysis of some items. I'm trying to find a way to quantify the difficulty and discrimination of a particular set of items. What is the best way to do this with this module? Geometrically, I think I'm looking for the value of OCC at theta=0 and the slope at P(theta)=0.5, but I'm not sure how to figure this out from what's generated by the library.


title: Stability of beta coefficients

text: If I have 24 temporal covariance matrices (say monthly covariance matrices computed from daily returns of all the SP500 stocks), and I compute 24 betas for variables x and y for every month for past 2 years. Then,   * What is then the best estimator of beta (e.g. equally or exponentially weighted average of these betas?    * How can I analyse the stability/variability of the individual beta as well as the estimator of the beta?    * By looking at these monthly betas can I make any conclusions of the relationship between x & y?  I would very much appreciate, if you have any standard references that clarify my concerns.


title: volume augmented garch(1,1) model in matlab

text: Actually I want to add volume traded of a stock in my Garch(1,1) model to forecast the volatility.In Matlab I can specify the model as garch(1,1) and then use estimate and forecast commands.But I am not sure how to specify my volume augmented garch(1,1) model in Matlab. Any help on this would be very appreciated. Thanks, Jayadeep Shitole


title: R implementation of some new Principal Component Pursuit methods

text: I'm looking for R packages implementing some new PCA methods. The first one is the Stable Principal Component Pursuit method of Zhou _et al._ (2010). The second one is the PCA via Outlier Pursuit algorithm of Xu _et al._ (2012). Both are highly cited papers in statistics, yet I could not find either of them on the robust or multivariate R task view and was wondering whether somebody more knowledgeable could point me in the right direction.   1. Zihan Zhou, Xiaodong Li, John Wright, Emmanuel Candes and Yi Ma (2010). Stable Principal Component Pursuit.   2. H. Xu, C. Caramanis, S. Sanghavi (2012). "Robust PCA via Outlier Pursuit", IEEE Transactions on Information Theory, 58(5), pp 3047-3064, 2012.


title: Unsupervised Decision Trees

text: As far as I know, decision trees are always used with labeled data to learn rules which differentiate between different labels. But is there anything called unsupervised decision tree? Say I have an unlabeled dataset and I want to find rules of the form $x_1 > 2\; AND \; x_2 > 4$ which _best_ cluster the data. I found this paper and its related patent. Do they talk about what I need? Are there any other works talking about similar things?


title: Taking the average p value from a set of simulated p values

text: I have 149 locations that are lined up from east to west. I have the geographical distances between each location and the adjacent location going west. I want to test whether the locations are randomly distributed or not. Therefore, I take the furthest west location and the furthest east location, and generate 145 random locations within this space and find the distance between each consecutive location (again from east to west). I then test the actual distribution of distances against the randomly generated distribution of distances using Kolmogorov Smirnov to get a p value. However, if I then decide to do 1000 simulations, does it make sense to just calculate the average (or median) p value of the 1000 KS tests and report this?


title: In R, what is the best graphics driver for using the graphs in Microsoft Word?

text: I use R to make neat graphs that I use in Microsoft Office documents. According to this page, the best quality is obtained with the PDF driver. Unfortunately, Word doesn't support importing PDF figures. What should I use?


title: Determining which variables have been dropped using cv.glmnet in R

text: I have a large amount of vegetation data that has been broken down into 13 habitat classes. I am trying to determine which vegetation tends to fall into or is absent from which habitat with any sort of significance. I have been put onto running a multinomial logistic regression, specifically using glmnet (as I have approximately 200 variables, and only about 260 observations). Running cv.glmnet using the code:               cv<-cv.glmnet(data,Class,family="multinomial",nfolds=50,standardize=FALSE)      I get a list of numbers that I am struggling to understand, however I found the code:               coef(cv, s=cv$lambda.1se)      Which returns the coefficients for each variable for each habitat class for the lambda that is 1 SE larger than the minimum Lambda value (which as far as I can tell the generally accepted lambda value).               (Intercept)                                              0.7914263664        Salix                                                    0.0000000000       Mash                                                     0.0000000000        Pin                                                      0.0000000000        Choke                                                    .               Betula                                                   0.0025260258        Ideae                                                    0.0000000000        Leather                                                  0.0000000000      What I'm wondering, using these coefficients, is it possible to state that those values with the largest magnitude (either closest to -1 and +1) are the most important in defining that class, which those close to 0 are unimportant, and those with periods were removed during the cv.glmnet. So in this case the plant "Betula" would be more influential than all others, and "Choke" was so uninfluential that it was removed? Also, no idea what intercept means, but I imagine I can find that one on my own.


title: Supersmoother algorithm - tutorial?

text: I came across a reference to a "Supersmoother" algorithm that was applied to prices in a timeseries. Does anyone know of a tutorial in Python or whether Pandas has such a capability?


title: Lsmeans: Is this what I did?

text: after doing the linear mixed model with lme4, I have used lsmeans for pairwise comparison with this command:               lsmeans(lmer52, pairwise~color, adjust="tukey")      I am not sure if it is "multiple pairwise comparisons, using Least Squares Means (LSMEANS) and Tukey's adjustment". Anyone have any idea if it is exactly the same thing?


title: Is it ok to spit non-normal variables in tertiles and put them into multivariate regression models?

text: I am now reviewing a paper in which the authors decided to predict a DV through linear regression using, beyond other variables, dummy variables obtained from a tertile split of continuous variables, which were not normally distributed. In other words, for example, they took a not normally distributed variable, split the variable into three tertiles, created three dummy variables for each tertile (i imagine they assigned 1 to subject falling into the selected tertile in each variable), and put all the dummy variables in the regression model. Their regression models get a really high R^2 value (.90). Is it correct to do so?


title: Does using a kernel function make the data linearly separable?

text: I'm reading about SVM and I learned that we use a kernel function so the data become linearly separable in the high dimensional (vector?) space. But then I also learned that they use the soft-margin idea. But my question is why to use a soft-margin if the data is going to be linearly separable anyway in the high space? Or does that mean that even after mapping with the kernel it doesn't necessarily mean that it will become linearly separable?


title: How do you plot interaction effects in SAS?

text: I want to plot interaction between two variables in SAS. One is continuous and the other is binary.


title: How to convert mean and standard deviation to a single meaningful and quantifiable value?

text: I look at mean and standard deviation of various map like data and each give meaningful information but sometimes they provide different information. For example, let's take states in a country. One state has higher mean than others but the standard deviation by itself looks OK since individual points from the whole state tend to be higher. In another case, the mean looks average but the state has both higher and lower values equally distributed so the standard deviation is higher. Now how can you quantify both mean and standard deviation to a single meaningful value? Let say, the map color looks hot both when mean of each state is deviating from the mean of the country as well as standard deviation of the state is high.


title: Bayesian mixture model for univariate continuous random variable

text: I'm quite new to the mixture models and I hope you'll help me to understand how they work. Suppose I have a univariate continuous random variable **_x_** which represents time of a visit, and suppose that **_x_** is multimodal, i.e. it is described by more than one probability distribution. I want to find how many subpopulations are within the overall population of **_x_**. I know that I can use finite mixture models, where I need to specify a number of components and probability distribution of each component, such as Gaussians. However, what I would like is to learn a number of components from the data and to not make assumptions regarding distribution of each component, but ideally infer it from the model. I know that Bayesian mixture model can do that, however I'm struggling to understand how the model works. In particular, how Bayesian nonparametric mixture model can be applied to continuous random variable? Is there any assumption about distribution of each component and how I should choose priors?


title: Is my data distribution normal? (Tried Shapiro and Kolmogornov-Smirnov tests)

text: I have a 1D data set with 83163 data points, and I would like to know whether the data follows a normal distribution. I tried using shapiro.test and ks.test in R: d is a vector containing the data > shapiro.test(sample(d, 5000))                   Shapiro-Wilk normality test          data:  sample(d, 5000)      W = 0.9694, p-value < 2.2e-16      (Repeated several times. _Note subsampling_.) > ks.test(d, dnorm, mean=mean(d), sd=sd(d))                   One-sample Kolmogorov-Smirnov test          data:  d      D = 1, p-value < 2.2e-16     alternative hypothesis: two-sided           Warning message:     In ks.test(d, dnorm, mean = mean(d), sd = sd(d)) :       cannot compute correct p-values with ties      Both tests indicate that the data distribution is _not_ normal. So I tried plotting the data (black), and it appears to be "taller" than a normal distribution with mean and sd estimated from the data (blue). ![http://dl.dropbox.com/u/38050492/data- distrib.png](http://i.stack.imgur.com/2AiBl.png) I wondered if the variance is over-estimated due to outliers, so I tried calculating Winsorized variance. I heuristically matched the peak to the data distribution, but I cannot get a good fit (red). Edit: qqplot also suggest non-normality. ![http://dl.dropbox.com/u/38050492/qqplot.png](http://i.stack.imgur.com/UW6ab.png) Is there a distribution that may better model the data? The reason why I wanted to check normality is because others have done two- sample z-tests, and modeled the data using Gaussian distributions. To make the long story short, the math work out much nicely if the data distribution is assumed to be normal. As the normality assumption goes beyond simple application of parametric tests, I don't know how robust the results are when the data is not normally distributed... There does appear to a considerable deviation from the normality in terms of the kurtosis of the distribution. And this deviation is consistent from dataset to dataset...


title: what does the correlation of Random forest regression tool in R represent

text: I've built a random forest model (regression model) using randomForest package in R, and I calculate the correlation between the predicted values and the actual ones in order to know how the trained model is going to perform, which is very high in my case, so I was wondering in such case how this correlation is build, I mean is it leave one out correlation or any type of cross validation correlation or just random and can't represent the real performance of the model when tested on unseen new cases???? the following is a snapshot of my script for calculating the correlation where x is the data (observations) and y is the numeric values I want the model to learn/predict (in the testing cases):               mytr_all = randomForest(x, y, ntree = 500,corr.bias=TRUE)     cor(mytr_all$y,mytr_all$predicted)


title: Parametric modelling of variance of count data

text: I am looking to model some data, but I am not sure what type of model I can use. I have count data, and I want a model that will give parametric estimates of both the mean and the variance of the data. That is, I have various predictive factors and I want to determine if any of them influence the variance (not just the group mean). I know that Poisson regression will not work because the variance is equal to the mean; this assumption is not valid in my case, so I know there is overdispersion. However, a negative binomial model only generates a single overdispersion parameter, not one that is a function of the predictors in the model. What model can do this? Additionally, a reference to a book or paper which discusses the model and/or an R package which implements the model would be appreciated.


title: High censoring rate in survival analysis

text: I have some data which I've analyzed using Kaplan Meier estimation. However, I have a gut feeling that this estimator is biased due to the high censoring rate in my data (nearly 50% censored at later times). What are some ways to address this in an analysis?


title: How do I calculate error propagation with different measures of error?

text: I am calculating the age of lake sediments at the base of a sediment core by dividing the total sediment mass of the core ($\mathrm{mg} \ \mathrm{cm}^{-2}$) by the sediment accumulation rate ($\mathrm{mg} \ \mathrm{cm}^{-2}\ \mathrm{y}^{-1}$). Both the sediment mass and the accumulation rate have variation associated with them. The sediment mass is a mean of 3 samples and the accumulation rate is reported as $\pm 10\%$. It is my understanding that I can calculate the error associated with age as: $\sqrt{\left(\frac{\delta x}{x}\right)^2 + \left(\frac{\delta y}{y}\right)^2}$ where $\delta x$ and $\delta y$ are the relative error of the measurements being divided (i.e., $x$ and $y$). The error associated with the sedimentation rate is $\pm \ 10%$ but the error associated with the sediment mass in a standard deviation. Are these forms of error compatible within the above error propagation formula? Thank you.


title: Assumptions and contraindications of conjoint analysis

text: I'm working with a team that is pressing hard to do a conjoint analysis on what is essentially a 2x2 factorial design. I get the feeling, though, that this is just their favourite hammer and not necessarily a good idea, so I'd like to push back in an informed way. What assumptions does a conjoint analysis make about the data? When is it not a good idea? What would it make of a factorial design?


title: Name of phenomenon on estimated CDF plots of censored data

text: My dataset contains two (rather strongly correlated) variables $t$ (runtime of algorithm) and $n$ (number of examined nodes, whatever). Both are strongly correlated by design, because the algorithm can manage roughly $c$ nodes per second. The algorithm was run on several problems, but it was terminated if a solution hasn't been found after some timeout $T$. So data is right-censored on the time variable. I plot the estimated cumulative density function (or the cumulated count) of variable $n$ for the cases where the algorithm did terminate with $t<T$. This shows how many problems could be solved by expanding at most $n$ nodes and is useful for comparing different configurations of the algorithm. But in the plot for $n$, there are those funny tails at the top going right sharply, as can be seen in the image below. Compare the ecdf for variable $t$, on which the censoring was done. ## Cumulated Count of $n$ ![ecdf of n](http://i.stack.imgur.com/BlohK.png) ## Cumulated Count of $t$ ![ecdf of t](http://i.stack.imgur.com/V8EMk.png) ## Simulation I understand why this happens, and can reproduce the effect in a simulation using the following _R_ code. It's caused by censoring on a strongly correlated variable under the addition of some noise.               qplot(       Filter(function(x) (x + rnorm(1,0,1)[1]) < 5,              runif(10000,0,10)),       stat="ecdf",geom="step")      ![synthetic data](http://i.stack.imgur.com/Zjscp.png) **How is this phenomenon called?** I need to state in a publication that these fans are artifacts of the experiment and do not reflect the real distribution.


title: Interpreting interaction terms in logit regression with categorical variables

text: I have data from a survey experiment in which respondents were randomly assigned to one of four groups:               > summary(df$Group)            Control     Treatment1     Treatment2     Treatment3                  59             63             62             66       While the three treatment groups do vary slightly in the stimulus applied, the main distinction that I care about is between the control and treatment groups. So I defined a dummy variable `Control`:               > summary(df$Control)          TRUE FALSE             59   191       In the survey, respondents were asked (among other things) to choose which of two things they preferred:               > summary(df$Prefer)           A   B  NA's          152  93   5       Then, after receiving some stimulus as determined by their treatment group (and none if they were in the control group), respondents were asked to choose between the same two things:               > summary(df$Choice)       A    B      149  101       I want to know if the being in one of the three treatment groups had an effect on the choice that respondents made in this last question. My hypothesis is that respondents who received a treatment are more likely to choose `A` than `B`. Given that I am working with categorical data, I have decided to use a logit regression (feel free to chime in if you think that's incorrect). Since respondents were randomly assigned, I am under the impression that I shouldn't necessarily need to control for other variables (e.g. demographics), so I have left those out for this question. My first model was simply the following:               > x0 <- glm(Product ~ Control + Prefer, data=df, family=binomial(link="logit"))     > summary(x0)          Call:     glm(formula = Choice ~ Control + Prefer, family = binomial(link = "logit"),          data = df)          Deviance Residuals:          Min       1Q   Median       3Q      Max       -1.8366  -0.5850  -0.5850   0.7663   1.9235            Coefficients:                         Estimate Std. Error z value Pr(>|z|)         (Intercept)           1.4819     0.3829   3.871 0.000109 ***     ControlFALSE         -0.4068     0.3760  -1.082 0.279224         PreferA              -2.7538     0.3269  -8.424  < 2e-16 ***     ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1          (Dispersion parameter for binomial family taken to be 1)              Null deviance: 328.95  on 244  degrees of freedom     Residual deviance: 239.69  on 242  degrees of freedom       (5 observations deleted due to missingness)     AIC: 245.69          Number of Fisher Scoring iterations: 4      I am under the impression that the intercept being statistically significant is not something that holds interpretable meaning. I thought perhaps that I should include an interaction term as follows:               > x1 <- glm(Choice ~ Control + Prefer + Control:Prefer, data=df, family=binomial(link="logit"))     > summary(x1)          Call:     glm(formula = Product ~ Control + Prefer + Control:Prefer, family = binomial(link = "logit"),          data = df)          Deviance Residuals:          Min       1Q   Median       3Q      Max       -2.5211  -0.6424  -0.5003   0.8519   2.0688            Coefficients:                                      Estimate Std. Error z value Pr(>|z|)         (Intercept)                         3.135      1.021   3.070  0.00214 **      ControlFALSE                       -2.309      1.054  -2.190  0.02853 *       PreferA                            -5.150      1.152  -4.472 7.75e-06 ***     ControlFALSE:PreferA                2.850      1.204   2.367  0.01795 *       ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1          (Dispersion parameter for binomial family taken to be 1)              Null deviance: 328.95  on 244  degrees of freedom     Residual deviance: 231.27  on 241  degrees of freedom       (5 observations deleted due to missingness)     AIC: 239.27          Number of Fisher Scoring iterations: 5      Now the respondents status as in a treatment group has the expected effect. Was this a valid set of steps? How can I interpret the interaction term `ControlFALSE:PreferA`? Are the other coefficients still the log odds?


title: When is it valid to include interaction terms in a regression model?

text: I am using logistic regression to analyze some categorical data (binary response variable and categorical -- mostly binary -- predictor variables). For my model, I have something like `A ~ B` and a hypothesis that the respondent's `B` has some explanatory power over the choice of `A`. When I run this regression, only the intercept shows a p-value within the range of statistical significance. However, I have some other variable `C` that assesses some pre-existing conditions for each respondent. When I run a logit regression on `A ~ B + C`, `C` has a very low p-value (statistically significant). That is to say, the pre-existing preferences that each respondent has, as reflected by `C` appear to have an effect on their choice in `A`. My question then, is whether or not it is appropriate to add an interaction term for `B*C` to my regression in this case. When I run the logit regression `A ~ B * C` (or the equivalent to `A ~ B + C + B:C`), both `B` and `C` and the interaction term `B:C` have high statistical significance (low p-values). Is this statistically valid? Does it make sense for something to become statistically significant when an interaction term is added to the model?


title: What kind of residuals and Cook's distance are used for GLM?

text: Does anybody know what the formula for Cook's distance is? The original Cook's distance formula uses studentized residuals, but why is R using std. Pearson residuals when computing the Cook's distance plot for a GLM. I know that studentized residuals are not defined for GLMs, but how does the formula to compute Cook's distance look like? Assume the following example:               numberofdrugs <- rcauchy(84, 10)     healthvalue <- rpois(84,75)     test <- glm(healthvalue ~ numberofdrugs, family=poisson)     plot(test, which=5)       What is the formula for Cook's distance? In other words, what is the formula to compute the red dashed line? And where does this formula for standardized Pearson residuals come from? ![cook's distance](http://i.stack.imgur.com/QdjMg.png)


title: How can I correct for measurement error in the dependent variable in a logit regression?

text: I'm running a binary logit regression where I know the dependent variable is miscoded in a small percentage of cases. So I'm trying to estimate $\beta$ in this model: $prob(y_i) = 1/(1 + e^{-z_i})$ $z_i = \alpha + X_i\beta$ But instead of the vector $Y$, I have $\tilde{Y}$, which includes some random errors (i.e. $y_i = 1$, but $\tilde{y_i} = 0$, or vice versa, for some $i$). Is there a (reasonably) simple correction for this problem? I know that logit has some nice properties in case-control studies. It seems likely that something similar applies here, but I haven't been able to find a good solution. A few other constraints: this is a text-mining application, so the dimensions of $X$ are large (in the thousands or tens of thousands). This may rule out some computationally intensive procedures. Also, I don't care about correctly estimating $\alpha$, only $\beta$.


title: Why are the correlations in two groups less than the correlation when the groups are combined?

text: I have been running correlations for a set of data and several subsamples. During this analysis I ran into a situation where the $r^2$ for two groups was smaller in each individual group as opposed to when they are grouped together.   * Is there any straight forward explanation for how this could happen?


title: Robust GAM that covers Gaussian distribution

text: I was looking at CrossValidate archives as well as r-archives and crantastic...for a package that has a robust approach to generalized additive models. I found two packages "robustgam" and "rgam" but their implemented functions cover only binomial and Poisson distributions (pls correct me if I am wrong). I would greatly appreciate if anyone could share with us other R-packages or robust approaches of general additive modeling that might have a better performance with small data sets ($n<100$ records or 50 -100 records).


title: The difference between discrete and continuous variables

text: Is the number of hydrogen bonds or the number of rings in a molecule a discrete or a continuous variable ? Can I say that the number of rings: 1, 3, 4, $\dots$ is a continuous variable because in theory this number could rise up to infinity? On the other hand it can only take integer values so....is it discrete then ?


title: AR(1) parameter estimation

text: Given a time series, I'd like to estimate the parameters of an AR(1) model for it. As explained on wikipedia, there are different ways for doing that. What may be called a naive method is to compute the sample mean, variance, and autocovariance of the sample and then obtain the parameters of the AR(1) model using some simple equations. Alternatively, one can use more complicated things like maximum-likelihood estimation. What is the benefit of one method versus the other? Are there any _provable_ differences? For example, suppose my data really comes from an AR(1) process, is the "naive" method provably less accurate than others such as maximum-likelihood?


title: Time varying predictors at higher aggregation levels in multilevel survival analysis

text: **The case:** I am trying to estimate event history models (also known as survival models) with time-varying predictors at two different levels of (geographical) aggregation. More precisely, I am using a discrete time event history model (logit model on stacked data) to predict the odds of outmigration (mig) at the household-level. Each household is exposed to the hazard of migration over a certain period (in this example three years; exposure). I have a number of time-varying (e.g., wx = cumulative working experience of household head) and time invariant household-level predictors (e.g., fem = household head is female) to control for the effect of varies socio-demographic on the decision to migrate. However, the households in my sample are located in different municipalities (MunID). In my research I am interested in how a set of time-varying characteristics of the environment (Env1, e.g. rainfall decline) that operate at the municipality-level impact the odds of household-level outmigration. However, I also need to control for some time-invariant municipality-level characteristics (Env2, e.g., % land used for agricultural production). A simplified example of the data structure is presented in the below table. ![enter image description here](http://i.stack.imgur.com/GF5EC.png) **The problem:** Because I have two levels of aggregation (households clustered in municipalities), I was intending to use logistic multilevel models. However, I am not quite sure how to correctly specify my levels. I am using R and the lme4 package to estimate the multilevel models. **Possible solutions:**   1. Courgeau (2007) describes a multilevel event history model with three levels: Time (level-1) is nested within individuals (level-2), who are nested within states (level-3). However, Courgeau only mentions a time-invariant state-level predictor. In my case, I have the problem that a time-varying predictor at the municipality-level (e.g., Env1) would not be recognized by the model as operating at the municipality-level because the values within each aggregation unit vary across time. However, the standard errors of the estimate for Env1 will be biased if the model considers this variable as a level-1 predictor because at each time point all households within one municipality will have the same Env1 value.   2. As another option, I could use the combined MunIDy variable to specify my third-level. MunIDy combines the municipality ID (MunID) with the exposure year variable (exposure) and results in n=3*2=6 aggregation units at level-3. However, this solution seems to be also less ideal since, each level-2 unit would contain only household and municipality level values for one exposure year (e.g., one unit would consists of all cases/observations in a particular exposure year and a particular community), and I am not sure if this would cause problems for the event history model. Does anyone have an idea of how to correctly specify the levels in my analysis so that I can investigate the effect of time-varying predictors at level-3? Or can anyone point me to published work that uses a multi-level event history analysis with time-varying predictors at higher aggregation levels? Thanks a lot for any help! **References:** Courgeau, D. (2007). Multilevel synthesis: From the group to the individual. Dordrecht, The Netherlands: Springer.


title: What's a good transform for decaying function with a plateau

text: I have a dataset (y vs. t) that I think is best described by:   * Slight initial downward slope   * At maybe t = 1 or 2, steep downward slope   * Asymptote at y = 0 ![enter image description here](http://i.stack.imgur.com/85kvc.png) ![Actual, fuzzy data](http://i.stack.imgur.com/4gmOe.png) I would like to transform the data so I can build a strong linear regression. The second portion looks good for exponential decay (use ln(y) as the dependent variable), but how do I combine with the first part? **How should I transform my data before regression?** **Images below are intended to respond to commenters' requests** ![enter image description here](http://i.stack.imgur.com/gs15y.png) ![enter image description here](http://i.stack.imgur.com/LVr5c.png) ![enter image description here](http://i.stack.imgur.com/tOyIn.png)


title: How can I find the standard deviation of the sample standard deviation from a normal distribution?

text: Forgive me if I've missed something rather obvious. I'm a physicist with what is essentially a (histogram) distribution centered about a mean value that approximates to a Normal distribution. The important value to me is the standard deviation of this Gaussian random variable. How would I go about trying to find the error on the sample standard deviation? I have the feeling its something to do with the error on each bin in the original histogram.


title: Time-Series Data and Omitted Variable Bias

text: I understand that ,usually, time-series studies do not aim to provide a causal explanation of anything but rather aim to forecast. In which case it does make sense that most time series studies aren't concerned with omitted variable bias. However, I've come across a time series study that seeks to explain Soviet defense expenditures over a specific time period. The authors caution against using their model for forecasting purposes. Is it fair to say that omitted variable bias would be something we are concerned with in this specific case? Maybe the answer is obvious to most of you but just looking to verify my thoughts. Thanks! JL


title: How to compute $\mathbb P( 3 X_{(1)} \geq X_{(2)}+X_{(3)})$ for order statistics of a uniform distribution?

text: I am trying to solve a problem for my thesis and I don't see how to do it. I have 4 observations randomly taken from a uniform $(0,1)$ distribution. I want to compute the probability that $3 X_{(1)}\ge X_{(2)}+X_{(3)}$. $X_{(i)}$ is the ith order statistic (I take the order statistic so that my observations are ranked from smallest to biggest). I have solved it for a simpler case but here I am lost to how to do it. All help would be welcome.


title: Probability of overlap between independent samples of different sizes

text: Today, I was trying to use probabilities to make an 'everyday' (and not very important) decision... but I found that I'd forgotten how to approach this sort of problem (if I am correct in believing that I ever knew how to deal with a problem like this). Here is the form of the problem I was trying to solve: There are Q objects in a container, each with its own unique color. Agent1 takes a sample G of M unique objects out of the container, and then puts those M objects back into the container. If Agent2 takes a sample V of L unique objects out of the container, then...   1. What is the probability that the larger of the sets G and V will contain the smaller of these sets?   2. What is the probability that the sets G and V will have an overlap of T elements? To give you a rough idea of what I was trying to figure out, I was thinking: "Hmm... There are 10,000 'things' to choose from, and I assume (for simplicity!) that each one has an equal chance of being chosen. I've picked 900 'things'. My friend picked 1500 things... and he picked 890 things in common with me. How likely is it that my friend peeked at my 'picking' history, and copied me?"


title: Should I re-center variables when looking at moderator effect in men and women separately?

text: I want to see if an interaction variable in a multiple regression is significant for the whole sample, and then just for men and just for women. When I created the interaction variable for the whole sample, I centered the interaction components by subtracting the mean for the whole sample. **Now, when I want to look at men and women separately should I recalculate male and female specific centered and interaction variables, centering them with the respective male and female sample means for the interaction components?**


title: Comparison of dependent but unpaired data sets

text: I'm trying to find a statistical test that can help me compare the means of two dependent groups. My data:   1. parameter values from white matter region in an in vivo rat brain    2. parameter values from white matter region in the same rat brain, bur after the tissue has been fixated. I cannot align / register the two brains to each other, so the region I have outlined in vivo is not exactly the same as the region I have outlined in the fixated tissue. My wish:   * I would like to compare the means of the two white matter regions.  My suggestion:   * A paired t-test, but as I know that each pixel value in data set 1 does not exactly correspond to the pixel value in data set 2, the paired t-test is not exactly right to do.  Is there a test of dependent but NOT paired data?


title: HMM - Forward Algorithm - Initial State Distribution

text: In my case, I have a dataset of observations which have not been produced by an HMM meaning that I do not know what kind and how many hidden states should exist there. So, the only known factor are the observations. After the training of my model using a specific number of hidden states each time, I needed then to apply forward algorithm to evaluate the model. My problem is that since I do not know what kind of hidden states there are, I am not able to assign proper initial pi values, therefore I decided to assign same probabilities for all states equal to 1/N. Do you find my approach meaningful? Thank you in advance.


title: How can I test the fairness of a d20?

text: How can I test the fairness of a twenty sided die (d20)? Obviously I would be comparing the distribution of values against a uniform distribution. I vaguely remember using a Chi-square test in college. How can I apply this to see if a die is fair?


title: Is the approach to moderator analysis advocated by Hunter, Schmidt and Jackson (1982) valid?

text: There are several approaches, for example, moderated regression analysis and Hunter, Schmidt and Jackson. The HSJ (1982) approach invokes true variance computation based on subsets say male- vs female-led studies. According to Hunter, J. E., Schmidt, F. L. and Jackson (1982) _Cumulating Findings Across Studies_ , if data are broken into (two) subsets, then there are two ways that a moderator variable will show itself.   1. There should be large differences in the mean effect-size between subsets.    2. There should be reduction in variance within subsets.  Is this a valid approach for identifying the moderator factors or variables? See more in Hunter, J. E., & Schmidt, F. L. (2004). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings (2nd ed.). Thousand Oaks, CA: Sage


title: Why are Pearson's residuals from a negative binomial regression smaller than those from a poisson regression?

text: I have these data:               set.seed(1)     predictor  <- rnorm(20)     set.seed(1)     counts <- c(sample(1:1000, 20))     df <- data.frame(counts, predictor)      I ran a poisson regression               poisson_counts <- glm(counts ~ predictor, data = df, family = "poisson")      And a negative binomial regression:               require(MASS)     nb_counts <- glm.nb(counts ~ predictor, data = df)      Then I calculated for dispersion statistics for the poisson regression:               sum(residuals(poisson_counts, type="pearson")^2)/df.residual(poisson_counts)          # [1] 145.4905      And the negative binomial regression:               sum(residuals(nb_counts, type="pearson")^2)/df.residual(nb_counts)          # [1] 0.7650289      Is anyone able to explain, WITHOUT USING EQUATIONS, why the dispersion statistic for the negative binomial regression is considerably smaller than the dispersion statistic for the poisson regression?


title: Poisson regression on partially missing covariates

text: I'm reading an article where the author fits a Poisson regression on some data. Some of the covariates (but not the outcome) are partially missing. The author writes that he: > conducted the analysis using a valid, reasonably efficient method for > handling missing-at-random data. Can anyone help me understanding what is he referring to and how this model can be fitted in practice (i.e. pointers to R packages, SAS procedures, Stata commands...)? I was thinking of (Generalized) Structural Equation Modeling but I might be mistaken.


title: Calculating transition probabilities for Ehrenfest diffusion

text: > Two containers $A$ and $B$ are placed adjacently to each other and gas > passes through a small aperture joining them. > > A total of $N(>1)$ molecules is distributed among the containers. > > Assume that at each epoch of time one molecule, picked uniformly at random > from the $N$ available, passes through this aperture. > > Let $X_n$ be the number of molecules in $A$ after $n$ units of time. > > Work out the transition matrix and classify the states. I think that my state space will be the number of molecules in $A$ at time $n$ and so I can define $S = \\{0, 1, ..., N\\}$. So from here, my matrix is going to represent the number of molecules in $A$. So, for example $P_{11}$ is the probability of picking a molecule from $A$ and putting it back in $A$. This can't happen as the molecule must travel through the aperture. So I get that all $P_{ii} = 0$. Also, we know that if we have $i$ molecules in $A$ at time $t$, then we will either gain a molecule, as this one has been picked from $B$, or lose a molecule, as this has been chosen from $A$. The probability of picked a molecule is $\frac{1}{N}$ and so we can say $$P_{ij} = \begin{cases} \frac{1}{N} & \text{if } j = i + 1 \\\ 1 - \frac{1}{N} & \text{if } j = i - 1 \\\ 0 & \text{otherwise} \end{cases} $$ So using this, I got the matrix to be $$\pmatrix{0 & 1 & 0 & 0 & 0 & ...\\\ 1 - \frac{1}{N} & 0 & \frac{1}{N} & 0 & 0 & ... \\\ 0 & 1 - \frac{1}{N} & 0 &\frac{1}{N} & 0 &... \\\ . & . &. &. &. &. \\\ . &. &. &. &. &. \\\ 0 & 0 & 0 &0 &... & \frac{1}{N} \\\ 0 & 0 & 0 & ... & 1 & 0}$$ But in the answers, instead of having $1$'s in the numerator, they've got $i$'s everywhere. How comes?


title: Number of points crossed by their best fit line

text: My lab teacher asked this question in class, but i find no way to work it out. If I have $n$ points with their uncertainties, I know that they follow a linear expression and I find the best-fit line of them, is there a way to predict how many points will be crossed by the line? I thought that this number should be related to the $\chi^2$ of the points that should be $\sim n$. Someone can help me even if I probably didn't explain myself well? Thanks!


title: Face Recognition approach DCT features

text: Face Recognition approach based on entropy estimate of the nonlinear DCT features proposes to use maximum entropy estimate of the DCT of the pixels. My question is maximizing entropy would mean minimizing information content since entropy is negative to information. So, how come the Authors say that maximum entropy means maximum information? Can somebody please help in clearing the confusion regarding their techniue ? Thank you in advance


title: Don't understand how they've calculated the transition probabilities

text: > Suppose the weather condition classified as "sunny" (S) or "rainy" (R). > Assume the condition on day $n + 1$ is dependent on the days $n$ and $n-1$ > only. Work out the transition matrix from the table below. > > $$\pmatrix{\mathrm{Yesterday} & \mathrm{Today} & \mathrm{Tomorrow} \\\ R & R > & R \\\ S & S & S \\\S & R & R \\\R & S & S \\\\}$$ So far, what I have done is said that if we let $X_n$ be the weather on day $n$, then $X_n$ isn't a discrete time markov chain as it doesn't satify the markov property. To turn it into a DTMC, lets say that $Y_n = (X_{n-1}, X_n)$. This is now a DTMC, with the state space $S = \\{1, 2,3, 4\\}$ where $1 = (R,R), 2 = (R, S), 3 = (S, R), 4 = (S, S)$. Now I want to work out the transition probabilites. So first lets calculate $P_{11}$. So this is the probability of going from state 1 to state 1, i.e if it rains yesterday and today, what is the probability it will rain tomorrow. So I am working out $$P\\{Y_{n+1} = 1 | Y_n = 1\\} = P\\{X_{n} = R, X_{n+1} = R | X_{n-1} = R, X_{n} = R\\}$$ I then said from here, using the law of total probability, we get $$\frac{P\\{X_{n} = R, X_{n+1} = R, X_{n-1} = R, X_{n} = R\\}}{P\\{X_{n-1} = R, X_{n} = R\\}}$$ But in the answers, it says this line should read > $$\frac{P\\{X_{n} = R, X_{n+1} = R, X_{n-1} = R\\}}{P\\{X_{n-1} = R, X_{n} = > R\\}}$$ Then for some reason they go from here to > $$P\\{X_{n+1} = R | X_{n-1} = R, X_{n} = R\\}$$ First, where does the $X_n$ go from the line above this, and secondly, how do they make the jump to the last line? EDIT: Ok, this is the data in the table > $$\begin{matrix} \mathrm{\underline{Yesterday}} & \mathrm{\underline{Today}} > & \mathrm{\underline{Tomorrow}} & \mathrm{\underline{Prob.}} \\\ R & R & R & > 0.6 \\\ S & S & S & 0.8 \\\ S & R & R & 0.5 \\\ R & S & S & 0.75 > \end{matrix}$$ I tried to work out $P_{12}$ and got my formula to be $P(X_{n+1} = S | X_n = R, X_{n-1} R)$. How would I solve this using the table? Also, in the answers, they say $P_{13} = 0$, why would this be?


title: Using Kolmogorov–Smirnov test

text: A newbie question (please provide detailed answers): I am trying to use the Kolmogorov–Smirnov test. I managed to calculate the difference between the empirical points and the theoretical distribution $D$ (following Wikipedia). But then I am a bit confused about the test:   1. Is the null hypothesis that the empirical data is or is not distributed according to the theoretical distribution?   2. How do I determine the critical level $\alpha$ when I have $D$? I can use this class to calculate the Kolmogorov Smirnov distribution. I think I have all the ingredients, but I am not sure how to put them together.


title: Differences between groups

text: I am currently doing a research on effects of writing styles on the perception of reader. Currently I have the following 4 groups               Group 1: Pre Test - Treatment 1a - Post Test     Group 2: Pre Test - Treatment 1b - Post Test     Group 3: Pre Test - Treatment 2a - Post Test     Group 4: Pre Test - Treatment 2b - Post Test          Treatment 1 refers to a normal writing style and Treatment 2 refers to the enhanced writing style.       I understand the most researchers will assume that pretest for all 4 groups are the same due to randomly chosen participants. However, what it can show is only that the treatments influence the participants. Is there any statistical tests I can use to test that treatment 1 influences the participants more than treatment 2? Thank you! any help is greatly appreciated.


title: Longitudinal regression: problem with listwise deletion (relative importance estimation, R, package RelaImpo)

text: I have 3 measurement points, N=4000, N=2000, N=3000 (week 0, week 6, week 12). My goal is to estimate the relative importance of 14 regressors on one dependent variable. Now I can do that separately for each time point, but the N are comparably small to estimate relative regressor importance with such a huge number of regressors. So I thought about simply "pooling" participants, pretending the data are cross-sectional, and using N=4000+2000+3000. Obviously this violates the assumption of independence of observations, so I should not do that. Alternatives? I can't use a regression in which I simply add time as regressor because the bottleneck week 6 (only N=2000) will lead to listwise deletion of 50% of the subjects.


title: Multiple categorical Variables and Multiple Hierarchical Counts- how to infer the effects?

text: I have the following categorical/count data :               Category A1 | Category A2 | Category B1  | Category B2 | Category C1  | Category C2 | Category C3  | User Population 1 |  Video Count 1  |  Ad Count 2           0          |     1       |     1        |   0         |   0         |  1           |   0          |      15000       |      7000          |      45000       where either one of A1 and A2 can be chosen and either one of B1 and B2 can be chosen and either one of C1 , C2 and C3 can be chosen and the above example would mean 15,000 users in A2 and B1 and C2 from population 1 and a count behavior on Videos and Ads being 7000 and 45000 respectively-as achieved by those 15000 users. Note- Importantly we have population counts for three different kinds of user behaviors (User Populations and Video, Ad Counts emanating from that user population) for each combination of A, B, C categorical variables. Zero Counts: Also-importantly we can have an entry where in categories A, B, C - all of them or any combination of them can have all zeros and a corresponding user-population and Video/ Ad counts that are a positive number indicating that they chose that behavior of not choosing A,B,C in that respective combination. Intuitively: There is a user population which can choose/not choose the paths through A,B and C as required. Now if I have such sample rows and corresponding user populations I would like to figure out the effect of A1 vs A2 with respect to the sub categories in B and the user populations etc..what is the way to model and solve this and come up with insights?


title: Which books would you recommend for mixed distributions (continuous and discrete)?

text: What is a good book that covers mixed distributions? Most statistics books either only briefly mention them or do not cover the topic at all. I'd like to have a comprehensive resource covering issues of joint distributions with both discrete and continuous variables, conditional and marginal distributions in the mixed case, etc.


title: Does augmented Dicky-Fuller test test complex unit roots?

text: The augmented Dicky-Fuller test, as described in Tsay's Financial Time Series: ![enter image description here](http://i.stack.imgur.com/JABsG.png) The regression done in the test seems to only produce real value $\hat{\beta}$, so it seems to me the test only tests existence of root 1 or -1. (Am I missing something?) A AR(p) model can be nonstationary because it has a complex unit root, so I wonder if the augmented Dicky-Fuller test tests existence of all unit roots (including complex ones, i.e. non-real unit roots)?


title: Is calculating goodness of fit part of model checking in time series analysis?

text: In time series, after fitting a AR(p) model to a time series, when checking the fitted model, we will apply Ljung-Box test on the residual series. In Tsay's Financial Time Series, there is a goodness-of-fit measure $R^2$: ![enter image description here](http://i.stack.imgur.com/rEynU.png) ![enter image description here](http://i.stack.imgur.com/DK5oJ.png) I wonder if calculating $R^2$ is also part of checking a fitted model? or is it for something else? Thanks!


title: Is this method for mixing doubly Truncated Normal distribution acceptable?

text: Say I have to mix **n** double Truncated Normal distributions $\textit{TNormal(μ,σ$^{2}$,0,1)}$. Furthermore, each distribution has a weight **w**. **w** represents the influence of the distribution in the mixture: the greatest **w** , the greatest is its influence in the mixture. The problem is to calculate the μ and σ$^{2}$ for the resulting distribution (I am always using _0_ and _1_ as the bounds to truncate the Normal distribution). This is the procedure I am using to mix the distributions: 1) For each _TNormal_ , I get a sample that represents the population "well enough". For this purpose, I am getting a large sample (i.e., 100000 samples). 2) I use the following equation to calculate the resulting **μ$_{y}$** : ![Equation to calculate the resulting **μ$_{y}$**](http://i.stack.imgur.com/I0X7l.png) Where **X$_{i}$** is a sample of a distribution to be mixed and **w$_{i}$** its weight. The R code for this is: `for (i in 1:100000) {Y[i] = ((wA*A[i])+(wB*B[i]))/(wA+wB}` where **Y** is the resulting _μ_ , **A** and **B** are the distributions to be mixed, **wA** is the weight for distribution **A** and **wB** is the weight for distribution **B**. In other words, **μ$_{y}$** is a set of values calculated given samples of the mixed distributions. 3) Select a variance ( **σ$^{2}$** ) for the resulting distribution. 4) Generate a doubly Truncated Normal distribution with the calculated mean and selected variance $\textit{TNormal(Y,σ$^{2}$,0,1)}$ I tested using the R package **truncnorm**. This procedure is based on this article.


title: Identifying patterns in data streams with minimal state

text: I could not pick a more generic title for this question I guess. Here is what I am dealing with: I have a time series data stream coming in from multiple measurement endpoints. The number of endpoints is between 2 to 200. The resolution is 10 seconds but I am using the 1 minute aggregate. The data is CPU utilization, the unit is percentage. Something like:               [endpointA: 10, endpointB: 12, endpointC: 5, ... endpointZ: 12]     [endpointA: 6, endpointB: 20, endpointC: 17, ... endpointZ: 5]     [endpointA: 3, endpointB: 9, endpointC: 19, ... endpointZ: 20]      I need to identify misbehaving endpoints. The undesired behavior is the have different CPU utilization pattern than the rest of the nodes. It quite obvious just by looking at the graph, but I need to translate that obvious human recognizable pattern to a software. My first approach was to generate the STD for every minute and use the 1.4 * (STD + mean) for misbehaving nodes but this captures too many false positives because the data is very spiky. I am wondering what is the best approach to avoid false positives but capture all of the bad guys. My current ideas: Every iteration (minute) just feeds into a hashmap that has counters for each node it sees. If you see a node alarming you increment the counter, if you don't see it decrement the counter. If a node reaches count number 3 you identify it as a bad node. Instead of using the STD for every minute, I keep an accumulated STD across N nodes (N<5). If you have any better way to deal with spiky data streams please share it. Thank you in advance.


title: can an ARMA process with complex unit roots be made stationary by differencing?

text: If an ARMA process (or just a AR(p) process) has real unit roots (i.e. 1 or -1), then differencing it repeatedly will make the differenced process weakly stationary. An ARMA process (or just a AR(p) process) may have complex unit roots. If that happens, can repeatedly differencing make the differenced process weakly stationary? Thanks!


title: Monte-Carlo estimation of the mean chord length in a polygon

text: This is a more "statistical" version of my  Math.SE problem . **Question** Let $\mathbf{P}$ be a convex polygon of $m$ sides. Pick two of its edges at random, and further pick a random point on each of these edges, and compute the length of the chord we get. If we repeat this process $n$ times,and average the chord lengths we thus obtained, we get an estimate $\hat{\ell}_n$ of the expected chord length $\ell$ of $\mathbf{P}$. Then,how large should $n$ be to ensure that $\hat{\ell}_n$ differs from $\ell$ by (say) at most 5%,with a confidence level of (say) 95%, and precisely how should we collect our samples? Note that determining the expected chord length analytically leads to intractable integrals as shown in my SE post. Also,note that there are multiple inequivalent ways to define the expected chord length of $\mathbf{P}$; the one we need here is $\ell = \lim_{n\rightarrow \infty} \hat{\ell}_n$.


title: Reconciling different parameterizations of the same nonlinear model

text: a journal article has a method for designing experiments to be fit to a 4-parameter logistic model. The model used is $y= D + \frac{A - D}{1 + (\frac{x}{C}) ^ B}$   A = upper asymptote   B = maximum slope   C = x value when y = 50% of maximum (i.e. 1/2 of upper asymptote)   D = lower asymptote   Using pilot data, further experiments are optimally designed by plugging the preliminary parameter estimates into equations presented in the article. However, the nonlinear modeling software that I have access to parameterizes the 4-parameter logistic model differently. The model used is $y = D + \frac{A - D}{1 + e^{B(x-C)}}$ Once I estimate the parameters from the software, how do I translate these to the exponential parameterization used by the software? Thank you.


title: Estimating parameters for a spatial process

text: I'm given an $n\times n$ grid of positive integer values. These numbers represent an intensity that should correspond to the strength of belief of a person occupying that grid location (a higher value indicating a higher belief). A person will in general have an influence over multiple grid cells. I believe that the pattern of intensities should "look Gaussian" in that there will be a central location of high intensity, and then the intensities taper off radially in all directions. Specifically, I'd like to model the values as coming from a "scaled Gaussian" with a parameter for the variance and another for the scale factor. There are two complicating factors:   * the absence of a person will not correspond to a zero value, because of background noise and other effects, but the values should be smaller. They can be erratic though, and at a first approximation might be difficult to model as simple Gaussian noise.    * The intensity range can vary. For one instance, the values might range between 1 and 10, and in another, between 1 and 100.  I'm looking for an appropriate parameter estimation strategy, or pointers to relevant literature. Pointers to why I'm approaching this problem the wrong way altogether would also be appreciated :). I've been reading about kriging, and Gaussian processes, but that seems like very heavy machinery for my problem.


title: Correlation and collinearity in regression

text: I did a correlation analysis for my variables. All of them are associated (the coefficient is above 0). However, there is no collinearity problem in my regression analysis. I do not know how to explain it? I think I am confused about correlation and collinearity analysis...but if coefficient is above 0, it shows there is an association, why no collinearity problem shown in regression analysis? By the way, what is the difference between regression analysis and correlation analysis?


title: Could Robust Subspace Clustering be used to remove outliers?

text: I have two samples set, one is positive, and the other is negative. But, in each of both sets, there are some outliers that don't belong to it. Can the Robust subspace clustering be used to help me refine the two datasets and remove the outliers. I just want those most probable samples to be left. Maybe the Mixture of Gaussian could do this thing. In this task, what are the differences between RSC ang GMM? Thanks.


title: A question about resolving Simpson's paradox by normalization

text: I recently asked a question about Simpson's paradox. Suppose we are concerned about making some choice about an arbitrary element of a population. Recall that Simpson's paradox arises when the answer suggested by the aggregate data differs from the answer suggested when the data is split into two sub- populations and looked at separately. Refer to my previous question for details. Both answers I got suggested I that side with the conclusion given by the sub- populations. They rightly remarked that the aggregate data is confounded and justified choosing the conclusion suggested by the sub-populations using a normalization scheme. An example normalization is given at the end of this answer. The normalization approach as justification for siding with the sub-population conclusion is discussed in this Stanford Encyclopedia of Philosophy entry. > One way to arithmetically counter this difficulty is by ‘normalizing’ the > representations of data from sub-populations and only pooling the normalized > representations of the data. Normalizing data counters the effects of > skewing by providing constant denominators for the fractions that represent > the data, and by representing the sub-populations that are compared as if > they were of equal sizes in the relevant respects in terms of which they are > compared. However, Simpson's Reversals show that there are numerous ways of > partitioning a population that are consistent with associations in the total > population. A partition by gender might indicate that both males and females > fared worse when provided with a new treatment, while a partition of the > same population by age indicated that patients under fifty, and patients > fifty and older both fared better given the new treatment. Normalizing data > from different ways of partitioning the same population will provide > incompatible conclusions about the associations that hold in the total > population. It seems, then, that siding with the conclusion suggested by the sub- population on this basis is completely arbitrary. Different partitions produce different conclusions. We can slice and the dice the data any way we like -- without prior information, there seems to be no principled way to prefer a conclusion suggested by a partition by age to the conclusion suggested by a partition by gender, for example. Am I correct in thinking that using the normalization approach does not resolve Simpson's paradox in favor of the sub-population conclusion, for the reasons stated above? Edit: I realize now that there's nothing particularly problematic about this. If you have access to both the age and gender splits, then use both to get the best result. I was concerned about a hypothetical case where Doctor A only had the age split and Doctor B only had the gender split, so they made different inferences. But it is not surprising this is the case, since they are inferring using different information. Also it appears that I have slightly misinterpreted the answers I was given. My apologies.


title: Find number needed for confidence level from population standard deviation

text: I need to know how to get the number of people needed in a survey for a 99% confidence level, within 1 person? And the only given is the population standard deviation of 15.4.


title: Conceptual question on mutual information and entropy

text: What does mutual information (MI) convey? Looking for good reference books on information theory


title: How to apply properly k-nn algorithm when having several attributes

text: Let assume I have a dataset like this dataset where there are several textual attributes even continuos attributes like age. I have always encountered cases where k-nn is applied on just two attributes and all of those ones numerical. How can I properly apply k-nn in a dataset like that?


title: Using AIC to determine best ARIMA Model

text: I'm trying to fit an ARIMA model to housing data set. Playing around with the p's and q I was able to get an ARIMA Model (2,1,2,)(2,0,0) with an AIC value of AIC=4946.76 I used auto.arima to see if I picked the best model. auto.arima picked the (2,1,3)(2,0,0) model that had an AIC value of AIC=4948.21 . Then I looked at the values for both models to see with the difference was between the two. The ARIMA (2,1,2)(2,0,0) model had an error               Warning message:     In sqrt(diag(x$var.coef)) : NaNs produced      My question is why did auto.arima pick the (2,1,3)(2,0,0) model instead of (2,1,2)(2,0,0)?


title: Which statistical test to use for Likert scale questionaire

text: I need help on determining what tests to use in SPSS. Let me provide some background on my research:   1. I am doing a research on system automation of leave management in a company (which is still using a manual paper form to apply for leave).   2. My hypothesis validates the automation of the different processes (independent variables) on the overall improvement of the leave management process (dependent). My control variables are age, gender and dept.   3. I have developed a prototype of a system and provided a questionnaire in the form of a 5-point Likert scale to the participant. (Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree).   4. The questions measures their response on the statement provided on the test evaluation of the system. These questions are categorised based on the hypothesis. Where do I start?


title: Complications of having a very small sample in a structural equation model

text: I am running a structural equation model (SEM) in Amos 18. I was looking for 100 participants for my experiment (used loosely), which was deemed to be probably not enough to conduct successful SEM. I've been told repeatedly that SEM (along with EFA, CFA) is a "large sample" statistical procedure. Long story short, I didn't make it to 100 participants (what a surprise!), and only have 42 after excluding two problematic data points. Out of interest, I tried the model anyway, and to my surprise, it seemed to fit very well! CFI >.95, RMSEA < .09, SRMR <.08. The model is not simple, in fact, I would say it is relatively complex. I have two latent variables, one with two observed and the other with 5 observed variables. I also have four additional observed variables in the model. There are numerous relationships between the variables, indirect and direct, with some variables being endogenous to four others, as an example. I am somewhat new to SEM; however, two individuals that I know who are quite familiar with SEM tell me that as long as the fit indicies are good, the effects are interpretable (as long as they are significant) and there is nothing significantly "wrong" with the model. I know some fit indicies are biased for or against small samples in terms of suggesting good fit, but the three that I mentioned earlier seem fine, and I believe not similarly biased. To test for indirect effects I am using bootstrapping (2000 samples or so), 90 percent bias corrected confidence, monte carlo. An additional note is that I am running three different SEMs for three different conditions. I have two questions that I would like some of you to consider and please reply to if you have something to contribute:   1. Are there any significant weaknesses to my model that are not demonstrated by the fit indices? The small sample will be highlighted as a weakness of the study, but I am left wondering if there is some huge statistical problem that I am completely oblivious to. I plan on getting another 10-20 participants in the future, but this will still leave me with a relatively small sample for such analyses.    2. Are there any problems with my use of bootstrapping given my small sample, or the context in which I am using it? I hope these questions are not too "basic" for this forum. I have read a number of chapters on SEM and related matters, but I find people are very dispersed in terms of opinions in this area! Cheers


title: How to evaluate "external" quality of clustering?

text: Let's say you want to cluster some objects, say documents, or sentences, or images. On the technical side, you first represent these object somehow so that you could calculate distance between them, and then you feed those representations to some clustering algorithm. Externally, however, you just want to group similar (in _some sense_ \-- and that's where things become pretty vague for me) objects together. For example, in case of sentences we want for clusters to contain sentences about similar topic/concept; we feel that sentences "oh look at this pic of a cute lolcat" and "facebook revealed new shiny feature tonight" should be in different clusters. What are the usual approaches for measuring this "external" quality of clustering? I.e. we want to measure how well our clustering procedure groups initial objects (sentences, images); we're not interested in internal measures (like averaged cluster radius, clusters sparseness), since those measures deal with objects' representations, not with real objects. Meaning, the chosen representation may be awful, and even if internal measures is great, externally we'll end up with clusters that are complete junk from our vague, subjective, "some sense"-ish point of view. P.S. Having limited knowledge in clustering domain, I suspect I may be asking about really obvious thing, or my terminology may sound strange to clustering experts. If so, please advice what should I read on the subject. P.P.S. Just in case, I asked the very same question on Quora: http://www.quora.com/How-to-evaluate-external-quality-of-clustering


title: Plotting confidence bands around fitted values from a binomial GLMM

text: I have some parameter estimates and confidence intervals estimated from a set of model-averaged binomial GLMMs: two main effects and their interaction. I would like to plot [population level] fitted values for the main effects and their interaction, as well as 95% confidence bands. I can plot the population level fitted values, but I'm not sure how to combine the parameter estimates with their CI's to plot confidence bands. Code for plotting the fitted values, and resulting graph, is below. I'm plotting rat caputre probability within forest patches (`Rat`) against distance from forest edge (`Dist`) and whether the forest patch was grazed or not by livestock (`Grazed`). Distance was square-root transformed in the model (`Dist_T`). Could someone please tell me how to combine the parameter estimates with their CI's to plot confidence bands? Thank you for your help! Jay Parameter estimates from the model were: Intercept: -1.557; CI=-3.034, -0.08 Dist_T: 0.097; CI=-0.017, 0.212 GrazedY: -1.898; CI=-4.17, 0.375 Dist_T:GrazedY: 0.182; CI=-0.038, 0.401               # estimated abund where grazing=N:     PlotData <- data.frame(Dist_T=seq(from=min(Rat$Dist_T), to=max(Rat$Dist_T), by = 0.1))                g <- -1.557 + 0.097*PlotData$Dist_T                                                    Rat.fitted <- exp(g)/(1+exp(g))                                                      Dist.bt <- PlotData$Dist_T^2     plot(Rat$Dist, jitter(Rat$Rat, amount=0.02), ylim=c(0,1), cex=0.7, cex.lab=0.7, cex.main=0.4, ylab="Rat capture probability",      xlab="Distance from forest edge (m)", pch=ifelse(Rat$Grazed=="Y", 1,2), col=ifelse(Rat$Grazed=="Y", "grey","black"))     lines(Dist.bt,Rat.fitted, lty=1, col="black")            # estimated abund where grazing=Y:     par(new=T)     PlotData <- data.frame(Dist_T=seq(from=min(Rat$Dist_T), to=max(Rat$Dist_T[Rat$Grazed=="Y"]), by = 0.1))#only plotting within range; where Grazed=Y     g <- -1.557 + (0.097+0.182)*PlotData$Dist_T -1.898*1 #additional grazing term and interaction term                                                   Rat.fitted <- exp(g)/(1+exp(g))                                                      Dist.bt <- PlotData$Dist_T^2     plot(Rat$Dist, jitter(Rat$Rat, amount=0.05), ylim=c(0,1), xlab="", ylab="", type="n")      lines(Dist.bt,Rat.fitted, lty=1, col="grey")         ![enter image description here](http://i.stack.imgur.com/os581.png)


title: How does randomization testing for correlating two variables work?

text: I was going through OpenIntro Statistics[http://www.openintro.org/stat/textbook.php] and came across two case studies in the first chapter   1. Discrimination against women during promotion - Page 42   2. Stanford Heart Transplant study - Page 65 How can shuffling of cards determine independence. In the first example, I can understand that when simulated with people for 100 times, the overall randomisation will take care of chance based promotion. Or, am I getting this wrong and just cards are shuffled? However, in the second example there are people who are dead of alive. How can cards be shuffled and create randomisation? How is randomisation performed to determine the independence or correlation of variables?


title: Proof of distribution $\chi_n ^2$?

text: I have the problem of proofing that $$X=(1/\sigma^2) \sum_{i=1}^{n} Y_i ^2$$ where $Y_i \sim N(0,\sigma^2)$ is $\chi_n ^2$ distributed with $E(X)=n$: My proof that $E(X)=n$: $$E(X)=E((1/\sigma^2) \sum_{i=1}^{n} Y_i ^2)= $$ $$\sigma^2=\sum_{i=1}^{n}(E(Y) - \bar Y)$$ $$E(X)=E((1/\sum_{i=1}^{n}(E(Y) - \bar Y)) \sum_{i=1}^{n} Y_i ^2)= $$ here I am stuck and do not get the transition to $E(X)=n$ My question are: How to get to $E(X)=n$? How would you prove that $X$ is $\chi_n ^2$ distributed? Is is possible to imagine some range of numbers and draw it? I really appreciate your answer!!!


title: Regressing volatility of one variable on volatility of other variable

text: I am a finance student and in finance we measure the risk of different financial items like stock prices, earnings, and cash flows etc. we measure this risk either through standard deviation or through ARCH/GARCH and we can generate its series. the main idea behind my question is that I want to check whether risk associated with one financial item transfers to other financial item. this standard deviation or ARCH/GARCH series is called volatility series for the given financial item. hence I am regressing one financial item's volatility on another financial item's volatility and because I am not an econometrics student so I want to ask. Is it fine to regress one variable's volatility on another variable's volatility. What is the procedure in panel data? the model is something like this: $$\mathrm{vol}(\text{cash flows}) = a + b \times \mathrm{vol}(\text{earnings}) + \epsilon$$ definitely my question includes the inquiries as mentioned by COOLSERDASH: how far this model makes sense economically; how well will it work as a prediction method; whether a linear functional form is a good model.


title: How to combine multiple imputed datasets?

text: I need a single imputed dataset (e.g. to create a country group dummy from the imputed country per capita income data). R offers packages package for creating multiple imputed data (e.g. Amelia) and combining results from multiple datasets (as in MItools). My concern is if I can average all the imputed data to obtain a single dataset. If so, how can I do it in R?


title: Is it sound to take the average of a cumulative average?

text: Sorry, I'm a huge statistics noob. I'm looking to find the probability that a certain event occurs in a binomial experiment. I take a few thousand samples per test, and then calculate the probabilities. The problem is that the experimental probabilities keep varying by a small amount. So I've started plotting the experimental probabilities as a function of sample size. When I look at the plots for these "cumulative averages" the trend I observe is that the experimental probabilities vary wildly at first and then taper off and oscillate around the true probability. However, I can't get enough samples per test to observe a straight line-like convergence. My question: can I average out these cumulative averages I've plotted for a more precise estimate of my true event probability? And if I can, would it be better to use a weighted average - like give more weight to probabilities using a higher number of samples and lower weight to probabilities using a lower number of samples? Or maybe even take the average of only the experimental probabilities that begin to converge very finely. Thanks


title: Machine Learning Process for detecting edges of overlapping objects with OpenCV

text: I'm quite new to machine learning and a bit unsure about the whole process and the interpretation of the results. **The Task:** I have images with some objects of somewhat the same color and shape which overlap and I want to segment the single objects using machine learning to get the edges of adjacent objects. **Current process:** For each image there is a label file with the pixels marked which represent the edge between two overlapping objects and another marking for the whole object areas. For each pixel (positive edge pixel and negative object pixel) I calculate the feature values and start training i.e. a Boosting Tree using OpenCV using ~80% of the images. With the trained classifier I predict the 'object-to-object-edges' for the 20% left and interpret the prediction result with my human vision. During post-processing I use some algorithms to segment all objects one by one using the prediction result. **The problem:** The process is more or less a command line tool reading and writing config files etc. I want to build a framework with a GUI to train different classifiers with variable parameters, features and validation. I'm not sure about the prediction results OpenCV offers. At first I thought I would face an image with either false or true dependant on whether the pixel is an 'object-to-object-edge' or not but I got images 8bit images with probabilities. In order to automatically validate a trained classifier refering to measurements like sensitivity and specificity, I would have to choose i.e. a simple threshold to define a pixel to be true or false and compare the result to the ground truth/label file. I think I have three possibilites:   * Start a bunch of training processes, predict, look at the results by myself   * Start a bunch of training processes, predict, threshold (maybe using a value based on some statistical meaurement since different classifiers have different intensity values), validate results (maybe cross-validation)   * Start a bunch of training processes, predict, do an amount of post-processing until the segmentation is done, validate Letting the application do the work would of course be nice but I guess its best to keep the post-processing as short as possible to prevent post- processing parameters to interfere with the validation. Has anyone experience with OpenCV ML or any tipps on how to build up the process ? How would you face such a task ? Best regards!


title: Binary panel logistic regression (xtlogit fixed effects) is not converging in Stata, how to resolve?

text: I have a panel dataset with a sample of 800 groups, each having between 200-500 observations. The data looks like this: ![enter image description here](http://i.stack.imgur.com/UNfSo.png) The dependent variable is binomial: `close_gp30_f30`. The independent variables are continous growth rates. An example summary of one of these is:                                         close_g1     -------------------------------------------------------------           Percentiles      Smallest      1%    -.0789325      -.9908884      5%    -.0396762      -.9907975     10%    -.0256917       -.990625       Obs             2993902     25%    -.0096911      -.9904597       Sum of Wgt.     2993902          50%            0                      Mean           .0015124                             Largest       Std. Dev.      .3472223     75%      .009676         103.25     90%     .0253968       103.3333       Variance       .1205633     95%     .0399516          104.5       Skewness       436.1726     99%     .0841585       309.3899       Kurtosis       266732.5      I would like to run this experimental regression:               xtset ticker_id date     xtlogit close_gp30_f30 close_g1 close_g10 close_g15 close_g30 close_g60 close_g120 if ticker_grp == 0, fe      However, when I add more than about 5 variables the regression never converges and I seem to get stuck in a loop of "backed up iterations", like so:               . xtlogit close_gp30_f30 close_g1 close_g10 close_g15 close_g30 close_g60 close_g120 if ticker_grp == 0, fe     note: multiple positive outcomes within groups encountered.     note: 11 groups (272 obs) dropped because of all positive or           all negative outcomes.          Iteration 0:   log likelihood = -175837.76       Iteration 1:   log likelihood = -175015.93       Iteration 2:   log likelihood = -175006.84       Iteration 3:   log likelihood =  -175002.6       Iteration 4:   log likelihood = -175001.69  (backed up)     Iteration 5:   log likelihood = -175001.69  (backed up)     Iteration 6:   log likelihood = -175001.69  (backed up)     Iteration 7:   log likelihood = -175001.69  (backed up)     Iteration 8:   log likelihood = -175001.69  (backed up)     Iteration 9:   log likelihood = -175001.69  (backed up)     Iteration 10:  log likelihood = -175001.69  (backed up)     Iteration 11:  log likelihood = -175001.69  (backed up)     Iteration 12:  log likelihood = -175001.69  (backed up)     Iteration 13:  log likelihood = -175001.69  (backed up)     Iteration 14:  log likelihood = -175001.69  (backed up)     Iteration 15:  log likelihood = -175001.69  (backed up)     Iteration 16:  log likelihood = -175001.69  (backed up)     Iteration 17:  log likelihood = -175001.69  (backed up)     Iteration 18:  log likelihood = -175001.69  (backed up)     Iteration 19:  log likelihood = -175001.69  (backed up)     Iteration 20:  log likelihood = -175001.69  (backed up)     Iteration 21:  log likelihood = -175001.69  (backed up)     Iteration 22:  log likelihood = -175001.69  (backed up)     Iteration 23:  log likelihood = -175001.69  (backed up)     Iteration 24:  log likelihood = -175001.69  (backed up)     Iteration 25:  log likelihood = -175001.69  (backed up)     Iteration 26:  log likelihood = -175001.69  (backed up)     Iteration 27:  log likelihood = -175001.69  (backed up)     Iteration 28:  log likelihood = -175001.69  (backed up)     Iteration 29:  log likelihood = -175001.69  (backed up)     Iteration 30:  log likelihood = -175001.69  (backed up)     Iteration 31:  log likelihood = -175001.69  (backed up)     Iteration 32:  log likelihood = -175001.69  (backed up)     Iteration 33:  log likelihood = -175001.69  (backed up)     Iteration 34:  log likelihood = -175001.69  (backed up)     Iteration 35:  log likelihood = -175001.69  (backed up)     Iteration 36:  log likelihood = -175001.69  (backed up)     --Break--     r(1);      I have also rerun this regression with all debugging information enabled, this is a lot of information but may provide the answer on why it is not converging. Note that here I regressed on the standardized values of the independent variables, but this had exactly the same effect (for some reason I hoped that it would solve my problem). http://pastebin.com/YU0EEkJt My **main questions** are:   1. Why is it not converging?   2. How can I resolve this situation? **Update: multicollinearity checks**               . collin close_g1 close_g3 close_g5 close_g15 close_g30 close_g60 close_g70 close_g80 close_g90 close_g100     (obs=3146949)            Collinearity Diagnostics                                  SQRT                   R-       Variable      VIF     VIF    Tolerance    Squared     ----------------------------------------------------       close_g1      1.17    1.08    0.8564      0.1436       close_g3      1.22    1.11    0.8169      0.1831       close_g5      1.25    1.12    0.7979      0.2021      close_g15      1.35    1.16    0.7396      0.2604      close_g30      1.48    1.22    0.6767      0.3233      close_g60      1.87    1.37    0.5343      0.4657      close_g70      1.97    1.40    0.5074      0.4926      close_g80      2.06    1.43    0.4857      0.5143      close_g90      2.03    1.43    0.4915      0.5085     close_g100      1.89    1.38    0.5280      0.4720     ----------------------------------------------------       Mean VIF      1.63                                     Cond             Eigenval          Index     ---------------------------------         1     4.0898          1.0000         2     1.4492          1.6799         3     0.9965          2.0259         4     0.7954          2.2675         5     0.7143          2.3928         6     0.6716          2.4677         7     0.5801          2.6552         8     0.5096          2.8328         9     0.4174          3.1303         10     0.3970          3.2096         11     0.3791          3.2847     ---------------------------------      Condition Number         3.2847       Eigenvalues & Cond Index computed from scaled raw sscp (w/ intercept)      Det(correlation matrix)    0.0422      This too does not seem to be a problem. **Update 2: with gradient option and modified "limits":**               . xtlogit close_gp30_f30 close_g1 close_g10 close_g15 close_g30 close_g60 close_g80 close_g100 if ticker_grp == 0, fe ltol(0) tol(1e-7) gradi     > ent     note: multiple positive outcomes within groups encountered.     note: 10 groups (240 obs) dropped because of all positive or           all negative outcomes.          ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 0:                                                        log likelihood = -186484.82     Gradient vector (length = 4065.469):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -145.9671      -491.122     -628.1631      -548.698     -1291.774     -2406.543     -2847.761     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 1:                                                        log likelihood =  -185998.3     Gradient vector (length = 2373.377):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -93.28661     -296.7566     -370.1424     -292.3351     -675.5539     -1381.919     -1716.862     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 2:                                                        log likelihood = -185954.74     Gradient vector (length = 2226.909):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -88.19199     -278.7058      -347.356     -273.7588      -633.336     -1296.948     -1610.864     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 3:                                                        log likelihood = -185934.48     Gradient vector (length = 2157.759):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -85.68227     -270.0902     -336.5747      -265.113     -613.5868     -1256.742     -1560.826     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 4:                                                        log likelihood = -185929.57                                                                        (backed up)     Gradient vector (length = 2140.928):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -85.03234     -267.9852     -333.9495     -263.0359     -608.7964     -1246.943      -1548.65     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 5:                                                        log likelihood = -185927.13                                                                        (backed up)     Gradient vector (length = 2132.452):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -84.55817     -266.8118     -332.5234     -262.1772     -606.6421     -1241.936     -1542.494     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 6:                                                        log likelihood = -185926.38                                                                        (backed up)     Gradient vector (length = 2125.423):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -79.15747     -261.7075     -327.7218     -267.7253     -612.7412      -1235.74     -1536.582     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 7:                                                        log likelihood = -185925.91                                                                        (backed up)     Gradient vector (length = 2117.104):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -72.14189     -254.8101     -321.5761     -276.4171     -622.5654     -1228.224     -1528.416     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 8:                                                        log likelihood = -185925.59                                                                        (backed up)     Gradient vector (length = 2111.886):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -67.13769     -250.2303     -317.4594     -280.2288      -626.198      -1222.17     -1525.716     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 9:                                                        log likelihood = -185925.59                                                                        (backed up)     Gradient vector (length = 2111.886):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -67.13769     -250.2303     -317.4594     -280.2288      -626.198      -1222.17     -1525.716     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 10:                                                        log likelihood = -185925.59                                                                        (backed up)     Gradient vector (length = 2111.886):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -67.13769     -250.2303     -317.4594     -280.2288      -626.198      -1222.17     -1525.716     ---------------------------------------------------------------------------------------------------------------------------------------------     Iteration 11:                                                        log likelihood = -185925.59                                                                        (backed up)     Gradient vector (length = 2111.886):          close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:  close_g~f30:             close_g1     close_g10     close_g15     close_g30     close_g60     close_g80    close_g100     r1     -67.13769     -250.2303     -317.4594     -280.2288      -626.198      -1222.17     -1525.716      **Update 3:** I don't know if this helps, but when I do `xtdata indepvars, i(ticker_id) fe clear` followed by a `logit depvar indepvar` (which normally worked just fine), the logit seems to get stuck too. I therefore believe that it has something to do with fixed effects and/or panel data. Does this make sense? **Update 4:** This seems to be a problem with Stata's background to double recast, please see my follow-up question xtlogit: panel data transformation's recast to double makes model incomputable (STATA)


title: How to compute a measure of distance between sites with continuous variables?

text: I have a dataframe with each row being a different site (51 sites), and each column being mean values of a different continuous environmental variable (19 variables). I am trying to calculate a measure of environmental similarity/dissimilarity by using a distance calculation between sites. I would like to calculate either a standardized Euclidean distance or Mahalanobis distance. I have managed to get them to work with both the distance function in the package ecodist, and the `dist.quant()` function in the package ade4 in [R]. E.g.               AusEnvDist <- distance(AusEnvNum, method="euclidean", sprange=NULL)      However my outputs are the same regardless of how the dataframe is organized (i.e., sites being in rows or columns) – I get an output matrix of $19\times19$ instead of $51\times51$ – i.e., it's not calculating the distance between sites, but between variables. Any ideas on how to fix this? Or a better method for getting a singular "environmental" value for each site?


title: Find correlation when SD and mean are given

text: A guy got 88 in his test which has mean score of 64 with SD 8. He missed the second exam in which the meanscore was 45 and SD was 5. He got a prorated marks of 55. The prof used regression to get his second test marks. What was the corelation between 2 exams ? $$ \hat{z}_y = r z_x $$ and i calculated respect $$\hat{z}$$ values from the formula $$ \frac{x-\bar{x}}{s_x} $$ for x and y (predicted) in this case. The answer that i got is 0.66. Am i doing it right ? P.S: I would really appreciate if it has been broken down answer in detail.


title: Is my interpretation of Bayesian probability and inference correct?

text: I have the following interpretation of the Bayesian probability and inference (without referring to Measure Theory, I am still at the very beginning of learning it): Let's say we have five random variables $x_1,x_2,x_3,x_4,p$ given as in the following graphical model: ![enter image description here](http://i.stack.imgur.com/E5Bnr.png) Let's say we have already observed $x_1,x_2,x_3$ and we want to make a Bayesian inference on the variable $x_4$. We assume a prior probability density function $p(p)$ for the parameter $p$ and we assume that each $x_i$ is generated from the distribution $f(x_i|p)$. Then according to our assumptions about the prior distribution $p$ and the generative distribution $f$, we assume a sample space $\Omega$ which contains all possible tuples of $(x_1,x_2,x_3,x_4,p)$ and each single outcome $\omega$ has a probability value assigned to it, which corresponds to our degree of belief about this single outcome, according to the Bayesian interpretation of probability. Then we have the event set $\mathcal{F}$ as well, according to the definition of a probability space. This space and its probability measures entail the prior distribution $p(p)$ and the distributions $f(x_i|p)$ and the independence of $x_i$s given $p$ . I assume that one of these outcomes, $\omega$ is selected and all events in $\mathcal{F}$ which contain $\omega$ are assumed as "occured". In practice, we cannot ever observe the value of the parameter $p$ in the selected outcome, $\omega$. But we are able to observe some of the $x_i$s which constitute our data, for example $D=(x_1,x_2,x_3)$. Then the broadest inference we can do about $x_4$ is to calculate the posterior distribution of $p$ given the data and then integrate over all possible single values of $p$, which might be generated the data $x_1,x_2,x_3$ and the target variable $x_4$ like in the following: $$ P(x_4|x_1,x_2,x_3) = \int_{p} f(x_4|p) P(p|x_1,x_2,x_3) dp$$ This more or less constitutes my imagination of how the Bayesian probability and inference works. Is this view correct? If not, what is wrong about it?


title: Should I keep the interaction term?

text: I am doing research on mortality. I am running Cox regression and I am adjusting for 40 variables which are proven in the literature to be related with mortality. My main exposure is X. In my final model I have 19 variables that are significant and my exposure is non significant. Than I try the last analysis including interaction terms between my exposure and the 19 covariates that I found significant in the final model, of which only one interaction term was significant. It changes the coefficient of my main exposure around 40% and makes it significant. Is this a good approach, and should I keep the interaction term in the model?