title: Mann-Whitney vs. t-test to compare two samples with different distributions

text: How should I test the statistical difference of a mean (or median?) concentration of an enzyme between two groups of patients where the distribution in one group is normal, whereas in other it is non-normal? The distribution of values for the entire study population is non-normal. Do I use a Mann-Whitney test, or Student's T-test?


title: Correcting for Type 1 error in multiple paired t-tests?

text: I'm wondering whether or not I should adjust the significance level of paired t-tests due to multiple tests (to avoid the possibility of Type 1 error), although the tests are independent. Here's what I'm trying to test: 4 groups of participants each underwent a different mood manipulation procedure. To test the effect of the mood manipulation, I'm using a word recall test, where the amount of correctly recalled positive words is compared to the amount of correctly recalled negative words. In other words, I'm conducting 4 paired t-tests (number of positive vs. number of negative words), one for each group. Should I correct for multiple tests - and if yes - which correction method would you recommend in this situation?


title: Testing for overdispersion in logistic regression

text: _R in Action_ (Kabacoff, 2011) suggests the following routine to test for overdispersion in a logistic regression: Fit logistic regression using binomial distribution:               model_binom <- glm(Species=="versicolor" ~ Sepal.Width,                        family=binomial(), data=iris)      Fit logistic regression using quasibinomial distribution:               model_overdispersed <- glm(Species=="versicolor" ~ Sepal.Width,                                 family=quasibinomial(), data=iris)      Use chi-squared to test for overdispersion:               pchisq(summary(model_overdispersed)$dispersion * model_binom$df.residual,             model_binom$df.residual, lower = F)     # [1] 0.7949171      Could somebody explain how and why the chi-squared distribution is being used to test for overdispersion here? The p-value is 0.79 - how does this show that overdispersion is not a problem in the binomial distribution model?


title: Assumptions of two-way ANOVA and k-fold cross validation

text: I want to compare 3 classifiers (kNN, SVM and CT) by using their classification accuracies on 10 folds, to highlight eventual differences between them. I think it could be done by a two-way ANOVA analysis, where classifiers=factors and folds=blocks, if some assumptions on data are verified. Following wikipedia, the assumptions are:   1. The populations from which the samples are obtained must be normally distributed.   2. Sampling is done correctly. Observations for within and between groups must be independent.   3. The variances among populations must be equal (homoscedastic).   4. Data are interval or nominal. I need an help on how to verify them in my case.   1. Do I have to verify that for every classifier, its 10 accuracies are normally distributed and/or that for each fold, the 3 accuracies are normally distributed?   2. Observations for within groups are independent because I use a different test set for every fold. Am I wrong? Observations for between groups are independent because I suppose classifiers to act in an independent way. Aren't they?   3. Do I have to verify that each group described in the first point has the same variance?   4. No problems in my case. Is there a quick way to verify all of the assumptions in Matlab?


title: Problems in performing Hausman test

text: I have a panel data of 21 countries over 12 years and when carrying the Hausman test to choose between the FE and RE I am having problems. When performing the Hausman test and I got this (V_b-V_B is not positive definite), so I have applied sigmamore and sigmaless and there was a problem about the rank so I have rescaled my variables but again I got this message when performing the Hausman test (V_b-V_B is not positive definite). I have chosen to use the usest in stata 12 and I having this . reshape wide pdgdp gcfdebt gdp inflation libor exchangerate tradeopenness, i(year) j(countrycode) (note: j = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21) no room to add more variables Up to 120 variables are allowed with this version of Stata. Versions are available that allow up to 32,767 variables. r(900); So, I have tried to increase the varlist but its saying maxvar not settable in this version of Stata After doing some research I have found out that I can use the xtoverid but I cannot really understand this whole thing about the IV estimations. Can someone please advice me about what I can do to get over these problems and get the right test to choose between FE and RE. I am not really at ease with econometric, so I will really appreciate a simple answer.


title: Looking for agreement amongst respondents

text: I have a small study of 21 respondents, who rated some items from 0-5, with 0 being unimportant and 5 being very important. I know beforehand that the items are related (they are a published list of items from a document); and I do not want measures which show how related these items are. The list of items is fixed. Is there some non-parametric way of showing some form of agreement in the ratings between the respondents? I've seen a study which uses Kendall W, but there are a lot of ties and perhaps this is not the most useful way. Just to clarify, I'm looking for some form of agrement measure for each item within the data, not specifically one item.


title: Neural network training - Features/variables have missing value

text: I have a binary outcome and my features are mostly continuous. However most of the features have missing data and are coded as missing. Missings are coded as 9999, 999999 etc. How do i scale my features if i have something like this ? When the model gets implemented and in real time we will have missing values. So i am advised to keep the missing's and build the model.


title: GBDT and model building: How am I overfitting?

text: Here's my situation: Binary classification and I've got a training set of roughly 250k samples and 10 features, and a validation set of roughly 100k with the same number of features. I'm fitting GBDT to the data with subsampling, so BGBDT I guess? Anyhow I took the training set and split it 80/20 and did a grid-search over the parameter space (using cross-validation) on the 80% and then fit a new model to the 20% using the paramaters obtained from the grid-search and used log loss to determine the error. The 20% are held-out and are trained separately from the 80%. Ok fine. I'm getting good results wrt log loss on the held-out model for the 20%, and I'm monitoring the out of bag estimate of the deviance during the tree building process which is decreasing throughout the run. Overall I'm training roughly 150 trees, subsampling at 0.25, with a learning rate of 0.1, and maximum depth of 10. The results from training the held-out samples this way give good results... if there was a problem with overfitting I would expect that the held-out samples wouldn't perform well. But when I retrain the model on the entire training set and then predict on the validation set my results are very poor. One thing to note, the training and validation sets are uniformly sampled from the same distribution and the classes are balanced. Suggestions?


title: Determining number of clusters with SSE scree plot with Gower's coefficient of similarity

text: I am researching cluster analysis, and I am interested in variables that are both categorical and continuous, for which I have read that a Gower's similarity coefficient is a good proximity measure. I am interested in first using an average linkage algorithm, and have found that some have recommended looking for the 'elbow' in the sum of squared error (SSE) scree plot as a guideline for deciding how many clusters to retain. I was wondering if the Gower's similarity coefficient (being non-metric and non-Euclidean) would allow me to create an SSE scree plot, or if that didn't make sense statistically.


title: Interpreting Durbin-Watson results

text: I have fitted a glm to my data set and used to the Durbin-Watson test to check model fit. I have obtained the result. How can i interprete it?               lag    Autocorrelation     D-W Statistic    p-value             1       0.7750748     0.4466024       0      Alternative hypothesis: rho != 0


title: How do you apply a linear regression built in SPSS to new data and generate prediction intervals

text: I am trying to use SPSS to build a linear regression on historical data (dependent and independent variables) and then apply this to new data (independent variables only) to generate predicted values and associated prediction intervals. I've looked in detail at the documentation on the `REGRESSION` procedure within SPSS, and while it is obvious how I would get the prediction and interval for the data used to build the regression (using a `/SAVE` subcommand to save the temporary variables `PRED`, `LICIN` and `UICIN`) I'm not seeing any functionality that would allow me to apply this to new data. Essentially I'm looking for the equivalent of `PROC SCORE` in SAS, or `predict.lm` in R.


title: Model-based learning algorithm for recommendation engine

text: Can you please suggest me a good model-based learning algorithm to recommend items to the user? Is there any open source implementation available on model based learning algorithm? I am sure Apache Mahout doesn't implemented any model based learning algorithms.


title: Logistic regression - power and predictor values

text: What is power in logistic regression? Is it the ability of the test to reject the null hypothesis when it is actually false? Second, if you're trying to maximize your statistical power when doing a logistic regression, is it better to use predictor values that are only high or low or a range of predictor values?


title: Is there any way to know if my clusters are meaningful or meaningless?

text: > **Possible Duplicate:**   >  How to tell if data is "clustered" enough for clustering algorithms to > produce meaningful results? I have used hierarchical clustering, e.g, Ward's method, single,complete, etc. I have the same problem, I do not know how to assess my clustering algorithm. How would I know that the clusters are meaningful? I do not know what FPC software is, can any one use FPC, and, what is the programming language of this software? If I'm not familiar with FPC, is there another means of assessment?


title: usefulness of k-means clustering on high dimensional data

text: I wonder what is the usefulness of k-means clustering in high dimensional spaces, and why it can be better (or not) than other clustering methods when dealing with high dimensional spaces.


title: Covariance pattern models versus generalized estimating equation models

text: Can somebody please explain the major differences between covariance pattern models (Hedeker and Gibbons, Chapter 6, 2006; Jennrich and Schluchter 1986) and generalized estimating equation models (Hardin and Hilbe, 2012; Liang and Zeger, 1986). My presumption is that the main difference is around ML/REML estimation in the covariance pattern models case; versus, quasi likelihood estimation in the GEE case. Is this correct? And also, the applicability of covariance pattern models to Gaussian response data whereas, GEE is more applicable to response data of many other distributions (Gaussian, Binary, Binomial, Poisson, etc). Are covariance pattern models with Gaussian response and some identity link and GEE (identity link, Gaussian response) similar/identical?


title: Dependent variable is the amount of remittances in a survey

text: My thesis is on the microeconomic and macroeconomic determinants of remittances, and i am doing the research by using primary data taken directly from the expatriates who are working abroad.My dependent variable is the amount of remittances coming to the country. Most of the papers that i have gone through have discreet dependent variables or have categorized their dependent variable. I don't want to lose the information, therefore i wish to use the exact amount of remittances which is being sent by the expatriates as my dependent variable. Can i use it for my regression? My independent variables are discreet, and can you please also suggest appropriate models which i could use to analyze my survey set? I am trying to see whether self interest, altruism,self insurance etc have an impact on the amount of remittances sent by the expatriates.


title: Confusion related to bayesian spatial scan statistics

text: I was reading this paper related to bayesian spatial scan statistics. http://books.nips.cc/papers/files/nips18/NIPS2005_0513.pdf. It's said that when we use bayesian spatial scan statistics, we don't need to use the randomization testing or significance testing of the found potential cluster. Why is it so? I didn't get it. Any insights on this?


title: Static Bayesian networks using p-values

text: In your opinion, what is the best way of handling Bayesian networks using continuous data, in this particular case, p-values? I have read about several discretization techniques, Gaussian approaches, etc., but I'm quite lost due the controversies each approach has. Also, while I found a lot of literature regarding how to handle temporal series for both static and dynamic Bayesian networks, I didn't find a good reference for handling p-values.


title: MCMC estimation in mixed models: is there something like "significance"?

text: I am a bit confused in determining whether to keep or remove some of the random coefficients in a mixed model. There are quite few level two units, therefore I have to use MCMC estimation. As result, standard errors for most variables in the random part are about 60% of the corresponding point estimate. On the other hand, DIC for the whole model is substantially reduced when adding each variable separately to the random part. As far as I understand modeling with ML, if a standard error greater than about half of the point estimate value, it is a sign that the respective coefficient is insignificant. Can I interpret the standard error in MCMC the same way? If so, what takes precedence, DIC or estimate/standard error ratio? I am a beginner with respect to mixed models, so any references for self- education will be appreciated along with the eventual explanation. Thank you! ADDENDUM: To make the question more clear with an example: Case one Case two In the first case I am inclined to accept the parameter, given that its mode is far from zero and 95% credible intervals are 2.2 and 11.1. In the second case I am inclined to reject the parameter, particularly looking at the kernel density graph. Zero has almost as high probability as the mode. However, 95% credible intervals are 0.002 and 0.012. If I only look at these I feel I should retain the parameter. The left bound is close to zero, but the estimate itself is not that large; and it is an interaction effect, so this size makes sense. In addition, DIC for the whole model goes down by about 50, which is quite a lot. Mixed signals, so to speak, at least they seem to me.


title: Evaluating skill level on a particular scale

text: I want to evaluate the skills of a trainer who has taught 4 different computer languages to a batch of 150 students. I have prepared a questionnaire, which I will give to each of the students. But I am yet to decide the scale on which the students will be evaluating the skill of the trainer for those 4 languages. I feel a scale of 10 is a very comfortable and students can provide the feedback for the trainer based on this scale. But I have seen many a times people evaluating performance on a scale of 5. Even once I was evaluated on the scaler of 5. Is there any significance of choosing a particular scale? I mean is there any thing like good scale , bad scale, best scale for a particular kind of feedback? Appreciate your time for replying.


title: Logarithm of percentage values in time series

text: I know that is preferable to logarithmize variables in absolute values in time series for various reasons. But do variables in percentage values (inflation rate) need to be logarithmated as well, as I've seen it in a study, but I'm not sure if it's ok. Thanks!


title: Repeated comparisons over time between two groups

text: I have to compare the mean between two groups over time. The difficulty here is that I have to make this coparison about 80 times. And it is not enough to know whether there is a difference, but I also have to say _when_ it occurs. For the context: a behavioral marker measured in 2 groups of 5 mice each, measured over and over again in different conditions, in day- and night-time, before and after some treatments etc.


title: How do I interpret this log linear regression coefficient?

text: Correction on previous post. I have a log linear regression model where the dependent variable is logged and an independent variable is linear. The slope coefficient for a key independent variable is negative: -.0564. Not sure how to interpret. Do I use the absolute value of .0564 and then turn it into a negative like this: ((exp.(.0564)-1) * 100 = -5.80) OR Do I plug in the negative coefficient like this: ((exp.(-.0564)-1) * 100 = -5.48) In other words, do I use the absolute figure and then turn that into a negative or do I plug in the negative coefficient? As you can see they produce 2 different answers.


title: Which test for small sample sizes?

text: I have got a total sample size of 12 soil samples which are splitted into 4 groups of $n=3$. A number of different variables were measured like content of organic carbon e.g. These data are logistically difficult and expensive to collect, so collecting more data is not possible.   Now I want to compare the means of the 4 groups ($n=3$) by testing. But there is the problem that normal distribution (Shapiro-Wilk) and homoscedasticity (Brown-Forsythe/Levene) is not given in each sample. So which test should I prefer? What do you think? Is it even useful to use test statistics in this case? Are there any alternatives beneath descriptive statistics? * * * OK, here are some details. The soil samples are from a long-term experiment site with four different treatments (plough, no tillage and so on). From each of these four treatments I have 3 soil samples. The different variables are measured in metric data. For example - content of organic carbon [g/kg]: (Group 1: 12.1 // 13.2 // 13.5), (Group 2: 13.1 // 13.9 // 13.5), (Group 3: 8.9 // 10.2 // 11.9). Now I wanted to test if there are any significant differences between the median of these groups. And another information is that concentrations of different substances in soils are generally not normal distributed.


title: Modeling binary outcomes - inaccurate model when using logistic regression?

text: I am trying to model the probability of a binary outcome with the independent variable being an hour variable. I understand that linear regression is not the correct method for this type of task (I wont go into my simplified understanding of why, if someone wants to add a succinct explanation I'm all for it). The problem I am running into is that logistic regression does not seem to accurately capture the trend in my data, which is inverse parabolic (low probability early, increasing to a peak at the middle of day and decreasing at the end). My test data is as follows, I just supplied the `ggplot` and `stat_smooth()` curve I am looking at vs. the model itself:               testset <- data.frame(                  hour=c(rep(100,100),rep(200,100),rep(300,100),rep(400,100),rep(500,100)),                    data=c(rbinom(100,1,.25),rbinom(100,1,.3),rbinom(100,1,.5),                           rbinom(100,1,.25),rbinom(100,1,.1)))          ggplot(data=testset,aes(hour,data))+stat_smooth(method=glm,family=binomial)      Notice that is I allow the stat_smooth function to default to a fitting/smoothing method (loess in this case), the representation is closer to the actual data. I know I am missing something terribly simple, or misunderstanding something in a silly way, but that is why I am asking the question, search didn't help.               ggplot(data=testset,aes(hour,data))+stat_smooth()


title: Do "true" multi-level models require Bayesian methods?

text: I've been recently learning about mixed effects models (e.g. via Fitzmaurice, Laird, and Ware 's book _Applied Longitudinal Analysis_ ) as well as Bayesian hierarchical models (e.g. via Gelman and Hill's book _Data Analysis Using Regression and Multilevel/Hierarchical Models_ ) One curious thing I've noticed: The Bayesian literature tends to emphasize that their models can handle covariates at multiple level of analysis. For example, if the clustering is by person, and each person is measured in multiple "trials," then the Bayesian hierarchical models can investigate the main effects of covariates both at the subject and trial level, as well as interactions across "levels." However, I have not seen these kinds of models in the textbooks introducing frequentist methods. I'm not sure if this is a coincidence, or an example of where Bayesian methods can do "more complicated things." Is it possible to use mixed effects models (e.g. the lme4 or nlme packages in the R statistical software) to investigate interactions of covariates across "levels" of analysis?


title: Profile likelihood of N in binomial model

text: Let $(n_1,...,n_k)$ be a sample from a Binomial$(N,p)$ where both parameters are unknown. In many cases, the profile likelihood of $N$ is asymptotic in the sense that it never decays to $0$. An example of this is given here. The following code shows the profile likelihood of $N$ for this data set.               data = c(16, 18, 22, 25, 27)     prof = function(n) sum(dbinom(data,n,mean(data)/n,log=TRUE))     n = points = seq(25,500,1)     for(i in 1:length(n)) points[i] = exp(prof(n[i]))     m = max(points)     plot(n,points/m)      What can we do from a classical viewpoint in such scenario? Shall we just say that a likelihood-confidence interval is $(N_0,\infty)$? Note: Using the Bayesian approach, the posterior of $N$ is going to decay to $0$ for any prior, due to its integrability. The issue here is that the inferences are strongly influenced by the prior given that, in vague words, the likelihood has little information about $N$.


title: Accept rate in Metropolis–Hastings algorithm

text: In the Metropolis–Hastings algorithm for sampling a target distribution. let   * $\pi_{i}$ be the target density at state $i$,   * $\pi_j$ be the target density at the proposed state $j$,   * $h_{ij}$ be the proposal density for transition to state $j$ given current state $i$,    * $a_{ij}$ be the accept probability of proposed state $j$ given current state $i$. Then by the detailed balance equation, after choosing the proposal density $h$, the accept probabilities $a$ is computed as $$ a_{ij} = \min(1, \frac{\pi_{j} h_{ji}}{\pi_{i} h_{ij}}). $$ If $h$ is symmetric, i.e. $h_{ij}=h_{ji}$, then $$ a_{ij} = \min(1, \frac{\pi_{j}}{\pi_{i}}). $$ When $h_i$ is a Guassian distribution centered at state $i$ and has the same variance $\sigma^2$ for all $i$, $h$ is symmetric. From Wikipedia: > If $\sigma^2 \,$ is too large, almost all steps under the MH algorithm will > be rejected. On the other hand, if $\sigma^2 \,$ is too small, almost all > steps will be accepted. I wonder why the accept probability changes in the reverse direction of the change of variance of the proposal density, as mentioned in the above quote? Thanks!


Find the least squares estimator of the parameter B (beta) in the following regression model: y= B + u What is the variance of the estimator?


title: Graphical and Statistical Tests for Robustness of Sharp RD

text: I'm doing sharp regression discontinuity design with my treatment variable $$ D_i = \begin{cases} 1 \enspace \quad \text{if $x_i \geq \overline{x}$} \\\ 0 \quad \text{otherwise} \end{cases} $$ where $\overline{x}$ is the threshold value of my forcing variable. The treatment is given from a certain income level and the outcome $Y$ is some health related status of the patient. I have done my analysis and gotten pretty good results but now I would like to test the robustness of my results. Perhaps other unobserved factors can affect the outcome (I have many different variables on the patients) or patients could manipulate the threshold value of their income because the threshold is known in advance. How else can I show robustness of my results? So far I have used polynomial RD and RD with local linear regression and the results are stable. Is there more? Should I use some data visualisation methods?


title: Number of components in PCA

text: I believe I have a problem understanding PCA: I would like to use this technique to reduce the number of features of my problem. I originally have 10,000 features and 500 samples. However, the use of PCA will limit my number of principal components to the smallest between the number of samples (columns of my data matrix) and the number of features (rows of this matrix). 100% of variance could therefore be explained by 500 components. But 500 components is far smaller than 10,000 features... How can all the variance be explained by less than the number of samples (which has nothing to do with the number of features)?


title: Stepwise binary logit regression - help for bootstrapping in Stata

text: I am running a stepwise binary logit regression in Stata using 14 independent variables. Two of the independent variables are dummies (assuming a value of 0 or 1). I've tested the independent variables for multicollinearity and adapted them by standardizing or using the natural logarithm of their values in order to mitigate this issue (VIF<2.5). The normal model runs smoothly; however, when I want to bootstrap the sample (# of observations: 73) with 1000 replications I receive p-values of 1.0000. Furthermore, the results conclude with the note: "one or more parameters could not be estimated in 314 bootstrap replicates; standard-error estimates include only complete replications." Two questions: 1\. Is the VIF threshold that I used correct (VIF<2.5)? Which other ways are there to get rid of multicollinearity, without dropping one of the variables? 2\. Since I don't assume that multicollinearity is an issue anymore, what else could I have done wrong concerning my bootstraping methodology? Many thanks in advance for your answer(s)! Best! Tim


title: Clustering prior to multiple linear regression?

text: Assume I have:   N = 100, 5 continues predictor variables and 1 continues response. Now I want to find a mapping between predictors and response using multiple linear regression. Lets assume I know that my 100 samples consist of x subgroups, where for any subgroup I could find a match with very low MSE but for the whole population my MSE would increases drastically. Is there a standardized way to find out whether these subgroups exist?


title: How to compare series of foreclosure rates wrt to time and place

text: I have three categories of foreclosure rates in NYC (rates are number of foreclosures per 1000 homes). The three categories are: overall rates of foreclosure, rates for homes within a flood zone, and rates for homes outside of flood zones. The rates are monthly for the last 2.5 years. The rates are provided at the borough/county level (Bronx, Brooklyn, Manhattan, Queens, Staten Island). What statistical process can determine if there is a significant difference in the rates of foreclosure both city-wide and across boroughs for homes situated in a flood zone compared to homes situated outside of a flood zone? Any pointers in the right direction would be appreciated. Thanks. Example data (sorry for the formatting)   Overall   Month   Bronx Brooklyn Manhattan Queens Staten Island New York City   12-Jan  0.91    0.99        0.20            0.83      0.55               0.77   12-Feb  1.08    0.86        0.17            0.84      0.78               0.78 In Flood Zone   Month   Bronx Brooklyn Manhattan Queens Staten Island New York City   12-Jan  0.96    0.90        0.15            0.81      0.43               0.71   12-Feb  1.20    0.66        0.37            0.81      0.64               0.67 Outside Flood Zone   Month  Bronx Brooklyn Manhattan Queens Staten Island New York City   12-Jan 0.91    1.01        0.21            0.83      0.56               0.78   12-Feb 1.08    0.91        0.14            0.84      0.80               0.80


title: Difference of two independent gamma distribution

text: Given two independent random variables $X\sim\Gamma(s,r)$ and $Y\sim\Gamma(t,u)$, what is the distribution of the difference, i.e. $D=X−Y$? I assume that $s$ and $t$ are integers. How can I obtain the skewness of the difference distribution?


title: How did Karl Pearson come up with the chi-squared statistic?

text: How did Pearson come up with the following Pearson chi-squared statistics in 1900? $$ K = \sum \frac{(O_{ij} -E_{ij})^2}{E_{ij}} $$ that $$ K \sim \chi^2 $$ Did he have chi-squared in mind and devise the metric $K$ (bottom-up approach), or did he devise the statistic and later prove that it follows the chi-squared distribution (top-down)? I want to know why he chose that specific form and not others such as $\sum(O_{ij} -E_{ij})^2$ or $\sum|O_{ij} -E_{ij}|$, and also why he divided the square with the denominator.


title: Coverage Proof of Confidence Intervals

text: The confidence interval for the mean of a random variable $Y$ has coverage $1-\alpha$ which I am trying to show. Starting from $$\widehat{E(Y)} - q_{1-\frac{\alpha}{2}}\sqrt{\frac{\widehat{Var(Y)}}{n}} \leq E(Y) \leq \widehat{E(Y)} + q_{1-\frac{\alpha}{2}}\sqrt{\frac{\widehat{Var(Y)}}{n}} $$ I re-arranged this to $$- q_{1-\frac{\alpha}{2}} \leq \sqrt{n} \left( \frac{\widehat{E(Y)}-E(Y)}{\sqrt{Var(Y)}} \right) \leq q_{1-\frac{\alpha}{2}} $$ Then by the central limit theorem $\sqrt{n} (\widehat{E(Y)} - E(Y)) \rightarrow N(0,Var(Y))$. Then I use the continuous mapping theorem. Given that $\widehat{Var(Y)} = \frac{1}{n}\sum^{n}_{i=1} Y_i^2 - (\frac{1}{n}\sum^n_i Y_i)^2$, define the function $m:(x,y) \rightarrow x - y^2$. Then $$x \equiv \sum^{n}_{i=1} Y_i^2 \rightarrow E(Y^2)$$ and $$y \equiv \sum^{n}_{i=1} Y_i \rightarrow E(Y)$$ by the weak law of large numbers. Then $\widehat{Var(Y)} = g(x,y) \rightarrow g(E(Y^2),E(Y)) = Var(Y)$ is a consistent estimator of the variance. Hence $$\sqrt{n} \left( \frac{\widehat{E(Y)}-E(Y)}{\sqrt{Var(Y)}} \right) \rightarrow N(0,1)$$ But now I am stuck in how to proceed to obtain the final coverage rate. If someone could give me a hint on how to get there, this would be greatly appreciated. @AlexH Thanks for pointing out the typo. I think I understand the hint. This should be $$ \begin{align} Pr \left(\sqrt{n} \left( \frac{\widehat{E(Y)}-E(Y)}{\sqrt{\widehat{Var(Y)}}} \in (-q_{1-\frac{\alpha}{2}}, q_{1-\frac{\alpha}{2}} \right) \right) &= Pr(N(0,1) \in (-q_{1-\frac{\alpha}{2}}, q_{1-\frac{\alpha}{2}} )) \newline &= Pr(N(0,1) \leq q_{1-\frac{\alpha}{2}}) - Pr(N(0,1) > q_{\frac{\alpha}{2}}) \newline &= \Phi (q_{1- \frac{\alpha}{2}}) - \Phi (q_{\frac{\alpha}{2}}) \newline &= 1 - \frac{\alpha}{2} - \frac{\alpha}{2} \newline &= 1 - \alpha \end{align} $$


title: Polynomials and NSA

text: I'm looking for some applications of criteria of irreducibility of integer polynomials inside and outside mathematics. I was reading the CV of Filaseta, a great researcher in this area, and he has gained some grants from the NSA. This makes me wonder if there are applications of this matter to computer science. Which are the applications of criteria of irreducibility such as Eisenstein, Dumas criterion, etc. to the real world or other areas of mathematics? Thanks.


title: Adding time variable?

text: Good day, guys. I'm new to SPSS and I plan to determine if there is any relationship between grades and time spent in watching tv/video game playing using Spearman's rank correlation. Question is, how do I input time variable? Do I convert say, 2 hours and 30 mins to mins so that I would put 150 under the time tab? My research led me to this but I don't really get it: "The answer is making the data into range whereby all the range has equal measurement( I mean for example 45mn in a range so that I would be [0-45] [46-90] and so on....so you have already done with data arrangement. Last but not least give integer to them (1=[0-45],2=[46-90].......) and you can finally enter to spss." Any help would be extremely appreciated. :) (And if possible, with explanation. Thank you soooo much :D)


title: How to propositionalize a relational data set for clustering analysis?

text: I am working with a data set of students and their courses for a single semester, attempting to cluster based on the courses & various other attributes where "courses" are the "many" side of a one-to-many relationship. If it matters, I have R, JMP and RapidMiner at my disposal. I can propositionalize the data set in into a single table structure in a few ways, but nothing that makes sense to me for one reason or another: I can structure the data set with multiple polynomial attributes that can take the value of any course name, e.g.,               STUDENT course1 course2 course3 course4 course5 GPA TEST1 TEST2 TEST3        1     ENGL2   MATH1   HIST1   ACCT1   BIOL2  3.2  87    342   18        2     ENGL1   CHEM2   COMP1   HIST1    N/A   3.5  93    312   22      etc. But the order of the courses doesn't matter, e.g., it doesn't matter if HIST1 appears in course3 or course4. Alternatively, I could split out all courses into binary values, e.g.,               STUDENT  GPA TEST1 TEST2 TEST3 ACCT1 BIOL2 CHEM2 COMP1 ENGL1 ENGL2 HIST1 MATH1        1     3.2   87   342   18     1     1     0     0     0     1     1     1        2     3.5   93   312   22     0     0     1     1     1     0     1     0      However, in the full data set this would result in about 400 attributes. That seems pretty inefficient, and data sets with far fewer attributes have choked my system's 16gb of RAM. So, I'm not sure how to proceed. Any insight on how such one-to-many structures are commonly dealt with would be useful. Thanks!


title: Why are intervals for means narrower than intervals for individual observations?

text: I am looking at problem 5.01 relating to Galton's data on adult heights of fathers and sons, from **Chapter 5 of Statistical Modelling: A Fresh Approach** _link to this book:_ http://www.mosaic- web.org/go/StatisticalModeling/index.html _Can anyone explain me why Confidence interval is so different that coverage interval?_ **Prob 5.01. The mean of the adult children in Galton’s data is**               > mean( height, data=Galton )      [1] 66.76069      > Had Galton selected a different sample of kids, he would likely have gotten > a slightly different result. The confidence interval indicates a likely > range of possible results around the actual result. > > Use bootstrapping to calculate the 95% confidence interval on the mean > height of the adult children in Galton’s data. The following statement will > generate 500 bootstrapping trials.               trials = do(500) * mean(height, data=resample(Galton) )       > **(a) What is the 95% confidence Interval on the mean height?** > > **A. 66.5 to 67.0 inches.**   >  B. 66.1 to 67.3 inches.   >  C. 61.3 to 72 inches.   >  D. 65.3 to 66.9 inches. > > **(b) A 95% coverage interval on the individual children’s height can be > calculated like this:**               qdata(c(0.025,0.975), height, data=Galton)      2.5% 97.5%       60   73      > **Q: Explain why the 95% coverage interval of individual children’s heights > is so different from the 95% confidence interval on the mean height of all > children?**


title: How to deal with invalid data values such as with age (e.g., -99, 0, F1)?

text: I have a data set that consists of 15 age values. I want to clean the data before doing anything further. I have a few questions about data cleaning and data integrity. What is the best treatment if the dataset contains an illegal value (like '${\rm F1}$', e.g.)? I thought taking the mean of the 14 remaining values and doing a mean substitution might be a good idea. If there are values that cannot be right, such as $-99$ (no one can be $-99$ years old), is the best course of action a subjective treatment? Or should I use a statistical method, such as mean substitution? It seems logical to me to turn the $-99$ to a $+99$, and treat it as a data entry error. Can $0$ be an age? The data set contains a $0$ value. I am mostly looking to know if there is any formalised approach to dealing with these type of problems.


title: General linear model with a quantitative covariate in the presence of a repeated-measures factor

text: Suppose that we have a response variable that is measured on each subject. There are 2 categorical variables (factors): one between-subject factor Group with two levels (control and treatment) and one within-subject factor Category with three levels (high, moderate, and low). Such a 2 $\times$ 3 mixed ANOVA can be analyzed through a general linear model (GLM) by dummy coding the factors. To demonstrate the GLM formulation, assume there are six subjects, three in each group, and we have the following equation with effect coding: $$ \begin{bmatrix} y_{11} \\\ y_{12} \\\ y_{13} \\\ y_{21} \\\ y_{22} \\\ y_{23} \\\ y_{31} \\\ y_{32} \\\ y_{33} \\\ y_{41} \\\ y_{42} \\\ y_{43} \\\ y_{51} \\\ y_{52} \\\ y_{53} \\\ y_{61} \\\ y_{62} \\\ y_{63} \end{bmatrix} = \begin{bmatrix} 1 & 1 & 1 & 0 & 1 & 0 & 1 & 0 & 0 & 0\\\ 1 & 1 & 0 & 1 & 0 & 1 & 1 & 0 & 0 & 0\\\ 1 & 1 & -1 & -1 & -1 & -1 & 1 & 0 & 0 & 0\\\ 1 & 1 & 1 & 0 & 1 & 0 & 0 & 1 & 0 & 0\\\ 1 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 0 \\\ 1 & 1 & -1 & -1 & -1 & -1 & 0 & 1 & 0 & 0 \\\ 1 & 1 & 1 & 0 & 1 & 0 & -1 & -1 & 0 & 0\\\ 1 & 1 & 0 & 1 & 0 & 1 & -1 & -1 & 0 & 0\\\ 1 & 1 & -1 & -1 & -1 & -1 & -1 & -1 & 0 & 0 \\\ 1 & -1 & 1 & 0 & -1 & 0 & 0 & 0 & 1 & 0 \\\ 1 & -1 & 0 & 1 & 0 & -1 & 0 & 0 & 1 & 0\\\ 1 & -1 & -1 & -1 & 1 & 1 & 0 & 0& 1 & 0 \\\ 1 & -1 & 1 & 0 & -1 & 0 & 0 & 0& 0 & 1 \\\ 1 & -1 & 0 & 1 & 0 & -1 & 0 & 0& 0 & 1 \\\ 1 & -1 & -1 & -1 & 1 & 1 & 0 & 0& 0 & 1 \\\ 1 & -1 & 1 & 0 & -1 & 0 & 0 & 0& -1 & -1 \\\ 1 & -1 & 0 & 1 & 0 & -1 & 0 & 0& -1 & -1 \\\ 1 & -1 & -1 & -1 & 1 & 1 & 0 & 0& -1 & -1 \end{bmatrix}\\\ \begin{bmatrix} \beta_0 \\\ \beta_1 \\\ \beta_2 \\\ \beta_3 \\\ \beta_4 \\\ \beta_5 \\\ \beta_6 \\\ \beta_7 \\\ \beta_8 \\\ \beta_9 \end{bmatrix}+\begin{bmatrix} \epsilon_{11} \\\ \epsilon_{12} \\\ \epsilon_{13} \\\ \epsilon_{21} \\\ \epsilon_{22} \\\ \epsilon_{23} \\\ \epsilon_{31} \\\ \epsilon_{32} \\\ \epsilon_{33} \\\ \epsilon_{41} \\\ \epsilon_{42} \\\ \epsilon_{43} \\\ \epsilon_{51} \\\ \epsilon_{52} \\\ \epsilon_{53} \\\ \epsilon_{61} \\\ \epsilon_{62} \\\ \epsilon_{63} \end{bmatrix} $$ The $F$-statistic for each effect (e.g., main effects and interactions) is formulated with terms each of which is associated with a partial model with some columns removed from the model matrix. Suppose I would like to also consider each subject's body weight as a covariate. I barely see any discussions of modeling such a covariate under the GLM framework. This is not really an ideal approach because it would presume that all the three categories have the same body weight effect. So what is exactly the difficulty here when a covariate is incorporated in a GLM? Broken orthogonality or something else?


title: Distribution of arrival times to server for an M/M/1 queue (what the server experiences)

text: In an M/M/1 queue, we know that inter-arrival times are exponentially distributed, and that service times are the same. What is the distribution of to-server inter-arrival times (aka service start times)? Put another way, what is the distribution of times between when a server starts helping a customer? Some intuition: if $\lambda \ll \mu$, then there is rarely a wait, and the server just experiences arrivals at the same rate as arrivals to the queue. if $\lambda \approx \mu$, then the server is almost always busy and experiences arrivals at the same rate of service/arrivals. However, when $\lambda < \mu$, there will be cycles where customers arrive to both empty and occupied lines. So, the server will at times see the true arrival rate $\lambda$, and at other times, arrivals to them will be $\mu$ Burke's Theorem shows that the distribution of departure time from a server is the same as arrival times. One proof this theorem use weighted sums, and the probability that the server is busy. I think there may be a similar approach to solve this problem.


title: Bonferroni correction for two different tests on the same dataset

text: I am having trouble figuring out the correction factor to choose for a Bonferroni correction. Let me explain. I have two datasets, control ($x$) and data from left ($y_1$, affected limb) and right limb ($y_2$, non-affected limb) of a pathology group. As $y_1$ and $y_2$ were from the same subject (paired data), I have performed a Wilcoxon signed-rank test to compare if these data were statistically different. However, when I compared the data between $x$ and $y_1$ or, between $x$ and $y_2$, I used a Wilcoxon Rank-Sum Test. As there are multiple tests, should I use correction factor of 3 ($p<=\frac{0.05}{3}$)? Or ($p<=\frac{0.05}{2}$) for group ($x$ and $y_1$) and ($x$ and $y_2$) only and no correction for group ($y_1$ and $y_2$)? I would really appreciate your kind reply. Thank you.


title: When to report proportions and when to report median + IQR for ordinal variable

text: A colleague brought to my attention documentation for a standardised questionnaire which recommended using the median and interquartile range to describe summary variables created by combining responses to multiple questions. In most instances these summary variables had 20+ possible values (e.g. the sum of responses to 5 questions each on a 0-4 scale resulting in a 0-20 scale). Strangely, this advice also applied to single item summaries of yes/no questions - i.e. the new summary variable is just the response to a yes/no question. Clearly, for a binary variable such as this the median + IQR gives much less information than reporting the proportion of 'yes' and 'no' responses. My question is - how many levels would a variable require for the median + IQR to be an informative summary of the central tendency and spread of that variable?


title: Issue With class imbalances in classification problem

text: I am working on a word sense disambiguation problem. Specifically, I am using decision lists to classify the ambiguous word. Decision lists work in the following way. The quantity $$log(\frac{P(sense_1 | collocation)}{P(sense_2 | collocation)})$$ is calculated for the ambiguous word. And if my math is correct, $$\frac{P(sense_1|collocation)}{P(sense_2|collocation)}=\frac{\frac{count(collocation,sense_1)}{count(collocation)}}{\frac{count(collocation,sense_2)}{count(collocation)}}=\frac{count(collocation,sense_1)}{count(collocation,sense_2)}$$ So if the absolute value of the log of this quantity is above a certain threshold then a decision rule is made (based on the sign on the log of this quantity). The issue that I have is that this quantity does not take into account the total number of training examples of each class. For example, what if sense_1 has 100 examples and 99 of them have 'collocation' in them and sense_2 has 10000 examples and 105 of them have 'collocation' in them. In that case, 'collocation' would more likely be associated with sense_2 which would not be correct intuitively. Would it be better to scale these counts by the total size of the particular class in the training data?


title: Distribution family for a ratio dependent variable in a generalized estimating equation

text: I have several dependent variables that are measures of racial disproportionality; I've calculated them as: % of events caused by racial minority group / % of events caused by racial majority group I have a dependent variable for each racial minority group in my sample. I am running longitudinal Generalized Estimating Equations (GEE) on these models, however I am somewhat stumped as to which family is appropriate for these dependent variables. The probability range for my ratios are truncated at 0, as it's not possible to have negative values in my DVs. This makes me question the validity of using a Gaussian family for my models. The idea behind these variables is that a ratio greater than 1 indicates some level of greater burden of events that a given racial minority is bearing compared to the racial majority, and a ratio less than 1 indicates the opposite.   * What would be the most appropriate family to use for my GEE regressions? EDIT: I misspoke about the racial disproportionality measure I was using. The correct formula is: % events by minority / % of total enrollment that is minority OVER % events by non-minority / % of total enrollment that is non-minority Because they are ratios, the number of observations with value less than 1 is comparable to the number of observations greater than 1, with the lower bound being 0 and the upper bound being non-bounded. Looking at the histograms of my response variables, they definitely seem to fit a negative binomial distribution better than the normal. The QIC (GEE adjustment to AIC) confirms this suspicion. My questions now are:   * Can I trust this evidence to move forward with the negative binomial family?   * If so, how do I possibly interpret the exponentiated coefficients from the resulting models? They don't see to be Incidence Rate Ratios, as one would interpret them to be from count variables...


title: Are short time series worth modelling?

text: Here is some context. I am interested in determining how two environmental variables (temperature, nutrient levels) impact the mean value of a response variable over a 11 year period. Within each year, there is data from over 100k locations. The goal is to determine whether, over the 11 year period, the mean value of the response variables has responded to changes in environmental variables (e.g. warmer temperature + more nutrients would = greater response). Unfortunately, since the response is the mean value (without looking at the mean, just regular inter-annual variation will swamp the signal), the regression will be 11 data points (1 mean value per year), with 2 explanatory variables. To me even a linear positive regression will be hard to consider as meaningful given that the dataset is so small (does not even meet the nominal 40 points/variable unless the relationship is super strong). Am I right to make this assumption? Can anyone offer any other thoughts/perspectives that I may be missing? Thanks! PS: Some caveats: There is no way to get more data without waiting additional years. So the data that is available is what we really have to work with.


title: Measuring distance between geographic coordinates among other variables

text: I am setting up a quasi-experimental design and I need to compare each treatment account to all potential control accounts within a certain geographic region. I would like measure the distance between a treatment account and each neighboring control account along with their energy usage to help select the most similar pair. Obviously the variables (EW coord, NS coord, and energy usage) are all on different scales, does this mean I have to standardize each variable first? Also, do I need to account for a correlation among variables—-I would think geographic coordinates would get close to zero weight because they orthogonal to each other. If I choose to use a Mahalanobis distance or some other statistical measure, how would I compute the covariance matrix? I know how to in a typical multivariate setting, but I am looking at a handful of control 3-tuples relative to one treatment 3-tuple. Would I just treat the treatment 3-tuple as the mean vector and use those deviations to compute a covariance matrix?


title: Need help for which test to choose?

text: I am studying on cognition and behaviors. I am lecturing linguistic courses and teaching linguistic terminology with visuals. In my research, just before each course, I asked participants 20 behavioral yes-no questions like "have you ever been bitten by a dog", or "have your ever watched Seinfel". Then I start my couse and throughout the course, I showed them some pictures which is related with linguistic terminology. These visuals are in accordance with these questions like "a picture of a pittbull" or "a scene from Seinfeld". Then at the end of the course, they are required to place these pictures on certain linguistic terminology they are related. I wonder about their retention, which pictures they remember and place correctly. My hypothesis is that 20 questions that I asked them is effective on their choices of visuals. For instance, watching seinfeld make it easier to remember it again. I prepared my data in SPSS, yes-no questions are 1-0, and their retention test is as wrong and correct, 1-2. How can I measure the cause and effect relationship between these dichotomous variables, which test should I use to measure the effect of 20 behavioral items on correct and wrong retention. I have 16 participants. Please, I really need help, I know I cannot bivariate. Thank you very much.


title: Combining standard errors of fit parameters

text: I have fitted a 3 parameter (mean, sigma & tau) model to my data and have also computed the standard error for each of them. The statistic of interest for my data is the sum of mean and tau. My question is how do I estimate the total error for this new statistic? I know I can't simply add up the errors for mean and tau. Any help would be greatly appreciated. Thanks in advance.


title: Nested factor variable: How to balance the number of observations with number of levels of a factor?

text: I am building a regression model and I'm having an issue with how to incorporate geographical information into the model as a predictor in the model. I have multiple geographical levels of granularity: ZIP, CBSA (Core Based Statistical Area), and state. The idea is to reduce geography to as granular a level as possible without splitting up geography into a bajillion different factors. Ideally, we'd just break everything down to the ZIP level, but we don't have enough observations at that level. Basically the problem is balancing the number of factors with observations per factor. Is there a mathematical way to figure out the right number of observations at the ZIP / CBSA / State level to lead to the best model performance? Right now I'm doing a hackish way trying different combinations of observations per geographical level to see what leads to the best predictions.


title: Standard error of proportion

text: In the first example of http://statisticshelp.com/confidence-interval Standard error of proportion is $$\frac{\sqrt {p(1-p)}}{n}$$ Why ?


title: Differentiation in support vector machine

text: ![enter image description here](http://i.stack.imgur.com/yzcGx.png) My Problem is We want to minimize 1/2 Norm(w)^2 under suitable constraints y(i){[wTxi+b}-1}..we write Lagrangian and we get the above equations .. I have a problem in differentiating with respect to w can somebody elaborate on that...how do we get first equation.


title: Numerical differentiation (derivative) and downsampling

text: I have some time course data which I would like obtain the first derivative of. As it seems quite difficult to model, I do not intend to fit a function to it, but rather compute the first derivative numerically (taking the difference of each measure from the one before via the `numpy.diff` function). Now, I also want to down-sample my data to get rid of some noise. I have tried to calculate whether it makes a difference if I derive first and downscale that or downscale first and derive that. But however I try to look at it I can't seem to put the same variables on both sides of my equation (first side - how I believe differentiation and subsequent down-sampling work, and on the second side the converse). $ \frac{(b-a)+(c-b)}{2} = \frac{c+d}{2}-\frac{b+a}{2} $ Could you help me out determine whether it matters in what order I perform my operations?


title: What is the fastest unsupervised feature learning algorithm?

text: I took a look at several unsupervised feature learning algorithms. Most of them (restricted Boltzmann machines and sparse auto-encoders) have very long training times even on small datasets like MNIST. I wonder if there are similar algorithms that can be trained in less time? Similar algorithms that seem to be promising might be sparse filtering and reconstruction cost ICA (RICA). Are there maybe more? Another problem with some of these algorithms is: most of them require batch training with algorithms like L-BFGS. In an online setting it would most probably be hard to train them without caching a batch of instances for training. Are there alternatives? Is there anything in the literature? I could not find anything.


title: Regression without obvious dependent variable

text: Say I have 4 variables: Anxiety, Depression, Memory and Attention. I've given tests to subjects and now have a list of scores for each variable. Say I want to know the relation between cognitive capacities (Memory and Attention) and mood (Depression and Anxiety). The problem here is that there is no way to select a theoretically supported dependent variable in my case. Bad cognitive faculties lead to worse mood, but bad mood also causes worse cognitive functioning. So now I'm wondering: how do I model this thing? Should I just select the 2 mood variables as dependents and run Memory and Attention as predictors? If I reverse this does it matter for the results? Or does it only matter for the interpretation of the results? I feel like I'm struggling with a very obvious problem but cannot figure out what to do. My only other thought was that regression simply does not apply to a model without direct effects and I'm doing this thing the whole wrong way.


title: LIBSVM evaluation functions for probability estimation

text: I found this tool that is providing some well known evaluation functions like precision and recall. Unfortunately this tool supports only binary-class C-SVM. I was wondering where I can find similar evaluation functions that are known to be appropriate for probability estimation SVM? If there is a LIBSVM extension tool like the above one would be even better. Otherwise I'd be happy to read more about evaluation functions for probability estimation SVM. Thanks!


title: Number of parameters in Markov model

text: I want to use BIC for HMM model selection:               BIC = -2*logLike + num_of_params * log(num_of_data)      So how do I count the number of parameters in the HMM model. Consider a simple 2-state HMM, where we have the following data:               data = [1 2 1 1 2 2 2 1 2 3 3 2 3 2 1 2 2 3 4 5 5 3 3 2 6 6 5 6 4 3 4 4 4 4 4 4 3 3 2 2];     model = hmmFit(data, 2, 'discrete');     model.pi = 0.6661    0.3339;     model.A =          0.8849    0.1151         0.1201    0.8799     model.emission.T =          0.2355    0.5232    0.2259    0.0052    0.0049    0.0053         0.0053    0.0449    0.2204    0.4135    0.1582    0.1578     logLike = hmmLogprob(model,data);     logLike =  -55.8382      So I think:               Nparams = size(model.A,2)*(size(model.A,2)-1) +                size(model.pi,2)-1) +                size(model.emission.T,1)*(size(model.emission.T,2)-1)     Nparams = 13      So at the end we have:               BIC = -2*logLike + num_of_params*log(length(x))     BIC = 159.6319      I've found a solution where the formula for `num_of_params` (for simple Markov model) looks like:               Nparams = Num_of_states*(Num_of_States-1) - Nbzeros_in_transition_matrix      So what's the right solution? Do I have to take into account some zero probabilities in transition or emission matrices? ====Updated since 07.15.2011==== I think I can provide some clarification on the impact of data dimension (using “Gaussian mixture distribution” example) X is an n-by-d matrix where (n-rows correspond to observations; d-columns correspond to variables (Ndimensions).               X=[3,17 3,43        1,69 2,94        3,92 5,04        1,65 1,79        1,59 3,92        2,53 3,73        2,26 3,60        3,87 5,01        3,71 4,83        1,89 3,30 ];     [n d] = size(X);      n = 10; d =2;      The model will have the following number of parameters for GMM:               nParam = (k_mixtures – 1) + (k_mixtures * NDimensions ) + k_mixtures * Ndimensions  %for daigonal covariance matrices     nParam = (k_mixtures – 1) + (k_mixtures * NDimensions ) + k_mixtures * NDimensions * (NDimensions+1)/2; %for full covariance matrices      If we treat X as **1-dimentional data** , than we have `num_of_data = (n*d)`, so for the **2-dimentional** data we have `num_of_data = n`. **2-dimentional** data: nParam = 11 ; logLike = -11.8197; BIC = 1.689 **1-dimentional** data: nParam = 5 ; logLike = -24.8753; BIC = -34.7720 I have a very little practice with HMM. **Is it normal to have HMM with (5000, 6000 and more parameters)?**


title: ANOVA with some paired and some unpaired subjects

text: I am a biologist rather than statistician so I might lack some insight. I have measurements on two paired groups (baseline and follow-up) along with a separate group of healthy controls. In addition the design of the study is unbalanced (n = 12 for B and F and n = 5 for HC). ### Question   * Is there an ANOVA technique that would allow me to carry out a fair ANOVA taking the inherent pairing into account for groups Baseline and follow-up? I've done some analysis already with paired t-tests for baseline versus follow-up and then unpaired tests for each group versus the healthy controls but I wondered whether I was missing a more elegant technique.


title: How to get expectation (E-value) for a dataset?

text: For an examination, scores for 10 students (all from class 4B) were obtained. I want to convert each score to E-value. If I understand correctly, to calculate E-value I have to determine an unrelated population first, so, I also got scores for 30 students in Class 4A. For discussion, I have generated two datasets here:               class4B <- c (0, 10, 20, 30, 50, 40, 70, 80, 90, 99)     class4A <- c(1, 3, 3, 3, 5, 6, 7, 2, 1, 90, 12, 14, 15, 44, 66,  88, 67, 78, 90, 45,                   67, 99, 100, 12, 45, 56, 100, 23, 45, 29)        * Would you mind to let me know how to get the TEN E-values for each students in Class4B with R?   * Should we consider the population in class4A?    * Is it a statistically sound approach to get E-value based on an unrelated population or so called "random population"?


title: How do I determine which set of measurements is better?

text: I am measuring protein in humans using two different types of measurement techniques, X and Y (measured on different scales). I have two replications for type X and four for type Y. I average the two measurements for X and average the four measurements for Y to get the 'true' protein average. There are about 120 subjects. There is large variation in the replication measurements and, surprisingly, the relationship between the average X and the average Y values is not very strong (r = .36). I am trying to determine which measurement technique is better. I would like to use the subject's level of protein as a predictor in a regression model. I have thought of looking at the coefficient of variations and Cronbach's alphas. Should these two results be consistent? Do they apply here? What would you recommend? Example Data: ![enter image description here](http://i.stack.imgur.com/l6jfL.png) Update: Thanks everyone for the thoughtful responses. I've found out more about the data (it was kind of just thrown at me). The protein measurements were done on consecutive days (so X had two days and Y had 4 days). So naturally someones protein varies from day to day which explains the variability in the measurements. The average is then used as an estimate of their current protein intake. Right now this a common approach for measuring a subjects protein level, since there is no way to know what someones current level of protein intake truly is. I don't like choosing the protein measurement that correlates strongest with the outcome. I don't think the reviewers will like this justification. Looking at various reliability measurements seems like a better approach to me (I prefer the coefficient of variation). I always thought cronbach's alpha is used for reliability of a latent construct (you could make an argument here I suppose that protein is latent). Also the correlation is indeed low between X and Y. However I computed that correlation ignoring the the measurement error evident in both X and Y. Correcting for this will lead to a stronger correlation. I have also learned that data collection was sloppy, so not having a strong correlation is not surprising.


title: Advantages of approaching a problem by formulating a cost function that is globally optimizable

text: This is a rather general question (i.e. not necessarily specific to statistics), but I have noticed a trend in the machine learning and statistical literature where authors prefer to follow the following approach: **Approach 1** : Obtain a solution to a practical problem by formulating a cost function for which it is possible (e.g. from a computational standpoint) to find a globally optimal solution (e.g. by formulating a convex cost function). rather than: **Approach 2** : Obtain a solution to the same problem by formulating a cost function for which we may not be able to obtain a globally optimal solution (e.g. we can only get a locally optimal solution for it). Note that rigorously speaking the two problems are different; the assumption is that we can find the globally optimal solution for the first one, but not for the second one. Other considerations aside (i.e. speed, ease of implementation, etc.), I am looking for:   1. An **explanation** of this trend (e.g. mathematical or historical arguments)   2. **Benefits** (practical and/or theoretical) for following Approach 1 instead of 2 when solving a practical problem.


title: Spearman or Kendall correlation?

text: I would like to assess the correlation between a 7-category ordinal variable (X) and a number of other variables some of which are ordinal with 3-6 categories, others are continuous and a couple are dichotomous. The dataset includes only 24 observations and lack of normality and tied observations are therefore issues to take into account.   * Would it be best to use the Kendall coefficient to assess the correlation between X and each of the other variables? If so, which one (i.e. tau-a or tau-b; I know that Roger Newson favours the former)?    * Would it be reasonable to use Spearman as well?    * And what would be the best test to assess the correlation with each of the dichotomous variables?


title: Robust correlation in R, directionality

text: I'm trying lmrob, cor.mve and cor.mcd in R for robust correlation of two number series (of the same length) A and B. I get different results if I put A~B or A,B than if I put B~A, or B,A. Might someone guide me as to why? thanks


title: Automatic threshold determination for anomaly detection

text: I am working with a time series of **anomaly scores** (the background is anomaly detection in computer networks). Every minute, I get an anomaly score $x_t \in [0, 5]$ which tells me how "unexpected" or abnormal the current state of the network is. The higher the score, the more abnormal the current state. Scores close to 5 are theoretically possible but occur almost never. Now I want to come up with an algorithm or a formula which automatically determines a **threshold** for this anomaly time series. As soon as an anomaly score exceeds this threshold, an alarm is triggered. The frequency distribution below is an example for an anomaly time series over 1 day. However, it is **not** safe to assume that every anomaly time series is going to look like that. In this special example, an anomaly threshold such as the .99-quantile would make sense since the few scores on the very right can be regarded as anomalies. ![enter image description here](http://i.stack.imgur.com/Ieo7b.jpg) And the same frequency distribution as time series (it only ranges from 0 to 1 since there are no higher anomaly scores in the time series): ![time series](http://i.stack.imgur.com/0ZPoe.jpg) Unfortunately, the frequency distribution might have shapes, where the .99-quantile is **not useful**. An example is below. The right tail is very low, so if the .99-quantile is used as threshold, this might result in many false positives. This frequency distribution **does not seem to contain anomalies** so the threshold should lie outside the distribution at around 0.25. ![enter image description here](http://i.stack.imgur.com/gZw9a.jpg) Summing up, the difference between these two examples is that the first one seems to exhibit anomalies whereas the second one does not. From my naive point of view, the algorithm should consider these two cases:   * If the frequency distribution has a large right tail (i.e. a couple abnormal scores), then the .99-quantile can be a good threshold.   * If the frequency distribution has a very short right tail (i.e. no abnormal scores), then the threshold should lie outside the distribution. /edit: There is also no ground truth, i.e. labeled data sets available. So the algorithm is "blind" against the nature of the anomaly scores. Now I am not sure how these observations can be expressed in terms of an algorithm or a formula. Does anyone have a suggestion how this problem could be solved? I hope that my explanations are sufficient since my statistical background is very limited. Thanks for your help!


title: Values for integral of square of standard Brownian process

text: I am trying to generate values in a table for the following function: $$ W = \int_0^1 [B(t)]^2 dt $$ Where $B(t)$ is a standard Brownian motion. Example: $W_{0.05} = 1.656$, $W_{0.025} = 2.135$. This is taken from a table C.6 in 'Survival Analysis' by Klein & Moeschberger. (They don't give a reference as to how values were computed). Am using `R` for this:               BM2 <- function (x) (rnorm(n=1, mean=0, sd=x))^2      But of course the standard `integrate(BM2, lower=0, upper=1)` doesn't work for this: `evaluation of function gave a result of wrong length`. Looked at this similar question is similar but it's clearly not as simple as: $$ W_t = \frac{t^3}{3} $$ I'm guessing the function is integrated by Ito's method but can't seem to find any worked examples similar to the formula above. Any help or pointers (particularly in `R` which doesn't seem to have an existing function for this) would be greatly appreciated, particularly if the numeric values given above could be reproduced. UPDATE 1/29: Thanks @ThePawn. Question corrected. I like the resampling method and it does produce _reasonable_ approximations. At first I thought this was due to the sample size or loop iterations being too small. Function doesn't appear to 'vecorize' easily in `R` so I rewrote it in C++. Using samples >10,000 or dt <0.0001 doesn't tend to improve things much (which I should probably have guessed) and gets rather slow. The method doesn't seem to give precision beyond 1 significant figure (although this is certainly good enough for a practical application):               require(rcpp)     require(inline)          src <- "     arma::colvec b1 = Rcpp::as<arma::vec>(B);     arma::colvec w1 = Rcpp::as<arma::vec>(B);     double nb1 = b1.size();     double dt1 = as<double>(dt);     double dtrt1 = pow(dt1, 0.5); // square root of dt     int lt1 = as<int>(lt);          for (int i=0; i<lt1; i++){     RNGScope scope; // reset random number generator each time     arma::colvec rn1 = rnorm(nb1, 0, dtrt1);     b1 += rn1;     w1 += pow(b1, 2) * dt1;     }          return Rcpp::wrap(w1);     "          f1 <- inline::cxxfunction(signature(B="numeric",lt="integer",dt="double"),      src, plugin = "RcppArmadillo")          n <- 1e4 # sample size     B <- rep(0, n) # hold result     dt <- 1e-5     T <- seq(0, 1, dt)     lt <- length(T)          B <- f1(B,lt,dt)          sum(B >2.135)/n # approx = 0.025     sum(B >1.656)/n # approx = 0.05      @probabilityislogic - not sure what limiting process I'm using, open to suggest there. Will try implementing the formula you suggest but looks like it will be another approximate solution... * * * Am still wondering if there's an exact method for this - or do you think the values were generated from a resampling approach like the above?


title: Analyzing historical incident rates and rating future performance

text: I am analyzing a large of dataset (n>100) of incident rates, with the aim of forming a normal distribution. Then I will know if a future incident rate (x%) is either close to a historical mean or not, and can score/rate it accordingly with an already created formula. The data is positively-skewed, as most data points cluster around or near zero percent. I HAVE to transform this data into a normal distribution, correct? What is the preferred method when dealing with percentages (these will always be between 0 and 100%)? Are there alternative non-normalizing methods I can use to reach my desired output? Anyway, let's say I've transformed the data and it follows a normal distribution. Now I can find the mean and std dev, then plot these in Excel using z-scores. Then I should be able to determine if incident rate x% is in the top 10%, top 20% of values, and score it with my formula accordingly. Any problems with this method?


title: Are sample statistics relevant when comparing census data across time periods?

text: I've had people tell me on numerous occasions that you can / should do tests normally reserved for sample statistics (e.g., difference of means etc.) on census data when you are comparing figures over time. The claim usually made is that this is because each census is a 'sample in time'. My understanding is that it is not appropriate to extend sampling theory in this way. Specifically, the explanation below (copied from census statistical techniques) describes my view quite well. > When you have a sample you use inferential stats to generalise to the > population. When you have a census you already have data for the whole > population, so there is no need to generalise. > > For example, if you used sampling, and there is a 3% difference between > groups, then you have to use inferential stats to decide whether that 3% > difference is real, or just due to random chance when you did the sampling. > > But if you did a census, and there is a 3% difference between groups, well, > then there's definitely a 3% difference. That 3% difference is not due to > random chance in sampling, because you have data for the whole population. > However, even with a census you will still need to use your own judgement to > think about why there is a 3% difference (for reasons other than random > chance in sampling), and whether the 3% difference is large enough to have > any practical significance for the work you are doing. > > So basically, just use descriptive stats. Correlations are fine, but you > only need the r value to show the strength of the correlation, not the p > value which is related to random chance in sampling. > > A lot of people don't get the difference between sample stats and census > stats, and will complain that you didn't do the stats properly. I've had > cases where I ended up having to do inferential stats on census data just > because people complained so much that there were no p values on anything! > > If you have a lot of missing data from a census sometimes you need some > fancy inferential stats to fill it in. I doubt this will apply to you, but > it does apply to the US population census because (for some bizarre > libertarian reason) completing the census survey in not mandatory in the US. However, **my question relates to comparisons across time** rather than between groups. The argument I've been given is that when you look across time, each census parameter is actually a single-point sample from all possible parameters across time. I have a couple of problems with this:   1. differences in census figures across time won't be attributable to chance. They must be attributable to changes in the underlying environment / population and    2. even if the 'samples in time' argument is correct, you only ever have very few data points sampled from this 'infinite population'--maybe parameters across four or five censuses you want to compare--which means $n$ is so tiny as to be useless for generating sample statistics. Surely others have addressed this situation more formally, but I've been unable to find material covering this situation. My question therefore is: **Can anyone here point me in the direction of material discussing this issue or offer an explanation of the 'samples in time' argument that provides a more formal foundation for accepting that it is indeed appropriate to use sample statistical tests on census parameters?**


title: How to model multivariate time series

text: I have a set of $n=1000$ samples of 4 dimensions (multivariate) where each measurement obtained from GPS tracking data is taken at a time interval representing spatial coordinates $(x,y)$, velocity. So, the data set contains 5 columns (another dimension for time) of 1000 tuples. Since there are 4 correlated vectors, then there should be 4 variables in the model each representing a measurement. I am really struggling to determine:   1. How to begin modelling with multiple inputs and single output   2. Which model to choose?   3. Does the modelling choice depend on the probability distribution? If so then what is the relation?   4. Should it be linear or non-linear models?  Illustration with a small example will really be helpful.


title: Is the chance of independent events just summed, when calculating overall chances of any event?

text: I'm not a mathematician or statistician (my last exposure was over 30 years ago with stats classes in school). My problem is if events are independent do you just add them together to get the overall chance of the event (my school exposure at least makes me think that maybe it is not that simple). **The actual problem:**   I'm about to have a heart operation.   There are chances of serious complications.   1% risk of A (cardiac perforation)   1% risk of B (stroke)   1% risk of C (pulmonary vein stenosis)   1% risk of D (vascular damage)   0.2% risk of E (mitral valve damage)   So, is my chance of a serious complication 4.2% (A + B + C + D + E)?   Or since each event has no influence on occurrence of other events is chance of a complication more like 1% being the highest risk value?   Or something in between? I did try searching here for an answer but everything rapidly got into stats maths and complex formulae, then I got lost, so please keep it simple for me <:-)


title: Why is maximum number of orthogonal contrasts equal to the number of groups minus one and what does mutually orthogonal mean?

text: 1. I was wondering why, for $g$ groups, it is always possible to construct $g - 1$ mutually orthogonal contrasts and $g - 1$ is the maximum number? A counterexample I have is that for two groups with means $\mu_1$ and $\mu_2$, $g=2$, and $\mu_1$ and $\mu_2$ themselves are two orthogonal contrasts.   2. The definition of orthogonal contrasts I have learned is a set of contrasts in which, for any distinct pair, the sum of the cross-products of the coefficients is zero. So it is a pairwise concept. What does "mutually orthogonal contrasts" mean, and is it not defined pairwise? Thanks!


title: Scatter Index question

text: I have a rather simplistic question, but maybe i mis-understood. For simulation purposes the use of the scatter Index (SI) is a good tool, but i had the idea that the smaller the scatter index the better the simulated results. In a recent paper i was reading they presented a SI of 0.90 as a good result, while my own similar results tend to have a SI 0.3-0.4 , which is better do you think. p.s I am referring to the relationship of actual measurements and results from simulation in atmospheric models. Thank you in advance


title: Least angle regression packages for R or MATLAB

text: I am looking for a Least Angle Regression (LAR) packages in R or MATLAB which can be used for **classification** problems. The only package that I currently know which fits this description is glmpath. The issue with this package is that it is a little old and somewhat limited in its scope (I am forced to rely on logistic regression for classification problems model). I am wondering if anyone knows of other packages that allow me to run LAR on different types of classification models, such as Support Vector Machines (see The Entire Regularization Path for the Support Vector Machine). The ideal package would allow me to run LAR-type algorithms for different types of classification models and also provide a function that can produce the full regularization path.


title: R package for multinomial naive Bayes text classification?

text: I am looking for a multinomial naive Bayes text classification package in R that accepts a term document matrix (from tm) as input for training and classifies new text based on that. I have about 50.000 short documents, which will belong to around 30 different topics (topics are assigned by LDA) and I would like to get a probability score for new documents.


title: Forecast and errors, dependent or independent

text: Assume the demand follows an MA(1): $$d_t=c+e_t-\theta e_{t-1},$$ Apply now, SES(single exponential smoothing)as a forecasting method. We know that the demand, $d_t$ is not independent. I was wondering if the forecasts and errors (demand-forecast) are also not independent?


title: Multivariate correlation in sensitivity analysis

text: I have a vector of parameters $\theta=(n,ph,pt,\gamma)$ which are fed into a process which produces a result. I would like to quantitatively report the sensitivity of the result to the variation of a parameter(s). Univariate correlation isn't what I'm after because $\theta$ is sampled from a distribution. What do you think is my best bet? Let me know if more information is required. Thanks. **EDIT** The process is stochastic. Basically a person moves round a room touching surfaces ( **n times** ) stochastically, based on observed probabilities. By touching each surface they pick up an isotope with probability **$\gamma$** . How much isotope they pick up depends on a variable **pt** , calculated experimentally.   After they finish touching surfaces, they wash their hands with probability and efficacy ( **ph** ). Therefore the output is a scalar quantity of isotope and a vector of surfaces touched. **EDIT 2** The isotope ($I$) on the person's hand is calculated in an additive manner after each surface contact so: $$I=\lambda_i \, pt_i V_i$$ Where $V_i$ is the surface concentration of isotope. Let's assume we're dealing with one surface so $V$ is constant. Then the total sum of isotope on the person's hand after $n$ touches with the same surface would be: $$I=\sum_{i=1}^n\lambda_i \, pt_i V_i$$ They then wash their hands with some efficacy $ph$. The left hand picture is a frequency density histogram of $I$. The right hand picture is a scatter graph between $I$ and $n$. ![enter image description here](http://i.stack.imgur.com/Q8Bp8.png) However $n$ is drawn from a distribution of observed data. How can this be incorporated into sensitivity analysis?


title: Can I use correlated variables in a mixture model?

text: I want to fit a mixture model with continuous (input) variables to cluster my data. Some of the variables are correlated with each other. Should I remove the correlated variables and retain only one of them (per group of correlated variables) or can I keep all of them?


title: Property of KL-divergence

text: Let $p_1$ and $p_2$ be two distinct probability distributions. Define $$ L(q)=D(q||p_1)-D(q||p_2) $$ where $D$ is the usual Kullback-Leibler divergence. Assume the support of $p_2$ is included in the support of $p_1$. Is it true that $L(q)$ achieves a (global) maximum at $q=p_2$? Where can I find a proof of this fact (if true)? **EDIT** : I assume (a) discrete PMF (b) I look for the maximum in the subset $$ \Delta = \\{ q| supp(q)\subseteq supp(p_1)\\}. $$ Many thanks,


title: Pooling data with unequal sample sizes

text: I have searched far and wide for a clear answer to my question (and am aware that a clear answer may not exist) and I'm hoping someone might be able to help me: I have survey data for 4 samples of unequal sizes (n=500, 490, 101, 103). I want to pool the data for regression (logit and tobit specifically) analysis, but am not clear how to account for the different sample sizes and their effects on the model outcomes. Any ideas what I should do? Thanks so much!


title: Distance metric and curse of dimensions

text: Some where I read a note that if you have many parameters $(x_1, x_2, \ldots, x_n)$ and you try to find a "similarity metric" between these vectors, you may have a "curse of dimensioality". I believe it meant that most similarity scores will be equal and not giving you any useful information. In other words almost all partner vectors will have some medium distance score which isn't useful for categorisation or clustering etc. Do you know where I can learn more in detail about that? Are there metrics which suffer less from this effect?


title: Can I use a non-linear mixed model for data containing both linear and quadratic relationship?

text: I have dataset with both linear and quadratic relationships for my response variable among individuals. My dataset includes individuals sampled from two populations (9 individuals from population A and 8 from population B). For each individual I have measured stable nitrogen-isotopes from 9 sequentially grown wing feathers (time series). I have two hypotheses:   1. their are differences in the mean isotope values across the wing for individuals between the two regions   2. their differences in the variation of the isotope values across the wing for individuals between the two regions I must admit I am a novice to mixed models. Originally I had falsely assumed that my data for each individual were 'linear'. Thanks to Ben Bolker I now know how to test these assumptions and have discover one individual from the 17 has a quadratic relationship. I had built the following GLMMs using the 'nlme' package in 'R' before discovering my error:               model1 <- lme(Delta15N ~ factor(Population), method = "REML", data = Data, random = ~ 1 | Individual, correlation = corAR1(form = ~ 1 | Individual))          model2 <- lme(Delta15N ~ factor(Population)*Feather, method = "REML", data = Data, random = ~ 1 | Individual, correlation = corAR1(form = ~ 1 | Individual))      Can you please suggest a reference or example code that I might use to correctly model my data?


title: How to determine significant differences between frequency spectra?

text: I want to determine whether the mean of frequency spectra are statistically different from each other. I have a set of trials in different drug conditions and I want to test whether a drug had an effect on the frequency distribution and/or power across frequency range. This is opposed to determining whether power in a particular frequency band is different across conditions. For example, below are two experiments. I want to determine whether the red line differs from the blue. Standard error is marked by grey shading. Thank you for any suggestions and pointers! ![enter image description here](http://i.stack.imgur.com/Ek1mG.png)


title: Using neural networks for multi target prediction

text: I have a spatial dataset with some xs and ys at different spatial locations. I want to learn a non linear regression function using neural networks. I looked in to the training data and the outputs are different locations i.e. ys are high correlated. So I was thinking of modelling a multi target prediction, instead of learning a separate regression for each case. I was thinking of learning a joint one using neural networks. I tried to use `nntool` of Matlab, however it lets me model only one output at a time. What should I do to model multiple outputs at the same time?


title: two level input and output in fuzzy logic implementation in matlab

text: I am new to matlab, i am doing simple project uisng fuzzylogic in matlab. my project having 2 levels. In first level i have 4 indivutual fuzzy inferrence system(FIS) and in 2 level i have one FIS which takes the input from outout of all four FIS in first level. I am getting output for the fist level of 4 FIS. Now i want to give these 4 output of four FIS into input of single FIS. I dont know how to do this. I goolged but most of all using the single FIS for their application. My case is too different form them..i need to give output of 4 FIS into input of another FIS. Help me i have to complete my project soon. Thanks in advance.


title: Application of LSA/LSI; Is it common to include the use of an edit distance?

text: I have been using Latent Semantic Analysis (LSA) or Latent Semantic Indexing (LSI) to identify whether different email addresses belong to the same individual by matching on names used for each email address; An email address represents a 'document', and the set of all names (as separate words) are the terms in that document: "John Doe" < johnd@example.com > is converted into the document 'johnd@example.com' with terms ['john', 'doe', 'johnd']. The typical LSA/LSI application more or less applies these techniques:   * Construct term-document matrix   * Apply tf--idf model   * Compute SVD; Perform rank/dimension reduction.   * Compute cosine similarity There have been multiple studies showing the above technique is quite successful in matching documents. However, my data does not exist of actual (English) document, but only a few words; Namely the names of the people that have sent emails. To give you an idea, my data set consists of 99,012 terms and 76,580 documents. The average number of terms in a document is 2.70. To create a relation in the LSA/LSI application between two documents that are mutually exclusive but should match (e.g. as a result of misspelling), I have decided to augment the term-document matrix with an edit distance such as the Levenshtein distance. I have searched the web and have not been able to find any website that describes the use of LSA/LSI in combination with edit distance. Is there a reason this has not been applied or described before? Does anybody have more information on combining LSI/LSA and edit distance? I can hardly believe I am the first that has applied the combination of these techniques. I did find two duplicate questions which never received an answer:   * Combining tf/idf with a similarity metric ie edit distance, affine gap of jaro winkler   * Duplicate detection/classification using TF/IDF and cosine similarity metrics


title: Correlation among explanatory variables in survival analysis

text: I'm conducting survival analysis in Stata with data structured as multiple records over time. To illustrate, let's say I'm studying patient mortality. **I'd like to check whether two explanatory variables, say weight and blood pressure, are correlated.** I could run a pairwise correlation using `pwcorr weight bloodpressure`, but this wouldn't take into account the fact that the observations can come from the same patients. How should I test for correlation between `weight` and `bloodpressure` while controlling for clustering on `patientid`? I've seen the Variance Inflation Factor (VIF) used for testing correlations in survey data, but I'm not sure how to transfer it to survival data.


title: Computing expectation of random variable (with known distribution and joint density) conditional on a sum

text: Given two identically distributed random variables $X$ and $Y$ (need not be independent) from the same (known) distribution (and so, known $\newcommand{\E}{\mathbb E}\E(X)$ and $\E(Y)$) and joint density, $f(x,y)$ and the conditional expectation $\E(X \mid Y)$, is it possible to compute $\E( X \mid X+Y )$?


title: Frequency of data: is quarterly data better than half-year data? Why?

text: Hi have a dataset which I constructed using quarterly observations (from bank accounts). I could have also used half-year or yearly data, but I chose quarterly because I thought that higher frequency data could help me obtain more reliable estimates. The question is, from a theoretical and/or practical perspective, how would you justify the use of higher frequency data? Any book or paper reference is highly appreciated.


title: Are the digits of $\pi$ statistically random?

text: Suppose you observe the sequence: 7, 9, 0, 5, 5, 5, 4, 8, 0, 6, 9, 5, 3, 8, 7, 8, 5, 4, 0, 0, 6, 6, 4, 5 , 3, 3, 7, 5, 9, 8, 1, 8, 6, 2, 8, 4, 6, 4, 1, 9, 9, 0, 5, 2, 2, 0, 4, 5, 2, 8 ... What statistically tests would you apply to determine if this is truly random? FYI these are the $n$th digits of $\pi$. Thus, are digits of $\pi$ statistically random? Does this say anything about the constant $\pi$? ![enter image description here](http://i.stack.imgur.com/iBeId.png)


title: Ensembling regression models

text: I'm working on a securities pricing project and have a bunch of models I'd like to stack/ensemble together. I've been using simple linear regression in R (the `lm()` function) so far but the results are over fitting pretty badly. Does anyone have any suggestions for whether some other stacking method might be better or any papers/articles that describe how to stack regression models (as opposed to classification models).


title: Weird MLE Problem based on Dirichlet Function

text: This question was on a HW in my Statistical Theory class and I find the professor's answer and explanation to be unsatisfactory. Please give me some guidance as to why   * $\bar{x}$ is the MLE if this is the case, or    * Let me know if I am correct to think that both $\bar{x}$ and $1-\bar{x}$ maximize the likelihood function and therefore the MLE is not unique, or   * If the problem is even well defined as it is posed. I feel like this could be the case also. I understand the definition of the MLE in more regular circumstances, as detailed in the Principles section of http://en.wikipedia.org/wiki/Maximum_likelihood, but the weird form of the PDF brings up issues with supremums that I am not used to dealing with. **Question:** Let $X_1,...,X_n$ be an i.i.d. sequence of 0-1 valued RV's with the probabilities $$ P(X_1=1)=\begin{cases} \theta, & \theta\in\mathbb{Q}\\\1-\theta, & \theta\notin\mathbb{Q} \end{cases} $$ where $\theta\in(0,1)$. Does the MLE of $\theta$ exist? **Outline of Professor's solution:** This is the main idea of my professors solution. The likelihood function is $$ L(\theta|x_1,...,x_n)=\\{\theta^{\sum{x_j}}(1-\theta)^{n-\sum{x_j}}\chi_{\theta\in\mathbb{Q}} +\theta^{n-\sum{x_j}}(1-\theta)^{\sum{x_j}}\chi_{\theta\notin\mathbb{Q}}\\} $$ where $\chi_A$ is the indicator function of the set $A$. We have $$ \begin{eqnarray} \underset{\theta\in[0,1]}{\sup} L(\theta|x_1,...,x_n)&=&\underset{\theta\in[0,1]}{\sup} {\\{\theta^{\sum{x_j}}(1-\theta)^{n-\sum{x_j}}\chi_{\theta\in\mathbb{Q}} +\theta^{n-\sum{x_j}}(1-\theta)^{\sum{x_j}}\chi_{\theta\notin\mathbb{Q}}\\}} \\\&=& \max\\{\underset{\theta\in\mathbb{Q}}{\sup} \\{\theta^{\sum{x_j}}(1-\theta)^{n-\sum{x_j}}\\},\underset{\theta\notin\mathbb{Q}}{\sup} \\{\theta^{n-\sum{x_j}}(1-\theta)^{\sum{x_j}}\\}\\} \\\&=& \max\\{\bar{x}^{\sum{x_j}}(1-\bar{x})^{n-\sum{x_j}},(1-\bar{x})^{n-\sum{x_j}}\bar{x}^{\sum{x_j}}\\} \\\&=& \bar{x}^{\sum{x_j}}(1-\bar{x})^{n-\sum{x_j}} \end{eqnarray} $$ **Professor:** At this point, the professor argues that the supremum in the second term,$\underset{\theta\notin\mathbb{Q}}{\sup} \\{\theta^{n-\sum{x_j}}(1-\theta)^{\sum{x_j}}\\}\\}$ is not attained since $\bar{x}$ is a rational number. Since the data consists of rational numbers, the supremum of the first term , $\underset{\theta\in\mathbb{Q}}{\sup} \\{\theta^{\sum{x_j}}(1-\theta)^{n-\sum{x_j}}\\}$ is attained at $\hat{\theta}_1=\bar{x}$ and that this is the MLE of $\theta$. **Me:** It seems like we should consider the supremum over the closure of the sets $\mathbb{Q}$ and $\mathbb{R}\backslash\mathbb{Q}$, which would be $[0,1]$, in which case $\sup L(\theta|x_1,...,x_n)$ is achieved at both $\hat{\theta}_1=\bar{x}$ and $\hat{\theta}_2=1-\bar{x}$. Otherwise, we are essentially assuming that $\theta$ is rational and ignoring irrational $\theta$. Is this the case? If so, is this an undesirable property of the Likelihood Principle in weird cases like this? Is there any plausible situation where issues like this occur? Should I stop worrying about weird problems like this? As an aside, considering $[0,1]\backslash\mathbb{Q}$ has Lebesgue measure 1 and $[0,1]\cap\mathbb{Q}$ has Lebesgue measure 0, it seems like $\bar{x}$ is a bad estimator, since it is an estimate of $\theta$ if it is in a very small set. Also, if $\theta\in\mathbb{R}\backslash\mathbb{Q}$, $\hat{\theta}_2=1-\bar{x}$ is consistent, so I can't think of a good reason why $\bar{x}$ is better. **Edit** As @cardinal pointed out, the $x_i$ are obviously rational, so this is not an issue. This addressed my first (silly) misunderstanding, which involved assuming the estimator $\bar{x}$ could be irrational or rational.


title: SVM using RBF and nearest neighbor classification method

text: SVM using RBF kernel is claimed to be similar (equivalent) to the K nearest neighbor classification method. I am not very clear about the analysis process of building this kind of relationship. Thanks for explanations.


title: What is meant by the term "formal distance metric"?

text: I am familiar with the idea of metrics in software and I know something about using distance functions in clustering, but I don't really know what is meant by "formal distance metric". What is the difference between a distance function and a formal distance metric?


title: When is it acceptable to use only two items (variables) per factor in factor analysis?

text: I have a set of 20 variables that I have put through factor analysis in SPSS. For purposes of the research, I need to develop 6 factors. SPSS has shown that 8 variables (out of 20) have been loading with low weights or have been loading equally on several factors, so I have removed them. The remaining 12 variables have loaded in pairs of 2 into the 6 factors, which is perfect--just as I wanted, but now, one of the professors working with me wants me to find justification why (or under what conditions) it is appropriate it is appropriate to use only 2 items per factor, since it is commonly known that factor analysis is used with 3 or more items per factor. Can anyone help me out with this issue, preferably with a published reference as well?


title: Finding outliers without assuming normal distribution

text: I have small datasets of size 40-50 points. Without assuming that the data is normally distributed I wanted to find out the outliers with 90% confidence at least. I thought boxplot could be a good way to do that but I am not sure. Any help appreciated. Also with boxplot implementations I could not find a implementation which besides drawing the plot explicitly spits out the outliers.


title: Hypothesis Testing using dichotomous and 7-point Likert scale

text: I am working on testing the following hypothesis: H1: People who are worried about their health will consume yogurt H0: People who are worried about their health will not consume yogurt I have interviewed 46 respondents using a 7-point Likert scale for the independent variable ('I am worried about my health' 1)Totally disagree 2) Disagree 3)slightly disagree 4)nor agree nor disagree, 5)slightly agree, 6) agree, 7) totally agree). The other question is dichotomous - Do you consume yogurt? 1)Yes 2)No. I would like to test this hypothesis using the variables above even though one is in Likert scale and the other is dichotomous. Any suggestions? Thank you in advance! T.


title: Is excess mass estimation smooth enough to bootstrap? At what rate might a bunching estimator converge?

text: The recent public finance literature often estimates relative excess mass around specific points of the earnings distribution ("kink points" or "notches" of tax schedules, say), and then bootstraps to get standard errors. This is done to nonparametrically estimate local elasticities (responsiveness). I know kernel density is itself problematic for bootstrap, maybe better with subsampling. And this method relates approximations of mass below histograms or empirical distribution functions to local densities. Does the estimator "sound" smooth enough to bootstrap? What rate of convergence is the estimator likely to have (for any method)? Maybe someone has some intuition. I am not aware of any statistical theory for this problem. Many papers directly use the code from this paper both for estimation and to bootstrap, though the theoretical idea and motivation comes from this paper. A little more detail is in a comment of mine on the blog Normal Deviate or a question of mine on Statalist.


title: Pre-post Logistic regression

text: A friend of mine approached me to help her to interpret her multinomial logistic regression model. They had measured people as 1 of 2 states at 2 time periods. So, each person can have 1 of 4 configurations: start at state 1, end at state 1; start at state 1, end at state 2; start state 2, end state 1; start state 2, end state2. They had performed an analysis where the outcome had 4 levels corresponding to each of these configurations. This struck me as a somewhat unnatural way to do this. I feel that this is a situation similar to if you had a continuous outcome, measured at 2 time points. If you were interested in the change from baseline, you can model the end value and adjust for the baseline and other covariates. Does this translate to logistic regression? Can you model the end state, and adjust for the beginning state with other covariates? I couldn't think of a good reason why this would not work, but I also had this problem. Is this better than modeling the outcome as 4 possible categories?