title: Regression, Independence & Normality of x and y variables (not residuals)

text: I am trying to predict productivity of inbound process of a warehouse using y-value = productivity in terms of pcs/hr x-value = # pcs and ratio pcs/SKU. (where SKU = stock keeping unit) Basic hypothesis --> (1) the more work there are (busier people are), the more they rush and the better the productivity. (2) In addition, since people handle 1 SKU only once, the more pcs/SKU, means the less time you have to spend per piece (ie if it takes 1 min to input information for 1 SKU, you could do 1 pcs in one minute at 1pcs/SKU or 100 pcs in one minute at 100pcs/SKU). Questions --> (1) does y variables and both of the x variables have to be normally distributed? I understands the residuals need to be normally distribution, but does the input and output itself also have to be normally distributed? (2) The number of pcs is contained in both x1 and x2, does this pose a problem. I believe this will create dependency problem? Thanks a lot for your help.


title: Interpretation of log transformed negatively skewed dependent variable

text: I am running multiple regression analysis and my dependent variable is not normally distributed (skewness=-1.794 and kurtosis: 4.643). In order to correct this, I applied log transformation but because it is negatively skewed I used this formula: $log(1+max value of DV - DV)$. The inversely and log transformed DV was closer to normally distributed (though inverse transformation was much closer, I still chose log transformation as there is literature on how to interpret log transformation, while none on how to interpret inverse transformation is available to me). When I run the regression on the transformed DV, I got negative values for unstandardised b which implies that increase in one unit of the predictor would lead to beta*100% decrease in DV. But it is very well known in literature that the increase of that independent variable cannot cause decrease in the dependent variable. I also see reversal of the signs of beta coefficients for some of the other independent variables (as I would expect also to see the increase in both dependent and indepenent variables, not one increase and other decrease). Does anyone know whether this formula applied to the negatively skewed variable can actually produce such a result (reversal of signs), or something else is wrong? Thank you.


title: Why does max-t methods use only the maximum of generated t-values?

text: I am wondering why max-t methods use the maximum of the t-values to generate the distribution. Does this not make it overly conservative compared to other methods, since the critical value will be decided by the 0.05 quantile of only the maximum of the generated sample? This question also encompasses the analog of the minimum p-value estimation techniques.


title: Problems estimating anisotropy parameters for a spatial model

text: I am looking for someone with experience in how to handle the anisotropy- parameters in the `likfit()` function which is part of the GeoR package in R. I am using `likfit()` to generate the `obj.m`-parameter for the function `krige.conv()`. The data I use consists of scattered points (5 to 30) on a ~50*50 grid. In `likfit()`, I want the parameters `psiA` and `psiR` to be estimated from the available data. This works fine by stating:               fix.psiA = FALSE, fix.psiR = FALSE      However, the estimated values show quite a huge range for `psiR`, given that the datasets I used are not fundamentaly different from each other. (It is a set of soil moisture measurements with values from 0 to ~45.) `psiR` ranges from 1 to about 8000... can that be right? Most of the values are in a range from 1 to 10, but I can not tell why some datasets produce these very large values for `psiR`. I was unable to find further information on what this parameter _exactly_ does. I do understand that this parameter regulates the dependency of an estimated value according to it's location relative to the ambient measured values. But I do not know in which way this is accomplished. I am sorry for the possibly bad english, not a native speaker. I am also sorry for not posting my code, but it is quite long and I felt that it is not necessary in behalf of my question. Thank you very much for your interest. I will post additional details, if required.


title: Given a fitted regression model, how to calculate the p-value of a new observation?

text: (apologies in advance if I abuse any terminology) I have a series of XY data points that I fit with a simple regression and I would like to be able to use this regression to obtain a p-value for the new data point. I have found a couple of sources describing how to calculate a Prediction Interval for Responses to a Particular x which seems to _almost_ answer this question. In fact, I think I can reformulate my question as how do I find the alpha of the smallest prediction interval containing my new data point.


title: How to determine the marginal pdf, the posterior?

text: How to get the marginal pdf of $p(y)$? Do you just integrate out $p({\sigma}^{2})$? Say, the following joint distribution for $y \in {{R}^{d}}$ and ${{\sigma }^{2}}\in {{R}^{d}}$ IG: means inverse Gamma $${{\sigma }^{2}}\sim IG(\alpha ,\beta )\propto {{({{\sigma }^{2}})}^{-(\alpha +1)}}{{e}^{-\beta /{{\sigma }^{2}}}}$$ $$y|{{\sigma }^{2}}\sim N(\mu ,{{\sigma }^{2}}\Sigma )$$ where $a\in R$, $b\in R$,$\mu \in {{R}^{d}}$,$\Sigma \in {{R}^{d\times d}}$ are known parameters. I know that $$p(y|\mu,\Sigma)\propto \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{1}{2}\sum\limits_{i=1}^{n}{{{\left( {{y}_{i}}-\mu \right)}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}({{y}_{i}}-\mu )} \right]$$ $$\propto \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{1}{2}\sum\limits_{i=1}^{n}{{{\left( {{y}_{i}}-\bar{y} \right)}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}({{y}_{i}}-\bar{y})-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}(\bar{y}-\mu )} \right]$$ $$\propto \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{n-1}{2}tr\left( {{({{\sigma }^{2}}\Sigma )}^{-1}}S \right)-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}(\bar{y}-\mu ) \right]$$ what I have done is using $$p{({\sigma}^{2})}{\times}p(y|{{\sigma }^{2}})$$ which gives me, $$\propto {{\left( {{\sigma }^{2}} \right)}^{-(\alpha +1)}}{{e}^{-\beta /{{\sigma }^{2}}}}\times \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{n-1}{2}tr\left( {{({{\sigma }^{2}}\Sigma )}^{-1}}S \right)-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}(\bar{y}-\mu ) \right]$$ This doesn't look like anything to me, am I even on the right track?! Anyways, if any experts knows, please point me out, thanks sooo much!


title: Basic question about cross validation

text: A very basic question about cross validation for Neural Networks. Do I have to create a new network for each fold or do I have to keep the network and incrementally train it with the k-th training set? http://www.faqs.org/faqs/ai-faq/neural-nets/part3/section-12.html says "You train the net k-times..." which let's me assume to use the same net every fold again. But when I don't create a new network for each fold, then I would use the k-th fold k-1 times. In this picture you see that without a new network, e.g. the first 4 data points would be used to train the model in the 2nd to 5th step. Can you give me a hint on the right interpretation?


title: Relationship between statistical power and p-values

text: As you have an increase in statistical power, does this decrease the p-value? If so, why does this occur?


title: Can i compare the strength of association that using phi & Cramer's V between the parameters?

text: I have a little question and need some clarification on chi square's strength of association (using the Phi & Cramer's V). Say i have 4 parameters, all of them show significant relationship with the e.g., Quality i.e., p<0.05 in chi square test, but i want to prove more result by knowing the strength of each parameter hence i use the phi and cramer's V then i compare all the four parameters' association strength. From this, i can know which one of them has the strongest relationship (most influential) as well as the least. Therefore, I would like to know whether or not such comparison between the parameters' strength is valid and allowable? Thank you and your comment is highly appreciated


title: How to specify and estimate the parameters of a model that is quadratic in several variables

text: I am trying to find how the quadratic model for a multiple featured dataset will be. Suppose that my training set is $X_1,...X_n$ with each $X$ of dimension $4$. Now suppose I want to fit a quadratic function to this data. Then my model will be $$h(X) = a_1 + a_2X + a_3X^2$$ where $a_1, a_2$ and $a_3$ are the parameters of my model to be estimated. Since the $X$s in the above equation are vectors that means that my parameters $a_2$ and $a_3$ also need to be vectors right? So how can I learn these parameters?


title: 2-dimensional moment and rotation

text: Is it possible to get a simple formula linking a central moment to the same moment in a rotated frame, such as the relation between the central moment and the moment about the origin? The formula I am using is http://en.wikipedia.org/wiki/Central_moment#Multivariate_moments


title: Machine learning with ordered labels

text: The usual method for adapting binary classifiers like various SVMs to multilabel data is one-vs-all, which assumes that labels are independent and in case of a prediction error we don't care what label the incorrect prediction should output. But suppose I want to predict a score from 1 to 5, and I prefer to err closer to the truth, how do I go about it then? $\lt k$ vs $\ge k$ comes to mind, because in this case labels which are closer together will have larger training data overlap, but is it theoretically sound? Specifically, is overall performance likely to suffer? More generally, what are state-of-the-art techniques I could employ in this situation?


title: Building the vocabulary in document classification

text: Assume I have a collection of documents and I want to use tf-idf as my document weighting measure. If I want my vocabulary to be of size 100, how do I choose those 100 words in the vocabulary? Considering training data as a whole and selecting the 100 top frequent words is not going to work here since my weighting system is tf-idf, right?


title: Should one remove all autocorrelation?

text: Or is an AC of say less than 5% (in the residuals) acceptable for continuing with usual Panel Data analysis?


title: Does it make sense to impute missing covariate data when the imputed value is a function of other covariates in the regression model?

text: We are building a model that adjusts for standard covariates (e.g., age, gender) and for the outcome at baseline. It would be ideal to adjust for each subject's baseline value like so: $$ Y = \beta_0 + \beta_1*{\rm age} + \beta_2*{\rm gender} + \beta_3*{\rm baseline} $$ However, we don't have baseline measures for everyone. There is a standard clinical estimation of $Y$ (current and baseline) as a nonlinear function of age and gender and also a measure $X$ (which we do have on everyone), i.e., $\text{Estimated Baseline} = f(X,\ {\rm age},\ {\rm gender}) $. Does it make more sense to model $Y$ by including the raw value $X$ or the function of $X$? For example: \begin{align} Y &= \beta_0 + \beta_1*{\rm age} + \beta_2*{\rm gender} + \beta_3*X \\\ Y &= \beta_0 + \beta_1*{\rm age} + \beta_2*{\rm gender} + \beta_3*f(X,\ {\rm age},\ {\rm gender}) \end{align} The estimated function is not a great estimate, but it is closer to the baseline measure.


title: Sample size function for McNemar test of repeated proportions in R

text: I have been scouring CRAN and Google for a function (that I wouldn't have to code myself) that would calculate the sample size needed for a pre-post design with a dichotomous outcome - to no avail. If I'm wrong, please let me know! I came across an R script from a university course and was wondering if someone wouldn't mind double-checking it for me - I've revised it a bit. (the URL is: http://www.statistik.lmu.de/institut/lehrstuhl/semwiso/bm- sose2009/CaCo/v8_4.pdf)               samsize.mcnemar <- function(pi.01, pi.10, alpha, beta, sided)                      {     pi.d <- (pi.01 + pi.10)     N <- (qnorm(1 - alpha/sided) * sqrt(pi.d) + qnorm(1 - beta) *         sqrt(pi.d - (pi.01 - pi.10)^2))^2/(pi.01 - pi.10)^2     return(ceiling(N))     }     # in my case, I am assuming that the 0 to 1 change will be 0.13 and that no      # one will change from 1 to 0           samsize.mcnemar(pi.01 = 0.13, pi.10 = 0, alpha = 0.05, beta = 0.2, sided = 2)      **First** , does this look correct? **Second** , since this is not a matched-pairs design, I am not assuming that the intervention will cause people to revert back (if they are already performing the healthy behaviour). Should I be more conservative? Thanks!


title: Chi-square test to see if set of dependent correlations are equal

text: I've got four dependent correlations (i.e., from the same sample) that involve predictor variables $(A,B,C,D)$ and an outcome variable $(E)$. $N=172$. $$\begin{array}{c|cccc}{\rm Statistic}&AE&BE&CE&DE\\\\\hline r&-.480&.276&.395&.327\\\p&< .01&< .01&< .05&< .01\end{array}$$ I am trying to run three chi-square tests on this set of four dependent correlations, where $H_0: r_{AE} = r_{BE} = r_{CE} = r_{DE} = 0$; and $H_1$ is simply "NOT $H_0$". Assuming $H_0$ is rejected, I will then be testing: $H_0: r_{AE} = r_{BE} = r_{CE} = r_{DE}$; with $H_1$ being "NOT $H_0$" again. Assuming that $H_0$ is rejected again, I will then be testing $H_0: r_{BE} = r_{CE} = r_{DE}$. SPSS cannot help with me comparing dependent correlations and Field's "Discovering Statistics..." (2013) only discusses comparing two dependent correlations, not an entire set. So, I have been directed to several places so far, with no luck:   1. "Multicorr" by Steiger (http://www.statpower.net/Software.html). I simply cannot figure out how to use it.   2. A paper by Steiger ("Tests for Comparing Elements of a Correlation Matrix", _Psychological Bulletin_ , 1980, _87_ (2)). But I cannot figure out how/where my numbers fit with the formulas he has provided there. Ideally, I'd just like the mathematical formula(s) I can use to get my chi- square observed value (I have the critical value tables already). Any guidance that could be offered would be most appreciated.


title: Computing Standard Errors in EM algorithm

text: I'm applying the EM to a hidden markov chain (the $\mathbf{Z}=\\{Z_1,...,Z_n\\}$ variable), with observations(the $\mathbf{Y}=\\{Y_0,...,Y_n\\}$ variable) dependent not only on the hidden markov chain, but also on the immediate past observation (the $Y_{j-1}$). $$Log(P(\mathbf{Z},\mathbf{Y}|\mathbf{\Theta)})=\log(\pi_{0z_1})+\sum^n_{j=2}\log(\pi_{z_{j-1}z_j})+\sum^n_{j=1}\log(P(y_j|y_{j-1},z_j;\mathbf{\Theta)})$$ In "McLachlan's Finite Mixtures", page 63 and 64, it talks about how to obtain the Observed Information Matrix, in terms of the complete data Log-Likelihood. My question is how, from the Observed Information Matrix, do we get the standard errors? (Why can one estimate the inverse of the covariance matrix of the MLE, by the Observed Information Matrix?) And could I obtain the std errors in a similar way for a hidden Markov model?


title: Compare density estimate to true pdf

text: I'm trying to compare various density estimation methods. My dataset $D$ is generated from a fixed mixture of Gaussians (which allows me to estimate the true pdf $p(x)$). Then, I compute the estimated density $\bar{p}(x) : x \in T$ of a test set $T$ from $D$ using Parzen windows and k-NN. I have two questions:   * $\bar{p}(x)$ seems to be scaled compared to $p(x)$. If I use these densities for classification, the scaling will not affect my decision, but if I want to compare the quality of the density estimate, I believe this "scaling" will be a problem. ![enter image description here](http://i.stack.imgur.com/ggkBf.png) ![enter image description here](http://i.stack.imgur.com/ZBCbT.png)   * How do I determine the quality of $\bar{p}(x)$? Could I just use the integrated squared error? $$ISE = \sum_{i=1}^{|T|} ( p(x) - \bar{p}(x) )^2 $$


title: Doubling the tails in two-sample permutation test

text: Suppose we have two samples and we wish to determine if they are drawn from the same distribution, the samples A,B being composed of some integers say. If we test this using a two-sample permutation test, specifically by looking at permutations where the differences in the means of the samples are as extreme as the difference observed: is there any reason to think that we can calculate the two-tailed p-value by looking at one tail and doubling the probability? This is what it appears to say in my lecture notes, but I don't understand why we could assume that the tails are symmetrical (or why it doesn't entail that assumption). Explanations were not forthcoming. Thank you.


title: Dealing with lots of ties in kNN model

text: I have a large data set (400k rows X 60 columns) that I'm trying to use to build a knn model. I'm using the `caret` package version of `knn` and the `forward.search` method from the `FSelector` package to eliminate variables via cross-validation. My problem is that once I use more than 20k lines of data I get a message about there being too many ties. Currently I'm only checking k-values between 1-19 (and only odd #'s as they supposedly shrink risk of ties) and only using variables with > 2 levels. Are there any other tweaks to using big chunks of data into a `knn`? EDIT: This is regression problem, not a classification problem.


title: When analyzing a difference for log-transformed data, when to take the log?

text: I am working with laboratory data that needs to be transformed due to skewness. It has been gathered at 2 time points and I am interested in the change from time #1 to Time #2. My question is, do I log transform the original values, then subtract the difference in the log-transformed data? Or do I calculate the difference from the raw values then log transform the difference? (That presents a problem in itself, since the difference can be negative, so could not be logged).


title: Forecasting with holiday dummy variables

text: I have an example of call center data for 2013. There are 261 days of data (excluding weekends).   For 2013, I have included a holiday dummy variable (`holiday`) for the days where there were no stats.   For 2014, I have also included a future holiday dummy variable (`holidayf`).   **My objective is to assess how accurate this code is in making predictions for 2014.** I tried this code below but when looking at `fc$fitted`, the forecasts don't seem to be correct. For the first 8 days of January 2014, it forecasts the exact number of calls that were received in the first 8 days of January 2013, which seems wrong. Also, where there is a public holiday in 2014, the future forecast for that day predicts a normal to high volume of calls, so it seems that the forecast is using the `holiday` variable and not the `holidayf` variable.               library(forecast)     y <- ts(calls,frequency=5)     z <- fourier(ts(calls,frequency=261),K=12)     zf <- fourier(ts(calls,frequency=261),K=12,h=261)     fit <- auto.arima(y,xreg=cbind(z,holiday))     fc <- forecast(fit,xreg=cbind(zf,holidayf),h=261)     plot(fc)      Data:               calls <-        c(0,145,175,129,266,219,156,184,167,241,218,194,192,162,236,219,212,191,162,216,       235, 218,180,150,245,209,210,211,151,236,197,217,140,164,200,156,152,153,141,224,178,       159,153,137,207,173,197,213,206,305,284,248,289,269,359,333,257,0,244,325,292,267,       206,0,0,360,261,327,284,385,377,317,327,271,372,191,320,268,261,376,320,280,251,200,       200,200,0,236,161,259,200,190,166,174,225,228,202,201,155,241,207,199,179,178,249,       243,230,177,181,264,250,219,204,178,244,249,185,184,164,0,253,216,217,165,170,185,       175,160,148,231,223,196,162,149,228,213,190,177,139,212,205,221,190,170,196,210,       198,192,131,220,185,199,153,166,240,176,200,145,0,255,202,220,220,181,250,171,164,       142,118,179,197,167,130,124,180,214,203,153,140,161,200,191,159,141,227,170,166,       166,106,131,0,176,156,109,196,175,175,174,161,230,191,159,150,91,180,188,173,157,       107,193,172,172,172,116,195,183,169,146,125,208,160,160,177,128,191,176,149,175,       136,217,162,178,130,99,158,154,135,146,106,155,148,119,137,96,161,106,114,139,84,       0,97,95,82,65,59,23,0,0,48,83,48)                    holiday <- c(1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,0,0,0,0,0,0,0,0,0,0,0 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,1,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,                  0,0,0,1,1,1,0,0,0)           holidayf <- c(1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,1,                  0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,                  0,0,0,0,1,1,0,0,0)


title: Modeling outliers of normal distribution

text: I am using a linear model to predict under-nutrition in children under 5. The common metric discussed is stunting (a binary outcome) which is defined as being more than two standard deviations away from a standard age-for height (I'm predicting the z-score which underlies the stunting status). Below are overlapping histograms of actual (blue) versus predicted (red) height for age scores. What I am really interested in predicting is whether or not an individual falls below a z-score of -2, and my distribution is very restricted. Any tips? Right now, my model looks like this:               Height.Age.Z ~ age + mothers.education + wealth + sanitation + (fixed effect on region)      Any tips, general and specific, are appreciated. Thanks, ![enter image description here](http://i.stack.imgur.com/gz57I.png)


title: Standardized dependent variable within a group in panel data models?

text: **Does standardizing of a dependent variable within the identifying group make sense?** The following working paper (Deforestation slowdown in the Legal Amazon; Prices or Policies?, pdf) uses a standardized dependent variable to analyze the effect of general policy change in Brazil on deforestation. Standardization is done as as follows:   $$ Y^{new}_{it} = \frac{Y_{it} - \overline{Y_i}}{sd(Y_{it})} $$ The authors argue, this serves to "consider relative variations in deforestation increments within municipalities". The authors use hereby a FE estimation (page 12) for panel data. Including a Post-policy-dummy for each following year after the a new law.   * How should coefficients be interpreted if the dependent variable was standardized in this way?    * Is standardization not unorthodox as it gives higher values to observations where the group/municipality experienced lower variations over time?


title: Is there a Project Euler-alike for machine learning?

text: I found Project Euler http://projecteuler.net/ to be incredibly useful in learning programming languages. Is there a similar site for Machine Learning? I did see http://www.kaggle.com/, but it is not nearly as accessible to beginners as Project Euler.


title: Test to rank methods by AUCs on various benchmarks

text: Suppose I have N methods and M benchmarks. I have an AUC statistic (and some other similar statistics) for each combination of method with benchmark. What test should I use to test if one method is better than the rest? I have seen some authors do pairwise comparisons using a one-sided Wilcoxon signed-rank test but I would prefer to test all methods at once. In any case I'm not sure the assumptions for the one-sided Wilcoxon signed-rank test hold. If the average AUC for each benchmark varies widely can you say the samples are from the same population? Also I'm not sure the distribution of the AUCs is symmetric around the median. Any advice would be welcome.


title: I have GARCH models, which one shall I choose?

text: 1- GARCH(5,5) **AIC= -50667.51** P-value of Coefficient not show               2- GARCH(1,2)   **AIC= -36935.9**     P-value of Coefficient calculated and show      3- GARCH(1,1) **AIC= -50410.11** P-value of Coefficient not show ## which one shall I choose ? GARCH(5,5) or GARCH(1,2) or GARCH(1,1)                ـ      Note: GRACH(1,2) the P-value of Coefficient /// a2 2.323e-15 not sig **, What do I do ??? ![enter image description here](http://i.stack.imgur.com/6Hvqc.png) ## \------------------------------------------------------------------------- GARCH (1,2) ![enter image description here](http://i.stack.imgur.com/YQt72.png) * * * ![enter image description here](http://i.stack.imgur.com/GFuW2.png)


title: Multiple definitions of node openbugs

text: I have some problem with this code in Openbugs. The model is sintatically correct and data are loaded, but when I compile, software output is "multiple definitions of node fn[1]". I don't know how to resolve the problem. This is the model:               model     {          for(i in 1:N)     {     x[i] ~ dgev(n[i],t[i],s[i])          n[i] <- fn[i]+sn[i]     t[i] <- ft[i]+st[i]     s[i] <- fs[i]+ss[i]          for(j in 1:N)     {          fn[i] <- beta0n+beta1n*lon[i]+beta2n*lat[i]     ft[i] <- beta0t+beta1t*lon[i]+beta2t*lat[i]     fs[i] <- beta0s          sn[i] ~ dmnorm(snmean[],tsn[,])     st[i] ~ dmnorm(snmean[],tst[,])     ss[i] ~ dmnorm(snmean[],tss[,])          tsn[i,j] <- inverse(covn[,])     tst[i,j] <- inverse(covt[,])     tss[i,j]<- inverse(covs[,])          snmean[i]<-0          covn[i,j] <- alfan*exp(-pow(sqrt(pow(lat[i]-lat[j],2)+pow(lon[i]-lon[j],2))/lambda, 0.5))     covt[i,j] <- alfat*exp(-pow(sqrt(pow(lat[i]-lat[j],2)+pow(lon[i]-lon[j],2))/lambda, 0.5))     covs[i,j] <- alfas*exp(-pow(sqrt(pow(lat[i]-lat[j],2)+pow(lon[i]-lon[j],2))/lambda, 0.5))     }     }          alfan <- 1/alfangamma     alfangamma ~ dgamma(1,12)     beta0n ~ dnorm(0,50)     beta1n ~ dnorm(0,60)     beta2n ~ dnorm(0,65)     lambda ~ dgamma(5,0.33)     alfat <- 1/alfatgamma     alfatgamma ~ dgamma(1,1)     beta0t ~ dnorm(0,33)     beta1t ~ dnorm(0,42)     beta2t ~ dnorm(0,53)     alfas <- 1/alfasgamma     alfasgamma ~ dgamma(1,0.04)     beta0s ~ dnorm(0,55)     }      These are data:               list(x=c(86.7,67.2,116.8,75.8,121.5,112.7,70,93.3,92,74.4,     64.1,82.9,63.5,96,104.8,85.4,78.7,86.2,85.5,115.3,     92.2,72,79.1,128.6,74,96.7,88.8,75.8,122.6,98.6,     75.3,104.5,97.7,89.5,70.9,70.4,68.7,83.4,61.7,69.1,     92,112.6,100.1,86.9,96.8,87.8,103.7,130,105.5,96.8,     76.3,99.7,125.2,99.8,75.8,75,75.6,77.9,84.4,98.2,     110.7,106,96,101.4,136,94.9,105.5,91.2,100.1,160.5,     117.4,82.7,106.8,82.8,201.5,97.5,64,94.2,59.4),     lon=c(661130,719070,758530,741785,693810,747735,749840,709325,     657250,659975,735320,701000,682010,733050,696730,672870,     715450,726370,695350,699790,708077,715095,733140,709480,     700550,676530,737730,703795,720330,712090,673790,709070,     686700,676325,707420,646900,692150,684590,665540,687800,     667290,700860,730875,671520,700540,721600,718900,755440,     690700,688698,678225,719940,744200,656680,657010,755720,     710800,686310,747861,731250,737700,715220,731025,747810,     654100,651280,738680,694760,676900,725245,725430,766485,     680340,677050,746130,699230,693849,685117,682720),     lat=c(233825,265660,250325,268332,273070,244475,263150,247050,     228850,267750,262270,285890,263780,248330,283130,251875,227440,     237220,255060,220375,278255,258260,252930,270170,237300,283550,250830,240680,275760,218325,269310,274350,240570,215325,227220,     267700,289940,226110,209848,290270,235375,272510,219535,257080,246940,236075,278900,233270,214450,282796,288730,225580,234920,     242110,220940,220430,225520,230590,254586,242140,227600,280130,267175,250410,259350,240080,241230,245160,246180,221680,268950,     248850,273225,280200,229570,259430,230708,248061,259340),N=79)      Thanks to everyone that will answer. * * * Now my code doesn't compile! I understand that the problem is to do inverse of matrix. How can I resolve this problem? Maybe is matrix too large (79*79)? Is there another method to calculate an inverse of matrix? Thanks


title: What language to use for genetic programming

text: As part of an assignment I'll have to write a genetic programming algorithm that does prediction of atmospheric pollutant levels. Since I have no experience, can anyone point me pointers to propositions of programming languages **in which evolved programs will be written**. **Clarification:** I'm not asking what will be the language I'll write the genetic algorithm itself (as I will be able to make the decision myself), I'm asking in what programming language the evolved programs should be created. My instructor suggested Lisp, but I don't like this idea --- first I would have to work on some kind of Abstract Syntax Tree, secondly reliably doing crossovers on tree structure can be hell of a mess. I'd rather use something that is dedicated for genetic programming like slash/A. SlashA does not require working on ASTs --- programs in bytecode are a just an array of ints that can be changed in any fasion necessary since every int array represents some slash/A program. Additional remarks:   * I'd like to avoid manipulating ASTs!    * This problem is hard (maybe not as hard as predicting stock values). This is due to the fact that (most probably) we don't have enough input information (there are some hidden parameters). Creating a model that has better performance that model that returns mean is somewhat of a challenge (mean models have 35% MAPE), most models have MAPE of about 25%, best have 20%.    * I'd like to have a language that manages datasets with many features with assumption that I'm not sure which are important. (Slash/A has a disadvantage here --- in this language input features are read sequentially --- so some features will be used with bigger probability).    * I'd like to be able to program this in Python, so python libs would be great --- but I can do bindings for C/C++ (no Java, no Matlab, etc). I'm conscious this is a survey question, so if it is to early for such question please close it, but I feel it is specific enough.


title: Why eigenvalues are greater than 1 in factor analysis?

text: Why we take eigenvalue greater than 1 in factor analysis to retain factors?   And how can we decide which variables are to be chosen as factors?


title: Are associative networks still pertinent for data mining applications?

text: Back in the 1990s, network topologies like Bidirectional Associative Memory and Hopfield Nets were all the rage. Largely, these were billed as having energy states and recurrents that would help them learn/store and recall specific patterns readily. These topologies are perhaps less pertinent now, but I'm wondering if they are still an optimal way to approach data mining? They seem like they would be pertinent for things like contextual analysis in NLP, with a network trained to observe what it has "seen" before. Will these topologies come back in, or have they been replaced by options with better accuracy rates?


title: Dealing with datasets with a variable number of features

text: What are some approaches for classifying data with a variable number of features? As an example, consider a problem where each data point is a vector of x and y points, and we don't have the same number of points for each instance. Can we treat each pair of x and y points as a feature? Or should we just summarize the points somehow so each data point has a fixed number of features?


title: Determine most likely values for two variables given the probability distribution of their differences

text: I am attempting to measure two variables directly, $A$ and $B$, as well as measuring their difference $C = A-B$. Each of these measurements have a degree of uncertainty. That is, suppose that the measurements for $A,B$ and $C$ are $\\{a_1,a_2,a_3,...,a_n\\}$, $\\{b_1,b_2,b_3,...,b_n\\}$, and $\\{c_1,c_2,c_3,...,c_n\\}$, respectively. Further, it is reasonable to assume the measurements all follow a Normal Distribution $N(x_a,\sigma_a^2)$,$N(x_b,\sigma_b^2)$ and $N(x_c,\sigma_c^2)$, if this is necessary. Especially in the case when $x_a - x_b \neq x_c$, how do I use this information to find the best estimate of A, B and C?


title: Finding inflection points in R from smoothed data

text: I have some data that I smooth using `loess`. I'd like to find the inflection points of the smoothed line. Is this possible? I'm sure someone has made a fancy method to solve this...I mean...after all, it's R! I'm fine with changing the smoothing function I use. I just used `loess` because that's what I was have used in the past. But any smoothing function is fine. I do realize that the inflection points will be dependent on the smoothing function I use. I'm okay with that. I'd like to get started by just having any smoothing function that can help spit out the inflection points. Here's the code I use:               x = seq(1,15)     y = c(4,5,6,5,5,6,7,8,7,7,6,6,7,8,9)     plot(x,y,type="l",ylim=c(3,10))     lo <- loess(y~x)     xl <- seq(min(x),max(x), (max(x) - min(x))/1000)     out = predict(lo,xl)     lines(xl, out, col='red', lwd=2)      ![enter image description here](http://i.stack.imgur.com/sZzNn.png)


title: SPSS Batch Processing Method

text: I have a more than 150 list of products that will be analyzed by regression method based on years. For example: products: `A,B,C ... N` (150th products) Then regression A based on month of years:               A(Jan, Feb, March, April... Dec)     B(Jan, Feb, March, April... Dec)     and so on...      Is there any method or formula to run this BATCH processing in SPSS, if there are some macros, where do I start?


title: Good machine learning models for confusable categories

text: I'm using the word confusable to represent similar looking glyphs in text. I'm building an optical character recognition tool with the primary goal of experimenting with machine learning – especially neural nets, which I already have some knowledge about. There is one issue that I just can't get my head around: Representing confusables. As an example `I` (capital I) and `l` (small L) looks the same in sans-serif fonts. If you extend it to the complete Unicode range, there are a lot of glyphs that look more or less the same. If I'm using supervised learning naïvely, I'm afraid that the training algorithm won't be effective because of these confusables. Maybe I need unsupervised learning. I could use pretraining on RBMs or stacked auto-encoders, but I'm not sure you can assume that all confusables are close in the output space. They might be. The task still remains to detect these areas and make a good representation. I'm basically looking for a good model to cluster confusables, which I can use to create an improved supervised learning model. I expect to do some post- analyses on the output of the classifier in order to resolve each confusable to a real Unicode character.


title: Posterior distribution as a distribution for a new random variable?

text: So in Bayesian framework one uses observed data $X=\\{x_1,...x_n\\}$ to update the prior $p(\theta)$. My question is it justified to say that $p(\theta|x_1,...,x_n)$ corresponds to a new random variable itself?


title: LibSVM: A few noisy labels turns good classifier bad. What is going wrong?

text: I'm using libSVM to train a pair of classifiers, each using an independent "view", or feature set, extracted from the same dataset, i.e. co-training. As part of the co-training method, I utilise a portion of the available labels to train each RBF SVM classifier, then have it predict labels and posteriors from the unlabelled set. The most confident results are added back into the training pool. In the initial case I begin with a large portion of training labels already known and synthetic features. In this case, test performance each iteration is good (~100%), and the selected labels for inclusion in the label pool are usually all correct (i.e. match ground truth). Around the 40th iteration, we begin to see a little noise in the selected labels (i.e. out of 5 labels selected, 2 or 3 may be incorrectly labelled). This small amount of noises causes the classifier to degenerate rapidly and on the next iteration of training, I get a classifier that only predicts class 1, so average accuracy between true pos and true neg is == 0.5, and therefore no good. I assume the line search cannot converge, so have tried different values of epsilon, turning off shrinking heuristics and different slack parameter settings. Meta-info: I'm using custom kernels, and caching my RBF kernel before passing it to libSVM. I don't alter the kernel at all besides specifying diffrent indices between runs. I've hacked my libSVM version to provide me with the objective function, however I am certain this has not lead to a bug in the code. I'm not sure what (if anything) I can do next to troubleshoot or fix the problem and not sure what is wrong besides failure to converge on the optimal solution (assuming it still exists). Edit: This actually just happened whilst ALL labels are correct. Same classifier and features.


title: Two groups are significantly different? Test to use

text: I have two groups of discrete data (integers):                 Group 1   Group 2           101       103           105       200           115       150            98       160           100       115           ...       ...      and I need to know if they are significantly different or not. For this kind of tests I know there are useful tests such as `t-test` or `Wilcoxon test`. However, I was told that this kind of statistically tests are for continuous data (not my case). Then, I used a `Chi-squared` test. However, assumptions are not met since there are a lot of cells with 0s. Even combining my data into bins (e. g. 1-19, 20-39, etc.), I have lots of cells with 0s. R throws this warning in this case:               Chi-squared approximation may be incorrect      I know as well there are the `Montecarlo simulation`. However, it is just a simulation and is always giving the same p-value, exactly the same p-value, for all my different datasets to be compared. I don't like this idea. Fisher test is practically impossible due to the size of my datasets. It is possible to use Fisher test if I group my data into bins of 1000, quite wide bins. However, I don't like this idea neither. In summary, do you know how can I deal with my data? **Notes** :   * My data is not paired.   * Group 1 has about 30.000 observations while group 2 hardly has more than 4.000.   * Extremely skewed data. Example with two of my datasets: ![Distributions of two of my datasets](http://i.stack.imgur.com/279Vx.png)


title: Why some non linear models can be fitted with the Gauss Markov assumptions?

text: Isn't a linear relationship between the dependent variable and the independent variable a necessary condition? So why semi-log and reciprocal models fit with this type of assumption?


title: Likelihood clarification

text: In a 20 station assembly line, each station has 0.5% chance of defect. at the end of the assembly line, quality control detects 80% of the defectives. they will be sent for rework. 95% of the rework can be fixed. everything will be shipped by then.   * what is the likelihood that a product goes to rework?   * what is the likelihood that a defective product shipped? Please show me how to compute this. I'm confused.


title: Feature selection based on information gain papers

text: I want to apply feature selection based on information gain: I have many features many of which are redundant. I am planning on selecting a feature and then iteratively add features that 'add the more information'. Is there any relevant paper I should read to get familiar with such methods?


title: Text Mining a single text document in RapidMiner

text: I am new to data mining and currently working on an online news article from TOI in RapidMiner. My aim is to get results which shows the most important things mentioned in the article or to find the valuable information hidden in it. I am done with document processing on the article "TOI manifesto" http://timesofindia.indiatimes.com/home/specials/lok-sabha- elections-2014/news/TOI-manifesto-An-agenda-for-the-new- government/articleshow/31973967.cms. I have applied tokenization, filtering and ngrams and got some result. Im stuck here. I don't know how to proceed further. Which technique/algorithm should i use in RapidMiner to get required results? Please give details, really need help with it.


title: How to think about iid observations

text: I'm trying to make sure my understanding of iid observations is rock solid and so I have two write-ups in which I've attempted to accurately explain the concept. Can anyone comment on the accuracy of each of these write-ups?               (1) One way to represent a population is through a random variable X, which would have     a pdf/pmf that modeled the relative frequencies of values with the population. If you     were to sample from the population such that each subject is equally likely to be     selected and so that selecting a particular subject has no bearing on the     likelihood of any other subject being selected, then each observation can be thought of     as a realization of the random variable X. As such, we denote the observations x_i.     Under these conditions the observations can be thought of as independent draws from the     random variable X, and so the observations are said to be iid (independently and     identically distributed) – i.e. the observations can be thought of as n realizations of     the random variable X.      Or               (2) One way to represent a population is through a random variable X, which would have     a pdf/pmf that modeled the relative frequencies of values within the population. If you     were to sample from the population such that each subject is equally likely to be     selected and so that selecting a particular subject has no bearing on the     likelihood of any other subject being selected, then each observation can be thought of     as a realization of the random variable X. As such, we denote the observations x_i.     Under these conditions the observations can be thought of as independent draws from     identically distributed random variables, X_1,…,X_n, where x_i is the realization of     the random variable X_i for all i. Hence, the observations are said to be iid     (independently and identically distributed).      There are a couple of portions that I want to get answers about, specifically.   * Could the statement "such that each subject is equally likely to be selected and so that" in the second sentence of each write up be removed without affecting the accuracy of the statement?   * Are both of these ways of thinking about iid draws correct?   * If both statements are correct, is it conventional to interpret the draws as being realizations from one RV, or each observation as a realization from separate but identical RV's?


title: rpart maximum input states

text: I am using rpart for decision tree models and if my dataset has columns with **around more than 25 factors/levels** , rpart gets stuck and runs indefinitely. So i just wanted to know if there is this is a known limitation or if there is way to make this work. I went through http://cran.r-project.org/web/packages/rpart/rpart.pdf but couldn't really find anything relevant P.S. I also tried the ctree function(party package) and it works fine even with 50 input states but I specifically want to use rpart


title: boosting with linear svm

text: I am working on boosting classifier. I am planning to use linear svm as the weak classifier. I am using liblinear for it. My question is how can I weight each instance of liblinear based on the boosting weights?


title: Help with proof that the expected value of $x_i$ is $\bar{X}$

text: I'm having a little trouble with the proof that the expected value of $x_i$ is $ \bar{X} $. What I have is $E[x_i]=\sum_{j=1}^{N}X_j Pr(x_i=X_j) $ Then, $Pr(x_i=X_j) = 1/N $ This is the bit I can't understand, how does that probability evaluate to that value. I know the denominator is how many ways you can choose n out N. I think that the numerator should be how many ways you can choose (n-1) out of (N-1). But I seem to have an extra n. Any advice ?


title: Difference between Gaussian Distribution and Cauchy Distribution

text: I have searched for the above topic but did not have an answer. Can someone please tell me the detailed difference between the multivariate Gaussian Distribution and multivariate Cauchy Distribution? How do they differ in sampling is Estimation of Distribution algorithm for optimisation.


title: How can find the slope of a GLM logit model?

text: I've created a logit model using GLM and I´d like to get the slope of the independent variables. I've read that using `coef()` in R it´s possible to do it, but the only thing I have back is the coefficients that I already have using `summary()`. I'm trying to get the slope at mean, like the one that is given when we use Gretl.


title: Is there a package that I can use in order to get rules for a target outcome in R

text: For example In this given data set I would like to get the best values of each variable that will yield a pre-set value of "percentage" : for example I need that the value of "percentage" will be >=0.7 so in this case the outcome should be something like: `birds >=5, 2 =< wolfs , 2<=snakes <=4`               dat <- read.table(text = "birds    wolfs     snakes  percentage     3         8          7         0.50     1         2          3         0.33     5         1          1         0.66     6         3          2         0.80     5         2          4         0.74",header = TRUE)      I can't use decision trees as I have a large data frame and I can't see all tree correctly. I tried the arules package as but it required that all variables will be factors and I have mixed dataset of factor,logical and continuous variables.Also I need "percentage" variable to be the only one that I would like to optimize. Here is my code :               library(arules)         dat$birds<-as.factor(dat$birds)         dat$wolfs<-as.factor(dat$wolfs)         dat$snakes<-as.factor(dat$snakes)         dat$percentage<-as.factor(dat$percentage)         rules<-apriori(dat, parameter = list(minlen=2, supp=0.005, conf=0.8))         rules.sorted <- sort(rules, by="lift")         inspect(rules.sorted)      Thank you


title: A good source to understand interaction in LMM?

text: I would like to know more about interaction in LMM using lmer. Can you recommend me any books, articles, websites?


title: Mlogit singularity issues in R when using both alternative and individual variables

text: My data describes the choices of visitors to a forest park between different regions within it. The data has both variables describing the region and the different individuals, and is in the long format (I wasn't able to import the data in a wide format). Here's a part of it:                     pId zoneID choice perShaded Avg_picnicDist Avg_searchDist Avg_roadDist Avg_density Avg_slope Avg_aspect NumTables No_visits purpose parks_visits zone_ religion     1.1   K10      1  FALSE  0.674165       34.28285       149.9919     47.68274  184.460488  4.647293  241.01953         2         2       2            2     6        1     1.2   K10      2  FALSE  0.518358       46.99878       699.8933    257.11613  180.035801  8.299255  267.68265         8         2       2            2     6        1     1.3   K10      3  FALSE  0.013443      238.19874      1895.3234    848.11628   40.816165  7.337484  211.19033         0         2       2            2     6        1     1.4   K10      4  FALSE  0.000008      118.35318      1933.9713    209.80435   88.661727  8.427688  239.75810         2         2       2            2     6        1     1.5   K10      5  FALSE  0.480388       85.79740       860.6304    162.50307  184.787958  9.016304  174.94392         3         2       2            2     6        1     1.6   K10      6  FALSE  0.390045       62.67671       809.3929     73.73866  652.461340  6.966383  270.83758         1         2       2            2     6        1     1.7   K10      7  FALSE  0.247667       88.19777       447.9074     50.38104  372.559662  6.661059  128.33136         0         2       2            2     6        1     1.8   K10      8  FALSE  0.138269      236.60154       891.2010    315.48513  141.750924  5.079428  179.26789         0         2       2            2     6        1     1.9   K10      9  FALSE  0.327164      152.13491      1187.2389    196.26516  246.658501  7.262928  249.06325         1         2       2            2     6        1     1.10  K10     10  FALSE  0.605936      127.08910      1934.2726    244.03761  101.317788  7.222477  263.82949         1         2       2            2     6        1     1.11  K10     11  FALSE  0.342743      262.25532      2010.6362    741.54838   50.244833  7.218908  264.98949         1         2       2            2     6        1     1.12  K10     12  FALSE  0.494176      115.57820      1140.5434    220.95758  194.244219  6.649306  236.02466         1         2       2            2     6        1     1.13  K10     13  FALSE  0.136447      269.81593      3276.1774    972.70061   17.044701  6.213433  193.13001         2         2       2            2     6        1     1.14  K10     14   TRUE  0.337722      232.84797      2884.2213    950.00891   44.063617  7.101079  109.28240         1         2       2            2     6        1     1.15  K10     15  FALSE  0.303521      110.34073      3785.6749   1501.10647   17.479083  9.837794  109.91601         5         2       2            2     6        1     1.16  K10     16  FALSE  0.202760      271.45326      4299.1473   2010.62657    4.260328  6.756867  140.85512         0         2       2            2     6        1     1.17  K10     17  FALSE  0.315446      490.10473      3601.0404   1871.80162    4.995194  7.776252   69.51587         0         2       2            2     6        1     1.18  K10     18  FALSE  0.000000      832.24798      5472.1534   3132.09387    1.090896  6.171129  123.73959         0         2       2            2     6        1     1.19  K10     19  FALSE  0.377565     1832.27872      6639.4049   4163.90939    0.000000  6.695283  219.51917         0         2       2            2     6        1     1.20  K10     20  FALSE  0.768519      240.50203      1317.7622    984.45432   55.064831  6.189258  235.94524         1         2       2            2     6        1     1.21  K10     21  FALSE  0.457788      370.39337      1638.3757   1186.05062   23.498703  7.899760  240.19327         1         2       2            2     6        1     1.22  K10     22  FALSE  0.000000      487.34645      3024.9163   1906.07303    4.894370  9.386472  204.43018         0         2       2            2     6        1     1.23  K10     23  FALSE  0.402916      810.55158      2390.3867   1598.35349    0.000000  7.234100  221.55110         0         2       2            2     6        1     1.24  K10     24  FALSE  0.000000      466.83673      4156.6922   1901.71082    1.665232  7.067360  221.62408         0         2       2            2     6        1     1.25  K10     25  FALSE  0.209725      621.64548      3936.0332   2509.64374    0.669637  8.861591  204.69257         0         2       2            2     6        1     1.26  K10     26  FALSE  0.173588      525.66404      4284.6076   1965.34834    2.222979  7.723666  168.04920         0         2       2            2     6        1     1.27  K10     27  FALSE  0.223391      331.13870      4460.3144   2175.40860    0.000000  5.132271  234.74961         1         2       2            2     6        1     1.28  K10     28  FALSE  0.544493      605.89578      4966.5343   2660.61470    0.000000  7.431034  182.73256         0         2       2            2     6        1     1.29  K10     29  FALSE  0.111995      892.92150      5915.3593   3599.12409    0.000000  4.675676  232.14156         0         2       2            2     6        1     2.1  K100      1   TRUE  0.674165       34.28285       149.9919     47.68274  184.460488  4.647293  241.01953         2         2       1            2     6        1     2.2  K100      2  FALSE  0.518358       46.99878       699.8933    257.11613  180.035801  8.299255  267.68265         8         2       1            2     6        1     2.3  K100      3  FALSE  0.013443      238.19874      1895.3234    848.11628   40.816165  7.337484  211.19033         0         2       1            2     6        1     2.4  K100      4  FALSE  0.000008      118.35318      1933.9713    209.80435   88.661727  8.427688  239.75810         2         2       1            2     6        1     2.5  K100      5  FALSE  0.480388       85.79740       860.6304    162.50307  184.787958  9.016304  174.94392         3         2       1            2     6        1     2.6  K100      6  FALSE  0.390045       62.67671       809.3929     73.73866  652.461340  6.966383  270.83758         1         2       1            2     6        1     2.7  K100      7  FALSE  0.247667       88.19777       447.9074     50.38104  372.559662  6.661059  128.33136         0         2       1            2     6        1     2.8  K100      8  FALSE  0.138269      236.60154       891.2010    315.48513  141.750924  5.079428  179.26789         0         2       1            2     6        1     2.9  K100      9  FALSE  0.327164      152.13491      1187.2389    196.26516  246.658501  7.262928  249.06325         1         2       1            2     6        1     2.10 K100     10  FALSE  0.605936      127.08910      1934.2726    244.03761  101.317788  7.222477  263.82949         1         2       1            2     6        1     2.11 K100     11  FALSE  0.342743      262.25532      2010.6362    741.54838   50.244833  7.218908  264.98949         1         2       1            2     6        1     2.12 K100     12  FALSE  0.494176      115.57820      1140.5434    220.95758  194.244219  6.649306  236.02466         1         2       1            2     6        1     2.13 K100     13  FALSE  0.136447      269.81593      3276.1774    972.70061   17.044701  6.213433  193.13001         2         2       1            2     6        1     2.14 K100     14  FALSE  0.337722      232.84797      2884.2213    950.00891   44.063617  7.101079  109.28240         1         2       1            2     6        1     2.15 K100     15  FALSE  0.303521      110.34073      3785.6749   1501.10647   17.479083  9.837794  109.91601         5         2       1            2     6        1     2.16 K100     16  FALSE  0.202760      271.45326      4299.1473   2010.62657    4.260328  6.756867  140.85512         0         2       1            2     6        1     2.17 K100     17  FALSE  0.315446      490.10473      3601.0404   1871.80162    4.995194  7.776252   69.51587         0         2       1            2     6        1     2.18 K100     18  FALSE  0.000000      832.24798      5472.1534   3132.09387    1.090896  6.171129  123.73959         0         2       1            2     6        1     2.19 K100     19  FALSE  0.377565     1832.27872      6639.4049   4163.90939    0.000000  6.695283  219.51917         0         2       1            2     6        1     2.20 K100     20  FALSE  0.768519      240.50203      1317.7622    984.45432   55.064831  6.189258  235.94524         1         2       1            2     6        1     2.21 K100     21  FALSE  0.457788      370.39337      1638.3757   1186.05062   23.498703  7.899760  240.19327         1         2       1            2     6        1          education age group_type group_type2 income former_visits parking playground tables weather group_ av_duration GPS research lake     1.1          2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.2          2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.3          2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.4          2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.5          2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.6          2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.7          2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.8          2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.9          2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.10         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.11         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.12         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.13         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.14         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.15         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.16         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.17         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.18         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.19         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.20         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.21         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.22         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.23         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.24         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.25         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.26         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.27         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.28         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     1.29         2  52          3           2      3             7       7          1      1       1      4           7   1        1    7     2.1          4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.2          4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.3          4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.4          4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.5          4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.6          4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.7          4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.8          4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.9          4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.10         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.11         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.12         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.13         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.14         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.15         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.16         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.17         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.18         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.19         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.20         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1     2.21         4  37          4           1      4             7       5          3      5       7      7           7   1        1    1      All the variables describing alternatives (up to NumTables) are continuous and some of them present multicolinearity. All the individual specific variables are either categorical or nominal. When I run the mlogit model using no individual specific variables, it works fine. For example:               Call:     mlogit(formula = choice ~ perShaded + Avg_roadDist + Avg_density +          Avg_searchDist | 0 | 0, data = DATA, method = "nr", print.level = 0)          Frequencies of alternatives:             1         2         3         4         5         6         7         8         9        10        11        12        13        14        15        16      0.0378151 0.1890756 0.0084034 0.0546218 0.0546218 0.2394958 0.0168067 0.0000000 0.0924370 0.0252101 0.0168067 0.1134454 0.0084034 0.0588235 0.0168067 0.0000000             17        18        19        20        21        22        23        24        25        26        27        28        29      0.0084034 0.0042017 0.0000000 0.0210084 0.0252101 0.0000000 0.0000000 0.0042017 0.0000000 0.0042017 0.0000000 0.0000000 0.0000000           nr method     6 iterations, 0h:0m:0s      g'(-H)^-1g = 3.76E-08      gradient close to zero           Coefficients :                       Estimate  Std. Error t-value  Pr(>|t|)         perShaded       1.59487156  0.40671788  3.9213 8.806e-05 ***     Avg_roadDist   -0.00152363  0.00026383 -5.7750 7.695e-09 ***     Avg_density     0.00255828  0.00035621  7.1819 6.877e-13 ***     Avg_searchDist  0.00034052  0.00014350  2.3730   0.01765 *       ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1          Log-Likelihood: -637.99      The model runs successfully even though some of the variables are highly correlated. The model also works when I run it using only individual specific variables, such as: `model1<-mlogit(choice~1|factor(No_visits)|0,DATA)` Whenever I try to use both types of variables, like in `model1<-mlogit(choice~perShaded|factor(No_visits)|0,DATA)`,I always get the               Error in solve.default(H, g[!fixed]) :        system is computationally singular: reciprocal condition number = 7.0136e-18      message, no matter in which part of the formula I place any of the variables. It's the first time I've tried to create such a model (and also started using R only recently), so maybe something I'm doing is fundamentally wrong, but I couldn't find any solution in other posts, except checking my data, which seems to be fine. I would appreciate any help.


title: Beginner's Question about Plotting in R

text: Alright, so I'm new to R, and I imagine what I'm trying to do should be very simple. I want to use plot(x,y), but then add the curve lm(y~x^2) over the plot. I'm sure I could add more detail this, but I am not sure what is needed, so I'll be vigilant to answer any questions you all may have. I appreciate any help I can get!


title: Why Bayes Rule in Naive Bayes compared to simple P(class|features)

text: I would like to improve on my recommendation system. Imagine I have training data of $M=7,000,000$ samples. Each training sample contains a variable number of words in the body, and a variable amount of tags assigned to the case. There are roughly 100 million unique words, and roughly 40,000 unique tags. The goal is to, given a set of words, to correctly classify the tag(s) when the tags are not observed. Intuition tells me to simply compute the probability for each word, and then use the conditional probability for each tag given the set of words for a tag. More concretely: Imagine one case has 5 unique words. Each word will have a $P(W_i)$. This is a constant value for the training set, and can be stored in memory. The probability of $T_1$ given $P(W_1, W_2....W_5)$, is then simply the number of occurrences in the intersection of $T_1$ for each word divided by $M$. Why does Naive Bayes suggest using (according to the Bayes rule) $\frac{P(W_1..W_5|T_1) * P(T_1)}{P(W_1...W_5)}$ when $P(T_1|W_1...W_5)$ seems easier to obtain, and more intuitive*? * _"What is the probability of a tag given some combination of words?"_ seems to be exactly what we want to know.


title: Extrapolate a proportion from unequal subsamples

text: I have a database of species records. Sample X species (number of individuals). Unidentified individuals were classified as "other". In 4% of these samples the category "other" was subsampled (unequal subsamples) and the individuals were identified at the species level. Now I'd like to estimate the size of the "other" category for the entire database. As it's only a subsample, I thought to calculate the proportion and define a discrete probability distribution to distribute the individuals for the "other" category into the newly defined species variables. Is this an adequate procedure? Are there more adequate techniques?


title: Selecting features manually and proving the rest are redundant

text: I'm working with a gesture dataset, where each gesture has a variable number of frames, and each frame has the 3d position of 20 joints, so that each gesture is represented by a matrix of size frames x 60. I know that some joints are redundant, since for example knowing the position of both shoulders pretty much determines the position of the chest and viceversa, at least for the poses in the gestures in my dataset. Running PCA on the matrix of all the gestures stacked horizontally, I get that with just 30 dimensions I can retain 99% of the variance, but of course this is in the eigenvectors space. How can I select a subset of the joints (equivalently, features) and prove that the rest are redundant, in a PCA sort of way? The simplest thing I could think of was to select some joints, use them as basis, project the frames onto the space they generate, and use the result as features, but a) the classification experiments I did with that didn't turn out well and b) I've no way of formally justifying the removal of features/joints with that approach.


title: factor analysis problem in structural equation model

text: > ![enter image description here](http://i.stack.imgur.com/E2C95.png) I am using an Extended Technology Acceptance Model and have adopted an instrument from a research paper. After performing factor analysis in SPSS, most questions related to different constructs (attitude, perceived usefulness, etc.) are shown in the same factor. The variables `Attitude`, `Perceived ease of use`, `Perceived usefulness` & `Innovativeness` are different but if a respondent agrees to my statements for `Perceived usefulness` with `STRONGLY AGREE` then he is also likely to answer for statements related to attitude with `STRONGLY AGREE` since there is a positive effect (theoretically and based upon lit. reviews) between all these variables in the model. What should I do if respondents are answering everything as `STRONGLY AGREE` or `AGREE`? Technically it is right, but how can I get my factor analysis results right? Excluding the dependent variable (`Behavioral Intention`) it should make 4 factors. After then I've got to learn Amos (as per my supervisor instruction) for analysis. I need guidance in easy wordings. I'm no good in SPSS and not familiar with Amos.


title: Which permutation test implementation in R to use instead of t-tests (paired and non-paired)?

text: I have data from an experiment that I analyzed using t-tests. The dependent variable is interval scaled and the data are either unpaired (i.e., 2 groups) or paired (i.e., within-subjects). E.g. (within subjects):               x1 <- c(99, 99.5, 65, 100, 99, 99.5, 99, 99.5, 99.5, 57, 100, 99.5,              99.5, 99, 99, 99.5, 89.5, 99.5, 100, 99.5)     y1 <- c(99, 99.5, 99.5, 0, 50, 100, 99.5, 99.5, 0, 99.5, 99.5, 90,              80, 0, 99, 0, 74.5, 0, 100, 49.5)      However, the data are not normal so one reviewer asked us to use something other than the t-test. However, as one can easily see, the data are not only not normally distributed, but the distributions are not equal between conditions: ![alt text](http://i.stack.imgur.com/5y0kx.png) Therefore, the usual nonparametric tests, the Mann-Whitney-U-Test (unpaired) and the Wilcoxon Test (paired), cannot be used as they require equal distributions between conditions. Hence, I decided that some resampling or permutation test would be best. **Now, I am looking for an R implementation of a permutation-based equivalent of the t-test, or any other advice on what to do with the data.** I know that there are some R-packages that can do this for me (e.g., coin, perm, exactRankTest, etc.), but I don't know which one to pick. So, if somebody with some experience using these tests could give me a kick-start, that would be ubercool. **UPDATE:** It would be ideal if you could provide an example of how to report the results from this test.


title: Is it possible to weight survey observations to more than one distribution?

text: I have a sample survey of a population whose distributions of certain characteristics are not identical to the distributions of the overall population. For example, the age of my respondents may be biased downward, or the incomes in my sample may be too high compared to the population (or my theoretical population distribution, if the actual population distribution is unknown). I know that it is possible to calculate a weighting coefficient that can adjust the distribution of the sample to match the population distribution on one dimension, but is it possible to adjust for two dimensions (e.g. age and income)? If so, couldn't there be a situation where there is no solution (no single coefficient that will adjust the sample distribution to the population distribution on both dimensions?)


title: Simple and multiple logistic regression

text: I have almost two questions. I need a single covariate logistic regression (LR) for each of my variables. Should I do it manually in SPSS, selecting each variable and do logistic regression? Is there a "for each" cycle to do it? I should switch to R language to have what I want. In the multivariables (multi covariates) LR, could I have missing values? Thanks!!


title: PCA on a heterogenous correlation matrix better than MCA?

text: I have a questionnaire of 80 questions that I need to do dimension reduction on. About half the questions are ordinal (Likert-style questions), and the other half are qualitative/nominal. I've been given a suggestion to perform Multiple Correspondence Analysis on the dataset, but I was wondering, would it be methodologically sound to perform PCA based on a heterogenous correlation matrix that contains polychoric and polyserial correlations (as produced by the hetcor() function in John Fox's 'polycor' R package). This makes sense to me, as PCA is basically a mathematical technique for calculating loadings and eigenvalues based on correlations. Any thoughts? Thanks!


title: Encoding categorical features to numbers for machine learning

text: Many machine learning algorithms, for example neural networks, expect to deal with numbers. So, when you have a categorical data, you need to convert it. By categorical I mean, for example: Car brands: Audi, BMW, Chevrolet... User IDs: 1, 25, 26, 28... Even though user ids are numbers, they are just labels, and do not mean anyting in terms of continuity, like age or sum of money. So, the basic approach seems to use binary vectors to encode categories: Audi: 1, 0, 0... BMW: 0, 1, 0... Chevrolet: 0, 0, 1... It's OK when there are few categories, but beyond that it looks a bit inefficient. For example, when you have 10 000 user ids to encode, it's 10 000 features. The question is, is there a better way? Maybe one involving probabilities?


title: Should I expect a Z-Score as an output from a Signed Rank test?

text: I have some data that is highly skewed. I want to test whether the median of the data is different from 1, so I decided to use a Wilcoxon Signed Rank test. The description and implementation of this test in Stata and SAS is given here: http://www.ats.ucla.edu/stat/sas/whatstat/whatstat.htm#1sampm http://www.ats.ucla.edu/stat/stata/whatstat/whatstat.htm#1sampm There's no Z-Score reported in SAS, only an 'S' stat and a p-value. I need to report a statistic, but the 'S' stat reported by SAS looks like a variance or something. Is there something else going on here? I think both the SAS and Stata examples on this page use the same 'hsb2' dataset available from http://www.ats.ucla.edu/stat/data/hsb2.sas7bdat so the results should be the same.


title: Frequency of combinations possible

text: I have an SPSS data set with 700+ respondents and 13 products that they could own. Each product has its own variable Product01 = 1 means they own product 1 Product02 = 0 means they dont own product 2 etc etc *Edit for clarity: For example, this would mean counting: How many respondents have product 1, 2, 3 and none of the other 10. How many respondents have product 5 and 8 and none of the other 11. etc. Does anyone have an spss syntax that could determine the frequency of the possible combinations found from the data? Similar to this: Computing combinations and this: http://pages.infinit.net/rlevesqu/Syntax/Combinations/FindAllCombinationsOf1upToNitemsOutOfMitems.txt


title: How to calculate the weighted covariance?

text: I'm trying to calculate the weighted co-variance by hand to better understand what is going on. I have read the Wikipeida article and I understand the concept. However, when plugging in numerical values I encounter the following problem: For example assume I have three observations as given in matrix A $A = \left[ \begin{array}{ccc} 1 & 0 & 0 \\\ 0 & 1 & 0 \\\ 0 & 0 & 1 \\\ \end{array} \right]$ each row vector of matrix A has a weight value associated with it as $w_i = (0.5, 0.33, 0.17) $ Using the equation in Wikipedia which is: $$ q_{jk} = \frac{\sum_{i = 1}^{N} w_i}{(\sum_{i = 1}^{N} w_i)^2 - \sum_{i = 1}^{N} w_{i}^2} \sum_{i = 1}^{N} w_i (x_{ij} - \bar{x}_j)(x_{ik} - \bar{x}_k) $$ if I'm to calculate $q_{21}$ the derivation would look like the following $$q_{21} = \left[ \frac{1}{1 - (0.25 + 0.1089 + 0.0289 )} \right]\left(w_i(1-0.5)(0 - 0.3) + w_i(0-0.5)(1 - 0.3) + w_i(0-0.5)(0 - 0.3)\right) $$ what should I plug in to $w_i$ ? Is the other parts of the substitution correctly done?


title: Logistic Regression and Inflection Point

text: We have data with a binary outcome and some covariates. I used logistic regression to model the data. Just a simple analysis, nothing extraordinary. The final output is supposed to be a dose-response curve where we show how the probability changes for a specific covariate. Something like this: ![enter image description here](http://i.stack.imgur.com/0I8uo.png) We received some criticism from an internal reviewer (not a pure statistician) for choosing logistic regression. Logistic regression assumes (or defines) that the inflection point of the S-shaped curve on the probability scale is at probability 0.5. He argued that there would be no reason to assume that the inflection point was indeed at probability 0.5 and we should choose a different regression model that allows the inflection point to vary such that the actual position is data driven. At first I was caught off guard by his argument, since I have never thought about this point. I did not have any arguments to why it would be justified to assume that the inflection point is at 0.5. After doing some research, I still don't have an answer to this question. I came across 5-parameter logistic regression, for which the inflection point is an additional parameter, but it seems that this regression model is usually used when producing dose-response curves with a continuous outcome. I'm not sure if and how it can be extended to binary response variables. I guess my main question is why or when it is OK to assume that the inflection point for a logistic regression is at 0.5? Does it even matter? I have never seen anybody fitting a logistic regression model and explicitly discussing the matter of the inflection point. Are there alternatives for creating a dose response curve where the inflection point is not necessarily at 0.5? Just for completeness, the R code for generating the above picture:               dat <- read.csv("http://www.ats.ucla.edu/stat/data/binary.csv")     dat$rank <- factor(dat$rank)     logit <- glm(admit ~ gre + gpa + rank, family = binomial(link = "logit"), data = dat)     newdata <- data.frame(gre = seq(-2000,8000,1), gpa = 2.5, rank = factor(1,c(1,2,3,4)))     pp <- predict(logit, newdata, type = "response", se.fit = TRUE)     plot(newdata$gre, pp$fit, type="l", col="black", lwd=2,ylab="Probability", xlab="Dose")      **Edit 1:** Just to add to what Scortchi said in one of the comments: The reviewer did indeed argue that biologically it might be more likely that the change in curvature occurs earlier than 0.5. Therefore his resistance against assuming that the inflection point is at 0.5. **Edit 2:** As a reaction to the comment by Frank Harrell: As example, I modified my model above to include a quadratic and a cubic term in `gre` (which is the "dose" in this example).               logit <- glm(admit ~ gre+I(gre^2)+I(gre^3)+  gpa + rank, family = binomial(link = "logit"), data = dat)     newdata <- data.frame(admit=1, gre = seq(-2000,8000,1), gpa = 2.5, rank = factor(1,c(1,2,3,4)))     pp <- predict(logit, newdata, type = "response", se.fit = TRUE)     plot(newdata$gre, pp$fit, type="l", col="black", lwd=2,xlim=c(-2000,4000),ylab="Probability", xlab="Dose")      ![enter image description here](http://i.stack.imgur.com/0JpC3.png) Despite the fact that it is probably not meaningful to add a quadratic and a cubic `gre` term in this case, we see that the form of the dose-response curve has changed. Indeed we now have two inflection points at about 0.25 and near 0.7.


title: GAM model for proportion data

text: I'm trying to model number of devices failed out of total produced using GAM. I want the value to be bound between zero and one. Couldn't find anything that can solve this problem. Any help will be greatly appreciated. (I have number of failures, total produced and the factors driving failure for each batch produced) I tried, Failure rate ~ s(factor1)+s(factor2)+s(factor3) But there is no condition mentioning 'Failure rate' should be bound between 0 and 1.


title: Use a clustering as a segmentation

text: I ran a cluster analysis on a population of customers. I used variables like:   * Lifetime   * Spent amount   * etc. Now I'd like to use these clusters ('Little buyer', 'Regular fan of the brand A', ...) for segmentation purposes. **Problem:** Future customers will appear, but I can't rerun a cluster analysis each day, so I need to assign the new customers to the current clusters. **Proposal:** I can assign the future customers to the nearest centroid of the current clusters. The new customers are assigned to clusters which have not been built for them, if the distribution of the new customers is not the same than the distribution of the current ones. Thus, the clustering is going to deteriorate. **Question:** How can I monitor the evolution of the quality of the clustering / segmentation? I was thinking about monitoring the evolution of the R-Squared (Coefficient of determination), because it's the metric I used to choose the actual clustering, but I'm not sure it's a best practice.


title: What does it mean that the true effect size is not known?

text: I just read this statement: > In neuroscience the true effect size is not known What does this mean? In contrast, what does it mean that where the true effect size is known?


title: 1 control group vs. 2 treatments: one ANOVA or two t-tests? (3 or more groups = ANOVA but explanation needed)

text: I have 3 groups, one control group and two treatment groups that have nothing to do with each other. I just want to see if any of those 2 treatment are different from the control group. Textbooks say 3 groups or more, use ANOVA to avoid type-1 errors. But I just don't understand why that applies here. On the one hand, it's only 2 t-tests so doesn't seem like the chance of error is greatly increased. And on the other hand, being that have no interest in differences between the treatment groups, by doing an ANOVA can it not be misleading? (if the two treatments are different from each other but not from the control). So, should I do an ANOVA and if the means are different perform a "Dunnett's"? Or just do the 2 t-tests?


title: Distribution function of maximum of n iid standard uniform random variables where n is poisson distributed

text: I am studying probability theory on my own and am trying to work the following problem in the book - Let $X_1, X_2, . . .$ be independent, $U(0, 1)$-distributed random variables, and let $Nm \in Po(m)$ be independent of $X_1, X_2, . . . .$ Set $V_m = max\\{X_1, . . . ,X_{Nm}\\} (Vm = 0 \ when\ Nm = 0)$. Determine (a) the distribution function of $V_m$, (b) the moment generating function of $V_m$. (c) Show that $E[Vm] \to 1$ as $m \to \infty$. (d) Show that $m(1 − Vm)$ converges in distribution as $m \to \infty$, and determine the limit distribution. I got stumped at the very first part when trying to find the distribution function of $V_m$. I know that the CDF of $X$ is given by - $F_X(x) = \begin{cases} 0, & x < 0 \\\ x, & 0 \leq x \leq 1 \\\ 1 & x > 1 \end{cases}$ This implies that the CDF of $V_m$ should be given by $F_{V_m}(x) = \begin{cases} 0, & x < 0 \\\ x^n, & 0 \leq x \leq 1 \\\ 1 & x > 1 \end{cases}$ where $n \in Po(m)$ Then by differentiating $F_{V_m}(x)$, we get the density function $f_{V_m} = nx^{n-1}$. Then using conditional probability, we want to find the CDF of $V_m$ and I did the following to get there - $P(V_m = x) = \sum_{n=0}^{\infty} P(V_m = x|N_m=n) . f_N(n) = \sum_{n=1}^{\infty} nx^{n-1} e^{-m} \frac{m^n}{n!} = me^{m(x-1)}$ The problem is when I try to integrate this over x from 0 to 1,the answer is not 1 which implies that this might not be the correct pdf and hence integrating this may not give me the correct CDF. Incidentally, when integrating this over 1 to x, I get $ e^{m(x-1)} - e^{-m}$ which is not the correct answer. I am not sure if I have made a big blunder in my understanding of something or a small goof up. I have looked at my calculations a few times to ensure that I have not made any silly error. I think once I am past this first part, I should be able to get to the rest. I would highly appreciate any help as I feel that this is essential for me to understand before I move forward with my course. Thanks to anyone ande


title: Combining probabilities for class prediction

text: Before I release a new web service, I am trying to develop my data analytics skills so I can monitor the performance of the service. I may end up using and external service (eg: Google Analytics) or an analytics library, but I need to understand how the algorithms work before I can be confident I am using and interpreting the analytics correctly. The best way I learn is to build software, but I am stuck on a (probably simple to experienced analysts) problem: I have a dataset of 10,000 items. There are 100 labelled with class A (the items can have multiple classes). I note that 60 of the class A items have attribute X. I also count the other significant attributes that seem to correlate with items of class A. I might then think that any new item with attribute X has a 60% chance of being a class A item (ignoring the influence of other attributes for the present). But I also calculate that 20% of the items of the whole dataset have attribute X. How do I calculate the change in my prediction probability? I suspect a Bayesian approach would be suitable but the results do not look right. If I make the same calculations with other classes and other attributes, how do I merge the probabilities? (eg: to find the probability of a new item with attributes X,Y having class A, I look up P(A|X)=0.3 and P(A|Y)=0.5, then...). I tried the p=ab/(ab+(1-a)(1-b)) merging formula but this seems wrong. Some gentle guidance would be appreciated.


title: Mutually independent i.i.d. random variable

text: If the demand for labour depends on the real wage, assuming that the error terms are mutually independent i.i.d. random variables, both with mean zero, how can we show that the real wage and the error term of the labor demand equation are correlated? Edit: This is a question with 2 equations. Labor demand and labor supply. Labor supply is vertical and does not depend on real wage whereas labor demand is dependent on real wage. Both equations have an error term at the end.


title: Asymptotic Least Squares question (with random regressors)

text: > Consider the DGP $y_i=x_i+\epsilon_i$, where $\epsilon_i \sim Z$. We > estimate $\beta=1$ by regression without a constant term, so in $y_i=\beta > x_i + \epsilon_i$. > >   1. Show that this DGP does not satisfy the stability assumption > ($\operatorname{plim}(\frac{1}{n}X'X) \rightarrow Q, Q$ invertible) and show > that the speed of convergence of $b$ to $\beta$ is $n\sqrt{n}$ in this case. > (Hint: $\sum_{i=1}^n i^2=\frac{1}{6}n(n+1)(2n+1))$ >   2. Let $x_i=\frac{1}{i}$. Show that again the stability assumption is not > satisfied and that the speed of convergence is $n^0$. (Hint: > $\sum_{i=1}^\infty \frac{1}{i^2}=\frac{1}{6}\pi^2$). > What I tried for (1): $\frac{1}{n}x'x=\frac{1}{n}\sum x_i^2=\frac{1}{6}(n+1)(2n+1)$. Which does not satisfy the assumption I guess? Also \begin{align*} n^p(b-\beta) &= (x'x)^{-1} x'\epsilon\\\ &= \left(\frac{1}{n^p} x'x\right)^{-1} x'\epsilon\\\ &= \frac{6n^p}{n(n+1)(2n+1)} x'\epsilon\end{align*} And I think that I need to show that if I would take the limit $n\rightarrow \infty$, the first term would evaluate to a constant if $p=\frac{3}{2}$. I don't however see why this is so ($p=3$ would follow from my derivation I believe). What I tried for (2). $\frac{1}{n}x'x=\frac{1}{n}\sum_{i=1}^n \frac{1}{i^2}$. Again this doesn't satisfy the assumption? And I'm not sure about the second part of this subquestion. Thank you in advance for your help :)!


title: Is there a function in R that takes the centers of clusters that were found and assigns clusters to a new data set

text: I have two parts of a multidimensional data set, let's call them train and test. And I want to built a model based on the train data set and then validate it on the test data set. The number of clusters is known. I tried to apply kmeans clustering in R and I got an object that contains the centers of clusters: kClust <\- kmeans(train, centers=N, nstart=M); Is there a function in R that takes the centers of clusters that were found and assigns clusters to my test data set? What are the other methods/algorithms that I can try? Thank you


title: Detecting the period of a signal with a length of only a few periods

text: > **Possible Duplicate:**   >  Fourier data with non-integer periods, correcting for phase bias I have a not perfectly but quit periodic signal, with a length of only a few periods. What methods are available for automatically measuring its period? I tried using the maximum of the Fourier transform, but this essentially only tells me how many integer periods are there in the full length of the signal. If I only have a few repetitions, this will give me an inaccurate estimate of the period. Let me illustrate: suppose we have a perfectly periodic function $sin(x)$ sampled at intervals of 0.01 between 0 and 30. The Fourier transform will tell me that the signal is about 5 periods long, which gives an inaccurate period length of 30/5 = 6, instead of the true one which is $2\pi \approx 6.28$. Since this example signal is almost perfectly periodic and sampled at a high density, it is possible to do much better than this. **What robust ways are there for period detection that might work in this scenario?** **EDIT:** Here is an example signal of 1000 sample points to test on. I am expecting a period of around ~150, by manual measurement (i.e. 6 repetitions in a length of ~898). You may argue that it should be double that value if you look closely at the alternating height of the small humps, but as the signal is not _perfectly_ periodic, for this application I need ~150.


title: Gamma Distribution MGF

text: ![enter image description here](http://i.stack.imgur.com/R0NYV.png) I'm not sure how to find part a, what formula can be used to find this mgf?


title: Implementation and Interpretation of Fixed versus Random Effects

text: I am reading an article which uses a simple least squares model to measure the effect of a prevention campaign on methamphetamine use (http://www.ncbi.nlm.nih.gov/pubmed/20638737). In its second equation, it uses state fixed effects in an OLS model to capture the effect of living in each state.               Meth ~ constant + Betas*Attributes + State1Beta*State1 + ... +              StateZBeta * StateZ      I have two questions:   1. Is the only difference between using random and fixed effects in this instance the distribution in which the values fall (ie, a random effect would have a mean of 0 and a normal distribution)? And,    2. How does this jive with the interpretation that random effects are intercept shifts, and fixed effects are slope shifts?


title: How should I analyze my data?

text: I am new here, so I hope this question is on topic. I understand a little about statistics, but my knowledge of what methods to use in real case scenarios is a little limited. I want to statistically analyse the accuracy of my developed system. This system is used to obtain a specific measure of an user, let's call it _Method A_. Only the user can use the system to obtain his/her own measure. To compare the results I can take that measure of an user using two other methods, but these are operated by myself ( _Method B_ and _Method C_ ). So what I have is a sample of 40 subjects and three measures of that distance, for each subject. Two that come from already existing systems and the third one comes from my developed system. Below there is a table that reflects the possible results of this test. Note that due to budget limitations I can't really acquire more data than this. Not even take more measures of a subject with my system.               Subject | Method A  | Method B  | Method C         1       | 122.6     | 123.6     | 125.6          2       | 102.6     | 104.4     | 105.2         (...)   | (...)     | (...)     | (...)          40      | 152.6     | 151.0     | 149.7      The two measures from the already existing system are also not considered correct, that is. They are the methods in which the community most relies to obtain that particular measure, but are not considered as "correct". Note that each subject has a different measure and the measure is not related inter- subjects. That is, any two subjects can have entirely different measures. How can I statistically analyse my data in order to prove that my system is accurate or not. Or at least how to show that the measures that I obtain from my method are close to the establish methods. Which kind of statistical analysis should I perform with my data to obtain either conclusion? Is there any modifications I can add to the design of my experiment in order to help prove the accuracy of my system? Thanks in advance.a


title: Question about weighted logistic regression

text: I'm helping a friend perform a logistic regression on field data pulled from several archeological sites. By chance the overwhelming majority of records in the analysis dataset were collected from a single site. In order to prevent the characteristics of this one site "overwhelming" the other sites, is it appropriate to apply weights (by site) to the records, say weight=1/(stnd error)?


title: What causes the discontinuity in the distribution of published p-values at p < .05?

text: In a recent paper , Masicampo and Lalande (M-L) collected a large number of p-values published in many different studies. They observed a curious jump in the histogram of the p-values right at the canonical critical level of 5%. There is a nice discussion about this M-L Phenomena on Prof. Wasserman's blog: http://normaldeviate.wordpress.com/2012/08/16/p-values-gone-wild-and- multiscale-madness/ On his blog, you will find the histogram: ![Histogram of published p-values](http://i.stack.imgur.com/EFtVY.png) Since the 5% level is a convention and not a law of nature, **what causes this behavior of the empirical distribution of published p-values?** Selection bias, systematic "adjustment" of p-values just above the canonical critical level, or what?


title: Is it ok to fit a Bayesian model first, then begin weakening priors?

text: When doing frequentist stats, there is a long list of big no-nos, like looking at the results of statistical tests before deciding to collect more data. I'm wondering generally if there are a similar list of no-nos for the methodologies involved in Bayesian statistics, and specifically whether the following is one of them. I've realized recently that for some of the models I've been fitting, my process has been to first fit the model with informative priors to see if it works or blows up, and then weaken the priors either to uninformative or weakly informative and refit the model. My motivation for this really has to do with the fact that I'm writing these models in JAGS/Stan, and in my mind I've been treating it more like a programming task than a statistical one. So, I do a first run, sort of rigging it to converge quickly by using informative priors, making it easier to catch errors in the model I've written. Then, after debugging the model, I refit it with uninformative, or weakly informative priors. My question is whether or not I'm breaking some serious rules with this process. For example, for my inferences to be valid, and to avoid exploiting researchers' degrees of freedom, do I need to commit to specific priors _before_ are start fitting any models?