title: LASSO vs AIC for feature selection with the Cox model

text: I have some questions about the Lasso.   * After using the AIC or BIC to select a model, the model is fit with the variables selected in order to get the standard errors of the estimates with CIs, p-values etc...   * When I use the Lasso approach, I try to do the same thing but the estimated values of the coefficients returned by the lasso are not the same as those returned by the cox model with only the variables selected by the Lasso.  If I understand the Lasso, it selects variables during the procedure estimating the coefficients, while the selection is done afterwards with the AIC. Is this right? Here are my questions:   1. Why I don't find the same values?   2. Can I use the Lasso to select variables, and then fit a cox model?


title: Likelihood-ratio test or z-test?

text: Consider the following two logistic regression models: $$ \begin{aligned} &\text{Model A: }&P(Y=1)&=\frac{\text{exp}\left(b_1+b_2X_2\right)}{1+\text{exp}\left(b_1+b_2X_2\right)} \\\ &\text{Model B: }&P(Y=1)&=\frac{\text{exp}\left(b_1+b_2X_2+b_3X_3\right)}{1+\text{exp}\left(b_1+b_2X_2+b_3X_3\right)} \end{aligned} $$ In model A, suppose that both the parameters $b_1$ and $b_2$ appear significant, i.e. $-1,96 > z_k$ or $1,96 < z_k$. When adding a dummy variable $X_3$ to create model B, there seem to be two options if I want to check whether the variable adds more explanatory power to the model:   1. Perform a Likelihood-ratio test between model A and model B by calculating the test statistic $G^2=-2\log \left(\frac{L_{\text{A}}}{L_{\text{B}}}\right)$ and compare it to the critical value of the chi-squared distribution with one degree of freedom.   2. Perform a z-test of $X_3$ in model B, calculating $z=\frac{b_3}{\text{se}\left(b_3\right)}$ and compare it to the critical values of the standard normal distribution. The question now is: Is it sufficient in this case to calculate the z-value, if I want to determine whether model B has significant more explanatory power than model A? Is it possible that $X_3$ in a z-test is significant, and at the same time the LR-test implies there is no difference between the two models? In summary: when adding only one variable to a logistic model, what is the best test to perform, if I want to investigate if the variable added significant explanatory power to the model?


title: Dealing with multicollinearity when removing a highly collinear predictor reduces significance

text: In my experience with dealing with multicollinearity, often removing one collinear variable from the model results in the other collinear variable(s) becoming significant (assuming that all the collinear variables are significantly correlated with the dependent variable bivariately. However, I've recently encountered a situation where I have a model shown below in which $X_2$ and $X_3$ are highly correlated ($r > 0.95$) and the tolerance scores for the two variables are below $0.1$ $$ Y = \beta_0\ + \beta_1X_1\ + \beta_2X_2\ + \beta_3X_3\ +\beta_4X_1X_2\ +\beta_5X_1X_3\ $$ * _all variables are continuous._ The results of the regression model show that all 5 slopes are significant (3 slopes for the main effects and 2 slopes for the interactions). One of the possible solutions is to remove one of the highly collinear predictors. If I remove one of them - say $X_2$, I get a new model as shown below. $$ Y = \beta_0^\prime + \beta_1^\prime X_1\ + \beta_3^\prime X_3\ +\beta_5^\prime X_1X_3\ $$ After $X_2$ is removed, while $\beta_0^\prime$, $\beta_1^\prime$, and $\beta_3^\prime$ are significant, crucially, $\beta_5^\prime$, the slope for the interaction term, is no longer significant. The same thing happens if I try to remove $X_3$. So I wonder what may be the reason that causes this pattern to arise and how it can be potentially dealt with. Thank you in advance!


title: How do I use student's-t distribution without the sample size?

text: Here is my question (homework obviously): **A sample from a normal population produced variance 4.0. Find the size of the sample if the sample mean deviates from the population mean by no more than 2.0 with a probability of at least 0.95.** So I'm trying to find $n$, the sample size, having only $\hat{\sigma}$, the sample variance, and a bound on the distance between $\bar{x}$ and $\mu$. My intuition was normally in this situation we need to use the t distribution since $\hat{\sigma}$ is an unbiased estimate for $\sigma$ (we did all the proofs in class).The problem is the t distribution changes depending on $n$, the sample size, so which distribution (how many degrees of freedom) should I consult when looking up the t-values containing 95% of the probability mass? I tried it for different values of $n$, and then squared the values to compare them to the d.f. of the t distribution - the closest I could get was 0.6 off. (I took the t-value at $\alpha = 0.025$ (right-tail) for 5 d.f., implying $n$ is 6, and squaring the t-value gave me 6.61, which is a discrepancy of 0.61 (isn't this large?). The reason I squared the t-values becomes apparent if you "normalize" the bound on the means into a t-statistic. Am I going about this correctly? This doesn't seem right... (edit - this is what I did): We have $P(\left| \bar{x} - \mu \right| \leq 2.0) \geq 0.95$, also given is $\hat{\sigma} = 2$, need to find $n$. So: $$-\sqrt{n}\frac{2.0}{\hat{\sigma}} \leq \sqrt{n}\frac{\left( \bar{x} - \mu \right)}{\hat{\sigma}} \leq \sqrt{n}\frac{2.0}{\hat{\sigma}}$$ filling in values: $$-\sqrt{n}\frac{2.0}{2} \leq \sqrt{n}\frac{\left( \bar{x} - \mu \right)}{\hat{\sigma}} \leq \sqrt{n}\frac{2.0}{2}$$ then: $$-\sqrt{n} \leq \sqrt{n}\frac{\left( \bar{x} - \mu \right)}{\hat{\sigma}} \leq \sqrt{n}$$ at this point I was pretty confused, so what I did was look up values for $\alpha = 0.025$ at different degrees of freedom for the t-distribution, and then since d.f. = n - 1, I took that t-value (which is $\sqrt{n}$ from what I've derived) and compared it to the value of n implied from the d.f. I was using...For example take the t distribution with 5 d.f. (implying sample size is 6). The t-value is 2.571 for my $\alpha$. Squaring this to get n, we get 6.61. So this n clearly $\neq$ 6 which was implied in the distribution I was using. All of this seems kind of ridiculous...Where did I fudge things?


title: Graphics encyclopedia

text: I have to build a multi-user webapp which is about traffic measurements, prognoses etc. At this point I know that I'll use bar and pie charts. Unfortunately, those chart types aren't rich in expressing all the data that I collect and compute. I'm looking for a collection of graphical charts. It is very ok if I have to buy a book or anything else. I need to find some graphical samples with explanations in order to inspire me. Do you know of any such resource? Do you have any advice for me?


title: What correlation coefficient and graph is appropriate with this data?

text: I'm not in Statistics field. I conducted the case study and collected the data as shown below I have data as shown in the table below: ![enter image description here](http://i.stack.imgur.com/mH1tC.jpg) ![enter image description here](http://i.stack.imgur.com/ExRl1.jpg) I would like to find correlation coefficient from this two table data(between **NOA and HVOC** , and between **NOA and HVOL** ). I conducted the case study with the source code. I measured software metrics named "NOA" and "HVOL" for all the method/function before I modified this source code. And then, after I modified the code, I again measureed the same metrics for all the method. **NOA Diff** field in the table is calculated from NOA (after modifying the code) minus NOA (before modifying the code). That is "NOA Diff = NOA(after)-NOA(before)". The same way was applied to HVOC metric; **HVOC Diff** = HVOC(after)-HVOC(before) My questions are   * What type of **correlation coefficient** should I use?   * What **kind of graph** should I create to illustrate my data?   * The table above is all data, i mean it's **population** not a sample, can I use the method that is used with a **sample**   * Is Spearman is for non normally distributed data?


title: How to get generalisation performance from nnet in R using k-fold cross-validation?

text: I'm doing some Machine Learning in R using the `nnet` package. I want to estimate the generalisation performance of my classifier by using k-fold cross-validation. How should I go about doing this? Are there some built-in functions that do this for me? I've seen `tune.nnet()` in the `e1071` package, but I'm not sure that does quite what I want. Basically I want to do the cross-validation (split into 10 groups, train on 9, test on the other 1, repeat) and then from that obtain some sort of measure of how well my classifier generalises - but I'm not sure what measure that should be. I guess I want to look at the average of the accuracies across the different cross-validation examples, but I'm not sure how to do that with the `tune.nnet()` function above. Any ideas?


title: What is tied data in the context of a rank correlation coefficient?

text: I am not in statistics field. I have seen the word "tied data" while reading about Rank Correlation Coefficients.   * What is tied data?   * What is an example of tied data?


title: Choosing seasonal decomposition method

text: **Seasonal adjustment** is a crucial step preprocessing the data for further research. Researcher however has a number of options for trend-cycle-seasonal decomposition. The most common (judging by the number of citations in empirical literature) rival seasonal decomposition methods are X-11(12)-ARIMA, Tramo/Seats (both implemented in Demetra+) and $R$'s stl. Seeking to avoid random choice between mentioned (or other simple methods like seasonal dummy variables) decomposition techniques I would like to know a basic strategy that leads to choosing seasonal decomposition method effectively. Several important subquestions (links to a discussion are welcome too) could be:   1. What are the similiraties and differences, strong and weak points of the methods? Are there any special cases when one method is more preferable than the others?   2. Could you provide general guides to what is inside the black-box of different decomposition methods?   3. Are there special tricks choosing the parameters for the methods (I am not always satisfied with the defaults, `stl` for example has many parameters to deal with, sometimes I feel I just don't know how to choose these ones in a right way).   4. Is it possible to suggest some (statistical) criteria that the time series is seasonally adjusted efficiently (correlogram analysis, spectral density? small sample size criteria? robustness?).


title: Multiple regression: What to do when the intercept is not significant?

text: > **Possible Duplicate:**   >  When is it ok to remove the intercept in lm()? I have some regression output from R:                  Coefficients:                  Estimate Std. Error t value Pr(>|t|)         (Intercept) -28.84775   29.71148  -0.971    0.341         sqft          0.17091    0.01545  11.064 2.48e-11 ***     lotsize       6.77770    1.42129   4.769 6.19e-05 ***     baths        15.53470    9.20827   1.687    0.104         ---      The intercept (and baths) looks useless, so I ran `confint()` to confirm it:                           2.5 %     97.5 %     (Intercept) -89.9205710 32.2250754     sqft          0.1391546  0.2026609     lotsize       3.8561912  9.6992184     baths        -3.3931660 34.4625741      And what do you know, it is. Now I am stuck on what to do here. I see that the intercept can have a zero slope, so can I treat the intercept like a predictor and drop it? After taking `baths` out and running the regression again, the intercept is still coming up as statistically insignificant:                            Estimate Std. Error t value Pr(>|t|)         (Intercept) -23.20585   30.51527  -0.760 0.453566       and                                2.5 %     97.5 %     (Intercept) -85.8180248 39.4063179      Any ideas?


title: Constant in OLS model

text: Sometimes in OLS model we have constant for example -2345 significant and doesn't have a mean. Why we must keep it in the model? Why when we drop it the results change? What does it mean? And sometimes it is insignificant. Why?


> **Possible Duplicate:**   >  When is it ok to remove the intercept in lm()? When shall I include an intercept in linear regression? Hi all, In general, given some data, how shall I decide if I should include an intercept in linear regression? Could you please give me some pointers? Thank you!


title: Significance of Regression Intercept (R lm model)

text: **Question:** Having performanced a linear regression in R with the lm function, I'm not sure how to interpret the results for the _Intercept_ (as shown below). It seems the probability of the intercept's relevance is low (i.e. _Pr(>|t|)_ is 0.845, and higher that 0.05). Does this mean I should drop the intercept from the model by forcing it through zero? Alternatively, does it mean that I should still keep the intercept but recognise that it's not significant? **Output:**               Call:     lm(formula = DI ~ II)          Residuals:      Min       1Q   Median       3Q      Max      -0.23960 -0.03306 -0.01116  008724  0.20568           Coefficients:                 Estimate    Std. Error   t value   Pr(>|t|)     (Intercept) -0.07952    0.39953      -0.199   0.845     II           0.86381    0.04593      18.809   8.23e-11 ***     ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1           Residual standard error: 0.1346 on 13 degrees of freedom     Multiple R-squared: 0.9646, Adjusted R-squared: 0.9618      F-statistic: 353.8 on 1 and 13 DF,  p-value: 8.23e-11       **Additional Background Information** My overall aim is to find a relationship between two datasets of mass that I have. So, if I have a value for DI, I am able to find out the corresponding value of II.


title: Estimation of TRANS_GUESS and EMIS_GUESS matrices in Baum-Welch algorithm

text: I would like to ask how do we estimate the probabilities in TRANS_GUESS and EMIS_GUESS matrices (see here http://www.mathworks.com/help/stats/hidden- markov-models-hmm.html#f8288 ) as an input for the Baum-Welch algorithm. Should they be calculated based on the data or are they completely random? I am a little bit confused about this issue.


title: How to include interaction terms in R/tree model?

text: I have read at many places that tree is good for uncovering complex dependencies among predictor variables. From Tree models in R: > The recursive structure of CART models is ideal for uncovering complex > dependencies among predictor variables. If the effect of, say, soil moisture > content depends strongly on soil texture in nonlinear fashion, CART models > of species occurrences have a better shot at detecting this than interaction > terms in GLMs or even GAMs. However the `tree()` function from the _tree_ package doesn't seem to accept the interaction term and reports error “trees cannot handle interaction terms”. **Is there a way to include interaction term in tree?**               > dat = read.csv("~/Downloads/treedata.csv")     > tree(cover~(elev+plotsize+disturb),data=dat)     node), split, n, deviance, yval           * denotes terminal node          1) root 8971 39740 4.006         2) elev < 1157.5 6875 29610 3.868 *       3) elev > 1157.5 2096  9567 4.460 *     > tree(cover~(elev+plotsize+disturb)^2,data=dat)     Error in tree(cover ~ (elev + plotsize + disturb)^2, data = dat) :        trees cannot handle interaction terms


title: nlme() convergence error code = 1

text: I was trying to use the nlme package in r to do a multilevel linear model. I have yield as response variable and rainfall as predictor variable for 60 years for 6 different locations (State). I am trying to see whether rainfall has same level of effect on yield in all locations or different effects. In principle, I am trying to see if slope of yield vs rainfall significantly varies between locations. Therefore rainfall is my random effect. I built my model like this:                # baseline model which only includes intercept     mdl1<-gls(yield ~ 1,data = data, method="ML")          #intercept as random effect     mdl2<-lme(yield ~ 1,data=data,random = ~1|state,method="ML")             # slope as random effect     mdl3<-lme(yield ~ rain, data = data, random = ~rain|state,method="ML")          ##compare the three model     anova(mdl1,mdl2,mdl3)     ##this shows me when I add slope as random effect, my model shows better fit compared to baseline model (mdl1)      this is all working fine. The problem starts when I do the same analysis using an another predictor variable (a count data)                # baseline model which only includes intercept: Works fine     mdl4<-gls(yield ~ 1,data = data, method="ML")          #intercept as random effect - works fine     mdl5<-lme(yield ~ 1,data=data,random = ~1|state,method="ML")             # include different predictor (break) this time instead of rain     mdl6<-lme(yield ~ break, data = data, random = ~break|state,method="ML")      when i run the mdl 6, this gives me the error               Error in lme.formula(res_yld ~ brk, data = data, random = ~brk | state,  :      nlminb problem, convergence error code = 1     message = iteration limit reached without convergence (10)      I have absolutely no clue why is this happening. Everything worked fine for my first predictor but this does not work on another predictor. What am I doing wrong here? I tried reading about this online but the posts are not very clear to me. I would really appreciate of anyone could me out on this. Thanks


title: Density values of densityMclust package in R

text: I'm using the Mclust package to fit a Gaussian mixture model (GMM) to a data set. The data has two possible labels, so I'm trying to evaluate two different GMMs, one for each label. Now I have new data point, and I want to label it by comparing the likelihood of it coming from each of the models. I'm using the denstiyMclust "density" field to get the density value in each point of the dataset, but I don;t know how to interpret it - sometimes it is even greater than one. Can anyone say what is the meaning of these densitiy values, and how can one compare two values from different models? Thanks, Yuval


title: Proper ways to perform time series and ARIMA

text: Note that I do most of my analysis using R and Excel. Let's take this data set for example. I modified it as the data itself is proprietary: the years are also different:               1967    2,033,407     1968    2,162,275     1969    2,159,640     1970    2,312,352     1971    2,554,449     1972    2,548,425     1973    2,101,225     1974    1,951,944     1975    2,106,250     1976    1,687,625     1977    1,636,496     1978    1,494,525     1979    1,606,825     1980    1,460,937     1981    1,310,494     1982    1,319,750     1983    1,263,643     1984    1,171,656     1985    1,194,950      What I usually do:   1. A linear regression   2. Some form of polynomial trending   3. Moving average and double moving average   4. Basic ARIMA using p = 1, q = 0.   5. I calculate the errors for all these as well   6. I average all the forecasts out and the error to have my final result. Note that I'm an engineer that wants to get into statistics and the ability to properly validate and calibrate my models. ## Question What is the correct way to forecast this to 5, 10, or even 15 future years? In a way I'm looking to move beyond the plugging data into a model and believe the data. Yes, I'm aware I can look at the errors. I mainly use RMSE or MAE. But I still am not confident when it comes to just predicting data the right way. ### Note this is also related to this question I posted here before.


title: Why would significance of F-value change in linear regression if you change the reference group?

text: For categorical predictors with k levels, it doesnt matter what you choose as your reference group. So why would the F-value significance change in the linear regression if you change the reference group? Also is it possible for the F-test to not reject the null hypothesis when it really should? When does this happen? For reference:                         The REG Procedure                                            Model: MODEL1                           Dependent Variable: HDL HDL cholesterol (mg/dl)                            Number of Observations Read                       2032                       Number of Observations Used                       2025                       Number of Observations with Missing Values           7                                                  Analysis of Variance                                                    Sum of           Mean           Source                   DF        Squares         Square    F Value    Pr > F                Model                     4     1380.42472      345.10618       1.89    0.1094           Error                  2020         368694      182.52193           Corrected Total        2024         370075                                  Root MSE             13.51007    R-Square     0.0037                        Dependent Mean       51.47259    Adj R-Sq     0.0018                        Coeff Var            26.24711                                                                                The Reg Procedure                                            Model: MODEL1                           Dependent Variable: HDL HDL cholesterol (mg/dl)                            Number of Observations Read                       2032                       Number of Observations Used                       2025                       Number of Observations with Missing Values           7                                                  Analysis of Variance                                                    Sum of           Mean           Source                   DF        Squares         Square    F Value    Pr > F                Model                     4     1333.76988      333.44247       1.83    0.1210           Error                  2020         368741      182.54503           Corrected Total        2024         370075                                  Root MSE             13.51092    R-Square     0.0036                        Dependent Mean       51.47259    Adj R-Sq     0.0016                        Coeff Var            26.24877      The first one is with group 1 as reference group. The second is with group 2 as reference group. Note that the two F values are different.


title: princomp relevant information matrix

text: I want to get $ relative error = \frac{x-x_{0}}{x}$ between the results of R and mine with pca, to do so I am doing               Data <- read.csv("C:\\dataset.txt",header=T,row.names=NULL)     pca  <- princomp(Data, cor = TRUE)     R_    <- pca$loadings[,1:25]      Although R_ contains a matrix when doing               max(abs(( R_ - MYRESULT) / R_ ))       I am getting numbers like `21671.33`... wich is not the case as the matrix are almost equal...What could be the error.               > typeof(R_)     [1] "double"     > typeof(MYRESULT)     [1] "double"      R_                         Comp.1       Comp.2      Comp.3      Comp.4       Comp.5      Comp.6      Comp.7      Comp.8      Comp.9       Comp.10     x1  -0.233694952  0.117256657 -0.26171589  0.11393704  0.254809586 -0.25022024  0.10366925 -0.10343966 -0.14203329 -0.1907500299     x2  -0.150092223 -0.023046932 -0.06350137 -0.16360607  0.135719926 -0.49301629  0.38433141  0.20287488 -0.14779720 -0.1557853588     x3  -0.208798702 -0.115620602  0.17171312 -0.03359685  0.050249728 -0.36797387  0.21069681 -0.10510851 -0.19520819  0.4458739074     x4  -0.231787481 -0.246610140  0.29624555 -0.16154665 -0.172802779 -0.04512977 -0.05205971 -0.14348860 -0.23209102 -0.0768192169     x5  -0.174773487 -0.331782564  0.20917948 -0.05296483  0.009748868  0.07303589 -0.01987149 -0.22041110 -0.32501098 -0.2363458857     x6  -0.040765485 -0.350314388  0.03876466 -0.03250258  0.207882569  0.33380077 -0.03750823 -0.05567884 -0.22270121  0.0395916149      ... MYRESULT                               V1          V2          V3          V4           V5          V6          V7          V8            V9          V10      [1,] -0.233694953 -0.15009222 -0.20879870 -0.23178748 -0.174773486 -0.04076548  0.22125345  0.18651088  0.2605342467  0.187890851      [2,]  0.117256657 -0.02304693 -0.11562060 -0.24661014 -0.331782566 -0.35031439 -0.14093683 -0.27160044 -0.1837792519 -0.189111703      [3,] -0.261715894 -0.06350137  0.17171311  0.29624555  0.209179478  0.03876467 -0.17828465 -0.24067596 -0.3419418288 -0.325710915      [4,]  0.113937042 -0.16360606 -0.03359685 -0.16154665 -0.052964823 -0.03250257  0.03040334  0.16702865 -0.1193355400  0.052946558      [5,] -0.254809590 -0.13571993 -0.05024973  0.17280278 -0.009748867 -0.20788256 -0.29811927 -0.10755628  0.1563905111  0.338866066      [6,]  0.250220236  0.49301631  0.36797388  0.04512977 -0.073035883 -0.33380078 -0.29614050 -0.02620513  0.0216369958  0.204902414


title: What is the procedure for a mixed effects analysis of covariance?

text: I have 31 observations for 9 subjects undergoing therapy (4 session for 4 subjects, and 3 session for the rest). I have implemented a mixed model to assess relationships between the observations using Fisher-transformed correlation coefficients. For example, to assess the effect of finger tapping on correct responses, I correlated the number of finger taps with the percentage of correct responses for each subject (3 pairs of observations for some subjects, and 4 for others). I then perform a Fisher-transform on the 9 resulting correlation coefficients to get 9 z-scores. Then it's a simple matter of testing whether or not those z-scores differ from 0. What I would like to do is control for subjects' age. However, I'm not clear on the best way to do this. It's easy to calculate the covariance between the z-scores and subject age, but how do I get the effect of finger taps on correct responses, controlling for age? (As a side note, I've implemented this as a MIXED model in SPSS, but it fails to converge, so I can't trust the results).


title: What distributions are for the slope and for the intercept in linear regression?

text: In linear regression from a dataset $\\{ (x_i, y_i), i=1,\cdots,N \\}$, I wonder what distributions are for the slope and for the intercept? In Excel output, t-tests are used to tell whether the slope mean is zero and the intercept mean is zero. But t-test assumes the distributions of slope and of intercept are normal distributions. So I would like to see whether this is true. Thanks and regards!


title: Is there a sample distribution so that the distribution of p-value is skewed towards 1?

text: I heard that:   * Under a null distribution, the distribution of p-value is flat over $[0,1]$.   * Under an alternative distribution, the distribution of p-value is skewed towards $0$. Is there a sample distribution so that the distribution of p-value is skewed towards $1$? Also does the first two cases assume the test rule is a reasonable one? What if the test rule produces a lot of errors? Thanks!


title: Factor analysis using spss

text: I got determinant value– 5.84E-007. What does it mean? Should I continue the work or not? What do I have to do? And also I obtain the following Kaiser-Meyer-Olkin value and Bartlett's Test results               Kaiser-Meyer-Olkin Measure of Sampling Adequacy.       .793     Bartlett's Test of Sphericity   Approx. Chi-Square  3.452E3                                                     df      253                                                   Sig.     .000      Bartlett's Test of Sphericity Approx. Chi-Square 3.452E3 - is this value is correct?


title: Percentage of overlapping regions of two normal distributions

text: I was wondering, given two normal distributions with std1,mean1 and std2,mean2, how can I calculate the percentage of overlapping regions of two distributions? I suppose this problem has a specific name, are you aware of any particular name describing this problem? Are you aware of any implementation of this (e.g., Java code)?


title: Understand the PACF Values in R / Correlation over 1

text: I have a basic problem with the `pacf` function in R. By applying this function to get the partial auto correlation functions of a time series, the values range in some cases over 1.   In my understanding the value of a correlation has to be between -1 and 1, so I am quite confused. So the question is: How can the value of the partial auto correlation function be outside of the intervall of -1 and 1? For further Informations: My timeseries have 55 or 180 different measurements in time and compare 5 different variables. I already tried to use only the random variables of the decompose function. EDIT: Data Set https://docs.google.com/spreadsheets/d/14O6UwBn2KC7NJwGIGlssXsClTocMQe3_zpeUBVy7K_o/pubhtml Each column is a different Data Input. I hope this is the right way to do it ...


title: Denormalizing Data

text: I am applying **Polynomial Regression** to my data, however the parameters theta were always **=0** , i noticed that my y data or output is too large in the order of 100000 so i **normalized y** , i got very good predictions. However what can i do if i need to get the predictions based on the **real y values** , meaning how can i **denormalize** ? wont the parameters be wrong afterwards somehow since i got them off of the normalized data ? Note: i also normalize my input x data


title: Family vs. class of distributions - definitions

text: What is the difference between a class of distributions and a family of distributions? The class of (a,b,0) distributions is defined as: ![enter image description here](http://i.stack.imgur.com/hOUbD.jpg) The Binomial, Poisson, Geometric and Negative Binomial meet these criteria. The exponential family is defined as ![enter image description here](http://i.stack.imgur.com/rKCpW.png) Is the classifier "class" reserved for those distributions that satisfy a recursive relationship and "family" reserved for those distributions that can be transformed into a specific functional form (i.e. not a recursive relationship)?


title: Finite population correction for calculating margin of error

text: Let's say that I have a population of 10000, and I have a sample of 300 that have responded to an online survey. Of 10000 invitations, 300 hundred have replied. Would I be able to use the finite population correction for margin of error? Population: 10000   Sample: 300   Response: 3%


title: Usage of PCA - how to scale observations?

text: I want to use PCA in this kind of situation. I have three variables:   1. how many times something happened for user - positive integer;   2. total "power" of all happened events for user - real number, can be negative   3. percent of "successful hits" - real positive number 0 < x < 1 Wikipedia states that "PCA is sensitive to the scaling of the variables." A problem is that "power" can be measured using various units. And choice of units will affect the results. I do not see a natural choice of units for the moment. **Are there any suggestions on how to scale observations for PCA?**


title: Padding a matrix - effect on statistical analyses

text: I have the time-series data for a lot of stocks from their specific groups (market indices), and I would like to perform some quantitative tests on them as a group. Let's say for example I have 30 stocks over a period of 10 years with daily information. It is stored as a cell (in Matlab) so basically a matrix with 30 columns and 2520 (10 x 252 trading days in a year) rows. Lets say I want to find the correlation matrix, i.e. the correlation between each pair of stocks. To do this you needs a regular (rectangular) matrix - but my problem is that, say 3 of the stocks appeared only 8, 6 and 5 years ago, so their columns are 2, 4 and 5 years shorter than the others, respectively. I have two options to get my rectangular matrix   1. chop of all data going further back that the oldest 'start-date' of a stock, which means losing 5 years of data in my example - not really an option.   2. fill out, 'pad', the shorter columns to make them the same length as the longer stocks. I have already removed one or two stocks as they are really young, and now want to pad the few remaining stocks that have shorter time series. My question is: to what extent will my results be affected/skewed/biased if I pad those columns and run the analyses (correlation etc.). Would the errors be negligible? Can I minimise them with my choice of what I pad them with? I have considered using 'NaN' in Matlab, as it functions neutrally in many other analyses, but here it would throw an error. My next best guesses would be to pad with zero, or with the mean value of that column (i.e. the mean stock price over the time series). Any other ideas, or is padding a complete no-no? Thanks in advance


title: Conversion rate clarification

text: This is a very simple question, but for some reason I cannot wrap my head around it... It's actually a bit weird because it's so simple that I'm kind of laughing that it's not making sense. On my site I have something that looks like this: ![](http://i.imgur.com/kVN7Dz0.png) It is roughly a measure of how 'good' a post on my website is, by measuring visitor and Facebook share data. Example: I create a post. Then I seed it initially with a single share on my own page. From there I watch it grow... people see the post on Facebook, they click or don't click. If they do click, they end up on my post, where they share it or don't share it. If they do share it, then people on their Facebook see it in their feed, click or don't click. If they do click, the cycle repeats.   * `visits` is the number of uniques from Google Analytics   * `shares` is the number of shares the post got from Facebook API   * `shares/hr` is `shares/post age in hours`, ie: `99/43.14` Here's where my question begins. You'll also see `(1.5%)`, now whether I'm thinking about this right, I'm not sure, but in my head this is the "conversion rate" of a share. Meaning if `1` person shows up, they have a `1.5%` chance of sharing it. I calculate this by taking `shares/visits*100`. Now, there is another metric which is the inverse of this so-called 'conversion rate', and that is `visits/shares`... or what would _seem_ to be, in english... "how many visits resulted from a single share". So in the example above, `6571/99` which is now the first column here: ![](http://i.imgur.com/1C69doQ.png) Now really, my question boils down to... I'm confused as to what the difference is between these two statistics: `shares/visits*100` and `visits/shares` Is one really useful and the other not? Are both useful? Are my understanding of what they are incorrect? `1.5%` seems to imply the post did terribly (which it did), but `66.4` visits per share seems to imply it did amazing when considered relative to other posts on my site (some posts which did orders of magnitude better have a "visits per share" value of something very low, like `4.4`). One final idea I was pondering (may be completely wrong...): maybe one of these values is measuring how well a _headline_ is doing, as that's only what the person sees on Facebook before they click. So if the headline is amazing, there could be a lot of clicks but not many shares. Now, could the other one possibly be measuring how well the _content_ is doing? So here, it could be that not many people click (because of a bad headline), but of the people that DO click, a large percentage of them share because the content is so good. I'm sorry I don't have a very clear question, mainly I just need help where the errors in my thinking are, and what do these values really mean? How can I use them effectively to determine how good a post is in the categories of title & content (shareworthiness). Edit: Also, I would **love** if you have an idea for some other statistic that would be very useful that I don't have on here, probably something a lot more complex.


title: Best Methods for data-mining neuroimaging data with 1000 subjects

text: I am part of a team tasked with performing exploratory analysis of a large data set containing neuro-imaging scans. For each scan I will likely calculate some variable that relates to brain function - for example functional connectivity. I may have 20 such statistical values per subject. The dataset has hundreds of subjects (1000 actually), so there is a lot of data. We are interested in finding relationships between various factors, such as age, health, socioeconomic status, there are 100s of measures taken for each subject. Lets say we want to data-mine this dataset to find what it is about a person's lifestyle that improves functional connectivity. Q1) What do you think the correct approach would be? I am thinking of splitting the data into a training set of data, in order to try to find patterns that relate to brain function, and a testing set, in order to test those models and hypotheses that are generated in the first phase. Anyone have any better ideas than this. Q2) If you think the proposed method of splitting the data set is a good idea, then which methods do you recommend for finding patterns? I have done a bit of searching and found articles on decision trees etc. My boss suggested just trying to correlate everything with everything and look for patterns but I suspect that this is a bad idea and I will likely miss many interactions and non-linear relationships. Q3) Which software do you think I should use? I am very familiar with MATLAB but don't mind learning R. Other than that, it would probably need to be free software as the department I work in is unlikely to want to pay for anything. Thanks


What is the difference between data mining, statistics, machine learning and AI? Would it be accurate to say that they are 4 fields attempting to solve very similar problems but with different approaches? What exactly do they have in common and where do they differ? If there is some kind of hierarchy between them, what would it be? Similar questions have been asked previously but I still don't get it:   * Data Mining and Statistical Analysis   * The Two Cultures: statistics vs. machine learning?


title: The relationship between machine learning, data mining and statistical analysis?

text: > **Possible Duplicate:**   >  What is the difference between data mining, statistics, machine learning > and AI? How to compare machine learning vs. data mining? data mining vs. statistical analysis? machine learning vs. statistical analysis?


title: How to find the distribution of data with data fitting

text: I need to do data fitting to find the distribution of given data. I need to find the PDF function of the distribution. I can use data fitting functions in MATLAB and Python. It looks like a truncated gamma. What if the data cannot fit the truncated gamma well? The QQ-plot (quantile-quantile) shows that truncated gamma is not a good fit. How to find the distribution parameters for truncated gamma such as alpha (shape) and beta (scale)? If data fitting cannot work here, what other methods I can use for that? Any help would be appreciated.


title: Relationship between McNemar's test and conditional logistic regression

text: I am interested in the modeling of binary response data in paired observations. We aim to make inference about the effectiveness of a pre-post intervention in a group, potentially adjusting for several covariates and determining whether there is effect modification by a group that received particularly different training as part of an intervention. Given data of the following form:               id phase resp     1  pre   1     1  post  0     2  pre   0     2  post  0     3  pre   1     3  post  0      And a $2 \times 2$ contingency table of paired response information: \begin{array}{cc|cc} & & \mbox{Pre} & \\\ & & \mbox{Correct} & \mbox{Incorrect} \\\ \hline \mbox{Post} & \mbox{Correct} & a & b&\\\ & \mbox{Incorrect} & c& d&\\\ \end{array} We're interested in the test of hypothesis: $\mathcal{H}_0: \theta_c = 1$. McNemar's Test gives: $Q = \frac{(b-c)^2}{b+c} \sim \chi^2_1$ under $\mathcal{H}_0$ (asymptotically). This is intuitive because, under the null, we would expect an equal proportion of the discordant pairs ($b$ and $c$) to be favoring a positive effect ($b$) or a negative effect ($c$). With the probability of positive case definition defined $p =\frac{b}{b+c}$ and $n=b+c$. The odds of observing a positive discordant pair is $\frac{p}{1-p}=\frac{b}{c}$. On the other hand, conditional logistic regression uses a different approach to test the same hypothesis, by maximizing the conditional likelihood: $$\mathcal{L}(X ; \beta) = \prod_{j=1}^n \frac{\exp(\beta X_{j,2})}{\exp(\beta X_{j,1}) + \exp(\beta X_{j,2})}$$ where $\exp(\beta) = \theta_c$. So, what's the relationship between these tests? How can one do a simple test of the contingency table presented earlier? Looking at calibration of p-values from clogit and McNemar's approaches under the null, you'd think they were completely unrelated!               library(survival)     n <- 100     do.one <- function(n) {       id <- rep(1:n, each=2)       ph <- rep(0:1, times=n)       rs <- rbinom(n*2, 1, 0.5)       c(         'pclogit' = coef(summary(clogit(rs ~ ph + strata(id))))[5],         'pmctest' = mcnemar.test(table(ph,rs))$p.value       )     }          out <- replicate(1000, do.one(n))     plot(t(out), main='Calibration plot of pvalues for McNemar and Clogit tests',        xlab='p-value McNemar', ylab='p-value conditional logistic regression')      ![enter image description here](http://i.stack.imgur.com/HC8YV.jpg)


title: notebook or desktop PC for datamining projects

text: I'm beginner in datamining. I will do datamining projects, tasks as data modeling(run different dm algorithms), preprocessing(ETL,complex SQL quieries). Is it sufficient for that tasks to use ordinary notebook(I consider quadcore i7 2ghz, 16GB ram, 256ssd -however dm desktop tools as spss,sass doesn't work in mac, but it's possible virtualize windows with eg. virtualbox) or is it highly recommended to use powerfull destkop PC. Which do you recommend ,which alternative is most used in practice ? * * * _Should I chose a desktop or notebook?_ I need to buy notebook, I consider between 2models macbook pro: cheaper(quadcore i7 2ghz, 16GB ram, 256GB ssd,iris pro) or expensive(quadcore i7 2.3ghz, 16GB ram, 512GB ssd, GeForce GT 750M) and price difference between this models is 400eur. In the near future I will be work on datamining projects(as I mention on data modeling, preprocessing tasks) so I'm finding some optimal hardware soultion which allow me to work on this projects so I'm thinking some options:   1. is it sufficient to buy cheaper macbook pro ? or when power of 1. isn't sufficient then(more expensive alternatives):   2. buy expensive macbook pro ?   3. or buy for ordinary day to day work cheaper macbook pro + for datamining purpose buy some powerful desktop PC (I think around 400-600eur it's possible to buy powerful desktop pc). Is it in practice on real projects mostly used separate powerful desktop PC for this datamining tasks or mostly use only ordinary powerful notebooks ? _Question around OS_ For my day to day work I prefer unix. The question is when is it sufficient for datamining tasks to use emulation of windows(eg. virtualbox,parallels..), or is it very noticably better separate machine with windows(I think some tasks eg ETL complex SQL quieries run longer time).


I am analysing a set of data where I try to predict an outcome (Level of women’s nutrition knowledge; whether it is High or Low) by using certain covariates (demographic characteristics of the sample). I have already done Chi-square analysis and now I am progressing to binary logistic regression. To avoid an overly complicated presentation of the results by inclusion of a large number of non-significant variables, demographic factors found in Chi- square test to be significantly associated (p<0.05) with women’s knowledge were entered into the logistic regression analysis. These factors are: Prior Pregnancies, Planned pregnancy, Education levels, Household income, First language and Having health and/or nutrition related qualification.   1. Dependent variable = Total score of nutrition knowledge of pregnant women; which coded as: a. Low : 0 b. High : 1 2.Independent variables: **a-Prior Pregnancies** has 3 levels and coded as:.Tow and more: 0 None: 1 One: 2 **b-Planned pregnancy** has 2 levels and coded as: N0:0 Yes: 1   **c-Education levels** has 4 levels and coded as: Some high school or less: 0 High school completed: 1 TAFE: 2 Tertiary education: 3 **d-Household income** has 3 levels and coded as: < $25000/yr: 0 $25000-50000/yr: 1 >$50000/yr: 2   **e-First language** has 2 levels and coded as: N0:0 Yes: 1   **f-Having health and/or nutrition related qualification** has 2 levels and coded as: N0:0 Yes: 1 (As seen above, the levels or class of the independents categorical variables were coded 0 for the lowest interest and 1 for the greatest interest in case of dichotomous variable and so forth for other variables). **My Questions:** 1- Which category, the first category (the one of the lowest value) or the last one, should I designate it as the reference category? 2- In case of variable like Age groups which has 4 classes and the first class has the lowest value but the last class has not the highest value; for example the age groups are as the following: Under 20 yrs (has the lowest score in the nutrition knowledge), 20-29 yrs, 30-39 yrs (has the highest score in the nutrition knowledge) and 40 yrs and above (the score of knowledge decreased) How could I choose the category with the highest value to be the reference category when I run the binary logistic regression analysis while it is not the first or the last category? 3- In regard binary logistic regression, which method is better: enter or one of the forward or backward elimination methods? What is the deference between them? Based on what should I choose the method?


title: What justifies the zero mean assumption for Gaussian processes?

text: Most internet resources--- papers, slides, etc--- on GP I've found take the mean function to be zero.   Chapter 15 of Murphy's book (Machine Learning: A Probabilistic Perspective) says this is because _'the GP is flexible enough to model the mean arbitrarily well, as we will see below'_.   I didn't see that he ended up explaining this comment. So my question is:   Why do you not lose generality when you make a zero mean assumption?


title: Forecasting Function R (Holt-Winters HW approach)

text: I was trying to forecast using the "hw" method in R. I have data which follows:               Period 1:      568     485     360     523     514     370     332     Period 2:      841     685     719     647     615     389     367     Period 3:      731     721     819     662     581     436     394     Period 4:      865     805     952     759     677     429     424     Period 5:     598     868     888     849     707     458     426     Period 6:     950     806     826     804     730     541     439     Period 7:     1,070     770     989     863     737     525     461     Period 8:     1,041     863     989     833     783     506     496      > >     [...] And more periods >   The period here represent weeks and each of the data in periods are quantities on a given day. So we can see a pattern that the first 3 days usually shows the largest quantities. I decided to forecast with HW method. So I did the following in R:               library(forecast)     x <- data$weeklydata     x <- ts(x, frequency = 7)     plot(hw(x, 6), byt = "1")      However, the graph I get ranges from -200 to 300, with my forecast being showed to near 0 to negative quantity. Am making a fundamental mistake here with my concepts, or is there something wrong with my R coding?


title: Maximum likelihood estimator (Gaussian errors, known SD)

text: Suppose that the random variables $Y_1, ..., Y_n$, satisfy $Y_i = \beta \cdot x_i + \epsilon_i$ for $i = 1,...,n$ where $\beta$ is a constant, $x_1,...,x_n$, are constants, and $\epsilon_1,...,\epsilon_n$, are independent and identically distributed random variables with $\epsilon_i \sim N(0,\sigma^2)$, where $\sigma^2$ is a known constant. (a) Determine the exact distribution of $Y_i$. (b) Find the maximum likelihood estimator $\hat{\beta}$ of $\beta$ and show that it is an unbiased estimator of $\beta$. (c) Determine the exact distribution of $\hat{\beta}$.


title: Is a p-value a sample statistic, or a population parameter, or neither?

text: I would also be interested to know why it is one or the other.


title: Theoretical objections to hypothesis testing

text: We've all seen how $p$-values can be misinterpreted to draw false inferences. However, I'm interested in learning more about theoretical/mathematical/philosophical objections to the hypothesis-testing- paradigm, ie. I'd like to learn why someone would prefer (say) a likelihood ratio over a hypothesis test. I've read some articles by Andrew Gelman on this topic, but I'm trying to find more literature. Any help would be greatly appreciated.


title: Can you overfit by training machine learning algorithms using CV/Bootstrap?

text: This question may well be too open ended to get a definitive answer, but hopefully not. Machine learning algorithms, such as SVM, GBM, Random Forest etc, generally have some free parameters that, beyond some rule of thumb guidance, need to be tuned to each data set. This is generally done some kind of re-sampling technique (bootstrap, CV etc) in order to fit the set of parameters that give the best generalisation error. My question is, can you go too far here? People talk about doing grid searches as so forth, but why not simply treat this as an optimisation problem and drill down to the best possible set of parameters? I asked about some mechanics of this in this question, but it hasn't received much attention. Maybe the question was badly asked, but perhaps the question itself represents a bad approach that people generally do not do? What bothers me is the lack of regularisation. I might find by re-sampling that that best number of trees to grow in a GBM for this data set is 647 with an interaction depth of 4, but how sure can I be that this will be true of new data (assuming the new population is identical to the training set)? With no reasonable value to 'shrink' to (or if you will, no informative prior information) re-sampling seems like the best we can do. I just don't hear any talk about this, so it makes me wonder if there is something I'm missing. Obviously there is a large computational cost associated with doing many many iterations to squeeze every last but of predictive power out of a model, so clearly this is something you would do if you've got the time/grunt to do the optimisation and every bit of performance improvement is valuable.


title: Calibration in Statistics

text: I would like to know how to, given a data set, obtain the following: 1/ The parametric probability density function that best explains the data 2/ The parameter values of such pdf Thanks Jonny


title: Conceptual understanding of root mean squared error and mean bias deviation

text: I would like to gain a conceptual understanding of Root Mean Squared Error (RMSE) and Mean Bias Deviation (MBD). Having calculated these measures for my own comparisons of data, I've often been perplexed to find that the RMSE is high (for example, 100 kg), whereas the MBD is low (for example, less than 1%). More specifically, I am looking for a reference (not online) that lists and discusses the mathematics of these measures. What is the normally accepted way to calculate these two measures, and how should I report them in a journal article paper? It would be really helpful in the context of this post to have a "toy" dataset that can be used to describe the calculation of these two measures. For example, suppose that I am to find the mass (in kg) of 200 widgets produced by an assembly line. I also have a mathematical model that will attempt to predict the mass of these widgets. The model doesn't have to be empirical, and it can be physically-based. I compute the RMSE and the MBD between the actual measurements and the model, finding that the RMSE is 100 kg and the MBD is 1%. What does this mean conceptually, and how would I interpret this result? Now suppose that I find from the outcome of this experiment that the RMSE is 10 kg, and the MBD is 80%. What does this mean, and what can I say about this experiment? What is the meaning of these measures, and what do the two of them (taken together) imply? What additional information does the MBD give when considered with the RMSE?


title: Support vector regression in R

text: I am searching tutorial for support vector regression in R. I found this and manual for e1071 package. But there is few explanation how to set parameters, like choose kernels, choose regression, not classification. Any material is appreciated.


title: Fisher information in a hierarchical model

text: Given the following hierarchical model, $$ X \sim {\mathcal N}(\mu,1), $$ and, $$ \mu \sim {\rm Laplace}(0, c) $$ where $\mathcal{N}(\cdot,\cdot)$ is a normal distribution. Is there a way to get an exact expression for the Fisher information of the marginal distribution of $X$ given $c$. That is, what is the Fisher information of: $$ p(x | c) = \int p(x|\mu) p(\mu|c) d\mu $$ I can get an expression for the marginal distribution of $X$ given $c$, but differentiating w.r.t. $c$ and then taking expectations seems very difficult. Am I missing something obvious? Any help would be appreciated.


title: Looking for ideas to build a predictive model

text: I'm new to predictive models and I have a problem at hand that I need some advice with. Basically for a clinical application we want to predict the outcome of a rating scale with a model built on top of outcomes of our new measurement device. My dependent variable, a clinical rating scale, is an integer between 0 and 10 (inclusive). Unfortunately I don't have a large sample ($n \approx 100$) and I have a lot features to select from ($p \approx 120$). Also many of these features are correlated. Nearly all of the features are continuous variables. I have a separate sample for validation ($ n \approx 40$). There are several issues I'd like have your advice about:   1. Should I go for regression or tree based methods?    2. Should I try ensemble learning methods or I'd better stick with a single model? Which methods should I try and why?   3. If it's better to go for a single model, how should I handle the model selection problem? Should I e.g. limit the number of predictors and go for methods like LEAPS with AIC or should I go for methods like LASSO?   4. If ensemble methods are suggested, which methods can handle cases with small $n$ and large $p$ better?   5. Discussing selected/influential features is important for me. Depending on the answers to previous questions, how should I go about it?  I have some understanding of regression modeling and model selection problems. I have used the bestglm package in the past. Currently I'm looking at the Caret package as it brings a large number of methods under the same interface. References get technical about the details of the models but so far I didn't find a good one to go over practical issues for problems with small n and big p. I appreciate your suggestions and help. Thanks, AlefSin


title: Linear model when both dependent and independent variables have unpaired replicates

text: I've got a set of experiments where 3 replicate measurements are done for rat cell lines and for human cell lines under different conditions using various compounds. I am trying to predict the response in human based on the response in rat. Assuming the replicates are not paired (replicate 1 in rat doesn't correspond to replicate 1 in human), is this the correct way to model this?                   rat <- matrix(rnorm(15),5)         human <- matrix(rnorm(15),5)         compound <- c("c1","c2","c3","c4","c5")              y <- NULL         x <- NULL         c <- NULL         for (i in 1:5) {             y <- c(y, human[i,1])             x <- c(x, rat[i,1])             c <- c(c, compound[i])             y <- c(y, human[i,1])             x <- c(x, rat[i,2])             c <- c(c, compound[i])             y <- c(y, human[i,1])             x <- c(x, rat[i,3])             c <- c(c, compound[i])             y <- c(y, human[i,2])             x <- c(x, rat[i,1])             c <- c(c, compound[i])             y <- c(y, human[i,2])             x <- c(x, rat[i,2])             c <- c(c, compound[i])             y <- c(y, human[i,2])             x <- c(x, rat[i,3])             c <- c(c, compound[i])             y <- c(y, human[i,3])             x <- c(x, rat[i,1])             c <- c(c, compound[i])             y <- c(y, human[i,3])             x <- c(x, rat[i,2])             c <- c(c, compound[i])             y <- c(y, human[i,3])             x <- c(x, rat[i,3])             c <- c(c, compound[i])         }              lme(y ~ x, random = ~1|c)      So I'm basically considering all combinations between the 3 replicates since they are not paired. Thanks! Erich


title: Generate distribution based on descriptive statistics

text: There's a hidden variable that I'd like to approximate. I know several summary statistics: min, max, mean, median, standard deviation, n; and that it's approximately normal. I can obviously do a plain normal distribution given the mean and standard deviation, but I know it's slightly skewed and limited tails. Obviously my approximation won't be perfect, but closer. I'll probably implement in `R`, but suggestions not tied to platforms are appreciated. Example:               Xbar <- 101.73     Xmedian <- 97.7     Xs <- 20.45 (standard deviation)     Xmin <- 50     Xmax <- 160     n <- 148


title: Repeated Measures ANOVA missing values - run separate models?

text: I am evaluating pre and post test data for 2 groups using repeated measures ANOVA. Given that I am missing a few data cells, the final n is reduced in the analysis. I'd like to know if I can run separate repeated measures ANOVA for each subtest (or combinations that have the same number of full data sets) rather than include them all in the same model to maximize the data analyzed. If I ran separate ANOVAs, are there implications for my interpretation of the output? Thank you for any advice.


title: Kruskal-Wallis test of groups with unequal sample sizes matlab

text: I can't seem to work out why I can't perform a 5 grouped Kruskal-Wallis test on unequal sample sizes in matlab. Any thoughts how to do this? Also as a post-hoc analyis using Wilcoxon Rank sum is this individually comparing the groups? Regards, EDIT: I observed the length of 5 types of nursing care however now have 5 groups of differing sample sizes, just because type 1 care took place more often. So when I run `[P,ANOVATAB,STATS]=kruskalwallis([rand(10,1) rand(30,1)])` > I get : Error using horzcat CAT arguments dimensions are not consistent. Why do unequal sample sizes matter and what should I do instead?


title: Good reference book for epidemiology

text: I'm looking for a good reference book in epidemiology. I have Rothman's Epidemiology: An introduction and Porta's A Dictionary of Epidemiology. Rothman's was a big dissapointment as it is very simplistic and lacks anything that is beyond obvious after basic studies. I've heard that his Modern Epidemiology is good but I'm slightly hesitant in buying it since I'm so disappointed with the introductory book. I've found Porta's to be very nice but it's a dictionary and I'm more interested in a text-book. I would be grateful for any suggestions.


title: How to calculate the p-value of parameters for ARIMA model in R?

text: When doing time series research in R, I found that `arima` provides only the coefficient values and their standard errors of fitted model. However, I also want to get the p-value of the coefficients. I did not find any function that provides the significance of coef. So I wish to calculate it by myself, but I don't know the degree of freedom in the t or chisq distribution of the coefficients. So my question is how to get the p-values for the coefficients of fitted arima model in R?


title: Deriving mathematical model of pLSA

text: After knowing how LSA works, I went on continue reading on pLSA but couldn't really make sense of the mathematical formula. This is what I get from wikipedia (other academic papers/tutorial show similar form) \begin{align*} P(w,d) = \sum_{c} P(c) P(d|c) P(w|c)\\\ &= P(d) \sum_{c} P(c|d) P(w|c)\\\ \end{align*} I gave up trying to derive it, and found this instead \begin{align*} P(c|d) = \frac{P(d|c)P(c)}{P(d)}\\\ P(c|d)P(d) = P(d|c)P(c)\\\ P(w|c)P(c|d)P(d) = P(w|c)P(d|c)P(c)\\\ P(d) \sum_{c} P(w|c)P(c|d) = \sum_{c} P(w|c)P(d|c)P(c) \end{align*} How does the summation appear at the last line? I am currently reading through some tutorial on Bayesian Inferencing (learnt basic probability rules and Bayesian theorem before but can't really see them being useful enough here). p/s: not sure how to write multiline equation in MathJax way, will fix when I figure out the right way...


title: Is the link function in Probit model canonical?

text: If I understand correctly, Probit model is a generalized linear model. I didn't see it listed in the table, so I was wondering if its link function is canonical for some distribution? Thanks!


title: Competing risk survival analysis with time-dependent covariates

text: Can anyone recommend an R package that handles left-truncation, right censoring, AND time-dependent covariates? I have a data set that consists of distance-to-event data rather than time-to- event as seen in traditional survival analysis. I am studying the disturbance effects of cruise ship travel in a Bay in Alaska to a small seabird that floats on the water when resting or feeding. In reaction to an approaching ship a bird can either 1.Flush from the water 2.Dive underwater 3. allow the ship to pass (treated as censored). Since birds are selected at a certain distance from the ship and this distance decreases as the ship advances I have transformed the distances measured so that at the ship progresses the distance increases rather than decreases (Subtracted each distance from the furthest distance observed for all birds). The distances at which the bird ultimately reacts will then be back transformed when presenting the results. This allows the data to be passed to typical Survival Analysis software as if it was time-to-event (increasing each repeated observation). I am exploring the effects of multiple different fixed covariates (weather, distance to shore, ship speed, etc.) but have one time (or distance rather) dependent covariate, the birds bearing off the bow in relation to the ships course. Unless the bird is directly in the line of the ship its relative bearing off the ships course will change as the ship approaches closer. Eventually if the bird remains on the water it will be at 90 degrees tot he bow and the ship will pass by the bird. Therefore I have repeated distance and bearing measurements on each bird allowing this variable to change over the course of the observation. I have considered this covariate to be binary (0-44 degrees and 45-90 degrees). I am mainly interested in how the covariates effect the competing events (Flush or Dive) and whether these two avoidance strategies change based on distance to the ship (preliminary analysis appears that they do).


title: data analysis about data clustering by vector correlation distance

text: I am working on data analysis. Given a group of data vectors, each of them has the same dimension. Each element in a vector is a floating point number.               V1 [  ,   ,   , … ]      V2[  ,   ,   , … ]      ...     Vn [  ,   ,   , … ]       Suppose that each vector has M numbers. M can be 10000. n can be 200. I need to find out how to partition the n vectors into sub-groups such that each vector in one subgroup can be represented by a basic vector in the subgroup. For example, W = union of V1, V2, V3 … Vn Find subgroup i, j, … t :               Gi = [  V1, V6, V3, V5, … , Vx ]     Gj = [V22, V11, V56, V45, … , Vy]     …     Gt = [V78, V90, V9, V12, … , Vz]      Such that : Union of Gi , Gj, … , Gt is equal to W and there is no overlap among all Gi , Gj, … , Gt. Also , each subgroup has a basic vector that has strong correlation with all other element vector in the subgroup. For example, in Gi, we may have vector Vx as the basic vector such that all other vectors have **strong (linear) correlation** with Vx. **Here, we measure the linear correlation betwwen two vectors not two data points.** Moreover, we need to minimize the number of the subgroups, here, it is " t " . It means that given 200 vectors ( n = 200), we prefer a subgroup G1, G2, …, Gt, and t is minimized. For example, we prefer t = 5 over t = 6. if t is more than 10, it may not be useful. My questions: What kind of knowledge domain this problem belongs to ? Is it a clustering analysis ? But, in cluster analysis, one data point is a number, but, here one data point is a vector. Are there some statistics models or algorithm can be used to do this kind of analysis ? Are there some software tools or packages that solve this problem ? If my questions are not a good fit for this forum, please tell me where I should post it. R packages do the clustering for data points not for data vector by correlation. Any help would be appreciated.


title: Enrichment of variables within clusters

text: I'm trying to find if some clusters in an arbitrary annotated dendrogram are enriched for a particular level of a categorical variable . At the moment, as you can see in the picture, I'm simply calculating a Fisher'Exact Test for all the eight variables (outcome 1/0) using the within cluster occurrence of that level vs. the outisde occurrence. I define the subclusters by cutting the dendrogram at different levels, and then I plot a pink line over the most enriched cluster (I don't correct the p-values via BH at each level of cut, in order to favor bigger cluster). ![enter image description here](http://i.stack.imgur.com/pmz5l.jpg) My **first** question is: what would be the correct test if the levels of a variable were more than two? My **second** question is: is there a more elegant method than cutting the tree at several levels (in my case, arbitrarily defined to 2:20) and check each time the enrichment in all the generated subgroups? I use R, so feel free to reply by pointing me towards R packages :-) Thanks!


title: Multicollinearity with Interaction (high VIF)

text: When I check the VIF of my independent variables with the dependent variable, it looks normal and less than 5 but when I add the interaction variables, the VIF increase to 48 for some variables. I read that centering or creating zscore (standardized values) for the variables are two of the solutions to the problem, but I can't seem to find any article that did that. I am doing panel data analysis with fixed model. Four continuous independent variables are interacted with a continuous variable and also with a year dummy in another model what should I do? thank you in advance


title: Which statistical test should I use when there are multiple latent predictors and multiple latent outcomes?

text: I am doing my research on Personality (the big five factors) and if personality can predict commitment to an organization. However, commitment is composed of three dimensions. What is the best method I can use when trying to link the big five factors of personality with commitment to an organization?


title: Where can I get reference formulas for truncated distributions?

text: I have been comparing tails of various distributions for an application where we will need the conditional expectation of the tail or a truncated tail: $E(X | a<X \leq b )= \frac{\int_a^b x f(x) dx}{F(b) - F(a)}$ There is some limited information out there on the web [such as http://en.wikipedia.org/wiki/Truncated_normal_distribution ], and I am capable of carrying out the integrals with difficulty, but ideally I would like to see worked-out formulas and approximations for the major continuous distributions. Is there a good reference work I am missing?


title: Logistic regression in R

text: Still on running logistic regression models and would like to ask a few questions around it. **Question 1** : Is there a simple way of getting the p-values of each independent factor in a logistic regression model. For example, I am running this model:               mymod3 <- as.formula(surv~as.factor(tdate)+as.factor(sline)+as.factor(pgrp)                                               +as.factor(weight5)+as.factor(backfat5)                                               +as.factor(srect2)+as.factor(bcs)                                               +as.factor(agit)+as.factor(uscore)                                               +as.factor(loco)+as.factor(teat2)                                               +as.factor(uscoref)+as.factor(colos)                                               +as.factor(tb5)+as.factor(nerve)                                               +as.factor(feed5)+as.factor(fos)                                               +as.factor(gest3)+as.factor(int3)                                               +as.factor(psex)+as.factor(bwt5)                                               +as.factor(presp2)+as.factor(mtone2)                                               +as.factor(pscolor)+as.factor(pmstain)                                               +as.factor(pshiv)+as.factor(ppscore)                                               +as.factor(pincis)+as.factor(prectem5)                                               +as.factor(pcon12)+as.factor(crum5)                                               +as.factor(pindx5))          sofNoMis3 <- apf[which(complete.cases(apf[,all.vars(mymod3)])),]     FulMod3 <- glm(mymod3,family=binomial(link="logit"),data=sofNoMis3)     summary(FulMod3)      I am using this to look at the significant level of each factor:               anova(FulMod3,test="Chisq")      and got this:               Analysis of Deviance Table          Model: binomial, link: logit          Response: surv          Terms added sequentially (first to last)                                   Df Deviance Resid. Df Resid. Dev  Pr(>Chi)         NULL                                 7791     7096.2                   as.factor(tdate)    15    50.71      7776     7045.4 9.215e-06 ***     as.factor(sline)     1    13.90      7775     7031.5 0.0001924 ***     as.factor(pgrp)      3     8.83      7772     7022.7 0.0316335 *       as.factor(weight5)   4     7.18      7768     7015.5 0.1268943         as.factor(backfat5)  4     3.86      7764     7011.7 0.4258074         as.factor(srect2)    1     0.15      7763     7011.5 0.6987832         as.factor(bcs)       3     6.46      7760     7005.1 0.0910745 .       as.factor(agit)      2    13.44      7758     6991.6 0.0012075 **      as.factor(uscore)    2     2.16      7756     6989.5 0.3401845         as.factor(loco)      2     1.58      7754     6987.9 0.4530983         as.factor(teat2)     2    25.45      7752     6962.4 2.980e-06 ***     as.factor(uscoref)   2     0.48      7750     6962.0 0.7861675         as.factor(colos)     1     1.06      7749     6960.9 0.3034592         as.factor(tb5)       4    49.22      7745     6911.7 5.265e-10 ***     as.factor(nerve)     2     0.99      7743     6910.7 0.6105452         as.factor(feed5)     4    11.79      7739     6898.9 0.0190170 *       as.factor(fos)       1    47.10      7738     6851.8 6.732e-12 ***     as.factor(gest3)     2    22.60      7736     6829.2 1.235e-05 ***     as.factor(int3)      2     6.61      7734     6822.6 0.0367298 *       as.factor(psex)      1     9.50      7733     6813.1 0.0020493 **      as.factor(bwt5)      4   348.42      7729     6464.7 < 2.2e-16 ***     as.factor(presp2)    1   106.23      7728     6358.4 < 2.2e-16 ***     as.factor(mtone2)    1    34.13      7727     6324.3 5.146e-09 ***     as.factor(pscolor)   1    12.57      7726     6311.7 0.0003928 ***     as.factor(pmstain)   1     0.30      7725     6311.4 0.5845095         as.factor(pshiv)     1    32.29      7724     6279.2 1.328e-08 ***     as.factor(ppscore)   1    16.71      7723     6262.4 4.351e-05 ***     as.factor(pincis)    1     0.02      7722     6262.4 0.8892848         as.factor(prectem5)  4   126.06      7718     6136.4 < 2.2e-16 ***     as.factor(pcon12)    1    17.88      7717     6118.5 2.350e-05 ***     as.factor(crum5)     4    15.25      7713     6103.2 0.0042137 **      as.factor(pindx5)    4    25.58      7709     6077.6 3.838e-05 ***     ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1       but it does not always agree with the final model after applying backward elimination: Example: these three factors were not significant above but they still appeared in the final model below               as.factor(weight5)   4     7.18      7768     7015.5 0.1268943         as.factor(backfat5)  4     3.86      7764     7011.7 0.4258074         as.factor(srect2)    1     0.15      7763     7011.5 0.6987832              step(FulMod3,direction="backward",trace=FALSE)      which gives:               Call:  glm(formula = surv ~ as.factor(tdate) + as.factor(pgrp) + as.factor(weight5) +          as.factor(backfat5) + as.factor(srect2) + as.factor(agit) +          as.factor(uscore) + as.factor(teat2) + as.factor(uscoref) +          as.factor(fos) + as.factor(gest3) + as.factor(int3) + as.factor(psex) +          as.factor(bwt5) + as.factor(presp2) + as.factor(mtone2) +          as.factor(pscolor) + as.factor(pshiv) + as.factor(ppscore) +          as.factor(prectem5) + as.factor(pcon12) + as.factor(pindx5),          family = binomial(link = "logit"), data = sofNoMis3)          Coefficients:                    (Intercept)  as.factor(tdate)2009-09-11  as.factor(tdate)2009-09-15  as.factor(tdate)2009-09-18  as.factor(tdate)2009-09-22                          1.34799                     0.18414                    -0.19490                    -0.15552                    -0.16822       as.factor(tdate)2009-09-25  as.factor(tdate)2009-09-29  as.factor(tdate)2010-01-26  as.factor(tdate)2010-01-29  as.factor(tdate)2010-02-02                          0.60046                     0.80784                    -1.03442                    -1.30562                    -1.01486       as.factor(tdate)2010-02-05  as.factor(tdate)2010-02-09  as.factor(tdate)2010-02-12  as.factor(tdate)2010-02-16  as.factor(tdate)2010-02-19                         -1.04438                    -0.89311                    -1.06260                    -0.79833                    -1.09651       as.factor(tdate)2010-02-23            as.factor(pgrp)2            as.factor(pgrp)3            as.factor(pgrp)4         as.factor(weight5)2                         -0.55411                     0.12659                    -0.04727                     0.21817                    -0.22592              as.factor(weight5)3         as.factor(weight5)4         as.factor(weight5)5        as.factor(backfat5)2        as.factor(backfat5)3                         -0.10143                    -0.31562                    -0.37656                    -0.19883                    -0.01188             as.factor(backfat5)4        as.factor(backfat5)5          as.factor(srect2)2            as.factor(agit)2            as.factor(agit)3                          0.08293                    -0.17116                    -0.18201                    -0.49145                    -0.36659               as.factor(uscore)2          as.factor(uscore)3           as.factor(teat2)2           as.factor(teat2)3         as.factor(uscoref)2                         -0.12265                     0.15334                     0.16575                     0.21520                     0.24166              as.factor(uscoref)3             as.factor(fos)2           as.factor(gest3)2           as.factor(gest3)3            as.factor(int3)2                         -0.24363                    -0.29506                     0.09747                     0.81894                    -0.25595                 as.factor(int3)3            as.factor(psex)2            as.factor(bwt5)2            as.factor(bwt5)3            as.factor(bwt5)4                         -1.21086                     0.20025                     0.30753                     0.29614                     0.56753                 as.factor(bwt5)5          as.factor(presp2)2          as.factor(mtone2)2         as.factor(pscolor)2           as.factor(pshiv)2                          0.86479                    -0.29270                    -0.40912                    -0.72782                    -0.33848              as.factor(ppscore)2        as.factor(prectem5)2        as.factor(prectem5)3        as.factor(prectem5)4        as.factor(prectem5)5                         -0.25958                     0.73842                     0.77476                     0.92158                     0.96269               as.factor(pcon12)2          as.factor(pindx5)2          as.factor(pindx5)3          as.factor(pindx5)4          as.factor(pindx5)5                          0.38119                     0.43199                     0.44496                     0.73458                     0.59771            Degrees of Freedom: 7791 Total (i.e. Null);  7732 Residual     Null Deviance:      7096      Residual Deviance: 6102         AIC: 6222      **Question 2** : I would like to calculate the standard errors of the odds ratio of each factor level               exp(NewMod3$coefficients)  #Odds ratios      **Question 3:** Lastly, to tell whether the levels of each factor are significantly different or not                              (Intercept) as.factor(tdate)2009-09-11 as.factor(tdate)2009-09-15 as.factor(tdate)2009-09-18 as.factor(tdate)2009-09-22                       3.8496863                  1.2021883                  0.8229141                  0.8559688                  0.8451676      as.factor(tdate)2009-09-25 as.factor(tdate)2009-09-29 as.factor(tdate)2010-01-26 as.factor(tdate)2010-01-29 as.factor(tdate)2010-02-02                       1.8229563                  2.2430525                  0.3554327                  0.2710041                  0.3624544      as.factor(tdate)2010-02-05 as.factor(tdate)2010-02-09 as.factor(tdate)2010-02-12 as.factor(tdate)2010-02-16 as.factor(tdate)2010-02-19                       0.3519109                  0.4093819                  0.3455567                  0.4500787                  0.3340336      as.factor(tdate)2010-02-23           as.factor(pgrp)2           as.factor(pgrp)3           as.factor(pgrp)4        as.factor(weight5)2                       0.5745817                  1.1349500                  0.9538339                  1.2437928                  0.7977835             as.factor(weight5)3        as.factor(weight5)4        as.factor(weight5)5       as.factor(backfat5)2       as.factor(backfat5)3                       0.9035410                  0.7293337                  0.6862173                  0.8196866                  0.9881871            as.factor(backfat5)4       as.factor(backfat5)5         as.factor(srect2)2           as.factor(agit)2           as.factor(agit)3                       1.0864697                  0.8426844                  0.8335940                  0.6117399                  0.6930936              as.factor(uscore)2         as.factor(uscore)3          as.factor(teat2)2          as.factor(teat2)3        as.factor(uscoref)2                       0.8845715                  1.1657233                  1.1802836                  1.2401126                  1.2733576             as.factor(uscoref)3            as.factor(fos)2          as.factor(gest3)2          as.factor(gest3)3           as.factor(int3)2                       0.7837753                  0.7444886                  1.1023798                  2.2681046                  0.7741829                as.factor(int3)3           as.factor(psex)2           as.factor(bwt5)2           as.factor(bwt5)3           as.factor(bwt5)4                       0.2979401                  1.2217088                  1.3600609                  1.3446543                  1.7639063                as.factor(bwt5)5         as.factor(presp2)2         as.factor(mtone2)2        as.factor(pscolor)2          as.factor(pshiv)2                       2.3745019                  0.7462454                  0.6642372                  0.4829602                  0.7128545             as.factor(ppscore)2       as.factor(prectem5)2       as.factor(prectem5)3       as.factor(prectem5)4       as.factor(prectem5)5                       0.7713779                  2.0926314                  2.1700692                  2.5132469                  2.6187261              as.factor(pcon12)2         as.factor(pindx5)2         as.factor(pindx5)3         as.factor(pindx5)4         as.factor(pindx5)5                       1.4640265                  1.5403203                  1.5604231                  2.0845978                  1.8179532       Example: I would like to have a table like this:               Factor           levels  Odds ratio          Parity group      (1)    1.00±standard error   a                        2     1.50±standard errror  b                       3-4    1.17±standard error   c                        >5    1.19±standard error   c      I would really appreciate your help on these 3 areas. Baz


title: Sorting answers, given overvotes and undervotes

text: At many question-and-answer sites, like StackExchange, people can upvote or downvote each answer. These sites also typically try to use the votes to sort answers, so the answers that are most likely to be helpful or accurate tend to appear nearest the top. Given the number of upvotes and downvotes for each answer, how should one sort the answers? * * * **Examples.** To illustrate why I think this question might have some non- trivial statistical content, let me survey a handful of methods one might consider and some shortcomings of them. In each method, we compute a _score_ for each answer from its upvotes and downvotes, and then sort the answers by score, so the only question is what method to use to compute the score.   * **Upvotes minus downvotes.** The additive difference is simple to compute. _Limitations._ Is an answer with 121 upvotes and 100 downvotes really better than an answer with 20 upvotes and 0 downvotes?   * **Upvotes divided by total number of votes.** This estimates the fraction of voters who held a positive opinion of the answer, which is arguably natural and meaningful in its own right. _Limitations._ Is an answer with 1 upvote and 0 downvotes really better than an answer with 8 upvotes and 1 downvotes?   * **Compute a confidence interval.** I suppose we could compute, for each answer, a confidence interval for the true fraction of people who would vote positively if they voted. But it is not clear how to extend this to a complete sorting scheme; what do we do when two answers have overlapping confidence intervals?   * **Laplace smoothing.** We could apply Laplace smoothing (additive smoothing) to the counts of upvotes and downvotes, then estimate the fraction of votes that are positive. If we have $u$ upvotes and $d$ downvotes for an answer, its score will be $(u+1)/(u+d+2)$. (For instance, this would declare that an answer with 3 upvotes and 1 downvote is equivalent an answer with 1 upvote and 0 downvotes. Does that seem reasonable? Hard to say.)


title: Measure for temporal "spread"

text: I basically have a timespan from January until August (or any other timespan) which is divided into active periods (denoted by a red line in the plot) and inactive periods (the rest). What I want to compute from these periods is a measure of _temporal spread_ (couldn't think of a better name). I've already come up with a very simple measure, the percentage of active periods, but this doesn't tell me whether active periods are _spread_ all over the timespan. For example, both plots below have a percentage of active periods of about 50%, but the latter should have more "continuity", or _temporal spread_. ![enter image description here](http://i.stack.imgur.com/0l7wP.jpg) ![enter image description here](http://i.stack.imgur.com/m6vJE.jpg) In the last plot, the overall percentage is even lower than 50%, but spread should be the highest of all three, because there are also active periods in May or June. ![enter image description here](http://i.stack.imgur.com/jzzY6.jpg) As you can see, I don't know what I am looking for exactly, so I was wondering whether there exists some sort of standard measure for this problem.


title: How to export SPSS graphs so they can be edited

text: I recently switched to SPSS v20. Now when I export graphs to a DOC file the graphs are images. When I export from my earlier version of SPSS the graphs can be edited. Can anyone tell me how to export editable graphs with V20? Thanks!


title: Measuring prediction quality given discrete predictions of a continuous dependent variable

text: Say we have a variable that can assume all values between 0 and 1 and we have a system that _predicts_ measurements of this variable providing estimates in terms of 6 discrete levels (let's say 0, 0.2., 0.4, 0.6, 0.8 and 1.0). Now we are getting the actual outcomes of measurements (about 100) of this variable on a continuous scale. Given the discreteness of the input what would be the most approriate statistic to assess the predictive quality of the system? A Spearman rank correlation, a (non-parametric) regression, an ANOVA + post hoc test perhaps (given that we have 6 groups of predictions and might want to see whether there's a significant difference between them), or something else?


title: How to do a classification with only one variable?

text: I would like to classify individuals of a database by a single quantitative variable. Is hierachical clustering suitable to do this? If it is possible, how the algorithm work? If use hierachical clustering is not right, which other procedures or techniques could I use? My purpose is not classify the variable, but individuals. So decision trees are not suitable.


title: Which clustering algorithm is suitable for one-dimensional Lists without knowing k?

text: I have a one dimensional List like this               public class Zeit_und_Eigenschaft     {         [Feature]         public double Sekunden { get; set; }     }          //...     List<Zeit_und_Eigenschaft> lzue = new List<Zeit_und_Eigenschaft>();     //fill lzue      lzue can be               lzue.Sekunden     1     2     3     4     8     9     10     22     55     ...      Goal is to find clusters in that list, ie elements that could form groups like f.i. in this example               lzue.Sekunden     1     2     3     4          8     9     10          22          55      Which clustering algorithm is suitable(I don't know the number of clusters k)? GMM? PCA? Kmeans? Other?


title: Classifier vs model vs estimator

text: What is the difference between a classifier, model and estimator? From what I can tell:   * an estimator is a predictor found from regression algorithm   * a classifier is a predictor found from a classification algorithm   * a model can be both an estimator or a classifier But from looking online, it appears that I may have these definitions mixed up. So, what the true defintions in the context of machine learning? Thanks in advance


title: Facebook users similarities

text: I'm searching for a similarity metric such that given two Facebook users it returns a value that reflects how similar the two users are. The similarity metrics must take into account (at the same time) these two contributions:   1. the user profile (age, hometown, location, religion, likes, etc.)   2. the "network factor", i.e how similar two users are in relation to their network (e.g. two users sharing a lot of common friends are more similar relative to two users with few friends in common) So I need of a _unique_ metric (a unique value) that tells us how similar two users are with regard to the users profile and the network factor. **Update:** Maybe, to well understand the rationale behind my problem, can be useful the idea developed in this paper: Network and Profile Based Measures for User Similarities on Social Networks. But in this work, first, it is not clear (to me) how to combine the network and the profile similarity and, second, the network similarity (due to certain reasons discussed in the paper) is not symmetric.


title: random sampling in a polygon

text: I would like to sample a uniformly random point in a polygon... If sample a large number they'd be equally likely to fall into two regions if they have the same area. This would be quite simple if it were a square since I would take two random numbers in [0,1] as my coordinates. The shape I have is a regular polygon, but I'd like it to work for any convex polygon. http://stackoverflow.com/questions/3058150/how-to-find-a-random-point-in-a- quadrangle


title: Compare Log-Normal Distributions

text: I am looking at distributions for the fuel consumption of vehicles in the US and I have two sets of data: **Data Set 1 -** This is a dataset for the fuel consumption of _all_ vehicles made available for sale from 1990 to 2010 (not sales weighted), which follows a log-normal distribution. Thus, I am able to define the mean and standard deviation for the entire population of vehicles available for sale each year. **Data Set 2 -** This just has the annual _average_ sales weighted fuel consumption of vehicles in the US from 1990 to 2010. Thus, I am only able to define the mean (no standard deviation) of sales weighted fuel consumption each year. Is there any way for me to infer a distribution for the sales-weighted fuel consumption with the data in Data Set 1? Put another way, am I able to establish the standard deviation for Data Set 2 using the data that I have available in Data Set 1? Any advice would be greatly appreciated and many thanks!


title: Spearman's $\rho$ between multiple variables

text: I need to find a way to measure association between several variables at the same time. For example consider a function $z=f(x,y)=x+y+1$ with following sample data.               x   y   x+y z     3   1   4   5     3   2   5   6     5   1   6   7     5   2   7   8      Yes, the dataset is too small for accurate Spearman correlation measure, but it demonstrates the point. Spearman $\rho$ between $y$ and $z$ is 0.4 ~ very low. Same goes for $\rho$ between $x$ and $z$. But for $(x+y)$ and $z$ it is equal to 1. I need to come up with an idea to deal with this kind of situation: measuring correlation between multiple independent variables and function output. I suspect that this is a well known thing. I just do not know how to formulate it properly.


title: HUGE number of degrees of freedom in MLM mixed model - is this correct?

text: We are doing a mixed multilevel modeling analysis, using SAS PROC MIXED with 145 subjects, and 255 observations per subjects. We used subjects as a random factor and had three continuous variables (and the interactions between these variables) predicting our DV. The degrees of freedom we have for each of the solutions for fixed effects (except the intercept) are HUGE: 33, 280. Does that seem right? I know there are a lot of observations in this analysis, but I want to make sure that this seems reasonable. Thanks so much!


title: Regression confidence on new data point

text: I'm in the process of building a prediction system, and one of its requirements is to be able to give the confidence of a prediction. That is, given a set of independent variables, it should be able to not only give the predicted dependent variable, but also some indication of "confidence". Classifiers are typically able to return a range between predictions (from 0-1, for instance), which gives you some information about how "confident" the classifier is about a given prediction. In a binary situation, a prediction of 0.51 might denote low confidence, while 0.98 might denote high confidence, given the training set. Regression changes things a bit, as it is used to predict a continuous variable anyway, rather than a classification. What are commonly accepted methods to determine a given data point's confidence? Confidence/prediction intervals somewhat accomplish this, but my data has the potential to be very non-normally distributed. Considering just one feature, I could end up with 20 pieces of data around x=50, and only 1 around x=1. I'd expect my bounds to be tighter around x=50, but the CI/PI wouldn't really account for this uneven distribution, as it assumes normality. Is there any way to account for this? Thanks for any help you can offer, and sorry about the relative airiness of this question!


title: Paired comparison chi square

text: I have a question related to the use of chi square test for paired data. I have read that McNemar tests would be an option but in some software like R it works only for 2x2 data (may be I do not know the correct way to do it) and my data is more than a 2 x 2. These are counts for individuals before and after applying an insecticide. Some people suggested a paired t test could work, but I am not too sure about it since my data are counts. Data looks like the table below. \begin{array}{|l|r|r|} &\text{Before}&\text{After}\\\ \hline \text{Species A}&30&12\\\ \hline \text{Species B}&30&7\\\ \hline \text{Species C}&30&6\\\ \hline \text{Species D}&30&2\\\ \hline \text{Species E}&30&4\\\ \hline \end{array}


title: Inconsistency in Two stage Maximum Likelihood Estimation

text: I want to maximize a log-likelihood function (L) that is a function of parameters $\beta_i$ for $i=1,..,k$ and $\alpha_1, \alpha_2$. Ideally, I want to perform the estimation of all parameters in one step. Unfortunately, I cannot perform one step estimation due to the form of the model I have. But for some fix values of $\alpha_1, \alpha_2$, I can find the maximum likelihood estimations of $\beta_i$ for $i=1,..,k$. So what I did is that I created a function like $f(\alpha_1, \alpha_2)$ and defined it as $f(\alpha_1, \alpha_2)=L(\alpha_1, \alpha_2,\hat{\beta})$ i.e. the log-likelihood corresponding to $\alpha_1, \alpha_2$. Then I maximized this function $f$ numerically with respect to $\alpha_1, \alpha_2$. Does this approach solve the inconsistency of the two stages estimation that I have? Is this a valid approach at all? If not, is there any other estimation method that I can use?


title: Quality of NER classification decreases dramatically when sliding window transformation is performed

text: I'm writing a NER classificator now. That performs quite well even without window transformation (~80% F1-score for non-English language is quite well, AFAIK), but the strange thing is that when I perform window transform, quality decreases dramatically. Does anybody have an idea why? NER classification task has "local structure" so knowledge of local context must help, isn't it? The only assumption I can make: I have not enough training data (~300 features versus 25k training examples), so when I perform window transformation even with window size = 1 (800 features vs 25k samples) that simply overfits. I've tried reducing dimension of data using different PCAs and RMBs, but while the data itself is already quite dense, that makes no sense and reduces classification quality as well. The classificator itself is the composition of Extremely Randomized Forest and Linear SVM. Thanks.


title: Is the Poisson distribution suitable for intermittent, clumpy events?

text: Can I apply the Poisson distribution on the following type of data set? I have two types of processes, each generating events. The actual data that I posses are sets of these timestamps. The occurrences are independent and random. Unfortunately, events are clustered because the process generating them pauses for random periods (short AND long periods of pause). It is not the pause periods that interest me, but rather the periods during which the process is on (therefore the pause periods of time should not influence my analysis). Since the data is not uniformly distributed over time, but rather comes in clumps, I am not sure whether I can use the Poisson distribution here. My ultimate goal is to analyze the difference in frequency between two variants of such processes. I will compute the probability distribution for each process, for the interval of a day and see how the two differ. My fear is that the pause intervals would skew the distribution so badly that they will become meaningless to my goals. The following are histograms for the timestamps, each bin representing one day: https://dl.dropboxusercontent.com/u/17450714/i-1.jpg https://dl.dropboxusercontent.com/u/17450714/i-0.jpg If Poisson is not good for this data, how may I analyze it by excluding the huge pause periods? Cluster analysis? Thanks


title: Why marginals in graphical models matter?

text: Many of the algorithms (like subset sum) find "marginals" in graphical models. But it is not really clear to me why marginals matter. I am asking this, since when one is doing inference they do on the full joint distribution of random variables. The only reason I can think of, is that, in case we observe some of the variables, we can easily marginalize them. But to me, it looks like there is something more than this.


title: Hypothesis testing? reject or fail to reject?

text: i have come across a question which says that 70% of all people are happy. then, it goes on to discuss about how 1000 people were randomly selected and how out of them 760 were found to be happy. now the question is, should i be rejecting or fail to be rejecting the null hypothesis? the significance level is 5%. i tried to figure some things out: null hypothesis = 70% of all people are happy alternate hypothesis = 70% percent of people are not happy sample statistic = 760 (?) critical value / cutoff value = (?) how to find this? significance level = 5%, (1.96)(? what is this?) so how do i figure out if i reject or fail to reject the null hypothesis? any help on this would be greatly appreciated!