title: Why does log likelihood function for a model use SSE/n and not SSE/df?

text: I'm trying to find out how log-likelihood function works for linear regression. I found the formula here and here. Making some experiments with it (see code below), I was quite surprised that the likelihood uses `SSE/n` instead of `MSE` (`SSE/df`). `MSE` was used everywhere up to now! I thought MSE is much better estimator of $\sigma^2$ mentioned in the formula in the 1st resource (page 6) - the actual residual variance. But the 2nd resource and my experiment clearly states that $\sigma^2$ is defined as `SSE/n` (where n is length of the outcome variable vector). Here is the code to play with:               set.seed(128)     y = c(rnorm(200, 20, 4), rnorm(300, 30, 4), rnorm(400, 40, 4), rnorm(500, 50, 4))     cat1 = as.factor(c(rep(1, 200), rep(2, 300), rep(3, 400), rep(4, 500)))     rand_order = sample(1:length(cat1))     cat2 = cat1[rand_order]     cat2y = c(rep(1, 200), rep(-2, 300), rep(3, 400), rep(-4, 500))     y = y + cat2y[rand_order]     m1 = lm(y ~ 0 + cat1 + cat2)          # logLik using residual degrees of freedom (-3941.94):     -length(m1$model$y)/2*log(2*pi) - length(m1$model$y)/2*log(sum((m1$residual)^2)/m1$df.residual) - 1/2*m1$df.residual          # logLik using N (-3941.931)     -length(m1$model$y)/2*log(2*pi) - length(m1$model$y)/2*log(sum((m1$residual)^2)/length(m1$model$y)) - 1/2*length(m1$model$y)          # real logLik (-3941.931)     logLik(m1)


title: What statistical measures are recommended for mixing of population samples?

text: I want to know what statistics should be used to measure the mixing of languages in cities in a country/state. I have samples from individuals and know the language they speak and the city they live in. The cities are not all the same size and there are 3 different language groups. The samples are taken uniformly from the total population so that larger cities have a proportionally larger number of samples in the data set. I want to avoid naive approaches such as; taking the value from A/B since 100/50 is the same as 2/1 and have equal contributions.


title: Why are the coefficients of the first principal component all of the same sign?

text: This is probably really simple but I can't figure it out... was trying for few days... If _all the coefficients of a positive-definite covariance matrix are positive,_ how can I prove that the coefficients of the first principal component are all of the same sign and the coefficients of all other principal components cannot all be of the same sign? I have tried to prove from the definition but can't get the idea... can any one help?


title: Two-sample T-test with weighted data

text: I want to perform a two-sample T-test to test for a difference between two independent samples which each sample abides by the assumptions of the T-test (each distribution can be assumed to be independent and identically distributed as Normal with equal variance). The only complication from the basic two-sample T-test is that the data is weighted. I am using weighted means and standard deviations, but weighted N's will artificially inflate the size of the sample, hence bias the result. Is it simply a case of replacing the weighted Ns with the unweighted Ns?


title: Ranking distributional data by similarity

text: The question in short: What methods can be used to quantify distributional relationships between data when the distribution is un-known? Now the longer story: I have a list of distributions and would like to rank them based on their similarity to a given base-line distribution. Correlation jumps into my mind in such a case and the Spearman correlation coefficient in particular given that it does not make any distributional assumptions. However, I would actually need to create the coefficient based on binned data (as this is done for histograms or densities) rather than the raw data and I don't know if this is actually a valid step or if I am just manufacturing data. In other words, if I have a 10000 point data set for each distribution, I would first create a binned distribution for each were each bin is of equal width and contains the frequencies of how many points each bin has. Just the way this is done for density plots or histograms. Each bin is on a discrete scale. The data is actually computer screen coordinate data and values are between 1 and 1024. Each pixel position could represent a bin (but larger bins are possible e.g. every 5 pixel being one bin). I would then compare the sequence of bins with each other rather than the raw data. The data set would look like this. bins: 1 2 3 4 .... 1024   dist#base:1 2 2 3 ..... 3   dist#1: 1 4 5 5 3   dist#2: 2 2 3 5 6   ...   dist#1000: 1 2 4 6 6   Does this make sense? Are there better ways of doing that? Are there better statistical methods? The goal of all this is to first) test how close are distributions from measure A to measure B and second) if I can predict one, if the other is missing. Best, Ampleforth


title: Is it legitimate to use factor analysis or clustering before regression

text: My goal is to make a logistic regression. The `DV` is a yes or no variable, and I already found 3 significant `IV` in my model. The problem is: I have 5 other categorial (yes or no) variables (they are nearly about the same topic) that I think they have an impact in the `DV`. Unfortunately, none of them were significant in my model. The question is: Is it legitimate to use a `factor analysis` or a `clustering technique` to construct one variable that summarize the information? and then use it as an `IV` in my model? if yes,can this variable be significant? and what kind of analysis do you suggest? Thanks in advance.


title: What are guidelines for SmartPLS boostrapping case size?

text: In SmartPLS, bootstrapping is used to generate the t statistic from which statistical significance can be judged. The two main bootstrapping parameters are case and sample size. Increasing the sample only seems to ensure that the t statistic is stable. Increasing the case size seems to greatly increase the t statistics. E.g. if I have 160 observations, my t statistic for one path is about 2. If I increase the case size to 2000, the t statistic is about 7. It seems almost like cheating to use a huge case number just to get statistical significance, but I'm not aware of standard practices. What are the guidelines for bootstrapping case size?


title: Relation between the phi, Matthews and Pearson correlation coefficients?

text: Are the phi and Matthews correlation coefficients the same concept? How are they related or equivalent to Pearson correlation coefficient for two binary variables? I assume the binary values are $0$ and $1$. Thanks and regards! * * * The Pearson's correlation between two Bernoulli random variables $x$ and $y$ $$ r_P = \frac{E[(x-E[x])(y-E[y])]}{\sqrt{Var[x] \cdot Var[y]}} = \frac{E[xy] - E[x]E[y]}{\sqrt{Var[x] \cdot Var[y]}} $$ $E[x]= \frac{n_{1\bullet}}{n}$, $Var[x]= \frac{n_{0\bullet}n_{1\bullet}}{n^2}$, $E[y]= \frac{n_{\bullet 1}}{n}$ $Var[y]= \frac{n_{\bullet 0}n_{\bullet 1}}{n^2}$, $E[xy]= \frac{n_{\bullet 1}n_{1\bullet }}{n^2}$ $$ r_P = \frac{E[xy] - E[x]E[y]}{\sqrt{Var[x] \cdot Var[y]}} = \frac{n_{1 1} n - n_{1\bullet} n_{\bullet 1}}{\sqrt{n_{0\bullet}n_{1\bullet} n_{\bullet 0}n_{\bullet 1}}} $$ * * * From Wikipedia, the phi coefficient > In statistics, the phi coefficient (also referred to as the "mean square > contingency coefficient" and denoted by $φ$ (or $r_φ$)) is a measure of > association for two binary variables introduced by Karl Pearson. This > measure is similar to the Pearson correlation coefficient in its > interpretation. In fact, **a Pearson correlation coefficient estimated for > two binary variables will return the phi coefficient**. ... > > If we have a 2×2 table for two random variables $x$ and $y$ > > ![enter image description here](http://i.stack.imgur.com/Lo3Ef.png) > > The phi coefficient that describes the association of $x$ and $y$ is $$ \phi > = > \frac{n_{11}n_{00}-n_{10}n_{01}}{\sqrt{n_{1\bullet}n_{0\bullet}n_{\bullet0}n_{\bullet1}}} > $$ Matthews correlation coefficient from Wikipedia: > The Matthews correlation coefficient (MCC) can be calculated directly from > the confusion matrix using the formula: $$ \text{MCC} = \frac{ TP \times TN > - FP \times FN } {\sqrt{ (TP + FP) ( TP + FN ) ( TN + FP ) ( TN + FN ) } } > $$ In this equation, TP is the number of true positives, TN the number of > true negatives, FP the number of false positives and FN the number of false > negatives. If any of the four sums in the denominator is zero, the > denominator can be arbitrarily set to one; this results in a Matthews > correlation coefficient of zero, which can be shown to be the correct > limiting value.


title: Mixed groups of participants

text: It is permissible to analyse data for a group of persons, which contains repeated measures data and independent measures data. Can I analyse this data as if it comes from one group? If not why not? Thanks for your answers


title: Only few Levels Of Dummy Variables are significant

text: When I run a mutiple regression on 3 independent variables(2 dummy variables with multiple levels each and a metric variable). In my result only few levels of a dummy variable are significant. The result of the levels for the same dummy variable are insignifcant. How do I interpret it? Is the whole variable with multiple levels inisignificant or do the rest of the levels exhbit the base category behaviour? Thanks


title: Var(AZ)=A Var(Z) A^T?

text: I am learning linear models, and I do not understand the following: $\text{Var}(AZ)=A \text{Var}(Z) A^T$ where $A$ is a constant matrix. I want to know the variance $\widehat{\beta}$ in a linear model: $y=A\beta + \epsilon$, and $\widehat{\beta}=(A^T A)^{-1}A^T y$. To show $\widehat{\beta}=\sigma^2 (A^T A)^{-1}$, I need to know $\text{Var}(AZ)=A \text{Var}(Z) A^T$. How do I show that?


title: Spearman's rank correlation max number of respondents?

text: So apparently, our subject research proposal was turned down because we used Spearman's correlation to determine if there is any existing relationship between the time spent in playing video games and the grades of 240 students. We were told that Spearman's correlation could only be used at a max of 120 students, which made us really too shocked to speak (because we didn't know anything like that and also we lacked evidence to disprove our 80-year old professor). We've been reading a lot of different books so far and have not seen anything about this. I would really appreciate it if you could help us clear this confusion and find some information about the limitations of Spearman's correlation. Thank you. :)


title: Run Many Small or a Few Big Simulations to Estimate the Mean?

text: I was just running some simulations on tossing a coin given certain conditions, to test out some ideas I had. I was trying to find the ratio $\frac{\mathtt{successful\ tosses}}{\mathtt{total\ tosses}}$. Coin tosses were iid. At first, I was running simulations of 10,000 coin tosses, and I would have the program print the proportion of successful tosses. Then I realized I could, at the expense of a few more seconds, run a simulation of 100,000 coin tosses. I knew that the standard error of the mean is inversely proportional to the square root of the sample size, so I thought that this might give me a more accurate result. However, then I also thought: if I take ten simulations of 10,000 coin tosses each rather than 100,000 coin tosses, I will be able to estimate a normal distribution using those ten datasets, which would allow me to home in on the mean and thereby on the true ratio. My question is: for the purpose of finding the aforementioned ratio, are these methods equivalent, or is there some fine difference between the two?


title: Get probability distribution from decision tree

text: I'm implementing decision tree based on CART algorithm and I have a question. Now I can classify data, but my task is not only classify data. I want have a probability of right classification in end nodes.   For example. I have dataset that contains data of classes A and B. When I put an instance of some class to my tree I want see with what probability the instance belongs to class A and class B. How can I improve CART to have probability distribution in the end nodes?


title: Power calculation of study using 2 sample z-test

text: This is not homework! I have two very different answers for a power calculation I have done on the below data -               n = 50 in total, 25 in each group          hit = if the number called is then thrown on a dice     probability of hit = 1/6     each person has 36 dice throws          control group = call random number then throw a dice     wish group = wish for a number,call it out and then throw dice          2 groups tested by 2 sample z-test because we know distribution parameters          i.e mean chance is 6 (1/6 x 36 trials), standard dev is 2.23          control group hit total is 150     wish group hit total is 180      I know the result of the 2 sample z-test (z=-1.9025). That is not the question. **Q.How do I calculate the power of the study** (given I am interested in the difference between the groups rather than deviation from chance)? I think my study is under powered but have done this calculation in the pwr package of R -               h0 <-6 #control group     ha <-7.2 #wish group     sigma <- sqrt(36*1/6*5/6)     d = (ha - h0)/sigma          pwr.norm.test(d = d, n = 25, sig.level = 0.05, alternative = "greater")           Mean power calculation for normal distribution with known variance                    d = 0.5366563               n = 25       sig.level = 0.05           power = 0.8504646     alternative = greater      I think this is wrong.


title: Is there a measure of covariance for larger dimensions?

text: Can you measure the covariance of three or more variables simultanesously?


title: R: result of Levene test correct?

text: I want to use the Levene test to quantify the homo/heterogeneity of the variances of two samples. The density plot looks like this: ![Density plot of samples](http://i.stack.imgur.com/446M5.png) But the Levene test for the data [`hov(average~window,data=df)`] yields 1.531-e13, i.e. the probability that the variances are not equal is very high. But given the density plot, is this result reasonable or have I used the function incorrectly? To me the samples look identical, thus the variance should not be so different.


title: Evaluating performance of machine learning models

text: I'm trying to predict when a location has enough wind to go sailing. I've built several models that look for correlations between different weather conditions and whether it ends up being windy X hours later. Ultimately, I want to produce something like "Based on current conditions, there is a 70% chance it will be windy 1 hour from now." I have several models that consider different factors, so my challenge is figuring out which one to use. In this particular location, I know it's windy 15% of the time (3 years of data). Therefore, a model that always predicts a 15% chance basically deserves an overall score of 0 -- because it's more or less a coin flip. On the other end, a model that always predicts 0% or 100% deserves a "perfect" score. But what about in between? Is that the right way to compare models? How should I calculate an overall score for each model? Thx in advance!


title: 2 sampled ks test in r

text: I need to do a two-sample Kolmogorov-Smirnov (KS) test in `R`, only I don't understand the formulae and how it works when I look it up. I suspect this is because I don't understand the maths of what goes on in this test and just know whether it will tell me if the data are normally distributed or not. Any help given (about the formulae, interpretation, and even the usage in `R`) would be brilliant.


title: How to blend multiple time series models?

text: I have three different linear, multi-variate time series models with a best fit against the same observed value $Y$ at 1 minute, 3 minutes and 10 minutes horizons respectively. Each model is using different predictors data. There is no serial correlation or ARIMA involved here. I want to blend the three models to predict $Y$ every second, on a rolling base. The prediction for the three models is thus overlapping. For example, at time $t_3$ minutes, we'll have the prediction for the first and second model colliding (third prediction for the 1 minute model and first for the 3 minutes model). How should I approach the problem?


title: Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?

text: The coefficient of an explanatory variable in a multiple regression tells us the relationship of that explanatory variable with the dependent variable. All this, while 'controlling' for the other explanatory variables. How I have viewed it so far: While each coefficient is being calculated, the other variables are not taken into account, so I consider them to be ignored. So am I right when I think that the terms 'controlled' and 'ignored' can be used interchangeably?


title: Statistical Treatment for Hypothesis test

text: What statistical treatment should I used to test my hypothesis (there is no significant relationship or interaction between the 2 variables with respect to their preferences) if I have three samples and my data are expressed in frequency?


title: Are interaction plots only useful for fixed effects?

text: To test whether you want to add an interaction term to your model, is an interaction plot only useful when both effects are fixed. How do you determine whether to add interactions between random factors and fixed factors?


title: Script for downloading Medline abstracts with PMID

text: I am looking for a script that will enable me to download a set of abstracts (or article meta-data of an article) from PubMed by supplying it with a list of PubMed ID (PMID) numbers (e.g., from a .csv file). The ideal script would be for R and then Python, but I am certainly open to any solutions or suggestions. PubMed can be found here. As a side note, I did find an older package called MedlineR, but this package never made it to Bioconductor, and there doesn't seem to be any recent chatter on it. Thanks in advance. Brian


title: Model for two way table

text: Suppose you have data in the following format:                              <60 60-69 70-79 80-89 90-99     No Exercise      5   10     2    4    1     Moderate         2   4      12   4    3     Heavy            4   5      18   10   2     Extreme          5   6       7    9   12      The cells represent the number of people who died at that age range. Beside a chi square test, are there any other models that apply here? Would a poisson model be good here? Or could I just run a standard ANOVA? I want to test differences in death at different exercise levels.


title: Regression or Manova?

text: I have the following problem: I have one manipulated IV (degree of co-creation) based on three surveys "no" "low" and "high" degrees of co-creation. Furthermore I have 4 DV (Purchase intention, loyalty, word of mouth, willingness to pay) I want to test whether the degree of co-creation has a direct effect on my dependent variables. What should I do? Regression by means of dummy variables or manova? Or can it be an added value to do both? Kind regards, Daan


title: How to calculate the confidence interval of a function of a combination of two linear models

text: We have two linear fits, one for each data-set (unfortunately they include weights but I'm willing to ignore that if there's a nice analytic solution to this). Data-set `d1` and `d2` are different in the sense that `x` is the same variable but includes different discrete points (so `x` in `d1` might be 100,120,500,... and `x` in `d2` might be 110,110,335,...), and their `y`'s are not the same. So in R jargon:               f1 <- lm(y~x,weights=w,data=d1)     f2 <- lm(y~x,weights=w,data=d2)      I then want to take the trend line of each of these fits and use it in the following equation:               z = 1 / exp( 0.02254*f1^2 / f2^2 )      I did that and plotted out this z, which is dandy. But! using the predict function in R, I managed to get the 95% confidence interval trend-lines of `f1` and `f2`, which makes a nice graph. I'd love to be able to calculate/plot the same confidence interval trend-lines for the `z` line. The two ways I thought would work are:   1. to get the lower CI for `z` I take the lower CI for `f1` and the upper CI for `f2` and plug them into the `z` equation. To get the upper CI I do the opposite. I doubt this is correct.    2. I generate random `y` data for specific `x` data from the `f1` model (I'm not sure how to do that) and then generate random `y` data for the same `x` data from the `f2` model. This way I have `y`'s for `d1` and `d2` that share the same `x` data. I then can plug those discrete `y`'s into the `z` equation and get my discrete `z` points. I then try to fit some model to that, though these discrete `z` points might not even follow a linear trend. this might be difficult or just silly. So, this is my problem... Any brilliant ideas?


title: Chi square test of association

text: I wanted to study the distribution of BMI by age. So I randomly made age groups and did a chi square test. Is this appropriate?


title: Multivariate regression in Stata

text: Given a data set of course grades, there is a female student dummy variable that is set to 1 if a student is female, and 0 if male. There is also a dummy for female instructors. How would I perform the Hypothesis test using T-test with 5% significance, that the grades in classes taught by female instructors are higher than ones taught by male instructors, and that these differences are larger for female students? I am trying to do this in STATA. I know how Hypothesis testing works, but I don't know how to set up this problem.


title: similarity of feature vector

text: What is a better measure of feature vector similarity, Euclidean distance or dot product/cosine similarity? I've read about cosine similarity being used with document vectors, but I've also seen distance being used in machine- learning/neural network methods.


Supposing a Bayesian classifier with multivariate normal densities, how do I find the error rate of the classifier when we have two classes? I am using this: When dimension $d = 1$: $$P(x | \mu , \sigma^2) = N(x, \mu, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}} $$ In $d$ dimensions: $$P(x | \mu, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2}} \cdot e^{-\frac{1}{2}{(x-μ)^T\Sigma^{-1}(x-μ)}}$$ where $\mu$ is in $\mathbb{R}^d$ and $\Sigma$ is a $d \times d$ variance- covariance matrix. What would be the algebra to get the error rate, or could you give an example? I would like to do this in `Matlab`. Lets say I have 2 classes and 2 atributes with 20 examples:               features:         1.8756    1.4236         2.0677    1.3759         0.8540    0.7782         0.5651   -0.3511         0.6103   -0.6901        -0.1945    1.6438        -0.2620    0.8022         1.5326    0.0188         0.3334    1.0578         0.8535   -1.8545         0.3066    1.9716        -1.4424    0.4216         0.3275   -0.2844        -0.0079    2.8506         0.0114    1.4001        -0.4049   -0.3981        -0.0913    2.2094         0.3376   -1.0467         0.3455    2.4960         0.3232   -0.5614               and targets               2          2          1          1          1          2          1          1          2          1          2          1          1          2          2          1          2          1          2          1      What is the process to classify them and get the error rate of this examples, so we divide the set in 15 examples for training and the other 5 for testing?


title: How can I specify a level of a factor while in an lme?

text: I have a dataset with repeated measures at different speeds. I've binned the speed ranges into 0-0.5, 0.5-1.5, and 1.5-infinity.               mydata`$`fact<−cut(mydata$section, breaks=c(0,0.5,1.5,Inf))      which gives the levels (0,0.5] (0.5,1.5] (1.5,Inf] I first used the lme to test all speeds together to see if x and y are speed- dependent. Now I want to isolate speed bins, but I want to do it in the model instead of re-defining the factor. I can't seem to get it to work. "Fact" is what the speed bins correspond to.               lme(y ~ x + factor(repeatedmeasures) + fact, random = ~1 | z, data=mydata, na.action=na.omit)      Thanks for your help!


title: Multivariate Linear Regression in Python

text: How to compute the overall standard error of a linear regression model using Python? Which library should I use? I am looking for something like this, however, I can't see how to get the overall standard error. Any ideas?


title: Identifying subsets for outlier detection in local outlier factor

text: I am trying to gain better understanding of the idea of _local outliers_ (as discussed in this pdf) and how the function is implemented. Here are the key passages from the pdf: >   * Local outliers: Outliers comparing to their local neighborhoods, instead > of the global data distribution >   * In Fig., o1 and o2 are local outliers to C1, o3 is a global outlier, but > o4 is not an outlier. However, proximity-based clustering cannot find o1 and > o2 are outlier (e.g., comparing with O4). 1 >   * Intuition (density-based outlier detection): The density around an > outlier object is significantly different from the density around its > neighbors > An here is the figure referenced: ![enter image description here](http://i.stack.imgur.com/EFB37.jpg) * * * **Specific questions:** Say the overall set of data points (which contains `c1,c2,c3`) as in the figure is:               SP : {P1,P2,P3,P4,P5,P6,P7,P8}        1. Should the set SP be sub divided into further sets so in order to find outliers?   2. From above image it seems that it should?   3. What determines if the set of points should be divided into subsets and then lof is applied against each of the subsets instead of the overall set?   4. In the figure/pdf, 4 local outliers are produced but are these outliers of a single set or is each outlier just an outlier of a subset of the overall set?    5. Perhaps each reachability density corresponds to a specific subset of the items?


title: proportions at two times with predictor

text: I wonder if anyone can help me. My data has 3 main variables: proportions at 2 time periods, and an additional predictor. For example:                Item Type    Y(t)      X        Y(t+1)       1             .05        .2          .7       1             .07        .14         .06       1              0         .11         0       1              0         .05         .01       1              0         .04         0       etc.       2             .2         .1          .1       2              0         .23          0       etc        I am interested to know how the 4th column depends on the 2nd and 3rd columns. The issues are:   1. columns 2 and 4 measure the same thing at two times    2. all data are within [0,1]    3. for each item, the values in each column sum to 1    4. 99% of the data has a 0 in both columns 2 and 4 I thought of first calculating the difference between columns 4 and 2, to represent the rise/fall of each proportion, and regressing that on the 3rd column, i.e. the predictor. This is conceptually what I'm after. But still, when the value in column 2 was zero -- which is 99% of the data -- then that difference cannot be negative, meaning that the d.v. will have many small positive values and a few large negative values. Is that OK for linear regression? Thank you for any advice! David


title: Which type of random walk (1? 2? 3?) can unit root test?

text: Campbell (1997) showed that there are three types of random walk hypotheses classified by the level of restrictions on $\epsilon_t$:   1. RW1 (random walk 1: increments are independent identically distributed)   2. RW2 (random walk 2: increments are independent but not identically distributed)   3. RW3 (random walk 3: increments are serial uncorrelated) Which type of random walk (1? 2? 3?) can unit root test test? (ADF test,PP test, KPSS test) Is there any reference or mathematical evidence? Reference: Campbell, J. Y. (1997). "Chapter 2: The predictability of asset returns," The econometrics of financial markets. princeton University press.


title: Short and long-run trend

text: I have a time series, let's call it $y(t)$. Augmented Dickey-Fuller and Phillips-Perron tests indicate no unit root but a deterministic trend. The series is strongly autocorrelated. I estimated the following short-run dynamic model: $$y(t) = \alpha + \beta\cdot y(t-1) + \gamma\cdot t + \varepsilon_t$$ where $\alpha$, $\beta$ and $\gamma$ are estimated coefficients. The estimated model has no statistical anomalies and the coefficients are significantly different from zero. The estimated value of $\gamma$ measures a trend. However, a long-run trend can be estimated by calculating $\gamma/(1-\beta)$. We can estimate directly the following static, long-run model: $$y(t) = \alpha + \gamma\cdot t + \varepsilon_t.$$ **Question** : What is the _best_ , _consistent_ model to apply in an analysis of a deterministic trend?


title: random effect or time series model?

text: I'm modeling longitudinal patient level health records data which will have the following structure: \begin{equation*} \text{Prob}(Y_{it}=1 \text{ at time } t \text{ and member } i)=\beta_{i0} +\beta_{i1}x_{i,t-1} + \beta_{i2}x_{i,t-2} + ... + \beta_{ik}x_{i,t-k} + \text{additional covariates for patient age, diagnoses, etc.} \end{equation*} where Y is an inpatient hospitalization indicator variable for time period t and patient i, and the x's measure frequency of home nursing visits during prior time periods. The predictor variables are significantly correlated at the patient level. I'm considering a logistic regression, however the strong correlation between the x's makes things a bit more complicated. Is a time series model appropriate here, or would a multilevel repeated measures model be better (say using proc mixed in SAS with a random intercept for each member)?


title: Probability and limits

text: I've been having some trouble with this story problem. Any help you could give me would be really appreciated. A store manager monitors the store's temperature by taking 4 independent temperature readings at 4 locations where, if the system is working correctly, the temp. is normally distributed with a mean of 68 degrees F and a standard deviation of 3 degrees. What's the mean and standard deviation of the average of these 4 readings if the system is working correctly? What upper and lower limits should the manager set for the average of these 4 readings for there to be a .01 probability of violating a limit when the system is working correctly? Thanks for any advice/help you can give me!


title: combining binomial and normal values into one significance test

text: I have conducted an experiment in which a participant completes memory tasks (3 levels, within) while exploring a virtual environment (2 levels, within). The performance on these memory tasks is considered a binomial dependent variable. In fact there are two learning periods, each with its own memory task. After exploring the environment in these two learning periods, the participant is asked to find the shortest paths to various landmarks in the environment. We usually consider this success rate as a normally-distributed dependent variable, because we look at the percentage correct out of several path judgments combined into a single measure. The overall questions driving the experiment are: 1) Does learning such an environment (they are unusual) interfere with memory performance? 2) Does a memory task hinder learning of the environment such that subsequent shortest- path performance will be worse? Thus far I have done a separate analysis on each dependent variable. Is it possible to somehow (in a statistically valid manner) combine these two scores (actually 3, considering that there are two memory scores to each shortest path result) into a single significance test? Please let me know if you need clarification on the design. Thanks, Eric


title: Assessment of a single predictive model

text: I am facing a situation where I need to assess the predictive ability of a single (regression) model. I do not need to compare its performance with some other model, I just need to show that it predicts well or not. I have the Mean Square Error and Mean Absolute error for both on the training set and a separate test set. I also have the actual predictions so I can do some comparisons of the RMSE values with the mean predictions (or something similar) if needed. I understand that this probably depends also on the domain and how large an acceptable error is, but I cannot find any reference that discusses this topic in any level, every discussion on model assessment refers to comparison of different models. Any help would be greatly appreciated.


title: ADF-how to choose from the results?

text: I am a student and I have a project to do but I have some questions about ADF test. I have 52 observations of quarterly data. I have done the ADF test in my variables in the levels. I tested with intercept, intercept and trend and also using lags.   1. The test with inercept and the one with intercept and lags have to agree if there is or there is not unit root? 2.The eviews can directly use SIC criterion to choose lags but Stata doesn't and I am choosing randomly.If the variable has unit root (in both tests with intercept, intercept and trend) with and without lags which is preferable the one with lags or the one with out? Thank you in advance


title: Model speedup using weights in R

text: I need to run a cox-regression on a huge data set (millions) with only a few unique rows (hundreds) and regressing on categorical variable `group`. I could use just `Surv(t, c) ~ group`, but this would take a very long time. I was thinking about including only unique rows together with their count as a `weights` argument. How would you do this for this specific example? Is it possible to do such a function for generic formula?


title: Testing large dataset for normality - how and is it reliable?

text: I'm examining a part of my dataset containing 46840 double values ranging from 1 to 1690 grouped in two groups. In order to analyze the differences between these groups I started by examining the distribution of the values in order to pick the right test. Following a guide on testing for normality, I did a qqplot, histogram & boxplot. ![enter image description here](http://i.stack.imgur.com/IdmVG.png) ![enter image description here](http://i.stack.imgur.com/7BSbc.png) ![enter image description here](http://i.stack.imgur.com/GYZ2h.png) This doesn't seem to be a normal distribution. Since the guide states somewhat correctly that a purely graphical examination isn't sufficient I also want to test the distribution for normality. Considering the size of the dataset and the limitation of the shapiro-wilks test in R , how should the given distribution be tested for normality and considering the size of the dataset, is this even reliable? (See accepted answer to this question) **Edit:** The limitation of the Shapiro-Wilk test I'm referring to is that the dataset to be tested is limited to 5000 points. To cite another good answer concerning this topic: > An additional issue with the Shapiro-Wilk's test is that when you feed it > more data, the chances of the null hypothesis being rejected becomes larger. > So what happens is that for large amounts of data even very small deviations > from normality can be detected, leading to rejection of the null hypothesis > event hough for practical purposes the data is more than normal enough. > > [...] Luckily shapiro.test protects the user from the above described effect > by limiting the data size to 5000. **As to why I am testing for normal distribution in the first place:** Some hypothesis tests assume normal distribution of the data. I want to know whether or not I can use these tests.


title: finding joint Moment generating function of $U,V$

text: suppose $X_1, X_2$ are independent random variable of $N(0,1)$. suppose $$U=X_1+X_2, V=X_1^2+X_2^2$$ how can I calculate joint Moment generating function of $U,V$


title: Analytical solution to linear-regression coefficient estimates

text: I'm trying to understand matrix notation, and working with vectors and matrices. **Right now I'd like to understand how the vector of coefficient estimates $\hat{\beta}$ in multiple regression is computed.** The basic equation seems to be $$ \frac{d}{d\boldsymbol{\beta}} (\boldsymbol{y}-\boldsymbol{X\beta})'(\boldsymbol{y}-\boldsymbol{X\beta}) = 0 \>. $$ Now how would I solve for a vector $\beta$ here? **Edit** : Wait, I'm stuck. I'm here now and don't know how to continue: $ \frac{d}{d{\beta}} \left( \left(\begin{smallmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{smallmatrix}\right) - \left(\begin{smallmatrix} 1 & x_{11} & x_{12} & \dots & x_{1p} \\ 1 & x_{21} & x_{22} & \dots & x_{2p} \\ \vdots & & & & \vdots \\ 1 & x_{n1} & x_{n2} & \dots & x_{np} \\ \end{smallmatrix}\right) \left(\begin{smallmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{smallmatrix}\right) \right) ' \left( \left(\begin{smallmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{smallmatrix}\right) - \left(\begin{smallmatrix} 1 & x_{11} & x_{12} & \dots & x_{1p} \\ 1 & x_{21} & x_{22} & \dots & x_{2p} \\ \vdots & & & & \vdots \\ 1 & x_{n1} & x_{n2} & \dots & x_{np} \\ \end{smallmatrix}\right) \left(\begin{smallmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{smallmatrix}\right) \right) $ $ \frac{d}{d{\beta}} \sum_{i=1}^n \left( y_i - \begin{pmatrix} 1 & x_{i1} & x_{i2} & \dots & x_{ip} \end{pmatrix} \begin{pmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{pmatrix} \right)^2$ With $x_{i0} = 1$ for all $i$ being the intercept: $ \frac{d}{d{\beta}} \sum_{i=1}^n \left( y_i - \sum_{k=0}^p x_{ik} \beta_k \right)^2 $ Can you point me in the right direction?


title: What benefit can be gained from finding total population size

text: I would imagine this has been asked before but I can't find it so feel free to link if you can! Lets say I have a sample size of 100 and an unknown population size. There's a great deal I can learn from this sample about the total population, for example the mean of the sample is likely to be close to the mean of the population. However if I can then find the total population size does this give me new data. Specifically I can find the SIZE of the total population, but not the corresponding results for each unique entity in the population. In my case I expect the population size to be many orders of magnitude greater than the sample size, is this significant in any way?


title: 'Uniformization'?

text: I am looking for a better term for what I call ' _uniformification_ ', where I change data to make it more close to uniformly distributed. I am doing a project in which I try to make the output of a neural network uniformly distributed over the output space. Let's call the neural network $f(x | \theta)$, where $\theta$ contains the weights and other tunable parameters. What I do is to change the parameters of the network $\theta$ such that the output is closer to a uniform distribution. You could say I am making the output more uniform by modifying the function $f(x|\theta)$. How would I call this process? I was thinking of the term ' ** _uniformification_** ', but it sounds kind of weird. What could be a better term for this? Or is there already a term for this in the literature? **PS** : in no case will I use a transformation on either $x$, $\theta$ or on the output $f(x|\theta)$ in order to make the output more uniform. I am only adjusting the weights $\theta$ in order to achieve that. **Edit** : According to this post on the suffixes -ise, -ate and -ify, the term should actually be ' **uniformization** '.


title: Is the multivariate Gauss the only pdf incorporating covariances?

text: I am wondering whether there is another probability density function known to literature which is similar to the multivariate normal distribution in the respect that the pdf incorporates the covariances between all variables, it has the covariance matrix as a parameter or is in some way defined by other dependencies between each two variables. However, it doesn't need to be similar in the respect that it has a bell shape, or that its support is unlimited ($\forall x : P(X<x)>0$). I suppose copulas could fulfill these requirements, but are there any other pdfs which satisfy the condition? I haven't found any in the literature I have read.


title: Fitting survival/hazard model to probability of default

text: I will very grateful with some help on the following problem: I need to forecast probability of default for portfolio of retail loans, depending on several factors, that can be divided into three groups: first group – is initial conditions during the perdio, when these loans were issued (these factors are fixed in time, but vary among different months of issuing), second group is macroeconomic factors that vary during the life of the loans, and third factor depends to time trend. I have aggregated data, with only total figures for loans issued in particular month (not for each loan separately). I have tried to use survival analysis to estimate this problem, but with little result so far. I used the following approach: for simplicity I dropped time varying components and focused on initial conditions and time factor. Loans issued in different months can be viewed as different portfolios (vintages), and as time goes the share of bad loans in each vintage is increasing with diminishing speed. The data structure is following: new bad loans in the portfolio issued particular month               (Month passed)  (Loans issued in Jan 2000) (Loans issued in Feb 2000) …     1                0.15%                     0.12%      2                0.09%                     0.10%     3                0.05%                     0.08%     4                0.03%                     0.06%      The shape of the curve for each vintage resembles some hazard/survival function, so I tried to fit is with Weibull/Gamma and some other probability function, but with little result. I quite newcomer in survival analysis or risk management, so maybe there are some methods or algorithms that are usually used to handle such problems that I’m not aware about. I will very appreciate any suggestions or links to material where I can find some solutions and it will also be great if there is some available methods for estimation in matlab or R.


title: Intercept parameter, $\beta_0$

text: I have a hypothetical data given below that consists of 11 pairs of points (xi, yi ), to which the simple linear regression mean function $\mathbb E(y|x) = β_0 + β_1x$ is fit.:                X     Y      10    8.04       8    6.95      13    7.58       9    8.81      11    8.33      14    9.96       6    7.24       4    4.26      12    10.84       7    4.82       5    5.68      I have got intercept parameter,$\beta_0=3.001$ But the plot of the data is not showing the y-intercept is $3.001$. Rather the y-intercept is more than $3.001$. **WHY?** ![enter image description here](http://i.stack.imgur.com/HtTTU.png) I have used `R software` to calculate the parameters, $\beta_0$,$\beta_1$ and also to produce the plot.                x1 <- c(10,8,13,9,11,14,6,4,12,7,5)      y1 <- c(8.04,6.95,7.58,8.81,8.33,9.96,7.24,4.26,10.84,4.82,5.68)           lm(y1~x1)           plot(y1~x1)      abline(lm(y1~x1))      **EDIT**                 ht <- c(169.6,166.8,157.1,181.1,158.4,165.6,166.7,156.5,168.1,165.3)       wt <- c(71.2,58.2,56.0,64.5,53.0,52.4,56.8,49.2,55.6,77.8)            lm(wt~ht)            windows(9,6)       par(mfrow=c(1,2))            plot(wt~ht)       abline(lm(wt~ht))            plot(wt~ht,xlim=c(0,180),ylim=c(0,75))       abline(lm(wt~ht))      ![enter image description here](http://i.stack.imgur.com/i1bqw.png) How can i get the y-intercept? By expanding the straight line(population regression line) to negative axis of Y ?


title: How to do external validation of regression models

text: Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas in the field:   1. modelS ~ A + B + C + D + E + F    2. modelH ~ modelS + G    3. modelG ~ A + B + C + H I want to perform an external validation of each. Of course, I have plugged in the variables and have thus calculated the predicted probabilities of malignancy for each patient.   1. Is there a more formal way to compare the performance of these prediction formulas? Of course I have derived my own formula (following Harrell's methodology and using package rms). How can I properly compare my formula with the previously published ones?    2. Is it valid to take the parameters the previous formulas have found to be "significant" and derive the coefficients from my data set, thus deriving newModelS, newModelH and newModelG and then run stats (like C, likelihood chi2, etc) on those? I can also compare the newly replicated formulas by AICc, but does that inform on the previously published formulas?  The more general question is "How do you perform an external validation of a logistic regression formula"?


title: 2x2x2 repeated measures anova: do I correct for type I error?

text: I have a 2x2x2 repeated measures design and am trying to decompose a 3-way interaction. As all my factors have 2 levels, I always reach the point that I am doing pairwise comparisons for the simple effects. I have not managed to find a clear answer as to whether I should correct my p.values acquired from these dependent t.tests. From what I am seeing happening in 2-way designs, it implies to me that I should not correct. For example, if in a 2x3 anova the interaction is broken down with 2 one-way anovas comparing the 3 levels of IV2, and the only correction is for the comparison between these 3 groups only and not for the levels above that (just like if simply a one-way anova with 3 levels was being carried out in the first place), then does this mean that if the same interaction was broken down with 3 t.tests there would be no type-I correction? If this inference is correct, I suppose that even in my 2x2x2 design, there should be no correction.. (?) If I am wrong and I need to adjust p, how can I compute how many comparisons emerge from a 2x2x2 anova so that I can correct accordingly? Thanks in advance


title: What can one tell a school kid about statistics and machine learning?

text: Next week we have an intern from a local school in the house. The concept behind his short internship is to get an idea how the real world works and what certain jobs deal with, how the daily work looks like etc. Now I wondered, what one can tell/show/demonstrate such a young kid about Statistics and Machine Learning so that he/she   * gets the basic idea of this areas   * is getting enthusiastic (assuming that the kid's prior is not too heavily weighted in favor of other interests)   * wont forget about it the next day I am primarily looking for sticking images, examples for demonstration etc.. The kid's background:   * 15-16 years old   * basic math concepts are known (what is a graph, rule of three, what is a variable (mathematically, not statistically speaking)) Since the hard part of this question is to explain your area to one without any background knowledge, this question might as well serve as reference for chats with relatives and friends. _Sidenote:_ I skipped the description of my job intentionally, so that this question is not too specific, this question is about the topic of this site in general.


title: Issues with fitting distribution to heavy-tailed data

text: I am currently trying to fit distributions to some heavy tailed data-set (see the data set below) and have a hard time producing good results:               v1 <- c(11.87,14.04,11.86,179.88,13.09,14.68,13.54,32.25,54.52,66.28,15.16,39.83,89.81,116.87,298.94,427,6249.42,3334.9,4503.93,4933.9,15.72,12.33,13.64,15.12,47.86,12.3,12.79,12.34,14.44,12.17,120.17,12.82,13.07,47.48,13.72,12.19,30.48,129.16,191.41,282.96,1076.53,4354.01,7882.12,12.45,13.9,12.12,16.63,12.26,12.01,17.87,12.62,11.81,12.68,12.33,12.04,15.64,12,13.1,62.44,13.21,13.28,12.99,13.52,23.47,13.36,11.81,11.86,13.74,58.72,12.72,12.02,11.92,44.75,77.96,23.53,11.81,23.46,12.52,29.47,12.31,12.54,11.86,12.42,11.89,11.94,12.44,13.48,12.37,16.3,11.93,28.1,18.85,11.96,11.94,18.64,11.79,12.1,13.82,29.8,19.6,11.79,12.77,11.77,14.92,20.59,12.24,19.39,12.79,20.97,13.01,32.79,12.73,24.56,11.92,13.08,12.41,14.46,27.26,25.51,13.8,32.35,15.32,27.82,15.26,13.71,29.8,12.72,14.7,12.28,14.55,12.13,12.86,18.36,36.18,12.1,14.56,29.34,11.82,12,12.21,27.98,17.86,14.13,13.45,17.53,14,13.08,12.23,14.94,14.12,13.19,20.14,40.22,21.19,14.46,35.38,14.89,31.77,3057.58,12.86,12.15,20.52,132.7,17.5,31.34,15.27,13.45,15.32,24.75,12.45,13.45,29.4,13.82,13.48,83.84,100.4,12.78,31.58,16.59,12.51,59.34,66.09,232.43,12.74,12.21,73.53,109.48,13.53,17.65,87.09,18.93,12.43,12.32,15.55,14.1,12.15,12.43,11.82,12.87,12.28,140,240.49,12.76,25.97,13.6,18.32,117.4,242.05,13.94,111,161.53,247.33,13.51,15.49,65.64,14.27,35.17,11.83,30.21,29.14,12.53,11.76,14.08,49.06,212.09,258.35,13,13.74,29.08,60.23,12.16,142.66,202.93,74.79,12.88,27.48,48.91,64.79,49.25,224.59,299.4,29.24,68.88,15.12,34.8,23.68,43.55,12.4,17.61,18.1,15.15,11.92,14.17,13.45,14.51,44.46,14.24,34.15,258.84,12.75,12.77,34.44,13.12,22.49,53.46,14.96,13.75,11.77,16.2,12.52,12.19,17.69,34.83,13.25,12.39,29.59,56.69,82.38,12.13,27.69,15.12,50.21,68.42,16.84,14.96,11.81,15.53,168.74,797.01,52.84,67.02,15.83,167.27,240.05,12.03,48.64,30.45,28.81,54.1,17.73,33.99,19.93,37.21,35.3,122.36,44.94,15.2,26.46,217.48,257.06,14.69,13.22,55.8,26.95,55.05,16.71,44.58,20.71,14.24,41.69,58.3,108.43,137.71,13.89,19.53,46.72,22.45,36.93,20.72,17.39,15.32,28.83,16.34,26.04,44.12,17.84,14.23,14.17,13.63,13.12,12.91,12.72,36.33,18.25,14.06,14.67,27.51,18.38,12.69,14.14,16.19,11.87,12.26,31.92,14.09,19.07,32.24,19.29,34.24,21.39,13.05,17.57,5651.61,6635.33,1666.81,6692.4,2161.37,15.63,37.85,61.85,68.92,252.31,16.45,28.21,57.45,93.8,70.53,178.19,239.22,270.67,419.6,11.93,11.88,14.38,51.44,54.91,81.9,112.63,3911.01,8625.72,9144.85,11.9,19.59,39.06,3153.42,8628.67,17.58,12.7,11.91,17.08,11.92,18.83,12.09,13.19,14.02,11.74,42.91,225.66,257.56,18.97,58.93,150.21,249.29,262.74,20.67,48.07,239.44,283.07,777.53,866.46,2570.59,5306.95,7773.85,8706.43,8730.16,21.43,86.28,12.22,103.45,120.04,197.63,502.12,580.07,19.02,18.98,12.3,13.49,50.26,76.13,14.69,44.07,73.74,180.95,13.37,15.37,58.62,60.12,228.92,251.56,268.03,11.77,16.83,50.36,63.02,107.3,234.99,261.7,18.09,58.17,75.96,220.08,250.4,16.36,14.1,61.36,140.59,278.06,417.68,797.12,1633.51,3911.59,3463.77,32.29,59.93,17.8,70.88,88.52,244.83,282.94,312.01,658.95,828.67,15.23)      I tried to fit a generazlied extreme value and a skewed student-t distribution to it:               library(fExtremes)          # generalized extreme value distribution     empFit <- gevFit(v1)     coefs <- slot(empFit, 'fit')$par.ests     qqplot(v1, rgev(1e4, xi = coefs['xi'], mu = coefs['mu'], beta = coefs['beta']),ylim=c(0,1e4))     abline(0,1)               # skewed-student t     empFit <- sstdFit(v1)     vpars <- empFit$estimate     qqplot(v1, rsstd(1e4,vparas[1],vparas[2],vparas[3],vparas[4]),ylim=c(0,1e4))     abline(0,1)      The qqplots do not look very promising. The gev-distribution is completely off and the student-t is not capable of capturing the tails very well. I would really appreciate any help / comments in this regard as I am no expert in hvy-tailed distributions. I already tried to just fit the values above the .95 quantile with a generalized pareto and the remaining part with something else. But I am not sure whether this is a good approach to take. Another observation: Also tried out some other distributions and packages but very often find that the optimizers for fitting the distribution have a hard time in dealing with the likelihood (that is why I also question the parameters obtained in the above one examples). Best regards, noclue


title: Proper way to stack models when some models aren't always applicable?

text: Suppose you have two (or more) models that you want to ensemble together. However, some of the models are trained specifically on very specialized subsets of your data. If you do the stacking with a simple linear regression where the independent variables are the model outputs, how should you represent that output of the specialized model? Should you just put a zero for the model output on cases where the model is not applicable? Apologies in advance if I'm fundamentally misunderstanding how to do stacking, this is the first time I've tried it.


title: How to draw valid conclusions from "big data"?

text: "Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, everybody seems to be happy just to get _anything_ out. But aren't we violating all classic statistical principles such as hypothesis testing and representative sampling? As long as we make only predictions about the same data set, this should be fine. So if I use Twitter data to predict Twitter user behavior, that is probably okay. However, using Twitter data to predict e.g. Elections completely neglects the fact that the Twitter users are not a representative sample for the whole population. Plus, most methods will actually not be able to differentiate between a true "grassroots" mood and a campaign. And twitter is full of campaigns. So when analyzing Twitter, you quickly end up just measuring campaigning and bots. (See for example "Yahoo Predicts America's Political Winners" which is full of poll bashing and "sentiment analysis is much better". They predicted "Romney has over a 90 percent likelihood of winning the nomination, and of winning the South Carolina primary" (he had 28%, while Gingrich had 40% in this primary). Do you know other such **big data fails**? I remember roughly that one scientist predicted you could not maintain more than 150 friendships. He actually had only discovered a cap limit in friendster ... As for twitter data, or actually any "big data" collected from the web, I believe that often people even introduce additional bias by the way they collect their data. Few will have all of Twitter. They will have a certain subset they spidered, and this is just yet another bias in their data set. Splitting the data into a test set or for doing cross validation likely doesn't help much. The other set will have the same bias. And for big data, I need to "compress" my information so heavily that I'm rather unlikely to overfit. I recently heard this joke, with the big data scientist that discovered there are approximately 6 sexes in the world... and I can this just so imagine to happen... "Male, Female, Orc, Furry, Yes and No". So what methods do we have to get some statistical validity back into the analysis, in particular when trying to predict something outside of the "big data" dataset?


title: Estimators for Bernoulli trials

text: I have failure data for an experience over $T$ years: at the beginning of each year I have $n_t$ subjects and $d_t$ of these subjects experience a "failure" at the end of the year. Now if I assume that the trials are iid Binomial and the probability of failure $p$ is homogenous (i.e. constant over time), the ML estimator is given by $$\hat{p}=\frac{\sum\limits_{i=1}^T d_t}{\sum\limits_{i=1}^T n_t}$$ However, there is an alternate estimator: $$\bar{p}=\frac{1}{T}\sum\limits_{i=1}^T \frac{d_t}{n_t}.$$ What would be the assumptions and the model so that the estimate is $\bar{p}$? =========================== **EDIT:** The question is not about the relative merits of the two estimators. For example, it is not necessary to assume homogeneity for the second estimator. I am wondering about the assumptions that would "naturally" lead to the second estimator. =========================== **EDIT2:** I am in a situation where $\bar{p}$ is imposed as an estimator. I need to come up with confidence intervals or credible intervals for the estimate. My idea was to determine the assumptions (possibly a hierarchical model) and do a MCMC using `JAGS`...


title: Covariance Matrices

text: I was wondering if anyone knows of any really productive ways to graph/map/plot/visualize covariance matrix data. I am currently using `levelplot()` in `R`. It's very basic, and does the job, but we're looking for something better. We're also using `R` Shiny so that there is more interaction between the user and the data, but that also seems pretty basic. Please keep in mind my matrix is 2600x2600, so very large! If you have any neat, creative ideas, or know of packages that I would be interested in, I'd really appreciate it! Thanks!


title: Estimation of simultaneous equations model

text: My model is as follows: $$ \begin{aligned} y&=a_0 + a_1x_1 + a_2x_2 \\ x_2 &= b_0 + b_1x_1 + b_2z \end{aligned} $$ I'm only interested in the effect of $x_2$ on $y$. More precisely, I want to see the combined effect of:   1. $x_2$ directly, and    2. $x_2$ implicitly (through $x_1$) After estimating the coefficients, I have the following procedure:   1. solve equation 2 for $x_1$    2. substitute the resulting term in equation 1.   3. look at the "new" $x_2$ coefficient: $(a_1/b_1 + a_2)$ The model comes from theory and there's no possibility to include other variables as instruments. I tried multiple equations OLS with HAC standard errors and got some good results, but I'm not at all convinced that this is the right way.


title: Effect size for Fisher's exact test?

text: Does anyone know a way to calculate the effect size of a Fisher's exact test? Does that even make sense?


title: What is the minimal amount of sample runs required for doing a minimum detection limit?

text: I have to set up an example of how to do an MDL, and the professor wants to know how many times I'd run the samples. My previous class used as little as 8, but I'm not sure if thats correct or if I should say 20 instead.


title: ANOVA - independence assumption

text: I have conducted a study in which I have gathered 3 sentences per participant. These sentences were then classified and the words per sentence were counted. I want to test if there are significant differences in the word count means for the different categories of the classification. However, because the sentences are not fully independent (each participant has entered 3 sentences), I am worried that ANOVA would not be the right analysis, since it assumes the independence of the cases. Would it be better to carry out non-parametric statistics on transformed word counts, or is it okay to do an ANOVA? PS I have asked a related question about using the $\chi^2$ test in a similar situation to identify interactions between factors.


title: Random Forest - understanding k fold cross validation

text: I am trying to improve my data science knowledge by solving problems available on the internet. I am currently using the R package `randomForest` to classify the data. By using all of the features available in the training set I get an AUC of about 87% and when I use my model on the test data I get an accuracy of about 77%. Now I am trying to understand k fold cross validation. I have the following questions   * What is the objective of this process?   * Can someone explain the output of `randomForest$error.cv`?    * Where should I use the 'k' that is obtained from this procedure? Thanks all. Any help would be much appreciated


title: Deriving covariance from "embedded" random variable

text: I'm trying to derive covariance from "embedded" random variable (r.v.). Say we have two Gaussian r.v.'s $$ X_1 \sim N(\mu_1, \sigma^2_1)$$ $$ X_2 \sim N(\mu_2, \sigma^2_2)$$ where $\mu_1$ and $\mu_2$ are also Gaussian r.v, $$\mu_1 \sim N(\theta_1, \delta^2_1)$$ $$\mu_2 \sim N(\theta_2, \delta^2_2)$$ with covariance $\delta_{1,2}$. Here we assume that $\sigma$, $\theta$, and $\delta$ are all known. I'm trying to get $\mathrm{cov}(X_1, X_2)$ following the definition, $$\mathrm{cov}(X_1, X_2) = E[X_1, X_2] - E[X_1]E[X_2] \>.$$ However, I'm not sure how to incorporate $\delta_{1,2}$ into $E[X_1, X_2]$ when marginalizing out $\mu_1$ and $\mu_2$. Or, vice versa, given covariance $\sigma_{1,2}$ between $X_1$ and $X_2$, can we derive $\delta_{1,2}$? Any hint would be appreciated!


title: Best method to analyse whole population data

text: I have data of an entire population (N=27). I would like to find out which variables (of many) have the greatest effect on one certain dependent variable, and how much of its variance can they explain. The possible independent variables could be correlating as well (some of them certainly do). They are all scale measured. What do you think would be the best method to use and what should I pay attention to? I'm limited to SPSS, and rather sketchy statistical knowledge. I have been trying to get something with linear regression but I fail to achieve anything interpretable, and it crossed my mind that I might need something entirely different since this is not a sample. Thanks in advance! EDIT: For this question this might not be important, but: I have all the variables 5 times for 5 separate years. Later on I am planning on examining how the effect of a variable changes over time. EDIT nr.2: After the first replies it seems the best to detail the database I have: The 27 cases are 27 European countries, and the dependent variable is the percentage of their population that participated in demonstrations in that year. I also have lots of possible independent variables like gdp, unemployment, happiness, etc. I have all these values for 5 separate years. And basically I'm trying to find something that I can write about in my thesis, like "gdp is the biggest factor and its twice as big as happiness blabla _...in these countries_ ". The reason why I'm saying its the entire population, because I am not planning on drawing general conclusions. I wouldn't be able to do that anyway as the countries aren't even representative for Europe.


title: Why may one use qbeta() to calculate CIs for binomial proportions that are actually based on the F-distribution?

text: I just barely grasped why large-sample confidence limits that assume approximately normal distributions are not the best approach when relative frequencies approach either one or zero - so that in these instances methods according to Jeffrey or Clopper/Pearson are preferable, and can be calculated with relative ease using R's `qbeta()` (e.g. discussed here: Confidence interval for a proportion when sample proportion is almost 1 or 0). Now wikipedia (http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval) and others appear to claim that this is allowed due to some relationship between the F-distribution and the Beta-distribution, but I am struggling to follow either this argument or why it would allow using `qbeta()` instead of another (more complicated) parametrization of either `qf()` or `pf()`. Is there a rather straight explanation for this equivalence that a non-mathematician might follow? Thanks!


title: Asymptotic distribution of MLE (log-normal)

text: Say we have a sample $X_{1},...,X_{n}$ from a log-normal distribution with parameters $\mu$ and $\sigma^{2}$. That is, $\ln(X)$~$N(\mu,\sigma^{2})$. Let $T_{n},Z_{n}$ denote the MLE's for $\mathbb{E}(X)$ and $\mathbb{D}^{2}(X)$ respectively (based on the sample of size $n$). The question is: What are the asymptotic distributions of these estimators? I know they are both asymptotically normal with means $m_{t}=\exp(\mu+\sigma^{2}/2)$ and $m_{z}=m_{t}^2(\exp(\sigma^{2})-1)$, but what about the variances?


title: Has anyone studied the properties of the following method of estimation?

text: In Bayesian inference, we usually sample from the posterior $f(\theta_1,\theta_2|-)$ via MCMC to compute point estimates for the parameters of interest. I am investigating an alternate form of arriving at a point estimate. Sometimes it happens that drawing from the full posterior, $f(\theta_1,\theta_2|-)$, is difficult but it is relatively easy to draw from the conditionals $f(\theta_1|\theta_2,-)$ and $f(\theta_2|\theta_1,-)$. Thus, I want to investigate the properties of the following sampler: `Initialize` $\theta_1=\theta_1^0$, $\theta_2=\theta_2^0$               Converged = False       Do While Not(Converged) $\quad$ `Set` $\theta_1^i$ = $E(\theta_1|\theta_2^{i-1},-)$   $\quad$ `Set` $\theta_2^i$ = $E(\theta_2|\theta_1^{i-1},-)$   $\quad$ `If` $|\theta_1^i-\theta_1^{i-1}|\le \epsilon$ `and` $|\theta_2^i-\theta_2^{i-1}|\le \epsilon$                   Converged = True         End If       End While Some preliminary simulations suggest that the above sampler converges very fast in the context of linear regression. Could someone point me to an existing literature which looks at theoretical properties of the above sampler?


title: measuring the predictive power of a (linear) model

text: In a relatively simple setting, I try to fit a simple linear regression model. I want to assess how good this model for prediction. basically I try to do something like K-fold cross validation or jackknifing (that is - fit the model on part of the data, and predict the rest to see what happens). However, I seek a single measure that captures the overall "predictive power". I guess there are some theoretical solutions to that, so any reference/link will be appreciated. In the meantime, what I tried to do is to do cross-validation, and for every case in my sample I now have the observed value, the predicted value (based on some subset of the data), and the residuals. In the full linear model we know that $\sum(y^2) = \sum(\hat{y}^2)+\sum(e^2)$ but when using the predicted values on a subset this is likely to change so I define: $$M = \sum(\hat{Y}^2)+\sum(e^2)$$ based on prediction and use $$F = \frac{(M-\sum y^2)}{\sum (y^2)}$$ as a measure of how much the prediction deviates from the full linear model, with 0 means the best possible. I'd appreciate any feedback on this method and/or references to similar analyses.


title: Statistics problem travel distance

text: I've tried to solve this but I have no idea; I tried it several times but I still didn't manage to get a correct solution. For a sample of 10 students, the following bivariate data represents the distance and the duration of their travel to school. $$\begin{array}{|c|c|} \hline \mathbb x & 1 & 3 & 5 & 5 & 7 & 7 & 8 & 10 & 10 & 12\\ \hline\\ \mathbb y & 5 & 10 & 15 & 20 & 15 & 25 & 20 & 25 & 35 &35\\ \hline \end{array}$$ (a) Determine the scatter plot and the correlation coefficient of the sample. (b) Does this sample show sufficient evidence for the positive linear correlation of the distance and the duration of travel in the case of all students? (c) Find the equation of the regression line. (d) Does the slope $b_1$ of the regression line show sufficient evidence to claim that $b_1 > 0$ at a significance level 0.05?


title: What is the best way to visualize relationship between discrete and continuous variables?

text: What is the best way to show a relationship between:   * continuous and discrete variable,   * two discrete variables ? So far I have used scatter plots to look at the relationship between continuous variables. However in case of discrete variables data points are cumulated at certain intervals. Thus the line of best fit might be biased.


title: Wilcoxon-Mann-Whitney critical values in R

text: I have noticed that when I try to find the critical values for the Mann- Whitney U using R, the values are always 1+critical value. For example, for $\alpha=.05, n = 10, m = 5$, the (two-tailed) critical value is 8, while for $\alpha=.05, n=12, m=8$, the (two-tailed) critical value is 22 (check the tables), but:               > qwilcox(.05/2,10,5)     [1] 9     > qwilcox(.05/2,12,8)     [1] 23      Of course I'm not considering something, but... anyone could explain me why?


title: How to find similarities between time series?

text: In the following example I have a data frame which consists of a time series of water temperature measurements recorded at 5 depths in the ocean where each value in `Temp` corresponds to the date in `DateTime` and the depth in `Depth`.               set.seed(1)     Temp <- rnorm(43800,sd=20)     AirT <- rnorm(8760,sd=20)     Depth <- c(1:5)          DateTime = seq(from=as.POSIXct("2010-01-01 00:00"),                 to=as.POSIXct("2010-12-31 23:00"), length=8760)     Time <- as.POSIXct(DateTime, format = "%Y-%m-%d %H:%M")          DatT <- data.frame(Temp)          ## bind together     FinalDat <- cbind(DatT, Date = rep(Time,5))          FinalDat <- cbind(FinalDat, AirT = rep(AirT, 5),                       Depth = rep(Depth, each = 8760),                       Doy = as.numeric(format(Time,format = "%j")),                       DecTime = rep(seq(1,366, length = 8760),5))      From this I would like to find the similarity between air temperature (`AirT`) and the water temperature at these different depths. I can calculate the correlation between air temperature and water temperature for the different depths which shows that the correlation decreases with depth, however, I am looking for a more sophisticated method (other than receiving a single value for each time series). Can anyone suggest an appropriate method for achieving this? Here is a thinned version of my data.frame (FinalDat):                      Temp       Date     AirT Depth     1    3.4069 2011-01-01  6.52710   0.5     2    3.2165 2011-01-08  6.77050   0.5     3    3.5630 2011-01-15  6.61410   0.5     4    3.8027 2011-01-22 -2.94440   0.5     5    3.7328 2011-01-29 -2.92900   0.5     6    4.1008 2011-02-05 12.74800   0.5     7    5.1825 2011-02-12  8.91900   0.5     8    5.0460 2011-02-19  4.43910   0.5     9    5.6156 2011-02-26  8.58090   0.5     10   5.5237 2011-03-05  1.56630   0.5     11   5.5110 2011-03-12  7.94470   0.5     12   6.5645 2011-03-19 -0.25485   0.5     13   7.8872 2011-03-26  4.11280   0.5     14   8.9096 2011-04-02 12.88700   0.5     15  10.8740 2011-04-09  6.34800   0.5     16  10.9600 2011-04-16  9.44190   0.5     17  14.6930 2011-04-23 12.37300   0.5     18  15.1330 2011-04-30 11.40200   0.5     19  14.2740 2011-05-07 14.48400   0.5     20  14.5670 2011-05-14  5.68590   0.5     21  14.0330 2011-05-21  8.36570   0.5     22  13.3870 2011-05-28 11.03900   0.5     23  16.5560 2011-06-04 13.10300   0.5     24  16.4140 2011-06-11  7.56720   0.5     25  16.2590 2011-06-18 12.02400   0.5     26  16.5120 2011-06-25 11.43100   0.5     27  17.9290 2011-07-02  8.75940   0.5     28  18.7150 2011-07-09 13.06100   0.5     29  19.4970 2011-07-16 14.96000   0.5     30  17.7090 2011-07-23 10.06100   0.5     31  19.5090 2011-07-30 11.76200   0.5     32  20.4780 2011-08-06 12.56800   0.5     33  17.8850 2011-08-13 16.93200   0.5     34  17.7440 2011-08-20 14.82700   0.5     35  18.1400 2011-08-27 12.14800   0.5     36  17.0460 2011-09-03 17.19700   0.5     37  16.5960 2011-09-10 18.45800   0.5     38  15.6430 2011-09-17 13.07600   0.5     39  14.9960 2011-09-24 13.19300   0.5     40  16.9880 2011-10-01 20.17800   0.5     41  14.5380 2011-10-08 10.92600   0.5     42  14.3770 2011-10-15  9.64180   0.5     43  12.6210 2011-10-22 11.71200   0.5     44  11.7810 2011-10-29  7.32770   0.5     45  11.8270 2011-11-05 11.15600   0.5     46  10.9790 2011-11-12 11.30000   0.5     47  10.3410 2011-11-19 10.77100   0.5     48   9.6291 2011-11-26  6.81090   0.5     49   8.5466 2011-12-03  9.11540   0.5     50   7.1007 2011-12-10  3.52720   0.5     51   6.3081 2011-12-17  0.32245   0.5     52   6.1304 2011-12-24  5.40000   0.5     53   6.1974 2011-12-31 11.93100   0.5     54   3.2471 2011-01-07  0.44508   4.0     55   3.4851 2011-01-14  8.43040   4.0     56   3.8537 2011-01-21 -2.26530   4.0     57   3.8496 2011-01-28  1.33950   4.0     58   3.8461 2011-02-04 10.18700   4.0     59   4.7892 2011-02-11  6.91330   4.0     60   5.1160 2011-02-18  5.30300   4.0     61   5.4236 2011-02-25 10.40700   4.0     62   5.3190 2011-03-04  3.63990   4.0     63   5.5318 2011-03-11  5.49690   4.0     64   6.0956 2011-03-18  2.12060   4.0     65   6.7955 2011-03-25  5.66700   4.0     66   8.5056 2011-04-01 12.13400   4.0     67   9.8450 2011-04-08  6.11440   4.0     68  10.7220 2011-04-15  9.85800   4.0     69  11.3970 2011-04-22 11.39300   4.0     70  12.6660 2011-04-29  9.54510   4.0     71  13.6370 2011-05-06 12.99600   4.0     72  14.7530 2011-05-13  8.83800   4.0     73  13.8210 2011-05-20  8.81900   4.0     74  13.5120 2011-05-27 10.80500   4.0     75  14.2630 2011-06-03 11.08700   4.0     76  15.6330 2011-06-10  7.20970   4.0     77  15.5290 2011-06-17 10.11400   4.0     78  16.6340 2011-06-24 11.29400   4.0     79  17.4820 2011-07-01 11.43900   4.0     80  17.8620 2011-07-08 13.29700   4.0     81  18.8970 2011-07-15 11.61900   4.0     82  17.4720 2011-07-22 12.73800   4.0     83  17.8640 2011-07-29 14.95600   4.0     84  17.6610 2011-08-05 16.48600   4.0     85  17.4880 2011-08-12 16.06000   4.0     86  17.5980 2011-08-19 11.38900   4.0     87  17.8150 2011-08-26 10.69200   4.0     88  16.7470 2011-09-02 14.71100   4.0     89  16.0230 2011-09-09 14.09600   4.0     90  15.5350 2011-09-16 13.03500   4.0     91  15.0200 2011-09-23 11.81800   4.0     92  15.1200 2011-09-30 14.49800   4.0     93  15.0590 2011-10-07 10.76800   4.0     94  14.5790 2011-10-14 13.91800   4.0     95  12.7810 2011-10-21  8.84760   4.0     96  11.9060 2011-10-28  8.37600   4.0     97  11.7150 2011-11-04 12.57400   4.0     98  11.1170 2011-11-11 11.26200   4.0     99  10.3560 2011-11-18 11.11100   4.0     100  9.8484 2011-11-25 12.91700   4.0     101  8.7573 2011-12-02  2.43560   4.0     102  7.2890 2011-12-09  5.87070   4.0     103  6.4547 2011-12-16  2.31850   4.0     104  6.0336 2011-12-23 11.12900   4.0     105  6.2363 2011-12-30  4.78220   4.0     106  3.4241 2011-01-06  2.92880   6.0     107  3.4075 2011-01-13  9.59030   6.0     108  3.9650 2011-01-20 -1.45990   6.0     109  3.9061 2011-01-27  3.49050   6.0     110  3.8072 2011-02-03  6.15450   6.0     111  4.7597 2011-02-10  8.87270   6.0     112  4.9920 2011-02-17  4.61980   6.0     113  5.1485 2011-02-24  7.66730   6.0     114  5.4556 2011-03-03 -0.73976   6.0     115  5.4305 2011-03-10  9.08700   6.0     116  5.9204 2011-03-17  6.57230   6.0     117  6.4393 2011-03-24  4.21580   6.0     118  6.7361 2011-03-31  9.55530   6.0     119  8.9556 2011-04-07 10.24900   6.0     120 10.7240 2011-04-14  8.15550   6.0     121 10.4810 2011-04-21  9.75940   6.0     122 10.6310 2011-04-28  6.43770   6.0     123 13.2840 2011-05-05 10.70900   6.0     124 14.1380 2011-05-12  9.11810   6.0     125 13.7850 2011-05-19  8.26810   6.0     126 13.6800 2011-05-26 11.65000   6.0     127 13.8950 2011-06-02 13.10500   6.0     128 13.8530 2011-06-09 11.09500   6.0     129 14.6830 2011-06-16 12.63000   6.0     130 15.3570 2011-06-23 12.66100   6.0     131 15.5860 2011-06-30  9.57130   6.0     132 16.2190 2011-07-07 13.84300   6.0     133 16.1140 2011-07-14 10.93200   6.0     134 17.1240 2011-07-21 13.44100   6.0     135 17.2110 2011-07-28 13.41700   6.0     136 16.9800 2011-08-04 14.81500   6.0     137 17.4610 2011-08-11 16.33900   6.0     138 17.5690 2011-08-18 10.04300   6.0     139 17.3600 2011-08-25 11.87600   6.0     140 16.6180 2011-09-01 11.71000   6.0     141 16.1330 2011-09-08 11.84600   6.0     142 15.6770 2011-09-15 12.17800   6.0     143 15.1640 2011-09-22 11.90200   6.0     144 14.8160 2011-09-29 17.07200   6.0     145 15.0100 2011-10-06 13.21700   6.0     146 14.4410 2011-10-13 12.01300   6.0     147 13.0510 2011-10-20  3.75340   6.0     148 12.0580 2011-10-27  8.76700   6.0     149 11.6430 2011-11-03 13.96900   6.0     150 11.1400 2011-11-10 13.03200   6.0     151 10.5090 2011-11-17  8.26000   6.0     152  9.8690 2011-11-24 12.02100   6.0     153  8.9167 2011-12-01  9.64050   6.0     154  7.3953 2011-12-08  4.99180   6.0     155  6.5533 2011-12-15  3.73240   6.0     156  5.9886 2011-12-22 12.69300   6.0     157  6.3341 2011-12-29  7.12550   6.0     158  3.4249 2011-01-05  5.66070   8.0     159  3.4232 2011-01-12  4.42930   8.0     160  3.9632 2011-01-19  0.04384   8.0     161  3.9203 2011-01-26  5.33010   8.0     162  3.7569 2011-02-02  4.74030   8.0     163  4.6606 2011-02-09  6.48010   8.0     164  4.9632 2011-02-16  5.92770   8.0     165  4.9742 2011-02-23  7.15490   8.0     166  5.3302 2011-03-02  1.52250   8.0     167  5.3380 2011-03-09  6.46400   8.0     168  5.7341 2011-03-16  6.49500   8.0     169  6.1663 2011-03-23  4.01320   8.0     170  6.1420 2011-03-30 10.23300   8.0     171  7.1226 2011-04-06 12.10300   8.0     172  8.4905 2011-04-13  5.66120   8.0     173  8.7548 2011-04-20  7.94430   8.0     174  8.7160 2011-04-27  9.66080   8.0     175 10.1360 2011-05-04  5.13940   8.0     176  9.6597 2011-05-11 12.00800   8.0     177 12.8610 2011-05-18 13.42900   8.0     178 13.1300 2011-05-25  7.96200   8.0     179 13.6080 2011-06-01  8.94920   8.0     180 13.4880 2011-06-08 10.58700   8.0     181 13.6130 2011-06-15 14.82200   8.0     182 13.4320 2011-06-22 13.28300   8.0     183 13.3850 2011-06-29 10.09900   8.0     184 13.1230 2011-07-06 14.30500   8.0     185 13.2270 2011-07-13 13.44700   8.0     186 13.3370 2011-07-20 13.27900   8.0     187 13.4860 2011-07-27 16.17600   8.0     188 14.1970 2011-08-03 15.40500   8.0     189 14.0740 2011-08-10 11.20800   8.0     190 14.2160 2011-08-17 10.92000   8.0     191 14.7250 2011-08-24 13.57000   8.0     192 16.2060 2011-08-31 12.35700   8.0     193 15.6070 2011-09-07 13.77500   8.0     194 15.7620 2011-09-14 13.05900   8.0     195 15.1750 2011-09-21 11.50800   8.0     196 14.7600 2011-09-28 15.44600   8.0     197 14.7480 2011-10-05 15.41900   8.0     198 14.3420 2011-10-12 12.46600   8.0     199 13.3520 2011-10-19  8.04520   8.0     200 12.1960 2011-10-26  9.85760   8.0     201 11.6640 2011-11-02  9.94480   8.0     202 11.1650 2011-11-09  9.83490   8.0     203 10.6510 2011-11-16  7.18460   8.0     204 10.0600 2011-11-23  3.36370   8.0     205  9.1110 2011-11-30  6.19500   8.0     206  7.7550 2011-12-07  5.57610   8.0     207  6.7252 2011-12-14  5.46150   8.0     208  5.9302 2011-12-21  6.02540   8.0     209  6.4421 2011-12-28  8.87860   8.0     210  3.5953 2011-01-04 -0.87434  10.0     211  3.3527 2011-01-11  7.57030  10.0     212  3.9915 2011-01-18  4.65410  10.0     213  3.8234 2011-01-25  7.24100  10.0     214  3.8257 2011-02-01  2.03130  10.0     215  4.6403 2011-02-08  4.28250  10.0     216  5.0439 2011-02-15  2.09610  10.0     217  4.9152 2011-02-22  4.52120  10.0     218  5.3290 2011-03-01  4.09410  10.0     219  5.4156 2011-03-08 -0.97393  10.0     220  5.7129 2011-03-15  5.77950  10.0     221  6.0155 2011-03-22  4.84410  10.0     222  5.9742 2011-03-29  4.10630  10.0     223  6.1641 2011-04-05 10.80000  10.0     224  6.4420 2011-04-12  7.07030  10.0     225  6.6906 2011-04-19  9.72580  10.0     226  6.8168 2011-04-26 11.22400  10.0     227  6.5933 2011-05-03  7.08340  10.0     228  7.7907 2011-05-10 10.99000  10.0     229  6.8976 2011-05-17 12.02200  10.0     230  7.1110 2011-05-24  9.37830  10.0     231 10.1990 2011-05-31  7.85870  10.0     232  9.9413 2011-06-07  9.30420  10.0     233 10.2710 2011-06-14 10.07000  10.0     234  9.6727 2011-06-21 15.00500  10.0     235 10.0280 2011-06-28 13.74500  10.0     236 10.3870 2011-07-05 17.47700  10.0     237  9.9664 2011-07-12 15.32100  10.0     238  9.8633 2011-07-19 13.27000  10.0     239 10.1500 2011-07-26 11.90700  10.0     240  9.6085 2011-08-02 17.43300  10.0     241 10.2650 2011-08-09 13.26100  10.0     242 10.2090 2011-08-16 13.34400  10.0     243  9.9657 2011-08-23 16.62600  10.0     244 10.1730 2011-08-30 13.30700  10.0     245 10.4790 2011-09-06 14.72500  10.0     246 10.5400 2011-09-13 14.77800  10.0     247 11.9570 2011-09-20 15.70200  10.0     248 12.4710 2011-09-27 11.22700  10.0     249 12.1980 2011-10-04 13.29700  10.0     250 13.8790 2011-10-11 16.68600  10.0     251 13.7680 2011-10-18  7.88400  10.0     252 12.1430 2011-10-25 13.14200  10.0     253 11.5050 2011-11-01 13.24000  10.0     254 11.1610 2011-11-08 10.43400  10.0     255 10.7390 2011-11-15  9.17300  10.0     256 10.0850 2011-11-22  9.06990  10.0     257  9.1183 2011-11-29 12.74600  10.0     258  7.9380 2011-12-06  4.43310  10.0     259  6.8017 2011-12-13  9.05250  10.0     260  5.9501 2011-12-20  7.67070  10.0     261  6.3240 2011-12-27 10.47900  10.0     262  3.6034 2011-01-03  0.44518  12.0     263  3.2966 2011-01-10  1.92110  12.0     264  3.8193 2011-01-17  5.50270  12.0     265  3.8413 2011-01-24  2.38370  12.0     266  3.8157 2011-01-31 -0.20332  12.0     267  4.3896 2011-02-07 10.31200  12.0     268  4.9550 2011-02-14  6.72160  12.0     269  4.9230 2011-02-21  4.14600  12.0     270  5.3725 2011-02-28  4.17540  12.0     271  5.4386 2011-03-07 -0.17011  12.0     272  5.6663 2011-03-14 -0.20653  12.0     273  5.6173 2011-03-21  9.40060  12.0     274  5.7879 2011-03-28  3.42890  12.0     275  5.8027 2011-04-04  5.26820  12.0     276  5.7847 2011-04-11  8.31470  12.0     277  5.9425 2011-04-18  8.85620  12.0     278  6.2409 2011-04-25  6.88140  12.0     279  5.9921 2011-05-02  9.72310  12.0     280  6.0906 2011-05-09 12.04900  12.0     281  6.0394 2011-05-16 12.44600  12.0     282  6.0922 2011-05-23  9.88280  12.0     283  6.4765 2011-05-30 12.69800  12.0     284  6.6011 2011-06-06 11.33400  12.0     285  6.6846 2011-06-13 15.29900  12.0     286  6.9803 2011-06-20  9.87070  12.0     287  6.6561 2011-06-27 19.08300  12.0     288  7.2964 2011-07-04 12.62800  12.0     289  6.5017 2011-07-11 11.53500  12.0     290  7.4560 2011-07-18 13.64600  12.0     291  7.4972 2011-07-25  9.44130  12.0     292  7.1581 2011-08-01 18.45400  12.0     293  7.4089 2011-08-08 11.82800  12.0     294  7.5630 2011-08-15 12.00700  12.0     295  7.5153 2011-08-22 12.84500  12.0     296  7.4694 2011-08-29 11.80500  12.0     297  7.5982 2011-09-05 11.41900  12.0     298  7.5733 2011-09-12 13.39300  12.0     299  7.7114 2011-09-19 13.35600  12.0     300  7.9359 2011-09-26 15.57900  12.0     301  7.7604 2011-10-03 15.83000  12.0     302  8.2146 2011-10-10 16.05000  12.0     303  8.8018 2011-10-17 11.35400  12.0     304 12.0390 2011-10-24 14.38700  12.0     305 11.3230 2011-10-31 14.43600  12.0     306 11.4140 2011-11-07  2.19910  12.0     307 10.9020 2011-11-14  9.79220  12.0     308 10.1070 2011-11-21  9.87410  12.0     309  9.2805 2011-11-28  4.62830  12.0     310  8.2556 2011-12-05  2.57670  12.0     311  6.9548 2011-12-12  4.16250  12.0     312  6.0980 2011-12-19  1.10680  12.0     313  6.1540 2011-12-26 12.25600  12.0     314  3.6293 2011-01-02  2.14550  14.0     315  3.3969 2011-01-09  4.79400  14.0     316  3.8152 2011-01-16 11.32500  14.0     317  3.9532 2011-01-23  0.58015  14.0     318  3.8262 2011-01-30 -0.96998  14.0     319  4.1869 2011-02-06 10.84800  14.0     320  4.7450 2011-02-13  6.02670  14.0     321  4.9848 2011-02-20  1.09370  14.0     322  5.2429 2011-02-27  5.34090  14.0     323  5.3107 2011-03-06  4.29060  14.0     324  5.5456 2011-03-13  6.86140  14.0     325  5.6181 2011-03-20  6.71190  14.0     326  5.6648 2011-03-27  7.01350  14.0     327  5.6326 2011-04-03  7.26700  14.0     328  5.7413 2011-04-10  9.25000  14.0     329  5.7310 2011-04-17  7.36030  14.0     330  5.6897 2011-04-24  9.62540  14.0     331  5.8051 2011-05-01  9.68150  14.0     332  5.6844 2011-05-08 14.67800  14.0     333  5.7530 2011-05-15  9.46050  14.0     334  5.7696 2011-05-22 10.65700  14.0     335  5.8255 2011-05-29 12.23400  14.0     336  5.8422 2011-06-05 12.08500  14.0     337  5.9016 2011-06-12  5.82010  14.0     338  5.8791 2011-06-19 12.82400  14.0     339  5.9407 2011-06-26 15.17000  14.0     340  5.9502 2011-07-03 11.82400  14.0     341  5.8969 2011-07-10 10.83500  14.0     342  5.9582 2011-07-17 12.44000  14.0     343  6.1421 2011-07-24 12.59700  14.0     344  6.0970 2011-07-31 15.59200  14.0     345  6.1043 2011-08-07 12.65400  14.0     346  6.1677 2011-08-14 13.57600  14.0     347  6.3980 2011-08-21 15.59100  14.0     348  6.4571 2011-08-28 12.29000  14.0     349  6.0783 2011-09-04 15.58600  14.0     350  6.5784 2011-09-11 14.65100  14.0     351  6.5329 2011-09-18  8.72590  14.0     352  6.5395 2011-09-25 14.24400  14.0     353  6.8765 2011-10-02 14.41700  14.0     354  6.5151 2011-10-09 15.62400  14.0     355  6.5328 2011-10-16 10.04000  14.0     356  6.7127 2011-10-23 11.33300  14.0     357  7.0375 2011-10-30 11.93500  14.0     358  7.1903 2011-11-06  6.52240  14.0     359  8.4702 2011-11-13  7.77260  14.0     360  9.5075 2011-11-20  5.02910  14.0     361  9.3919 2011-11-27 13.07300  14.0     362  8.3426 2011-12-04  7.68420  14.0     363  6.9961 2011-12-11  5.19810  14.0     364  6.2100 2011-12-18  0.97878  14.0     365  6.0532 2011-12-25 10.49000  14.0     366  3.5866 2011-01-01  6.52710  16.0     367  3.6081 2011-01-08  6.77050  16.0     368  3.5889 2011-01-15  6.61410  16.0     369  3.9475 2011-01-22 -2.94440  16.0     370  3.8560 2011-01-29 -2.92900  16.0     371  4.0730 2011-02-05 12.74800  16.0     372  4.6510 2011-02-12  8.91900  16.0     373  5.0101 2011-02-19  4.43910  16.0     374  4.8702 2011-02-26  8.58090  16.0     375  5.2720 2011-03-05  1.56630  16.0     376  5.5106 2011-03-12  7.94470  16.0     377  5.6273 2011-03-19 -0.25485  16.0     378  5.6357 2011-03-26  4.11280  16.0     379  5.6058 2011-04-02 12.88700  16.0     380  5.6570 2011-04-09  6.34800  16.0     381  5.6360 2011-04-16  9.44190  16.0     382  5.6088 2011-04-23 12.37300  16.0     383  5.6481 2011-04-30 11.40200  16.0     384  5.6099 2011-05-07 14.48400  16.0     385  5.6895 2011-05-14  5.68590  16.0     386  5.6614 2011-05-21  8.36570  16.0     387  5.6061 2011-05-28 11.03900  16.0     388  5.6570 2011-06-04 13.10300  16.0     389  5.6595 2011-06-11  7.56720  16.0     390  5.6647 2011-06-18 12.02400  16.0     391  5.6749 2011-06-25 11.43100  16.0     392  5.6809 2011-07-02  8.75940  16.0     393  5.6509 2011-07-09 13.06100  16.0     394  5.6534 2011-07-16 14.96000  16.0     395  5.6758 2011-07-23 10.06100  16.0     396  5.8929 2011-07-30 11.76200  16.0     397  5.8423 2011-08-06 12.56800  16.0     398  5.8163 2011-08-13 16.93200  16.0     399  5.8350 2011-08-20 14.82700  16.0     400  5.8429 2011-08-27 12.14800  16.0     401  5.8970 2011-09-03 17.19700  16.0     402  5.8458 2011-09-10 18.45800  16.0     403  5.9711 2011-09-17 13.07600  16.0     404  5.9550 2011-09-24 13.19300  16.0     405  6.1380 2011-10-01 20.17800  16.0     406  6.0951 2011-10-08 10.92600  16.0     407  5.9874 2011-10-15  9.64180  16.0     408  6.1145 2011-10-22 11.71200  16.0     409  6.1598 2011-10-29  7.32770  16.0     410  6.1276 2011-11-05 11.15600  16.0     411  6.2428 2011-11-12 11.30000  16.0     412  6.2374 2011-11-19 10.77100  16.0     413  7.8198 2011-11-26  6.81090  16.0     414  8.4979 2011-12-03  9.11540  16.0     415  7.0908 2011-12-10  3.52720  16.0     416  6.3069 2011-12-17  0.32245  16.0     417  6.1082 2011-12-24  5.40000  16.0     418  6.1375 2011-12-31 11.93100  16.0     419  3.5696 2011-01-07  0.44508  18.0     420  3.0958 2011-01-14  8.43040  18.0     421  3.8281 2011-01-21 -2.26530  18.0     422  3.7336 2011-01-28  1.33950  18.0     423  3.6466 2011-02-04 10.18700  18.0     424  4.4241 2011-02-11  6.91330  18.0     425  4.7528 2011-02-18  5.30300  18.0     426  4.6170 2011-02-25 10.40700  18.0     427  5.1216 2011-03-04  3.63990  18.0     428  5.3997 2011-03-11  5.49690  18.0     429  5.4961 2011-03-18  2.12060  18.0     430  5.4625 2011-03-25  5.66700  18.0     431  5.4426 2011-04-01 12.13400  18.0     432  5.4857 2011-04-08  6.11440  18.0     433  5.5036 2011-04-15  9.85800  18.0     434  5.5052 2011-04-22 11.39300  18.0     435  5.5101 2011-04-29  9.54510  18.0     436  5.4935 2011-05-06 12.99600  18.0     437  5.5244 2011-05-13  8.83800  18.0     438  5.5015 2011-05-20  8.81900  18.0     439  5.5059 2011-05-27 10.80500  18.0     440  5.5142 2011-06-03 11.08700  18.0     441  5.5896 2011-06-10  7.20970  18.0     442  5.5319 2011-06-17 10.11400  18.0     443  5.5354 2011-06-24 11.29400  18.0     444  5.5589 2011-07-01 11.43900  18.0     445  5.5371 2011-07-08 13.29700  18.0     446  5.5686 2011-07-15 11.61900  18.0     447  5.5670 2011-07-22 12.73800  18.0     448  5.6439 2011-07-29 14.95600  18.0     449  5.6940 2011-08-05 16.48600  18.0     450  5.6338 2011-08-12 16.06000  18.0     451  5.6322 2011-08-19 11.38900  18.0     452  5.6545 2011-08-26 10.69200  18.0     453  5.7226 2011-09-02 14.71100  18.0     454  5.7256 2011-09-09 14.09600  18.0     455  5.7185 2011-09-16 13.03500  18.0     456  5.7454 2011-09-23 11.81800  18.0     457  5.8026 2011-09-30 14.49800  18.0     458  5.8097 2011-10-07 10.76800  18.0     459  5.7557 2011-10-14 13.91800  18.0     460  5.8051 2011-10-21  8.84760  18.0     461  5.8297 2011-10-28  8.37600  18.0     462  5.7679 2011-11-04 12.57400  18.0     463  5.8322 2011-11-11 11.26200  18.0     464  5.8059 2011-11-18 11.11100  18.0     465  5.8491 2011-11-25 12.91700  18.0     466  7.4509 2011-12-02  2.43560  18.0     467  7.1933 2011-12-09  5.87070  18.0     468  6.3294 2011-12-16  2.31850  18.0     469  5.8866 2011-12-23 11.12900  18.0     470  6.1258 2011-12-30  4.78220  18.0     471  3.6297 2011-01-06  2.92880  20.0     472  3.3468 2011-01-13  9.59030  20.0     473  3.8969 2011-01-20 -1.45990  20.0     474  3.8649 2011-01-27  3.49050  20.0     475  3.7189 2011-02-03  6.15450  20.0     476  4.5689 2011-02-10  8.87270  20.0     477  4.8594 2011-02-17  4.61980  20.0     478  4.8374 2011-02-24  7.66730  20.0     479  5.2294 2011-03-03 -0.73976  20.0     480  5.3550 2011-03-10  9.08700  20.0     481  5.5199 2011-03-17  6.57230  20.0     482  5.5185 2011-03-24  4.21580  20.0     483  5.5391 2011-03-31  9.55530  20.0     484  5.5010 2011-04-07 10.24900  20.0     485  5.5107 2011-04-14  8.15550  20.0     486  5.5052 2011-04-21  9.75940  20.0     487  5.5200 2011-04-28  6.43770  20.0     488  5.5096 2011-05-05 10.70900  20.0     489  5.5201 2011-05-12  9.11810  20.0     490  5.5574 2011-05-19  8.26810  20.0     491  5.5375 2011-05-26 11.65000  20.0     492  5.5331 2011-06-02 13.10500  20.0     493  5.5330 2011-06-09 11.09500  20.0     494  5.5517 2011-06-16 12.63000  20.0     495  5.5645 2011-06-23 12.66100  20.0     496  5.5771 2011-06-30  9.57130  20.0     497  5.5920 2011-07-07 13.84300  20.0     498  5.5822 2011-07-14 10.93200  20.0     499  5.6023 2011-07-21 13.44100  20.0     500  5.6408 2011-07-28 13.41700  20.0     501  5.6308 2011-08-04 14.81500  20.0     502  5.6300 2011-08-11 16.33900  20.0     503  5.6460 2011-08-18 10.04300  20.0     504  5.6596 2011-08-25 11.87600  20.0     505  5.6706 2011-09-01 11.71000  20.0     506  5.6735 2011-09-08 11.84600  20.0     507  5.6815 2011-09-15 12.17800  20.0     508  5.6948 2011-09-22 11.90200  20.0     509  5.6955 2011-09-29 17.07200  20.0     510  5.7011 2011-10-06 13.21700  20.0     511  5.7241 2011-10-13 12.01300  20.0     512  5.7566 2011-10-20  3.75340  20.0     513  5.7440 2011-10-27  8.76700  20.0     514  5.7583 2011-11-03 13.96900  20.0     515  5.7599 2011-11-10 13.03200  20.0     516  5.7876 2011-11-17  8.26000  20.0     517  5.7958 2011-11-24 12.02100  20.0     518  5.8690 2011-12-01  9.64050  20.0     519  7.2601 2011-12-08  4.99180  20.0     520  6.4506 2011-12-15  3.73240  20.0     521  5.9143 2011-12-22 12.69300  20.0     522  6.2010 2011-12-29  7.12550  20.0     523  3.7158 2011-01-05  5.66070  22.0     524  3.3825 2011-01-12  4.42930  22.0     525  3.9394 2011-01-19  0.04384  22.0     526  3.8967 2011-01-26  5.33010  22.0     527  3.8006 2011-02-02  4.74030  22.0     528  4.6272 2011-02-09  6.48010  22.0     529  4.8906 2011-02-16  5.92770  22.0     530  4.8437 2011-02-23  7.15490  22.0     531  5.2401 2011-03-02  1.52250  22.0     532  5.2371 2011-03-09  6.46400  22.0     533  5.4725 2011-03-16  6.49500  22.0     534  5.5676 2011-03-23  4.01320  22.0     535  5.5634 2011-03-30 10.23300  22.0     536  5.5158 2011-04-06 12.10300  22.0     537  5.5711 2011-04-13  5.66120  22.0     538  5.5681 2011-04-20  7.94430  22.0     539  5.5749 2011-04-27  9.66080  22.0     540  5.5829 2011-05-04  5.13940  22.0     541  5.5900 2011-05-11 12.00800  22.0     542  5.5885 2011-05-18 13.42900  22.0     543  5.6147 2011-05-25  7.96200  22.0     544  5.6210 2011-06-01  8.94920  22.0     545  5.6379 2011-06-08 10.58700  22.0     546  5.6279 2011-06-15 14.82200  22.0     547  5.6317 2011-06-22 13.28300  22.0     548  5.6570 2011-06-29 10.09900  22.0     549  5.6468 2011-07-06 14.30500  22.0     550  5.6739 2011-07-13 13.44700  22.0     551  5.6677 2011-07-20 13.27900  22.0     552  5.6808 2011-07-27 16.17600  22.0     553  5.6797 2011-08-03 15.40500  22.0     554  5.7022 2011-08-10 11.20800  22.0     555  5.7044 2011-08-17 10.92000  22.0     556  5.7113 2011-08-24 13.57000  22.0     557  5.7130 2011-08-31 12.35700  22.0     558  5.7259 2011-09-07 13.77500  22.0     559  5.7374 2011-09-14 13.05900  22.0     560  5.7435 2011-09-21 11.50800  22.0     561  5.7419 2011-09-28 15.44600  22.0     562  5.7606 2011-10-05 15.41900  22.0     563  5.7488 2011-10-12 12.46600  22.0     564  5.7847 2011-10-19  8.04520  22.0     565  5.7825 2011-10-26  9.85760  22.0     566  5.7979 2011-11-02  9.94480  22.0     567  5.7828 2011-11-09  9.83490  22.0     568  5.8190 2011-11-16  7.18460  22.0     569  5.8245 2011-11-23  3.36370  22.0     570  5.8462 2011-11-30  6.19500  22.0     571  6.8035 2011-12-07  5.57610  22.0     572  6.6398 2011-12-14  5.46150  22.0     573  5.8833 2011-12-21  6.02540  22.0     574  6.0826 2011-12-28  8.87860  22.0     575  3.7401 2011-01-04 -0.87434  24.0     576  3.2508 2011-01-11  7.57030  24.0     577  3.8161 2011-01-18  4.65410  24.0     578  3.8529 2011-01-25  7.24100  24.0     579  3.7638 2011-02-01  2.03130  24.0     580  4.5356 2011-02-08  4.28250  24.0     581  4.6182 2011-02-15  2.09610  24.0     582  4.7871 2011-02-22  4.52120  24.0     583  5.1305 2011-03-01  4.09410  24.0     584  5.2040 2011-03-08 -0.97393  24.0     585  5.4503 2011-03-15  5.77950  24.0     586  5.4623 2011-03-22  4.84410  24.0     587  5.4744 2011-03-29  4.10630  24.0     588  5.4610 2011-04-05 10.80000  24.0     589  5.4958 2011-04-12  7.07030  24.0     590  5.4913 2011-04-19  9.72580  24.0     591  5.4967 2011-04-26 11.22400  24.0     592  5.5384 2011-05-03  7.08340  24.0     593  5.5223 2011-05-10 10.99000  24.0     594  5.5361 2011-05-17 12.02200  24.0     595  5.5549 2011-05-24  9.37830  24.0     596  5.5721 2011-05-31  7.85870  24.0     597  5.5697 2011-06-07  9.30420  24.0     598  5.5709 2011-06-14 10.07000  24.0     599  5.5695 2011-06-21 15.00500  24.0     600  5.5792 2011-06-28 13.74500  24.0     601  5.5783 2011-07-05 17.47700  24.0     602  5.5969 2011-07-12 15.32100  24.0     603  5.6055 2011-07-19 13.27000  24.0     604  5.6135 2011-07-26 11.90700  24.0     605  5.6078 2011-08-02 17.43300  24.0     606  5.6343 2011-08-09 13.26100  24.0     607  5.6386 2011-08-16 13.34400  24.0     608  5.6365 2011-08-23 16.62600  24.0     609  5.6493 2011-08-30 13.30700  24.0     610  5.6607 2011-09-06 14.72500  24.0     611  5.6561 2011-09-13 14.77800  24.0     612  5.6629 2011-09-20 15.70200  24.0     613  5.6921 2011-09-27 11.22700  24.0     614  5.6948 2011-10-04 13.29700  24.0     615  5.6826 2011-10-11 16.68600  24.0     616  5.7206 2011-10-18  7.88400  24.0     617  5.7105 2011-10-25 13.14200  24.0     618  5.6953 2011-11-01 13.24000  24.0     619  5.7325 2011-11-08 10.43400  24.0     620  5.7176 2011-11-15  9.17300  24.0     621  5.7275 2011-11-22  9.06990  24.0     622  5.7293 2011-11-29 12.74600  24.0     623  5.7733 2011-12-06  4.43310  24.0     624  6.7186 2011-12-13  9.05250  24.0     625  5.8346 2011-12-20  7.67070  24.0     626  5.9464 2011-12-27 10.47900  24.0


title: Supervised approaches vs. topic models in sentiment analysis

text: I am researching Sentiment Analysis over social media, particularly classifying online texts such as blog posts as positive, negative or neutral. Most of the approaches I have found for sentiment analysis are supervised (they need labeled data to train a classifier). However, I have also found a couple of papers that do it using joint topic-sentiment models (unsupervised) like this one. According to the results in the topic model papers, the main advantage of unsupervised approaches based on topic models is that they do no need any labeled data (apart from prior "general" sentiment information, i.e. a dictionary of positive/negative words). However, they do not reach the accuracy of a supervised approach (2% less of accuracy). Are there any other advantages/disadvantages for using topic-sentiment models for sentiment classification instead of supervised approaches? Thanks.


title: How can I test the likelihood that two Poisson data sets are drawn from the same distribution in R?

text: Suppose I want to test whether or not code patches created on weekends have a greater bug rate than those created on weekdays. (We might guess that this is so because people who are at work on weekends are more hurried etc.) If we follow the standard and model bug creation as a Poisson process, this means (I think) that we want to tell the probability that both data sets came from the same underlying distribution, i.e. $\lambda_1 = \lambda_2$. How can I do this? One way I thought of is use MLE to find the parameter which best fits one distribution, and then test the likelihood that parameter generates the second distribution. Alternatively, I could do two regressions and then use a likelihood ratio test. The problem with both methods is they are not testing whether one single $\lambda$ underlies both sets, but rather if the best-fit for one is the best-fit for the other.


title: How to determine number of independent variables in regression?

text: I have heard that having more than one independent variable is useful when predicting human behaviour since it can be influenced by a combination of several factors! Forgetting about the limitations like the number of observations, is there any way to decide how many independent variables to include in a model? I mean, suppose one has 20 observations and is doing a multiple regression with 2 independent variables. Can we criticize him just because he has 2 predictors?