title: SPSS Cross tabulation error messages when analysing a 3x2 contingency table

text: I would like some clarification on what I initially thought was relatively simple analysis. A single survey was given to 100 german show jumpers (horses) to complete What we would like to know is the following; Is there an association between LBP (answered as either yes (1) or no(2)) with the following questions?   1. and the apparent success or failure in a tournament (answered as either yes(1), no(2) or don't know(3))   2. and a change in the individual's ADL's (answered as either yes(1), no(2) or don't know(3))   3. and the aggravation of other existing pains during a tournament (answered as either yes(1), no(2) or don't know(3)) Now I believe that the best way forward is to perform a 3x2 contingency table analysis of 'association' between the nominal Dependant variable (LBP) and nominal Independent variables (Success, change in ADL & aggravation of other pain). The test you would run is Goodman & Kruskal Tau B test (the equivalent SPSS test is the Phi & Cramer's V test. The problem arises when I try and run the CrossTab test (placing LBP (DV) in the ROWS box and the IV's in the COLUMNS box) i get the following warning message _*_ >                    Warnings     >     No measures of association are computed for the crosstabulation of Low back pain * Influence on success. At least one variable in each     > 2-way table upon which measures of association are computed is a     > constant.     >          >     No measures of association are computed for the crosstabulation of Low back pain * Aggravation of pain due to competition. At least one     > variable in each 2-way table upon which measures of association are     > computed is a constant.     >          >     No measures of association are computed for the crosstabulation of Low back pain * ADL affected. At least one variable in each 2-way     > table upon which measures of association are computed is a constant.***      Now I may have missed something pretty basic here but I am not quite sure why I am getting this warning/error message when using SPSS (PASW Statistics 18)? Any help would be greatly appreciated and thank you all in advance. Andrew V


title: Practical thoughts on explanatory vs. predictive modeling

text: This question has been bugging me for some time, and I was going to write a blog post about it. However, I think it is better left for discussion in this forum. Back in April, I attended a talk at the UMD Math Department Statistics group seminar series called "To Explain or To Predict?". The talk was given by Prof. Galit Shmueli who teaches at UMD's Smith Business School. Her talk was based on research she did for a paper titled "Predictive vs. Explanatory Modeling in IS Research", and a follow up working paper titled "To Explain or To Predict?". Dr. Shmueli's argument is that the terms predictive and explanatory in a statistical modeling context have become conflated, and that statistical literature lacks a a thorough discussion of the differences. In the paper, she contrasts both and talks about their practical implications. I encourage you to read the papers. The question I'd like to pose to the practitioner community is: have you ever fallen into the trap of using one when meaning to use the other? I certainly have. How do you know which one to use? How do you define a predictive excercise vs an exlanatory/descriptive one? It would be useful if you could talk about the specific application.


title: Prediction vs. Explanation and its Effect on Statistical Methods

text: In layman's terms, what is the difference between predicting and explaining in statistics? I was looking for the differences between AIC and BIC and found this post with an answer stating: > My quick explanation is > > AIC is best for prediction as it is asymptotically equivalent to cross- > validation. BIC is best for explanation as it is allows consistent > estimation of the underlying data generating process. This makes me think that in the same vein as precision and accuracy there is some core distinction here that significantly effects when and how to use a lot of statistical procedures. I googled it and only found a variety of papers including this one, but this is far too rigorous for my question given my current knowledge. Could anyone provide an intuitive exposition of the difference and perhaps some examples of how it affects the use of statistical methods and tools?


title: Missing daily data for one month

text: I have a two years of daily data. The data for one month is missing as it was not recorded due to some problems at the company. How should I treat the data with no values for a month.


title: Combining n-grams

text: In text mining, if we've computed n-gram counts, for say $n=1\ldots4$, is there a _principled_ way to combine them, other than just concatenating the $tf-idf$ matrices for each one? (equivalent to an unweighted sum of kernels if we were to construct kernel matrices for each one). For example, google's n-gram viewer: http://books.google.com/ngrams/datasets shows that they calculated from unigrams up to 5-grams, but they don't say how they combine them.


title: If I have many positive but insignificant results, is there a statistic that lets me say with significance "at least n of these results are positive"?

text: Let's say I have run the same regression for 100 different individuals separately. My coefficients of interest are positive (and quite different from one another) but statistically insignificant in all 100 results (lets say every p-value = 0.11). Is there a way to combine these p-values in order to conclude "at least 80 of these results are positive" with greater significance than p=0.11? My online searches have only showed me how to say "at least 1 of these results is positive" through a Fisher or similar test, but I haven't been able to generalize that result. I want to test "H0 = all 100 effects are the same at 0" against "HA = at least 80 effects are positive." My goal isn't to say there is a positive coefficient on average, and nor is it to specifically measure the coefficient. My goal is to demonstrate, with significance, that at least 80 people individually faced some positive effect regardless of which 80, and regardless of the magnitude of effect felt by each individual. Thank you in advance!


title: glmnet over fitting

text: cv.glmnet is has been used by most research papers and companies. While building a similar function like cv.glmnet for glmnet.cr( a similar package that implements the lasso for continuation ratio ordinal regression) I came across this problem in cv.glmnet.               cv.glmnet first fits the model:                    glmnet.object = glmnet(x, y, weights = weights, offset = offset,                           lambda = lambda, ...)      After the glmnet object is created with the complete data, the next step goes as follows: The `lambda` from the complete model fitted is extracted               lambda = glmnet.object$lambda      Now they make sure number of folds is more than 3               if (nfolds < 3)      stop("nfolds must be bigger than 3; nfolds=10 recommended")      A list is created to store cross validated results               outlist = as.list(seq(nfolds))      A `for loop` runs to fit different data parts per the theory of cross- validation                 for (i in seq(nfolds)) {         which = foldid == i         if (is.matrix(y))            y_sub = y[!which, ]         else y_sub = y[!which]         if (is.offset)            offset_sub = as.matrix(offset)[!which, ]         else offset_sub = NULL     #using the lambdas for the complete data          outlist[[i]] = glmnet(x[!which, , drop = FALSE],                                y_sub, lambda = lambda, offset = offset_sub,                                weights = weights[!which], ...)       }     }      So what happens. After fitting the data to complete data cross-validation is done with lambdas from the complete data. Can someone tell me how this can possibly not be data over-fitting?. We in cross-validation want the model to have no information about the left out part of the data. But `cv.glmnet` cheats on this!!!!!!!!!!


title: When to use Poisson distribution?

text: I'm slightly confused with when I should use a Poisson distribution. If I have a case where:   * I have the sample mean of the number of events happening in a particular day (say it is 1.27).    * Five days are randomly selected from a sample. Can I use the Poisson process to find the probability that the sample mean now is more than 0.3?


title: Do you know any paper reporting only CI instead of p value when the analysis is from LMM in lmer4?

text: I'm finding articles that report only CI, instead of p value. All articles I found will report p value despite its issue is inconclusive. I would like to look at how they put the important information from their analysis in the tables and parenthesis so that I can follow. Cheers, Pete


title: Transfer logistic regression odds ratio (based on stratified sample) to the population odds ratio

text: In order to build prognostic model for predicting breast cancer risk, the sample of 321 women has been formed. Each woman answered the questionnaire, and the answers were used as predictors. The outcome was whether the women has breast cancer or not. We have collected 132 patients with breast cancer, and also 189 women without breast cancer answered the same questionnaire. So the incidence of breast cancer in the sample is about 0.41. But in population the incidence is about 0.0005 per year. Consequently, not only breast cancer, but each harmful factor has greater incidence in the sample than that in population. Now, I've obtained OR for smoking about 35. * Does it means, that in population OR will also be 35? If not, is there any way to transfer logistic regression odds ratio to the population odds ratio? * I can calculate relative risk by $RR\approx \frac{OR}{1-BR+BR \times OR}$. What should I use as a basis risk? If it is the risk that nonsmokers from population will have cancer, which is tiny, then $RR\approx OR$. If it is `exp(intercept)` of the model, then it is also close to 35. And if it is incidence among nonsmokers from sample, then $RR=2.7$. Is any of these options right?


title: Warnings in R after "fitdistr" used

text: I have just tried to fit a t-dist in R for some data, and did this by reading in a 21x1 csv file and converting to numeric (can show code used if you think it's important). It has produced parameters but with 27 warnings all saying `1: In log(s) : NaNs produced` etc up to 26: and one saying `In sqrt(diag(vc)) : NaNs produced`. Does this affect the validity of the parameters it has come back with? I also just noticed it has 2 values for mean, sd and df. One on top and one below in brackets? what do these mean?                       m               s               df             -0.0001337445    0.0034887033   10.0256862670       ( 0.0008246931) ( 0.0000891192) (          NaN)     There were 27 warnings (use warnings() to see them)


title: Discrete data & alternatives to PCA

text: I have a dataset of discrete (ordinal, meristic, and nominal) variables describing morphological wing characters on several closely related species of insects. What I'm looking to do is conduct some kind of analysis that would give me a visual representation of the similarity of the different species based on the morphological characteristics. The first thing that popped into my head was PCA (this is the type of visualization I'm looking to create), but after looking into it (particularly other questions such as: Can principal component analysis be applied to datasets containing a mix of continuous and categorical variables? ), it seems PCA may be inappropriate for discrete data (PCA is used in these types of studies in the literature, but always with continuous data). Ignoring the statistical background of why this data is inappropriate, PCA does give me relatively perfect results with regard to my biological question (hybrid groups of interest fall right in the middle of their paternal groups). I've also tried multiple correspondence analysis to appease the statistics (at least as far as my understanding goes), but I cannot seem to get a plot that is analogous to one I would get with PCA, where my observations (the biological individuals) are separated say by color to show the different groupings (different species, biologically speaking). It seems that this analysis is aimed at describing how the variables (here, my morphological characteristics) are related to each other, not the individual observations. And when I plot observations colored by group, I only get a single value (perhaps an average) describing the whole set of individuals. I've done the analysis in R, so perhaps I'm also just not R-savy enough to get my idea of the plot to work. Am I correct in trying this kind of analysis with my data, or am I way off track? If you could not tell, my statistical expertise is limited, so the equations happening underneath these analyses are all way over my head. I'm trying to conduct this analysis completely descriptively (I don't need to do any more downstream number crunching), and I've read that if this is the case, PCA will suffice, but want to make sure I'm not violating too many statistical assumptions. Any help or advice would be appreciated--Thank you in advance!


title: Methods & CRAN packages to predict probability using neural networks or others machine learning algorithms

text: I have a medical database containing 7 input variables (4 are binary) and a binary outcome variable (Survival: yes/no). My objective is to train and test an algorithm that predict probability of survival (not a binary output). I used nnet package. But it returned for me a binary output (using "raw" and "class" type). Which method can i use? Is there any R CRAN Packages to predict probability of survival using neural networks or others machine learning algorithms? Thank you very much


title: How can the robustness of observational studies be increased?

text: In a very general sense, what are the different ways in which a retrospective observational study can be made to be robust? Cross-validation? Well-defined exposures? etc.


title: Which type of regression fits better?

text: I am a newbie in data mining world. I have a general question. I have a data set which has 10 independent variables and one target variable named as category which has 9 values like: 1, 2, 3, 4, 5, 6, 7, 8, 9. the 10 independent variables have different kind of range of values. some of them have values between 0 - 5000, some have big range like 5,000,000 - 100,000,000 etc. Moreover there is no specific relation (linear etc.) existing between target and independent variables. I am basically trying to predict the target variable category by using all of these independent variables. Can someone suggest what should be my approach? I am very confused. Should I use regression models, decision trees or cluster analysis?


title: Adversarial noise in PCA

text: PCA is known to be quite sensitive to outlier noise (and this is why several Robust PCA techniques exists.) However, I am looking for a concrete example of sensitivity of PCA to adversarial noise that is a synthetic setting in which we can show that an adversary can severely affect the quality of eigenvectors obtained. Can anyone provide a simple example for this or, better still, provide a reference? I am particularly looking at **PCA of graph Laplacians** where a malicious adversary can add a small fraction of nodes and/or edges. Any insights will be most appreciated. Thanks!


title: Random Forest: what if I know a variable is important

text: My understanding is the the random forest picks randomly _mtry_ variables to build each decision tree. So if mtry=ncol/3 then each variables will be used on average in 1/3 of the trees. And 2/3 of the trees will not use them. But what if I know that a single variable is probably very important, would it be good to manually increase the probability that this variable be picked in each tree? Is it feasible with the randomForest package in R?


title: Predictions when multiple outcomes

text: # Background and Setting I have data of this format: on each subject the list of exposure to some subtances, some demographics and then a multiple response (whether the subject developed a disease or not and if so then what type of disease). For example:   Subject1 was 15 years old male living in city C; he was exposed to subtances A, B and C and the outcome: subject had skin cancer.   Subject2 .... outcome: subject had psoriasis...   Subject3 .... outcome: subject was healthy...etc * * * # Question I would like to be able to make predictions from these data, to estimate what disease a new subject is likely to develop given his demographics and exposure history. I tried to play around with logistic regression, but the response is not binary, it could be a multitude of outcomes/diseases (including cases of absence of disease). How can i proceed to have my predictions - if at all possible? Thanks


title: Why does $\overline{y} = \hat \beta_{0} + \hat \beta_{1} \overline{x}$ in simple linear regression?

text: Today, once again, I observed that the dependent variable was predicted to be its mean when the independent variable was set to its mean in simple linear regression.   1. Let $(\hat{y},\hat{x})$ be vectors and $(\overline{y},\overline{x})$ denote their means. Does the equation in the title hold in general for a simple linear regression of $\hat{y}$ on $\hat{x}$?   2. What are the mathematical reasons for this? EDIT: The reason I ask is because I read (Willett & Stampfer. Total energy intake: Implications for epidemiologic analyses. Am J Epidemiol 1986;124:17-27) that to adjust intake of a certain nutrient for total caloric intake, one can take the residuals from a simple linear regression with that nutrient as DV and total caloric intake as IV and add the "expected nutrient intake for a person with mean caloric intake" (so the obtained values are not centered at 0 and often negative, which is strange for something that physically should be strictly non-negative). So the question follows: Why did the authors not instead more simply say one should add the mean nutrient intake? It seems odd that these two highly distinguished researchers would not be aware of this equivalence. ![enter image description here](http://i.stack.imgur.com/0Gadq.png)


title: Inner loop overfitting in nested cross-validation

text: I have implemented nested cross validation in Matlab for a classification problem. I have 56 features and 408 cases. I am performing feature and model selection in the inner cross-validation loop, using 10 fold CV for both inner and outer loops. In the inner CV loop I employ a sequential forward feature selection procedure, nested inside a grid search to determine the optimum regularization parameters for a given regularized discriminant classifier model and selected feature set. I am finding very poor performance on the outer loop CV (when compared to a standard quadratic discriminant classifier, obtained with single loop CV), leading me to conclude I am overfitting in the inner CV loop. I have read papers by Cawley and Talbot that show that biased model selection protocols favour worse models, and furthermore, that the inner loop procedure alone produces a biased performance estimate. Does this explain the overfitting observed for the inner loop in nested CV? Are there any practical strategies to avoid this overfitting?


title: Understand how to test a Logit model on new data

text: I am having some difficulty understanding something. Let's say that I have data and construct a logit model on that data. Now, let's say I have a similar and newer data set with those same variables, and I'm wondering if the original model predicts the same on the new data. Are the values expressed in the first logit model also similar on the newer data. Right, so I was wondering how one does this in R. Is it just a matter of specifying the new data when attempting to predict. Not totally understanding the conceptual or application of this idea. Any help is appreciated.               df = data.frame(sell=c("0","1","0","0","1"), home=c("own","rent","rent","rent","own"),                     income=c(50,20,20,50,50), gender=c("M","M","F","F","F"))     df$sell = as.factor(df$sell)     df$home = as.factor(df$home)     df$income = as.factor(df$income)     df$gender = as.factor(df$gender)     str(df)     m1 = glm(factor(sell) ~ home + income + gender,              data=df, family=binomial(link="logit"))     summary(m1)          new.df = data.frame(sell=c("1","1","1","0","0"), home=c("own","own","rent","rent","own"),                         income=c(30,30,30,50,50), gender=c("M","M","F","M","F"))     m2 = glm(factor(sell) ~ home + income + gender,              data=new.df, family=binomial(link="logit"))     summary(m2)      So how do I answer the question of did the original model (m1) do a good job of predicting the values in new data (new.df). Thanks!


title: Class complexity vs mean absolute error vs mean squared error when doing regression on continuous vars

text: I am trying to do regression on the weight of fish caught given a number of continuous parameters, using Weka I'm using 1-12 parameters and a sample size of 5000-15000 A typical output is               === Evaluation on test split ===     === Summary ===          Correlation coefficient                 -0.0065     Class complexity | order 0           22812.6387 bits      8.5569 bits/instance     Class complexity | scheme            25229.8025 bits      9.4635 bits/instance     Complexity improvement     (Sf)      -2417.1638 bits     -0.9067 bits/instance     Mean absolute error                      1.9104     Root mean squared error                  2.3735     Relative absolute error                105.193  %     Root relative squared error            100.9444 %     Total Number of Instances             2666           Now, which measure is the best measure of how successful the regression is? Complexity improvement, mean absolute error or mean squared error? Complexity improvement is connected to differential entropy (I think, I asked another question on that) which seems like a good measure. But so far, I've only gotten negative complexity improvements, suggesting either that my regressions are very poor or that Weka's estimation of class complexity is (does is suffer from the sparseness of the data perhaps?) Root mean squared error and mean absolute error are two other candidates but how do I choose between them? What are the pro's and cons of each?


title: Geometric Mean of Uniform random variables convergence

text: I am doing some independent study in asymptotic statistics and point estimation and am aware that you can get from log transformations of uniform random variables (exponential) all the way up to chi-squared distributions but I came across an intriguing question and I'm not sure how to handle it. I was thinking CLT or Delta Method but am unsure of myself. The thing I am trying to sort out is: Given $X_1,\ldots X_n\overset{iid}{\sim}Uniform(0,\theta)$ and $Z_n=\left(\prod_{i=1}^nX_i\right)^{1/n}$ what is an example of a function of $Z_n$, i.e. $g(Z_n)$, so that $g(Z_n)\overset{d}{\rightarrow}\chi_{(1)}^2$, where $\overset{d}{\longrightarrow}$ denotes convergence in distribution and $\chi_{(1)}^2$ is a chi-squared random variable with 1 degree of freedom? Maybe something like $g(h(Z_n))$ where $h(Z_n)\overset{d}{\rightarrow}Normal(0,1)$ and $g(t)=t^2$? Any help is greatly appreciated.


title: comparing variation of a blended product from populations of different sizes (dilutions)

text: _Original question_ : If population 1 consists of blended piles of size X_1 and population 2 consists of blended piles of size 4*X_1. Should I expect the standard deviation of X_1 to be equal to 4*the standard deviation of X_2 ? How do I otherwise compare the standard deviation of these two populations? **Edit to reword the question (with different analogy).** **Population 1** : Huge buckets of many different coloured balls (56 possible colours), each bucket of size X1 **Population 2** : Tiny buckets of the same coloured balls (again 56 possible colours), each bucket of size X2. Because the balls are the same size / density, each bucket in this population has a much smaller quantity of balls than the buckets in population 1. The balls have the same density and are not mixed well in either of the populations. **Samples**   * I take n1 "handfuls" of coloured balls, each handful from a different bucket from population 1 and take a weighted measurement of some undefined attribute for each handful. I repeat the process for population 2 and get n2 measurements.   * I was told that because population 1 buckets are larger I should expect less variation in the n1 measurements than in the n2 measurements.    * It has been suggested that I apply normalisation factors of 1/sqrt(X1) and 1/sqrt(X2) to the standard deviations to make them comparable, however I cannot find any relevant theory or sources of information.


title: What mechanisms generate the emergence of particular structures in networks

text: I'm a new member and I have some questions about the distinct processes that explain the emergence of a particular structure in networks. I'm new in this domain and if I understand well, the actual theories are:   * If the degree distribution follows an exponential law, the mechanism of attachment is random.    * If the degree distribution follows an log-normal law, the mechanism of attachment is mixed (random and preferential).    * If the degree distribution follows a power law, the mechanism of attachment is by preferential.  If I'm wrong, can you give me some explanation and some references please?


title: The name data sparsity in different applications

text: I am recently surveying the techniques or algorithms which handle the data sparsity problems in various fields. And I find quite similar name "data sparsity" or "sparse data" is used including the recommender system, text mining, information retrieval, statistical language modeling as well as high-dimension data. However, they all carried quite different specific meaning for specific applications. For instance, the large proportion of missing values in user-item matrix is regarded as sparsity. The large proportion of zero value(rather than missing) in instance feature matrix is also called sparsity. Also, the increasing dimension of data will also leading to more sparse data. Some (not formal) definitions are given in previous works: >   1. In recommendation system, it is defined as inability to find a > sufficient quantity of good quality neighbors to aid in the prediction > process due to insufficient overlap of ratings between the active user and > his neighbors[1]. >   2. In high-dimension data, the sampling density ~$N^{1/p}$ where $N$ is > the sample size and $p$ is the data dimension can also serve as a sparsity > problem.[2] >   3. A quite formal definition of large proportion of zeros exist in feature > matrix can be found [3], I also regard this as sparse representation rather > than data sparsity. > In short, I am quite clear to understand what sparsity means in each applications. **However, I am confused whether such name has a universal explanation or definition particular mathematically.** Until now, to achieve the above goal, I attempt to come up with a sparsity measurement which can cover the above ones(But in my own view, the sparse representation which is widely used in text mining etc is different problem.) [1]:Deepa Anand and Kamal K Bharadwaj. Utilizing various sparsity measures for enhancing accuracy of collaborative recommender systems based on local and global similarities. Expert systems with applications, 38(5):5101–5109, 2011. [2]:Hastie, T., Tibshirani, R., Friedman, J., Hastie, T., Friedman, J., & Tibshirani, R. (2009). The elements of statistical learning (Vol. 2, No. 1). New York: Springer. Page 23. [3]:Duchi, J., Jordan, M., & McMahan, B. (2013). Estimation, optimization, and parallelism when data is sparse. In Advances in Neural Information Processing Systems (pp. 2832-2840).


title: SAS Proc Arima - DF, ADF, SIC, AIC, autocorrelation in residuals

text: I am using Proc Arima to produce the Dickey-Fuller and the augmented Dickey- Fuller tests. According to documentation Proc Arima uses the Dickey-Fuller method that tests the following hypothesis:               H0: psi=0     H1: psi not 0      in the three regressions:               DYt=psi*Yt-1 + ut       DYt=intercept+psi*Yt-1 + ut     DYt=intercept + time trend  + psi*Yt-1 + ut      If the series of the residuals (ut) in the above regressions are serially correlated then the regressions should be augmented with lags of the dependent variable (DYt) that is [Sigma (ai*DYt-1)] The number of lags will be decided upon: a) the values of information criteria such as AIC and SBIC (run a large number of increasingly augmented regressions starting with 1 augmentation and select the regression with minimum values of the criteria) or b) by the rule: Keep augmenting until ut of the regression are not serially correlated. Based on the above is there an option in Proc Arima that will produce in the output (together with the DF and ADF tests) a statistical test for the serial correlation of the ut (e.g. Darbin Watson) of the equations described above and the values of the Information Criteria (AIC and SIC)? Other econometrics software such as E-Views produces these statistics by default when asking for a unit root test (see http://www.hkbu.edu.hk/~billhung/econ3600/application/app01/app01.html) Until now i use a manual way to calcualte a) and b) which is very time consuming (e.g. Run the above regressions (and the augmentd cases) with proc reg asking for IC (after creating the DYt, Yt-1 and DYt-1, DYt-2... variables manually with Base SAS), save the residuals with the output statement and run autocorrelation tests (e..g Darbin-Watson).


title: Are the products of different independent random variables independent?

text: I'm confused about the independence of the product of independent random variables. Let `A` and `B` be independent of each other and `C` and `D` also be independent of each other, then I understand that `AC` and `CB` are not necessarily independent each other. However, `A`, `B`, `C`, and `D` are mutually independent of each other. In that case, are `AC` and `BD` independent of each other?


title: Statistical intuition/data sense

text: I am a second-year undergraduate student, studying Math, and I've been talking to one of my professors a good amount about the difference between mathematical ability and statistical ability. One of the key differences he brought up was "data sense" which he explained as a combination of technical ability while operating within a set of what I'll informally call "common sense restraints" i.e. not losing sight of the reality of the problem amidst a lot of theory. This is an example of what I was talking about, which appeared on Gowers's blog: > In several parts of the UK the police gathered statistics on where road > accidents took place, identified accident blackspots, put speed cameras > there, and gathered more statistics. There was a definite tendency for the > number of accidents at these blackspots to go down after the speed cameras > had been installed. Does this show conclusively that speed cameras improve > road safety? > > The same person who argued for the randomized strategy in the negotiation > game basically knew the answer to this question already. He said no, since > if you pick out the extreme cases then you would expect them to be less > extreme if you run the experiment again. I decided to move on quickly from > this question since there wasn’t a lot more to say. But I told people about > a plan I had had, which was to do a bogus telepathy experiment. I would get > them to guess the outcomes of 20 coin tosses, which I would attempt to beam > to them telepathically. I would then pick the three best performers and the > three worst, and would toss the coins again, this time asking the best ones > to help me beam the answers to the worst ones. People could see easily that > the performances would be expected to improve and that it would have nothing > to do with telepathy. What I'm asking is **how to learn more about this "data sense"** , through any publications on the subject, if they exist, or through what other users have found to be helpful in developing this skill. I'm sorry if this question needs clarifying; if so, please post your questions! Thanks.


title: When is there a representer theorem?

text: The case of regularization in a hilbert space is considered---an optimization problem with an error term and a Tikhonov-regularizer. In the article "When is there a representer theorem" it is stated on page 2 (2508), and proofs are referred to, that the representer theorem holds when the regularizer is a _nondecreasing_ function of the norm, i.e. square of the hilbert space norm. **My question:** Are there any other requirements on the problem, for example on the error term? Thank you for any help!


title: Are robust methods really any better?

text: I have two groups of subjects, A, and B, each with a size of approximately 400, and about 300 predictors. My goal is to build a prediction model for a binary response variable. My customer wants to see the result of applying the model built from A on B. (In his book, "Regression Modelling Strategies", @FrankHarrell mentions that it's better to combine the two datasets and build a model on that, since doing so adds power and precision --- see page 90, External Validation. I tend to agree with him, considering that collecting the type of data that I have is very expensive and time consuming. But I don't have a choice about what the customer wants.) Many of my predictors are highly correlated and also very skewed. I'm using logistic regression to build my predictive model. My predictors come mainly from mechanics. For example, total time the subject was under a stress higher than threshold $\alpha$ for time period $[t_1, t_2]$, for various values of $\alpha > 0$ and $0 \leq t_1 < t_2$. It is clear that just from their definitions, many of these total times are algebraically related to each other. Many of the predictors that are not algebraically related are related because of their nature: subjects that are under high stress during a time period $[t_1, t_2]$ tend to be under high stress during time period $[t_3,t_4]$, even if $[t_1,t_2] \cap [t_3,t_4] = \emptyset$. To reduce the dimension of the data, I clustered related predictors together (for instance, all the total stress times together) and used principal component analysis to represent each cluster. Since the variables were skewed, I tried two alternative paths:   * Before doing the PCA, I used a logarithmic transformation to reduce the skew in variables.    * I used Mia Hubert's ROBPCA algorithm, as implemented by the package rrcov in R, (PcaHubert), to find the robust principal components.  I am using the overall shape of the ROC curve, the shape of the precision- recall curve, and the area under the ROC curve (AUC) as my performance measures, and I'd like to get similar results for both datasets A and B. I was expecting to get a much better result from using the robust principal components, but to my surprise, the first method did better: better AUC value for both datasets A and B, more similarity between ROC curves, and more similar precision-recall curves. What is the explanation for this? And how can I use robust principal components, instead of trying to make my data look like normal? Are there any particular robust PCA methods that you'd recommend instead of ROBPCA?


title: SPSS syntax to produce two-tailed test for regression

text: I have some syntax for SPSS and it produces everything that I need perfectly, although for some reason it spits it out one tailed instead of two tailed. Is there any way of editing this syntax so that it uses a two tailed analyses?               REGRESSION     /DESCRIPTIVES MEAN STDDEV CORR SIG N     /MISSING LISTWISE     /STATISTICS COEFF OUTS CI(95) R ANOVA CHANGE ZPP     /CRITERIA=PIN(.05) POUT(.10)     /NOORIGIN     /DEPENDENT SumF1     /METHOD=ENTER Gender     /METHOD=ENTER Age     /METHOD=ENTER HighSchool UnderGrad PostGrad     /METHOD=ENTER Student BlueCollar WhiteCollar     /METHOD=ENTER Single Married DeFacto     /SCATTERPLOT=(*ZPRED ,*ZRESID)     /RESIDUALS HISTOGRAM(ZRESID) NORMPROB(ZRESID).


title: Meaning of a convergence warning in glmer

text: I am using the `glmer` function from the `lme4` package in R, and I'm using the `bobyqa` optimizer (i.e. the default in my case). I am getting a warning, and I'm curious what it means.               Warning message:     In optwrap(optimizer, devfun, start, rho$lower, control = control,  :       convergence code 3 from bobyqa: bobyqa -- a trust region step failed to reduce q      I searched "a trust region step failed to reduce q." Found some information in the minqa package, which said "Consult Powell for explanation." I did (you can too, if you want! see the references and links to them below), but I fail to understand. In fact, I failed to find anything about reducing q. M. J. D. Powell (2007) "Developments of NEWUOA for unconstrained minimization without derivatives", Cambridge University, Department of Applied Mathematics and Theoretical Physics, Numerical Analysis Group, Report NA2007/05, http://www.damtp.cam.ac.uk/user/na/NA_papers/NA2007_05.pdf. M. J. D. Powell (2009), "The BOBYQA algorithm for bound constrained optimization without derivatives", Report No. DAMTP 2009/NA06, Centre for Mathematical Sciences, University of Cambridge, UK. http://www.damtp.cam.ac.uk/user/na/NA_papers/NA2009_06.pdf. P.s. I know I can change the optimizer, and I will to see if I can get output without warnings or errors. I will also check the gradient and Hessian if I can, as per a comment/answer by Ben Bolker. I'm using `glmer` within `dredge` from `MuMIn` and I'm not sure if Ben's answer will work without some additional tinkering, but I'll work on it once my computer finishes what it's doing, anyway, I digress. ## Update As per Dr. Bolker's comment below, I began looking through the FORTRAN code (Here is the code for anyone interested in looking but not downloading it). "430" appears in the bobyqb.f portion of the code. Simply search for "430" or "reduce Q" to find the relevant code. This is my first encounter with FORTRAN code, but I think the code says that if the following conditions are met, produce the warning: `NTRITS` > 0, `VQUAD` >= 0, `IPRINT` > 0. "The integer NTRITS is set to the number "trust region" iterations that have occurred since the last "alternative" iteration." `VQUAD` appears several times, and I'm not yet clear on it's significance as its value seems to be dependent on a variety of other variables, the values of which sometimes depend on other variables.From bobyqa.f: "The value of IPRINT should be set to 0, 1, 2 or 3, which controls the amount of printing. Specifically, there is no output if IPRINT=0 and there is output only at the return if IPRINT=1.". So, it seems the task is to figure out the significance of `VQUAD` being >= 0 and, perhaps, understanding how / when `IPRINT` became > 0. I'll have to go back to the paper to have a look, but the math, or at least its symbolic expression, is a bit of a barrier for me. Unless, someone knows about the algorithm or has the desire to learn about it, I think I'll have to iteratively increase my understanding of the warning by going back and forth between the papers, the code, and the internet until I understand what it means.


title: Normalization across columns in linear regression

text: I have a data set I would like to normalize in two different ways before building the multiple linear regression model. My data set looks as follows: $$ x_{1} y_{1,1} y_{1,2}...y_{1,n-1}y_{1,n}$$ $$x_{2} y_{2,1} y_{2,2}...y_{2,n-1}y_{2,n}$$ $$... $$ $$x_{m} y_{m,1} y_{m,2}...y_{m,n-1}y_{m,n} $$ ...where each $x_{i}, y_{i,j}$ is a count, and each row $i$ represents a data set collected from a video with a variable length $k$. To make it so that all the rows have values with equivalent meanings, I normalize each row by dividing all of the counts by $k$, the length of the video. Now, instead of counts, I have counts per minutes. I also want to normalize across each column (variable) to be from 0 to 1, with the idea that I can then compare the relative importance of each variables' coefficient to other variable coefficients. I am wondering if this is even a valid normalization. Normalizing across each row is fine, but I'm having trouble figuring out whether normalizing across each column using a different normalization factor is valid. My instinct is that it isn't. If it is not valid, is there another way to achieve what I want with being able to relatively compare the importance of variables?


title: Missing-data in a rather small group of participants

text: Hi all and thank you for your valuable help all through my research work :) I am not by any means expert in statistics, I am a psychiatrist doing my phD in the area om medical education and I have come to this point of my work where I struggle with analysing my results. Here comes my question: I have a group of 32 students that filled in a knowledge test consisting of 11 multiple choice questions, before and after an educational moment as well as some months later. The last test was performed through a special platform in the internet. The first two were on site. I intent to do some graphs and t-tests to look at and compare the results before and after. My "problem" is that i have some drop outs during the last follow-up. Six of them did not fill in the last follow-up knowledge test. How would you handle a situation like this? Exclude those 6 persons from the analysis over-all? It feels like a waste, since I have a rather small sample to begin with... THANK YOU!


title: Prediction results for two response variable from random forest

text: Would any R expert explain the predic function in the randomforest package to me? I want to get two prediction results for numberic response variable A and B seperately from following regression               result <-randomforest(A + B ~ C + D + E, data = dataset)     predict(result)      I can get one prediction result. But prediction is neither A nor B. I can get the prediction results for both A and B from Mvpart and party package. Thanks in advance!


title: Resampling and Huber-White

text: I'm trying to implement the "Free Step-Down Resampling Method" described by Westfall and Young in "Resampling-Based Multiple Testing" (algorithm ~2.8 in the text). My goal is to perform a multivariate linear regression. So, I have an error estimate (from the original sample) like this (using OLS): $\epsilon = Y - (\beta_0 + \beta_1X_1 + \beta_2X_2 + \dots \beta_pX_p)$ [BTW, $X_1 \dots X_p$ are dummy variables.] In order to resample ($i$ times), I have to do: $Y_i^* = \epsilon_i^*$ where $\epsilon_i^*$ is a _with replacement_ sample from the original $\epsilon$. Here is the problem: In my dataset, responses (rows) are clustered (data come from related individuals); so, I would usually have applied Huber-White estimators to account for correlations in OLS-based linear regressions. I don't know how to proceed here... Should Huber-White estimators be used? If so, how? Apologies if my question is too simple, but I'm new in resampling methods... I guess the answer is simple, too. Suggestions are welcome.


title: Model comparison

text: Can anyone suggest the statistical tools to compare CART, conditional inference tree, and random forests? I use these three algorithm for regression analysis and want to choose the better one.


title: Ordinal regression with categorical covariates and predictors

text: I am trying to do an ordinal logistic regression (ordinal outcome variables with more than 2 categories) with nominal (more than 2 categories for some) predictor variables, as well as nominal (more than 2 categories for some) covariates/moderators. First of all, I would like to test for effect interaction to see which of my covariates/moderators are indeed moderators so that I can treat them as such. Second of all, I would like to know what's the best way to come up with the optimal model? Do I need to do dummy variables for the covariates? Can I do a stepwise for ordinal regression?


title: Gertensgabe and Werner plot

text: I was told about the Gertensgabe and Werner plot to assess the optimal threshold for a GPD model. However, I did not found neither any R package nor code to implement this. Does anybody have some code to implement it?


title: Quasi-poisson or negative binomial regression with continuous dependent variable?

text: My dependent variable is originally count data. Because of several corrections it became continuous variable (originaly my data are pellet-group counts (for estimating deer density), corrected for research plot slope, avereged across three seasons, converted to deer density (number per sq. km)). Can I still apply quasi-Poisson or negative binomial GLM? Especially nb-GLM appear to be appropriate for my data structure (histogram and mean/variance relationship are adequate) and R processes the models without any warning and gives reasonable results. Could there be any catch? Rounding the data to integer does not seem to be solution since I would lose part of information and all corrections would be meaningless. Thank you for your answers.


title: Proportion as Dependent Variable or Control for the Denominator in Regression Model

text: I am a little confused as to which model specifications to use for my question. I have number of technological failures (positive count variable) as dependent variable but I am supposed to control for total number of technologies patented (another positive count variable but higher than failures variable always). I have a panel of 100 firms for 10 years. Either I can take proportion of failures by dividing technological failures with total number of technologies patented and use this as dependent variable. I can use glm for this specifications, I guess. Or I can use negative binomial model with technological failures as dependent variable and control for total number of technologies patented in the model. Is one of these methods preferred over other? Please feel free to refer to any article or journal which throws some light on this. Thanks


title: Measures of consistency and magnitude for hourly data?

text: I have 3 months of data, collected on an hourly basis, that measures how many events (0-10k) occured in each hour. What is the most scientifically rigourous means by which I can generate statistics for this data? In the past, I have calculated in an effort to quantify how regular these events are..   * A) For a particular day of week and hour, the percent of non-zero records across the data set.   * B) For a particular day, the percent of all hours in that day with non-zero records.   * C) For a particular hour, the percent of all hours at that particular hour with non-zero records. My purpose is to be able to quantify, if I were to be present for: a particular hour of a particular day each week, all hours on a particular day of each week, or a particular hour of all days of each week... how likely is it that events will happen and what is the likely magnitude of those events given past measurements? I'm trying to identify patterns, or lack thereof, and be able to predict the likely magnitude. If someone could point me at the right area of statistics, I'd appreciate it. The Google searches I have tried have yielded little result for specifically analyzing hourly data in such a manner. Note: Cross posted, as I wasn't aware there was a stats.se when making the OP. http://math.stackexchange.com/questions/654568/measures-of-consistency-and- magnitude-for-hourly-data


title: Large Standard error with no multicollinearity

text: I have two issues. First, I have ran a OLS, Poisson, and Negative Binomial Regression. For the OLS regression, I have multiple IVs (5 control and initially 2 predictor). For one of the predictor variables the standard error is very high (over 4) in the initial regression model. The Tolerance is .963 and VIF is 1.034 which would indicate multicollinearity should not be a concern. This predictor variable has a significant correlation with one of the control/IV variables - but it is low, .127. Any idea why such a high SE? The SE is even larger for the Poisson and Negative Binomial Regressions. This brings me to my second issue - my DV is measured as a count of some phenomenon. Thus, it is ordinal. There is an upper limit to the count. I am uncertain about which analysis technique to use. I think the Negative Binomial may be the correct one, but I am unsure. Thank you for your help.


title: How to use Cox proportional hazards model in interval-censored data with time-dependent variables?

text: The question is self-explanatory. I have survival data (n=156) which, in about half of the observations, is right-censored or interval-censored. I'm am using R to do the analysis and I know I can use intcox() and survfit() to deal with interval-censored data. But, in my point of view, none of this functions can deal with time-dependent variables.   To deal with time-dependent variables without interval-censored data, I am using coxph(). How can I analyse this type of data (if it is possible)?


title: Probability extraction from random forest classifier

text: I have a random forest to perform classification. I need the real probability of the predicted class. They take a feature vector **X** and output a predicted class **C**. Additionally we can compute the confidence level **L** we have over the predicted class (my implementation calls that value PROBABILITY (I'm using OpenCV machine learning module) and it comes by returning the percentage of trees who voted for that particular class, say 80% of the trees voted for class C1 then L is 0.8)   * How can we relate the value **L** or 'Probability' to the real probability of C given the feature vector X?   * Can we simply use **L** as a reliable probability measurement?   * Are the classifier performance values (true/false positive rate, etc) somehow considered into the value **L** , or do I need to use **L** as a prior estimate to compute the real probability given the classifier performance parameters?


title: Limitations of Genetic Programming?

text: From the implementation and tests I performed I start to wonder if GP (or LGP) can be used for anything else than symbolic regressions and ant move problems (which is not really programming). Did GP has successfully been used to generate an actual program?


title: libsvm with diffrent count of Keypoints

text: I would like to use libsvm for a keypoint detection algorithm. Each keypoint has 36 features, but each sample of an Object has a diffrent count of keypoints... Is it even possible to train with different count of keypoints? my input array would look like: Object 1: (K1_F1,...K1_F36,K2_F1,...K2_F36, ... , K12_F1,...K12_F36) Object 1: (K1_F1,...K1_F36,K2_F1,...K2_F36, ... , K15_F1,...K15_F36) Object 2: (K1_F1,...K1_F36,K2_F1,...K2_F36, ... , K16_F1,...K16_F36) Object 2: (K1_F1,...K1_F36,K2_F1,...K2_F36, ... , K9_F1,...K9_F36) Please help!


title: Whether to test items for normality (and transform) when performing confirmatory factor analysis on a set of scales?

text: I received the following email: > My project involves rewording a questionnaire and seeing if the internal > reliability and CFA results improve. There are 30 items that are meant to > form 4 subscales. Each item is scored always, sometimes, never. I have heard > that I should check that each of the items is normally distributed before > putting them into CFA. I've found that all items are normally distributed > except 2 items. I was then going to transform these two items using a log > transformation, but it seems odd to me to transform individual items, > especially when the subscales themselves are normally distributed. I guess I > just want to check: **do I need each item to be normally distributed or not, > and if so, is a log transformation okay if I recode 0-1-2 to 1-2-3?** So in summary the questions are:   * **Do you need to check whether individual test items are normally distributed when performing CFA?**   * **If individual items are not normally distributed, should you transform them before performing CFA?**


title: Determining sample size when given %C.I. and margin of error in a finite population

text: I'm trying to determine n, the sample size. We have $N=580$ and we want to estimate population proportion, $p$, at a 95%-CI and a margin of error of $0.10$, then I get $$n=Npq / (N-1)D + pq$$ where $D= B^2 / 4$ to be about $n=86$. But what if instead we keep that same margin of error at $0.10$ and now only want a 90%-CI, if we use the same formula don't we get the same $n$? Am I using the formula wrong here?


title: What is the meaning of p values and t values in statistical tests?

text: After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It seems that students easily learn how to perform the calculations required by a given test but get hung up on interpreting the results. Many computerized tools report test results in terms of "p values" or "t values". How would you explain the following points to college students taking their first course in statistics:   * What does a "p-value" mean in relation to the hypothesis being tested? Are there cases when one should be looking for a high p-value or a low p-value?   * What is the relationship between a p-value and a t-value?


title: Meaning of Pr(>|z|) in GLMM output?

text: > **Possible Duplicate:**   >  What is the meaning of p values and t values in statistical tests? Someone know waht is the meaning of Pr(>|z|) in GLMM output? Thanks


title: Mann-Whitney U and KS tests in R are not producing significant results

text: I wish to compare two test data sets using the Mann-Whitney U and KS tests in R. As a start, I thought to compare two similar sets (an A/A test) expecting to see a highly significant indication of similarity. The two lists are these shown below: https://www.dropbox.com/s/9cbvzlltdohjoef/set1.csv https://www.dropbox.com/s/1p9fqdja2khrvi1/set2.csv The results for the Mann-Whitney U test (wilcox.test) were:               W = 22073, p-value = 0.1948     alternative hypothesis: true location shift is not equal to 0      The results for the KS test (ks.test) were:               D = 0.1021, p-value = 0.2058     alternative hypothesis: two-sided      Why are neither of these statistically significant (p-value being below 0.05)? Here's a display of the values for the two sets: ![stripchart set vs set2](http://i.stack.imgur.com/9VrMc.png)


title: Interpretation of the p-value of the y-intercept coefficient in a linear regression

text: I am trying to interpret one of the p-values in a one variable linear regression. Some of the answers I've seen for similar questions were not worded as thoroughly as I would have liked. My interpretation is deliberately verbose because it will aid my understanding if faults are found within it. From Microsoft Excel the linear regression formula from 90 samples of (x,y) pairs is `y = 0.514x + 0.00087` and the p-value of the first coefficient is 4e-16 (scientific notation) and for the second it is 0.0027. Would it be correct to say that the interpretation of the p-value of the 0.00087 term is: > Under the assumption that the true value of the y-intercept is zero and the > first coefficient is 0.514, random sampling of the same number of (x,y) > pairs, specifically 90, would result in a least squares best fit line with a > y-intercept at least as extreme as 0.00087, with a probability of 0.0027. If not, then what would be the correct interpretation? Not so importantly, but just to be complete, I am also inquiring if it would be more accurate and complete to put the relevant phrase as > "at least as extreme as 0.00087 in the same direction, that is, positive". **Edit** : The Excel funcion is `Tools > Data Analysis > Regression` in Office 2003 with service pack 2. Excel regression p-values on coefficients are 2 sided. **Edit** : Regarding differentiation from this question here: The most up voted answer there discusses the p-value of a hypothesis, which seems ill defined or at least not specific. I am not interested in that. I am interested in the p-value of a coefficient that is not the coefficient of an independent variable. I am being very specific.


title: Intuitive explanation of why p is not the probability of the null hypothesis

text: I've got to teach a basic course on the $\chi^2$ test soon, and I will of course insist on the fact that $p$ is not the probability of the null hypothesis to be true, and that the result of the test can't allow to accept the null. However, students of this course won't have any mathematical or statistical background, and I'd like to provide an explanation as intuitive as possible. I've read some interesting things in this ScienceNews article, with an example of a dog barking when he's hungry, but I still don't find this really clear enough. Did you already provide such an explanation, or have any idea or example that would be "sufficiently intuitive" ? Thanks a lot in advance.


title: Is my data normal?

text: Ive tested my data for normality in SPSS, and i have two sig. numbers from the Kolmogorov-Smirnov and Shaprio-Wilk.   * In the Kolmogorov-Smirnov test i have the data: **0.014, 0.013, 0.200 and 0.200**   * In the Shaprio-Wilk test i have the data: **0.029, 0.022, 0.444, 0.490** I was just wondering whether these sets of numbers mean my data is **normally distributed** & **level of measurement at the intervalscale**?


title: Difference between t value and p value

text: I am not clear about how exactly the p values and t values are used. I know how to calculate them but I am not clear about the basic concept behind them. Any help ?


title: Can I think of the level of a hypothesis test as being the probability the null hypothesis is true?

text: I am trying to understand the level of a test better and was wondering if the level of a hypothesis is essentially equal to the probability that the null hypothesis is true. I have been trying to think of a counter example but haven't been able to confirm nor deny this association. Could anyone be kind enough to shed light on my understanding? Thank you!


title: T test and P value

text: > **Possible Duplicate:**   >  What is the meaning of p values and t values in statistical tests? I currently start epidemiology class, I am very confuse of p value and t test, what do they mean? how to use them ? can anyone explain the differences between these two values in specific case. Thank you in advance


title: Serialcorrelation

text: After Regression, i ran the tests for serial correlation Durbin-Watson d-statistic( 7, 29) = .7938438 . estat bgodfrey ## Breusch-Godfrey LM test for autocorrelation               lags(p)  |          chi2               df                 Prob > chi2      \-------------+------------------------------------------------------------- ##  1 | 10.909 1 0.0010                                   H0: no serial correlation      Can i deduce a positive serialcorrelation> how should i correct for it>?


what does it mean for a variable to be statistically significant?? Could anyone give a complete answer to this question?


title: Alternate interpretation of p-value

text: I have got a p-value of .0166 on a hypothesis test on comparison of hourly wages for different countries. Would it be right to make a a conclusive remark that “We can be 98.34% confident [1-.0166] that rejecting the null hypothesis and concluding that hourly wages are different for countries is the right conclusion.”


title: Anderson-Darling critical value in R

text: I am using R and i need to do an Anderson-Darling test on my data. I am using the adk package ( http://cran.r-project.org/web/packages/adk/adk.pdf ). How do I calculate the critical value for a given significance level? Let's say I want a significance of 0.01, what should I do? In the package, there is a method called adk.pval which has to be used and the following is taken from the help: * * * Usage adk.pval(tx,m) Arguments tx   threshold for which the right tail probability of the standardized Anderson- Darling statistics T_m is to be calculated m   index of the standardized T_m statistic (see reference) Details This function first interpolates the upper T_m quantiles as given in Table 1 (see reference below) to the given value of m by fitting a quadratic in 1/sqrt(m) to the quantiles as tabulated for the upper quantile levels .25, .10, .05, .025, .01. Next a quadratic in the interpolated quantiles (for m) is fitted to the log- odds of the upper probability levels defining these quantiles and the fitted log-odds value at tx is converted back to the calculated upper probability value, i.e., the p-value. p-values outside the tabulated range [.01,.25] are obtained by linear extrapolation of the fitted quadratic. * * * I have to use this package, so help is appreciated, thank you!


title: p - values in the summary of lm model in R

text: I am having a hard time understanding the notion of p-values (looking of layman interpretation in words) that comes up in the model summary of lm models in R. I understand that a layman interpretation for t-values is that they serve like estimated z - scores and a z-score talks about how a data point varies from it's mean (in terms of standard deviation jumps). So, in case when we are looking to determine the significance of a regression variable if we assume that the variable has no affect then the t-score would be like an estimated z-score differing from zero. More the value of t-score, more it differs from our assumption that the variable has no affect. But I am stuck at the next step. How are p-values calculated? and what is the verbal interpretation for them?


title: Explaining p-value to a sophisticated layman

text: I think I understand the concept of p-value but unfortunately I still have to exert a lot of brain cycles to get my arms around it. I would like to get an explanation of the p-value that is rigorous enough for a sophisticated layman - something that would be intuitive.


title: Interpretation of p-value in hypothesis testing

text: I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values, about which I have two specific questions:   1. The p-value is technically $P({\rm observation}|H_{0})$, which, as pointed out by the paper, generally does not tell us anything about $P(H_{0}|{\rm observation})$, unless we happen to know the marginal distributions, which is rarely the case in "everyday" hypothesis testing. When we obtain a small p-value and "reject the null hypothesis," what exactly is the probabilistic statement that we are making, since we cannot say anything about $P(H_{0}|{\rm observation})$?   2. The second question relates to a particular statement from page 6(652) of the paper: > Since the p-value, or range of p-values indicated by stars, is not set a > priori, it is not the long-run probability of making a Type I error but is > typically treated as such. Can anyone help to explain what is meant by this statement?


title: t-test disagreeing with scatter graphs

text: I've been having a little trouble with some data I've been trying to analyse. I've got 2 populations of protein crystals, one subjected to a humidity stream and one not, with a view to see if there are any effects on data quality. For most of my tests there hasn't been any problem but for one set of populations the t-test (two sample, two tailed, assuming equal variance) is disagreeing with what I can see in my graphs and what seems obvious from the averages. Here are the 2 populations: Pop 1: 28.6, 22.4, 14.5, 28.3, 22.7, 22.2, 37.7, 28.7, 20.8   Average: 25.1   SD: 6.21   Pop 2: 49.3, 47.8, 38.8, 37.7, 34.5, 36.6, 32.2, 40, 27.4, 31.6   Average: 37.6   SD: 6.54 This data is represented in purple on the graph ![](http://i.stack.imgur.com/T9yMR.png) I've put the data into a quantile plot to test for normality and they appear normal and did an f-test to check for equal variance and all seems to be in order to do a t-test. But the t-value I get back from the t-test is 0.0009, much lower than the critical t-value of 2.26 for alpha(2) =0.05, suggesting they are not only the same population but almost identical. Any suggestions as to why my data is not behaving? A Mann-Whitney U-test has found them to be different populations.


title: Expectation of the Pareto distribution

text: I would like to know if my understanding of the following is correct. This has been tripping me up for a long time now. Compute $\lim_{x\rightarrow \infty}x^{1-\beta}$. This is part of a homework problem regarding the expectation of the Pareto distribution. It says everywhere that the expectation is for $\beta>1$ only, but what about $B=1$? Then doesn't the expectation change significantly because the limit at $\beta = 0$ evaluates to 1? Could someone please clarify my doubt? Thanks.


title: How bagging on CART (RPART) is different from CART with cross validation?

text: I am wondering is there any difference between the following two algorithms:   1. RPART (Recursive partitioning) in R, with cross-validation (xval = 10, default)   2. Bagging on RPART In the first case, Rpart uses 10-fold cross-validation on the original dataframe In the second, randomly select data from the original dataframe and then perform RPART. To me both the algorithms look same. Where am I wrong in my understanding?


title: Proofreading lots of documents based on small sample

text: I have gazillion documents (your tax return) that I need to check for correctness, but I don't the manpower nor the will power to read through all of it. Even if I do, I can not guarantee the quality and consistency of the proof reading process. The only thing I can do is to pick a sample collection of document to proof read and assign `accept` or `reject` to it. From that I want to determine the confidence level of certain confidence interval...I am clueless on what I should do next or I am using the right approach. I have no experience with this problem domain, perhaps someone with more QA experience can point me in the right direction. Like what question to ask ... Thank you for reading :)


title: Probability of an event occuring

text: > **Possible Duplicate:**   >  How to find percentiles of a Normal distribution? The weight of a given Africander breed is said to follow a normal distribution with mean 200 kg and standard deviation of 10. Calculate a) The probability that the weight of the Africander is less than 215 kg. b) The probability that the weight lies between 190 kg and 210 kg. c) Find the weight M kg such that the probability of X > M = 5% d) Given that there are 10 Africander whose weights are greater than M kg, Calculate the probability that there are exactly 2 Africanders whose weights are greater than M kg.


title: Summing "black-box" confidence

text: I have a table with observed and estimated values for political districts like this               BEZIRK  EDU_OBS EDU_WEI EDU_CI     101     157     5129    15      `BEZIRK` is an ID for the district, `EDU_OBS` is the number of actually observed people in a sample (lets say, inhabitants), `EDU_WEI` is the estimation based on some unknown, black-box, extrapolation, and `EDU_CI` is the 95% confidence interval in percentage of the estimated value, e.g. for district `101`, the estimation would be 5129 $\pm$ 770 with 95% confidentiality. Now, let's say I have another table with the same districts but other values.               BEZIRK  EDU_OBS EDU_WEI EDU_CI     101     180     5987    14      I would like to add the values (the estimates) of those with the first table, per district. What happens to the CI? I.e. how do I recalculate the new CI of the combined values without knowing how it was computed in the first place? Can I just average it? I don't think so.


title: What does the dashed bounds mean when plotting a contour plot with R GAM?

text: At the moment I'm trying to interpret the green and red dashed lines in a contour plot when visualizing a generalized additive model (GAM) with R. These two lines seem to be something like confidence bands, but I'm not sure how to interpret these dashed lines in a contour plot. Does anybody have experiences with contour plots using R, specifically when fitting GAM?


title: Does (matrix) t-distribution have a conjugate prior?

text: If my data Y follows matrix T-distribution, i.e. $ Y \sim T_{n,p}(\nu,0,\Sigma,\Phi)$, can I find an appropriate prior for $\Phi$ such that its posterior is in closed form? Inverse Wishart distribution is the standard prior for modelling covariance however it's not conjugate with t-distributed likelihood. I could use Laplace transformation to approximate the likelihood to be normal distributed, but if $\nu$ is too small it wouldn't be a good approximation...


title: Appropriate error bars for repeated-measurements designs

text: I conducted a t-test / ANOVA (both repeated measurements) and I want to represent the difference in the mean via a bar graph. There are several different views about the appropriate error bars for repeated-measurements: personal preference (Field, 2000), Root Mean Square Error (Estes, 1997) or rather Statistical Significance Bars (Schunn, 1999). What is the best solution?


title: procedure of statistical modeling of sample

text: I never did the statistical modeling of sample. Usually i was engaged in common statistical procedures(descriptive statistic, correlation, regression, factor analysis and so on). Now i decided find out how simulate any sample? for example, i need simulate sample of 200 people. Suppose, it's psychological research, i have any methodics and each of them has certain parameters(scales). imagine that each methodics has 5-10 scales and we have 5 methodics. Finally, we have to work with 25-50 parameters. What i must know and must do to simulate my sample? and how modeling procedure can be done using SPSS or statistical software? For a start, suppose that 100 people of sample are men, and 100 people of sample are women, and my hypothisis that mean values of each parameter are different between men and women (using Anova) and correlation between parameters are different in men and women


title: Finding average settling value after step response

text: This is crossposted from StackOverflow. Someone suggested I should post it here. Though I don't have enough karma to post images. I'm working with gathering data from a biological monitoring system. They need to know the average value of the plateaus after changes to the system are made. ![enter image description here](http://i.stack.imgur.com/9RdwF.png) This is data for about 4 minutes. There is decent lag time between the event and the steady state response. These values won't always be this level. They want me to find where the steady-state response starts and average the values during that time. My boss, who is a biologist, said there may be overshoot and random fluctuations... and that I might need to use a z-transform. Unfortunately he wasn't more specific than that. I feel decently competent as a programmer, but wasn't sure what the most efficient way would be to go about finding these values. Any algorithms, insights or approaches would be greatly appreciated. Thanks.


title: Interpretation of two-way clustering in R

text: I used the following example code from **latticeExtra** to understand two-way clustering in _R_               library(latticeExtra)     data(mtcars)     x  <- t(as.matrix(scale(mtcars)))     dd.row <- as.dendrogram(hclust(dist(x)))     row.ord <- order.dendrogram(dd.row)          dd.col <- as.dendrogram(hclust(dist(t(x))))     col.ord <- order.dendrogram(dd.col)          library(lattice)          levelplot(x[row.ord, col.ord],           aspect = "fill",           scales = list(x = list(rot = 90)),           colorkey = list(space = "left"),           legend =           list(right =                list(fun = dendrogramGrob,                     args =                     list(x = dd.col, ord = col.ord,                          side = "right",                          size = 10)),                top =                list(fun = dendrogramGrob,                     args =                     list(x = dd.row,                          side = "top",                          size = 10))))      and this is what I got ![enter image description here](http://i.stack.imgur.com/zxWV1.png) Joining of both row and column entities make sense to me but I'm confused with different color shades of heatmap. **Questions**   * Do the Joining of row variables also take into account the column variables and vice versa   * What does mean the different colors in heatmap for different row variables clustering as well as for column variables clustering. Specifically focus on **cyl** and **disp** row variables.


title: What test do I use / how do I show in the analysis that I measured everyone twice?

text: I have already asked a question on this forum about this, but as I changed my experiment a bit and I am still confused. I conducted an experiment in which 25 men and 25 women listened to an attractive conversation and picked a photo (between a woman with red and a woman with green shirt) and next, they heard a neutral dialogue and did exactly the same, picked a photo between an woman in red and a woman in green. My hypothesis is that men are much more attracted to women in red, in contrast to women. I was thinking of using repeated measures ANOVA as both men and women were 'examined' in the same experimental conditions. So, I guess that my columns are: `gender` 2 levels (0 for males and 1 for females), `attraction` 2 levels (0 for no and 1 for yes) and `color` 2 levels (0 for green-or no red and 1 for red). My problem is how do I show that each participant did this twice (i.e., there were two dialogues)? ( _Note: I am using SPSS._ )


title: What data should be given for the result of a design of experiments with response surface?

text: In my thesis I've used several experimental designs, most of them "central composite designs" with 3 variables and 20 runs (6 times central point + 8 points on the "cube corners" + 6 points on the axis). The experiments have been planned and analyzed with the software `JMP 7`, which creates quite lengthy reports and for each of my 5 _DoE_ 's I have about 20 Y-values. In the chapter "results" of my thesis, I'm showing graphs similar to those here ![enter image description here](http://i.stack.imgur.com/cBROu.png) and of course the significant interactions. I wonder which additional data I should provide in the annex for the reader. For sure, I'll add:   * "raw data" (the table with values which the response surface design was calculated with)   * the parameter estimator for each linear and quadratic effect as well as the "p"-value for the significance. I'm not an expert in statistics, so _I wonder_ which other data describing the models would be necessary or helpful to properly document my work. Can anybody help, please?


title: How to remove blank string values from my analysis outputs in SPSS?

text: I'm fairly new to the world of statistical analysis so please excuse me for asking a probable rookie question. I created a google forms survey and after I got the desired sample size I used the xls file to load my data to SPSS and experimented with different types of analysis. In my survey there are various node questions. If someone answered "Yes" he would continue with the survey, but if someone answered "No" he would be redirected to the end page (demographics). So you understand that there are many Variables (questions) with missing values (blank answers). Now when I try to do frequencies analysis for example, the outputs for some variables like the one below account correctly for the missing values and exclude them from the stats. ![valid-missing](http://i.stack.imgur.com/b79HP.png) But for most variables it does not recognize the blank values as missing and includes them in the valid answers thus affecting the stats as shown below. ![nomissing](http://i.stack.imgur.com/sFPnc.png) As you can see the first row of the table above has no value name (blank) and a frequency of 35. Those same 35 cases where excluded in the first table. Last but not least I noticed that the type of the variables that are analyzed correctly is Numeric whereas the type of the other variables is String so I guess it has something to do with that. Can you please help me fix this issue and exclude all missing values from my analysis?


title: paired t test vs linear regression vs ANCOVA

text: I have data on patients before and after an intervention, half have the treatment, half did not. A paired t-test seems a simple way to determine if there is a significant difference after treatment in either group. However there are other variables that I'd like to adjust for, such as age. Could I use linear regression, with the final measurement as the dependent variable, and the initial measurement, treatment group, and age as independent variables, and then see if the 'group' dummy variable is significant? I see in some papers that ANCOVA is used for this application, but it seems to be mathematically identical to multiple regression. Is there any point to using it? Lastly, one outcome variable is continuous, another is a scale from 1-10. Is linear regression appropriate for ordered categorical data like this, or is there a method that covers the gap between logistic and linear regression that lets me add in possible confounding variables (such as age)? Thanks in advance!


title: Can the Johnson-Neyman moderation interaction technique be used with a categorical moderator?

text: Can the Johnson-Neyman moderation interaction technique be used with a categorical moderator (3 categories)? 60%, 80% and 100%. I am guessing that it could approximate the interaction effect at hypothetical values of the moderator. Although I have three groupings of the moderator, the moderator in reality is continuous.


title: How to evaluate a curve considering outliers?

text: I have data on runners who run marathons; for each runner I have their final times on a number of races. I would like to predict how fast they are running considering outliers i.e.   * he's running faster than x hours with probability: > x | P(x) > > 1:30 | 0% > > 2:00 | 1% > > 2:30 | 30% > > 3:00 | 66% > > 4:00 | 3% So, I have a curve for each athlete for the probability he's running the marathon in x hours. How can I evaluate the curve? Background: Currently I'm doing my evaluation using RMSE i.e. I guess that he will run in 2:45h and evaluate the error. This doesn't make much sense for me, since in reality he's running usually in 2:30 hours but he has some outliers which influences the mean downwards to 2:45 hours. I would like to evaluate my curve, which encodes that I have downwards outlier with certain probabilities.


title: Normalization Factor divide or multiply

text: If I want to normalize some data using a median normalization or trimmed mean normalization, do I multiply or divide my data by the normalization factors? Does it matter?


title: Correct Stats to Run?

text: I am doing a study on body image perceptions and physical activity among adolescents. My hypothesis is adolescents who engage regularly in physical activity will have a more accurate perception of body image in relation to less active people. I have collected data on gender, age, bmi, bmi categories, physical activity levels, body image perceptions. What are the best stats tests to run??


title: Prior on parameters of a scaled-inverse-chi-squared distribution

text: I have the following problem. I'm trying to do a bayesian analysis using a hierarchical model where I have data $y_{i\\!j}$ having class-conditional normal distributions with mean $\theta_j$ and variance $\sigma^2_j$. I want to have a full hierarchy on both class means and class variances. Thus, I assume that the $\sigma^2_j$ are sampled from a population distribution which is scaled-inverse-chi-squared with degrees of freedom $\nu_0$ and scale $\sigma^2_0$. If my algebra is correct, the conditional posterior distribution of $\nu_0$ and $\sigma^2_0$ would then be \begin{align} p(\nu_o, \sigma^2_0 | \textit{else}) &\propto \prod_{j = 1}^{J} \textit{scaled-inv-} \chi^2 (\sigma^2_j | \nu_o, \sigma^2_0) \times p(\nu_o, \sigma^2_0) \\\ &= \left[\frac{ \left( \frac{\nu_o, \sigma^2_0}{2} \right)^{\nu_0 / 2} }{\Gamma(\nu_0 / 2)} \right]^J \prod_{j = 1}^{J} \left(\sigma^2_j\right)^{-(\nu_0/2 + 1)} \textit{exp}\left\lbrace -\frac{1}{2} \nu_o \sigma^2_0 \sum_{j = 1}^{J} \frac{1}{\sigma^2_j} \right\rbrace \times p(\nu_o, \sigma^2_0) \end{align} As far as I understand, an additional level in my data would be necessary to estimate $\nu_0$ and $\sigma^2_0$. Nevertheless, I have the following questions:   * Is is possible to set up a conjugate prior?   * If no conjugate prior is available, what would be a suitable prior specification? Somehow I have the impression that such an issue shouldn't be too exotic, however I couldn't find anything on the web. Thank you, Dominik


title: Feature selection for pattern mining

text: I must find frequent patterns in temporal data, using a method that was imposed to me. This tool has problems handling these data: processing is long and takes a lot of memory. So, I would like to reduce the number of attributes. Also, I want to keep working with the original attributes (no transformation or combination). In other words, I would like to perform a feature selection. By the way, the data contains both numerical and nominal (or categorical) attributes. The first thing that comes to mind is to identify groups of correlated (or, more generally: associated) attributes and only keep one to represent each group. But I suppose more advanced, and possibly more efficient, methods exist. However, the only articles I can find on the Web are related to using pattern mining to perform feature selection, which is not what I want to do. Moreover, when I look up the feature selection methods described in the literature, they are designed for classification problems. So, I was wondering if there are any works focusing on feature selection, specifically for pattern mining.


title: Precision and recall of a random classifier

text: My understanding of precision and recall tells me that there is a tradeoff between these two measures: you can improve one at the cost of the other. However, when I think of a random classifier (on a binary task) that outputs class $1$ with probability $p$, I don't observe any tradeoff. $Prec= \frac{TP}{TP+FP} = \frac{p \cdot P}{p \cdot P + p \cdot N} = \frac{P}{P + N}$ $Rec = \frac{TP}{TP+FN} = \frac{p \cdot P}{p \cdot P + (1-p) \cdot P} = p$ Hence, precision is constant (it is given by the prevalence of the positive class), and recall is a high as I want it to be. Is there anything wrong here?


title: Name of the "paradox" reported by Gelman

text: In Andrew Gelman's book "Red State, Blue State" he analyzes the fact that rich people within particular states tend to vote more Republican than poor people, but that wealthy _states_ tend to vote more Democratic than poor states. Is there a name for this paradox? It seems to me to be related to, but not identical, to the ecological paradox.


title: What are $\rho$-, $\beta$-, and $\alpha$-mixing conditions?

text: I have seen properties named $\rho$-, $\beta$-, and $\alpha$-mixing conditions in papers related to Copulas and Markov processes like this one: > In this paper, we identify conditions on $C$ that suffice for geometrically > fast mixing rates. Geometric $\beta$-mixing, equivalent to geometric > ergodicity for stationary Markov chains, is established under a rather > strong condition that excludes copulas that exhibit tail dependence or > asymmetry. Geometric $\rho$-mixing, which implies geometric $\alpha$-mixing, > is obtained under a much weaker condition. We verify this condition for > various parametric copula functions that are popular in applied work. > $\rho$-, $\beta$-, and $\alpha$-mixing conditions may be used as the basis > for a range of inequalities and limit theorems that are useful in > demonstrating the asymptotic validity of statistical methods; Beare, B., 2010, Copulas and temporal dependence, Econometrica, 78(1). I am an engineering student and I have trouble understanding these conditions in statistical texts. Can somebody explain them?


title: Clustering spherical-shaped data

text: I have a data set that consists of census data (5 attributes). Performing PCA, I found that the original data looks like a big chunk (please look at the first picture), and therefore, I decided to use spectral clustering, and the picture below is the representation of the data (the picture is obtained by plotting the highest three eigenvectors of the Laplacian). The shape looks interesting to me and it looks like these data form sort of a function in the new domain (I really would like to hear a better interpretation from experts). ![enter image description here](http://i.stack.imgur.com/xw7AZ.png) ![My data](http://i.stack.imgur.com/YycQH.png) **I now have 2 questions:** (1) what can we say about the data in the new domain? (2) What is the best clustering technique you suggest to perform the clustering Thanks


title: How do I interpret regression coefficients with autocorrelated residuals?

text: I am building a regression model of time series data in R, where my primary interest is the coefficients of the independent variables. The data exhibit strong seasonality with a trend. ![Original data](http://i.stack.imgur.com/GYxaU.png) The model looks good, with four of the six regressors significant: ![Model](http://i.stack.imgur.com/ZmoSd.png) Here are the OLS residuals: ![Residuals](http://i.stack.imgur.com/EIybo.png) I used auto.arima to select the sARIMA structure, and it returns the model (0,1,1)(1,1,0)[12].               fit.ar <- auto.arima(at.ts, xreg = xreg1, stepwise=FALSE, approximation=FALSE)     summary(fit.ar)          Series: at.ts      ARIMA(0,1,1)(1,1,0)[12]                              Coefficients:               ma1    sar1      v1       v2      v3       v4         v5           -0.7058  0.3974  0.0342  -0.0160  0.0349  -0.0042  -113.4196     s.e.   0.1298  0.2043  0.0239   0.0567  0.0555   0.0333   117.1205          sigma^2 estimated as 3.86e+10:  log likelihood=-458.13     AIC=932.26   AICc=936.05   BIC=947.06          Training set error measures:                        ME     RMSE      MAE       MPE     MAPE      MASE     Training set 7906.896 147920.3 103060.4 0.1590107 3.048322 0.1150526      My question is this: based on the parameter estimates and s.e. of the regressors, I believe that none of them are significant - is this correct, and if so, what does it imply if my goal is to interpret the relative importance of these predictors as opposed to forecasting? Any other advice relative to the process of building this model is welcome and appreciated. Here are the ACF and PACF for the residuals: ![ACF-PACF](http://i.stack.imgur.com/a3Gvy.png)               > durbinWatsonTest(mod.ols, max.lag=12)      lag Autocorrelation D-W Statistic p-value        1     0.120522674     1.6705144   0.106        2     0.212723044     1.4816530   0.024        3     0.159828108     1.5814771   0.114        4     0.031083831     1.8352377   0.744        5     0.081081308     1.6787808   0.418        6    -0.024202465     1.8587561   0.954        7    -0.008399949     1.7720761   0.944        8     0.040751905     1.6022835   0.512        9     0.129788310     1.4214391   0.178       10    -0.015442379     1.6611922   0.822       11     0.004506292     1.6133994   0.770       12     0.376037337     0.7191359   0.000      Alternative hypothesis: rho[lag] != 0


title: How do I explain correlation for my study proposal?

text: I am doing a study proposal for a project that looks at two different measures for evaluating the presence of a health condition, one clinical exam one diagnostic exam. the clinical exam will be performed by clinican A and the diagnostic exam will be performed by clinician A (newly trained) and clinician B (expert). For both the clinical exam and the diagnostic test the response will be limited to a yes/no answer on the presence or absence of the condition in question. The clinical exam will be compared to the diagnsotic exam to look for differences in ability to document the disease status in question and the performance and interpretation of the diagnsotic exam will be compared between expert and newly trained user. For the study proposal I need to explain how I am going to establish a correlation and I have no idea how to do that. Any assistance you can provide is greatly appreciated.


title: What is a nonreductive database?

text: A database like Genbank is said to be a nonreductive database. What does that mean ?


title: Simpson's paradox in exam question

text: I am preparing for my statistics final and have this question in the review package to answer: ![http://i.imgur.com/QRKGigB.png](http://i.stack.imgur.com/YYSHj.png) I understand the hypothesis portion of the question. How should I approach the first question: **Can the researcher conclude that alcohol consumption is related to gender?** Does the answer involve using Simpson's Paradox?


title: Cross validation and computing c index

text: I have seen the discussion in the page Computing c-index for an external validation of a Cox PH model with R. I am doing almost the same thing, using 3-fold sampling (with two thirds in the training set and one third in the test set) to do cross validation of cox regression model. When I was trying do `rcorr.cen`, it gave me the error message `Error in rcorr.cens(x = estimates, S = survmod) : y must have same length as x`. That is, I need to have same size for `x` and `s`. But `survmod` had been done on the training set and `x=estimates` on the test set. Do I need to use the same size for test and training to do that? Or can I continue with this set to calculate the c-index to communicate training and test data set?


title: R forecast package: How to combine fourier terms with another XREG matrix

text: I'm using R forecast package with a daily time series data, that has complex i.e. Multiple seasonality (weekly, Yearly, monthly). The fit/forecast process also needs to take into account certain day specific effects. I plan to:   * Use auto.arima function    * Set TS frequency to 7, to take care of the weekly seasonality   * Use Fourier terms for 'xreg' parameter to take care of monthly and yearly seasonality   * Use a regressor matrix for other effects e.g. Holidays.     1. Can someone help in providing a concrete example on how to use the fourier function, which can take both monthly and yearly seasonality into account?     2. Would like to see an example of how to combine a regressor matrix with the fourier result, so that it can be assigned to 'xreg' parameter or together? On both these questions, I have only found possibilities of the above mentioned by Dr.Hyndman, but concrete examples can really be useful to the community as well. Thanks.


title: How to get the median of a row in SPSS?

text: I have a data in SPSS structured like this: student_number age study_for q1 q2 q3 q4 q5 q6 q7 q8 (<\-- these are columns) There are students with numbers from 1 to 67. For each of those students I would like to get the median of q1 to q8, how could I do that in SPSS? And then I would like to get a median of that median if possible. Thank you in advance for your answer.


title: Good online intro to statistics for social science?

text: Could anyone recommend a good intro to statistics for social science, available online? I'm looking for something book-like, but online (i.e. not primarily made of video or audio - I'd like to read things, and do exercises). I need it to be online (free or downloadable) because I can't wait for a book to arrive in the mail, but if you do recommend a book, I can see if it exists in the New York Public Library. In terms of subject matter, I'm mainly looking for an introduction as to how to do stuff (e.g. making projections, testing correlations, other tasks a social scientist might do), as opposed to a treatment of how to execute the algorithms. That is, I want to know how to "do" social statistics, as opposed to how to code the software that would do it (I realise that the distinction isn't entirely tenable, but I hope it conveys my current concern). My background is in law and software engineering. I'm capable of "using" maths, but I'm an indifferent mathematician - this is a time I want to lean on the former, and not a time to remedy the latter, except as necessary. Thanks to everyone for looking at this, and apologies if it's off topic, or a recurring question. Edit: I'd appreciate even a run-down on the kinds of operations that are common in social sciences, which I can then research myself in more depth as necessary. Edit2: This search turns up a lot of material that appears relevant: https://www.google.com/search?q=statistics+techniques+for+social+science+pdf


title: Why is linear regression not able to predict the outcome of a simple deterministic sequence?

text: A colleague of mine sent me this problem apparently making the rounds on the internet:               If $3 = 18, 4 = 32, 5 = 50, 6 = 72, 7 = 98$, Then, $10 =$ ?      The answer seems to be 200.               3*6       4*8       5*10       6*12       7*14       8*16       9*18       10*20=200        When I do a linear regression in R:               data     <- data.frame(a=c(3,4,5,6,7), b=c(18,32,50,72,98))       lm1      <- lm(b~a, data=data)       new.data <- data.frame(a=c(10,20,30))       predict  <- predict(lm1, newdata=new.data, interval='prediction')        I get:                 fit      lwr      upr       1 154 127.5518 180.4482       2 354 287.0626 420.9374       3 554 444.2602 663.7398        So my linear model is predicting $10 = 154$. When I plot the data it looks linear... but obviously I assumed something that is not correct. I'm trying to learn how to best use linear models in R. What is the proper way to analyze this series? Where did I go wrong?


title: Tutorials for drawing an ROC curve

text: Are there tutorials you recommend that teach how to draw an `ROC` curve? Thanks.


title: Combining two time-series by averaging the data points

text: I would like to combine the forecasted and backcasted (viz. the predicted past values) of a time-series data set into one time-series by minimizing the Mean Squared Prediction Error. Say I have time series from 2001-2010 with a gap for the year 2007. I have been able to forecast 2007 using the 2001-2007 data (red line - called it $Y_f$) and to backcast using the 2008-2009 data (light blue line - call it $Y_b$). I would like to combine the data points of $Y_f$ and $Y_b$ into a imputed data point Y_i for each month. Ideally I would like to obtain the weight $w$ such that it minimizes the Mean Squared Prediction Error (MSPE) of $Y_i$. If this is not possible, how would I just find the average between the two time- series' data points? $$Y_i = w\cdot Y_f + (1-w)\cdot Y_b$$ As a quick example:               tt_f <- ts(1:12, start = 2007, freq = 12)     tt_b <- ts(10:21, start=2007, freq=12)          tt_f          Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec     2007   1   2   3   4   5   6   7   8   9  10  11  12     tt_b          Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec     2007  10  11  12  13  14  15  16  17  18  19  20  21      I would like to get (just showing the averaging... Ideally minimizing the MSPE)               tt_i          Jan Feb Mar Apr May Jun  Jul  Aug  Sep  Oct  Nov  Dec     2007 5.5 6.5 7.5 8.5 9.5 10.5 11.5 12.5 13.5 14.5 15.5 16.5      ![enter image description here](http://i.stack.imgur.com/VscVU.jpg)


title: Best functional form to describe a scatterplot with a z-shape appearance with noise

text: What might be the general form of the equation can be fitted to the below scatter plot? The result should look like an smooth **_Z_** ![enter image description here](http://i.stack.imgur.com/gLW6O.png)


title: Is it meaningful to test for normality with a very small sample size (e.g., n = 6)?

text: I have a sample size of 6. In such a case, does it make sense to test for normality using the Kolmogorov-Smirnov test? I used SPSS. I have a very small sample size because it takes time to get each. If it doesn't make sense, how many samples is the lowest number which makes sense to test? _Note:_ I did some experiment related to the source code. The sample is time spent for coding in a version of software **(version A)** Actually, I have another sample size of 6 which is time spent for coding in **another** version of software **(version B)** I would like to do hypothesis testing using **one-sample t-test** to test whether the time spent in the code version A is differ from the time spent in the code version B or not (This is my H1). The precondition of one-sample t-test is that the data to be tested have to be normally distributed. That is why I need to test for normality.


title: Understanding Feature Hashing

text: Wikipedia provides the following example when describing feature hashing; but the mapping does not seem consistent with the dictionary defined For example, `to` should be converted to `3` according to the dictionary, but it is encoded as `1` instead. Is there an error in the description? How does feature hashing work? > The texts: >   >   >     John likes to watch movies. Mary likes too. >     John also likes to watch football games. >   > > can be converted, using the dictionary >   >   >     {"John": 1, "likes": 2, "to": 3, "watch": 4, "movies": 5, "also": 6, >     "football": 7, "games": 8, "Mary": 9, "too": 10} >   > > to the matrix >   >   >     [[1 2 1 1 1 0 0 0 1 1] >      [1 1 1 1 0 1 1 1 0 0]] >