title: Regression fails if predictor variance is small but mean is nonzero

text: R linear regression seems to fail if my predictor variance is very small, but nonzero:               > reg = lm(V1~V2,data)     > summary(reg)          Call:     lm(formula = V1 ~ V2, data = data)          Residuals:     Min      1Q  Median      3Q     Max     -15.968  -4.898   1.627   5.218   8.468          Coefficients: (1 not defined because of singularities)                 Estimate Std. Error t value Pr(>|t|)     (Intercept)  11.7963     0.6036   19.54   <2e-16 ***     V2                NA         NA      NA       NA      Here is an excerpt from my data... (it comes out this way after boxcox...)               > print(data$V2,digits=15)      [1] -0.640196668095416 -0.640196668115515 -0.640196668075674 -0.640196668083867      [5] -0.640196668103316 -0.640196668073982 -0.640196668094188 -0.640196668081038     etc      And here is the regression working if I manually remove the mean value:               > shifted = data$V2+0.640196668     > shifted      [1] -9.541601e-11 -1.155150e-10 -7.567402e-11 -8.386702e-11 -1.033160e-10      [6] -7.398204e-11 -9.418799e-11 -8.103807e-11 -8.288503e-11 -9.411305e-11     etc          > reg = lm(data$V1~shifted)     > reg          Coefficients:     (Intercept)      shifted       6.308e+00   -5.771e+10      Can anyone tell me if I'm using `lm` wrong? Thank you...


title: Which model should I use to fit my data?

text: Here is a problem I'm struggling with for some days now on ! Here is the kind of data I have               dependent.variable = c(0,1/3,1/3,1,0,1/3,2/3,1,0,1/3,1/3,1,1/3,1/3,1/3,1,0,2/3,2/3,1)     independent.variable.1 = rep(c(10,20,50,100),times=5)     independent.variable.2 = rep(c('proust','sante','cheers','nazdrovia'),each=5)      I want to know which independent variables (and which interactions) affect the dependent variable. My dependent variable is a proportion of true over n possible things. In my example n=3, so that there is either 0/3, 1/3, 2/3 or 3/3 which are true and the rest is false. In other cases my dependent variable might be booloean or take many possible states (n=1000). I want to take into account that my dependent variable is ordinal. The independent variables might or might not be ordinal. They are everytime discrete. I also want to consider possible interactions between independent variables. **Which stats should I use ?** Thanks a lot !


title: Ideas for "lab notebook" software?

text: So this is an odd fit, though really I think it's an odd fit for any site, so I thought I'd try it here, among my data-crunching brethren. I came to epidemiology and biostatistics from biology, and still definitely have some habits from that field. One of them is keeping a lab notebook. It's useful for documenting thoughts, decisions, musings about the analysis, etc. All in one place, everything committed so I can look back on analysis later and have some clue what I did. But it would be nice to move that into the 21st century. Particularly because even though the lab notebook system is decent enough for one person and documenting decisions, it would be nice to be able to attach plots from EDA, emails from data managers discussing a particular data set, etc. I'm guessing this will involve rigging up my own system from an unholy union of many different bits, but is anyone currently using a system and have any recommendations?


title: Is LSA and topic clustering easier in European languages similar to English?

text: I was watching a talk on latent semantic analysis and the speaker described experience applying LSA and REALLY messy data. He concluded that it demonstrated the difficulty of disambiguation of meaning, in regard to context. This got me thinking about how large the English vocabulary is. Are other languages with smaller vocabularies, harder or easier to do LSA on?


title: Multivariate normal with multivariate normal mean

text: A random vector $\boldsymbol x$ is distributed as a multivariate normal $$ \boldsymbol x \sim \mathcal{N}(\boldsymbol m, \Sigma) $$ and the mean vector $\boldsymbol m$ itself is a multivariate normal $$ \boldsymbol m \sim \mathcal{N}(\boldsymbol\mu, \Delta) $$ Is there a simple way to calculate the marginal probability density $$ P(\boldsymbol x) = \int P(\boldsymbol x | \boldsymbol m) \; P(\boldsymbol m) \; \mathrm{d}\boldsymbol m $$ without calculating the integrals explicitly? It should come out that $\boldsymbol x$ is distributed as $$ \boldsymbol x \sim \mathcal{N}(\mu, \Sigma + \Delta) $$ but I'm having difficulties to prove it.


title: What is the role of the logarithm in Shannon's entropy?

text: Shannon's entropy is the negative of the sum of the probabilities of each outcome multiplied by the logarithm of probabilities for each outcome. What purpose does the logarithm serve in this equation?


title: Calculating McFadden's pseudo-$R^2$ in glmnet?

text: How do you calculate the McFadden's pseudo-$R^2$ using the `cv.glmnet` object from the glmnet package? The `deviance.glmnet` function extracts the deviances, but I need the null (intercept-only) likelihood and the likelihood for the model with lambda that minimizes the cross-validated error. Here's my code for running glmnet – what's the next step?               > fit_lasso <- glmnet(x, y, alpha=1, family="poisson", lambda.min.ratio=.001)     > fit_lasso_cv <- cv.glmnet(x, y, alpha=1, family="poisson", lambda.min.ratio=.001)


title: Interaction effects in network meta-analysis (multiple treatment comparison) in R?

text: I would like to use network meta-analysis (or multiple treatment comparison) to investigate the main-effect and interaction of two factors on a continuous outcome variable. As an example the effect of hair-color (dark/light) and gender (male/female) on a blood pressure (continuous). I have mean, variance and sample size from studies that compared dark-males to dark-females (studies 1 & 2), studies that compared dark-males to light-males (study 5), studies that compared dark-females with light-females, studies that compared dark- males with light-males and light-females (study 7). Most importantly I also have studies that compared males to females irrespective of hair-color (studies 3 and 4).               my_data <- data.frame(study = c("Study_1","Study_1", "Study_2", "Study_2", "Study_3", "Study_3", "Study_4", "Study_4", "Study_5", "Study_5", "Study_6", "Study_6", "Study_7", "Study_7", "Study_7"), treatment = c("dark_male", "dark_female","dark_male", "dark_female","male", "female","male", "female", "dark_male", "light_male", "dark_female", "light_female", "dark_male", "light_male", "light_female"), mean=c(10,12,11,14,11,21,14,25,10,12,14,20,12,14,20), std.dev=c(1,1.2,1,1.2,2,2.1,1,1.3,1.3,1,1,1,1.2,1.2,1.3), sampleSize=c(10,10,20,20,15,15,35,35,18,19,20,22,40,41,40))          my_data      I am not sure if I can include studies 3 and 4 since they include light- and dark-haired subjects. Is it possible to investigate the main effects of hair-color, gender as well as their interaction using one of the available r packages for network meta- analysis (gemtc, netmeta, metafor)? What would be a suitable way to calculate post-hoc tests?


title: Optimization of the regularized least squares with gradient descent

text: I have this regularized least square formula: $$\sum\limits_{i=1}^N (\omega^T x_i - y_i)^2 + \lambda \left\|\omega\right\|^2$$ And the gradient: $$2 \sum\limits_{i=1}^N ((\sum\limits_{j=1}^d x_{ij}\omega_j)x_{ik} - x_{ik} y_i) + 2\lambda \omega_k$$ I want to use gradient descent to find the vector w. I am using matlab. I though I would be able to make two loops and calculate the ws but my solution is very unstable and I need to use very small learning term a (a=0.000000001) in order to get not NAN solution. But I thought the values of w should head towards 0 when the lambda is large but it does not happen... My data set is a matrix X (400x64) and y (400x1). This is two class problem where contains the class labels (+1 for class 1 and -1 for class 2). Here is my matlab code:               function [ w ] = gradDecent( X, Y, a, lambda, iter )     % GRADIENT DESCENT          w = zeros(size(X(1,:)))';          for it=1:iter % For each iteration         for k = 1:size(w,1)             s = 0;            for i = 1:size(X,1)                s = s + (X(i,:)*w - Y(i))*X(i,k);            end            w(k) = w(k) - a*(2*s+2*lambda*w(k));         end     end      Am I making some stupid mistakes?


title: R: Missing levels of categorical variable in summary of linear model

text: In my formula I have several terms with the categorical variable `color` and `color`'s interaction with other variables. Color can be either white or red. When looking at the summary of my model, color is replaced by colorwhite with no mention of colorred. Can someone explain the mechanics behind R only showing one level of my categorical variable in the summary? example: `>summary(lm(foo~bar+color+color:bar)) coefficients bar colorwhite colorwhite:bar`


title: How would you analyse this problem?

text: Following the confusion (on my part) here, and as suggested, I am presenting my question in a neutral way below: **Problem:** I am looking at the relationship between "product research" (variable A) and "product purchase" (variable B). 'Product' means any consumer goods (e.g. toys, clothes etc.). A and B are composite scores (i.e. made up of responses to several questions). The compositie scores for A corresponds to a five point scale (going from no research to a lot of research), while scores for B also corresponds to a five point scale (going from decision not to buy to always buy). I need to better describe my scale labels, but they are based on a five point Likert scale. I am treating my composite scores as interval level data. I have checked the scatterplot and there is a linear relationship between A and B. A selection of my data set is as follows: **Variable A - Product Research:** 2.98 2.98 3.12 3.14 3.16 3.32 3.19 3.21 3.32 3.34 3.46 3.55 3.72 3.76 3.96 4.03 4.14 1.21 1.65 1.78 1.72 1.08 1.04 1.98 **Variable B - Product Purchase:** 3.54 2.53 2.52 3.14 3.09 3.34 2.57 3.11 2.99 2.89 3.45 3.39 3.26 3.77 3.68 3.86 4.03 1.22 1.34 1.68 1.65 1.09 1.32 2.09 **Question 1** How should I calculate if a relationship exists betwen product research and product purchase? **Question 2** How do I find out if certain demographic characteristics such as gender and age influence the relationship between product research and product purchase? **Note:** My focus is on the overall relationship between product research and product purchase and how this relationship is shaped by demographic characteritics. In other words, I am NOT interested in the difference between males and females on its own; rather on the difference (if any) between the relationship of A and B for males and females. (unsure if this is the same thing but to me, there is a subtle difference)


title: Effect size measure for doubly-repeated measures design?

text: Update: I haven't gotten any answers, so I am simplifying my question. Say I measure test scores of $100$ people in one of two conditions. Each test has $10$ items. On each item, each person can receive a score of $0-10$. Each person's test score is the average across items. My question is, is the effect size of the condition variable **better** calculated as the mean score difference between the two conditions divided by a standard deviation calculated from: 1) the $100$ test scores, or 2) the $10000$ item scores? I realize there are better effect size measures than the two presented above, and you're welcome to offer a suggestion, but what I really want to know is if it is defensible philosophically to ignore the within-person variation simply because "the" relevant measure is each person's average score across items (e.g., on the SAT, persons only see their total test score, not their performance on individual items). I would never ignore variation in most analyses, but my colleague insists that because all that matters is the total test score, the effect size measure should ignore within-person variability. * * * Original question: I have $10$ scores for each of $n$ subjects in each of $2$ time points. I want to compute an effect size for the increase in scores from the first to the second time point. My colleagues have previously reported $(M_2 – M_1) / \sigma_{pooled}$ as the standardized gain in scores, where $M_j$ was the sample mean for time point $j$, and $\sigma_{pooled}$ was a pooled estimate calculated from the **_sample standard deviations of the $n$ mean subject scores_** for each time point; that is, for each time point, they took the mean score for each subject and then computed an SD estimate based on those means and then pooled those SD estimates. This measure of a standardized score gain does not fit with my intuition because it ignores both within-subject variability and the correlation of scores across time points but within subjects. I am only concerned with the first issue at the moment. 1) In this context, should a measure of effect size take into account within- subject variance or not, and why or why not? 2) If it should, what are some sensible measures of the effect size of the increase in scores from the first to the second time point? Preferably it would be on the metric of standardized scores or standardized difference scores for ease of interpretation, but other types of effect sizes are okay too. Citations are welcome because I am not familiar with this area, but are not necessary. Finally, below is reproducible R output and code for the model I am using for the data. This might or might not be relevant to the computation of an effect size, but I am including it for completeness. It treats the effect of time (condition) as a fixed effect, and estimates two correlated subject-level random intercept variances – one for the first time point and one for the second time point.               library(lme4)     set.seed(5000)     ncondition <- 2     nsubj <- 200     ntrial <- 10     sd.resid <- 5     sd.cond.1 <- 2     sd.cond.2 <- 1     cor.1.2 <- .7     cov.1.2 <- cor.1.2 * sd.cond.1 * sd.cond.2     Sigma <- matrix(c(sd.cond.1^2, cov.1.2, cov.1.2, sd.cond.2^2), 2, 2)     Condition <- rep(rep(1:ncondition, each = ntrial), times = nsubj)     Condition.effect <- rep(rep(c(5.0,10.0), each = ntrial), times = nsubj)     Subject <- rep(1:nsubj, each = ncondition*ntrial)     Subject.effect <- rep(matrix(t(mvrnorm(n = nsubj, c(0,0), Sigma))), each = ntrials)     Trial <- rep(rep(1:ntrial, times = ncondition), times = nsubj)     Error <- rnorm(n = nsubj*ncondition*ntrials, mean = 0, sd = sd.resid)     Y <- Condition.effect + Subject.effect + Error     datn <- data.frame(Condition, Subject, Trial, Condition.effect, Subject.effect, Error, Y)     datn$Condition <- factor(datn$Condition)     datn$Subject <- factor(datn$Subject)     datn$Trial <- factor(datn$Trial)     (mod <- lmer(Y ~ -1 + Condition + (0 + Condition | Subject), data=datn))          #Linear mixed model fit by REML      #Formula: Y ~ -1 + Condition + (0 + Condition | Subject)      #   Data: datn      #   AIC   BIC logLik deviance REMLdev     # 24596 24634 -12292    24580   24584     #Random effects:     # Groups   Name       Variance Std.Dev. Corr       # Subject  Condition1  4.7814  2.1866              #          Condition2  1.1729  1.0830   0.682      # Residual            25.5615  5.0558              #Number of obs: 4000, groups: Subject, 200     #Fixed effects:     #           Estimate Std. Error t value     #Condition1   4.8272     0.1915   25.20     #Condition2  10.1398     0.1365   74.27


title: Distribution of the size of overlap between two random samples with duplicates?

text: Similar to this question, I want to know if the overlap between two samples is significant. However, my items are not unique; I have _c_ distinct colors of items, there are $m_i$ items of color $\text{i}(1<\text {i}<\text {c}) $, and $N=\sum m_i$ items total. Does this make a difference? I have figured out that drawing one sample of _n_ items (without replacement) follows the multivariate hypergeometric distribution. My question is: given two samples of sizes _n A_ and _n B_, what is the probability that they share exactly _k_ distinct items in common? And really, I'd like my samples to be _sets_ , so I have at most one item of each color in my samples, though this seems to be a harder problem.


title: Random coefficient models with fractional polynomial transformation selection method, which criterion to use?

text: I am trying to conduct a meta analysis for dose response studies where I am using fractional polynomial transformation from predefined family of powers. Now data fitting using all possible combinations will yield large number of possible models where some of them are clearly not good while there are few models that produce low heterogeneity. Now to choose the best model there is the problem of the dimensionality similarity. All models have the same number of parameters so using AIC or BIC is just a reflection for the likelihood. Any suggestions to proper model selection method in this case? Thanks


title: Compressing data using wavelets with R

text: I'm trying to compress data using wavelets and I wanted to do it with R. Could anyone tell me if there is such a tool? Thanks in advance.


title: On the  "strength" of weak learners

text: I have several closely-related questions regarding weak learners in ensemble learning (e.g. boosting).   1. This may sound dumb, but what are the benefits of using weak as opposed to strong learners? (e.g. why not boost with "strong" learning methods?)   2. Is there some sort of "optimal" strength for the weak learners (e.g. while keeping all the other ensemble parameters fixed)? Is there a "sweet spot" when it comes to their strength?    3. How can we measure the strength of a weak learner with respect to that of the resulting ensemble method. How do we quantitatively measure the marginal benefits of using an ensemble?    4. How do we compare several weak learning algorithms to decide which one to use for a given ensemble method?   5. If a given ensemble method helps weak classifiers more than strong ones, how do we tell a given classifier is already "too strong" to yield any significant gains when boosting with it?


title: Generalization of basic probability question

text: This is the motivation for my question: Suppose we have $n$ tickets in a bag, and we draw $k$ of them uniformly at random without replacement. Now, repeat the same procedure independently (same $n$ tickets, draw $k$ of them UAR without replacement). I can straightforwardly show that, if $S$ is the number of overlapping tickets between the two samples, then $\mathbb{P}\left(S=j\right)={k \choose j}\left(\prod_{i=0}^{j-1}\frac{k-i}{n-i}\right)\left(\prod_{i=0}^{k-j-1}\frac{n-k-i}{n-j-i}\right)$. Now, my question is a generalization of this--essentially, I'm curious about what happens to this overlap when we don't have the same probability of drawing each ticket. To investigate this, my approach is to assign ''weights'' to tickets by creating duplicates To make this more concrete, suppose we have $g$ groups, each with a different number of tickets $n_{i}$, where each $n_{i}$ corresponds to the ''weight'' we want to assign to a particular ticket type. Now we can draw $k$ tickets uniformly at random and without replacement, but instead of looking at the number of individual tickets that are shared between two samples, I am curious about the number of groups that are shared, which we can denote by $R$. That is, if both samples include tickets from group $i$ but the groups of all the other tickets in sample 1 are different from the groups of all the other tickets in sample 2, then we have $R=1$. How can I find $\mathbb{P}\left(R=j\right)$? To tell you the truth I'm having trouble even finding $\mathbb{P}\left(R=0\right)$. Also, if this is a well- known probability question, I would appreciate someone giving me the name of it, so that I could learn about it some more on my own. Thanks! Update: I've written some basic R code that can be used to simulate this kind of problem and check any analytical results people have:               nGroups <- 3     nPerGroup <- c(4, 2, 5)     tickets <- vector(length = sum(nPerGroup))     curPos <- 1     for(i in 1:nGroups){         tickets[curPos:(curPos + nPerGroup[i] - 1)] <- rep(i, nPerGroup[i])         curPos <- curPos + nPerGroup[i]     }               Group.Overlap <- function(tickets, k){       sample1 <- sample(tickets, size = k)       sample2 <- sample(tickets, size = k)       sharedGroups <- intersect(unique(sample1), unique(sample2))       return(length(sharedGroups))     }


title: Old empirical research that is wrong in light of modern theoretical work

text: What are examples of published empirical studies that can be proven wrong in the light of more recent theoretical results?


What is the right method for computing average of percentage values? Arithmetic average, geometric average, or something else? Here's an example of problems I'm talking about. Suppose for years 2009, 2010, 2011, the annual GDP growth rates are 1%, 2%, 3%, respectively. What is the average growth rate over these three years? I'm not talking about the domain of GDP growth in specific (it's just a made-up example). Instead, I want to know the way to compute average of percentiles in general (or at least somewhat smaller domain of average of certain growth rates). * Update #1 - another example Suppose I have percentages Pi=1 - Ni/Di (i=1, 2, ..., m). Ni is the execution time of program i after optimization, and Di is the execution time of the same program i before optimization. Then, Pi represents the rate of reduction in execution time after optimization. How can we compute the average rate of reduction in execution time for these m programs? The arithmetic average avg(Ni) is not meaningful, right? * Update #2 - change of title From "Average of percentage values" to "Average of rate of change in a set of independent change events" Change event ei: change from original value ai to new value bi, and change rate pi=f(ai,bi) What is the average change rate for a set of events {e1, e2, ..., en}?


title: How can I identify probability mass functions with known proportions of contribution in a series of histograms?

text: Suppose I have two real-valued PMFs $f(i)$ and $g(i)$ with $1\le i\le N$ and $N$ is about 1 billion. The functions $f$ and $g$ can be assumed to be continuous on the positive integers, one-sidedly so at the endpoints of course. (Meaning if i and j are close, then $f(i)$ and $f(j)$ are close.) Lets say I have several samples of more than 10 billion observations each. I tally each sample to get a series of histograms. The histograms basically give us functions $s_1(i)$, $s_2(i)$, $s_3(i)$ and so on. For each sample, I know the percentage of the observations that come from $f$ and the percentage that come from the distribution described by $g$. Can I use the samples to reconstruct the distributions $f$ and $g$. My first guess was to do linear regression. Although, I'm unsure if linear regression makes use of all the information.


title: What are the red lines in quantile regression plot (quantreg package)?

text: Using `plot.rq` in the `quantreg` package in R, we can plot the coefficient estimate distribution, and get something like this: ![enter image description here](http://i.stack.imgur.com/sXeVn.png) What are the dotted red lines? Extensive googling has revealed that the middle one is the average of all 99 estimates, but still don't know about the dotted red line.


title: Framework for reinforcement learning for games

text: [I hope this is the correct place to post - if not, please feel free to migrate the question to another site]. I would like to build a computer program that will play games effectively, and I want to use reinforcement learning techniques, namely, let two copies of the program play against each other, and use http://en.wikipedia.org/wiki/Temporal-difference_learning for learning an optimal strategy. Before I start coding - is there an existing framework for such tasks? Preferrably in Java, but not necessarily. I found some Java frameworks for reinforcement learning, but they are not geared towards games.


title: Algorithm convergence with logistic classifier

text: I am doing a college classification project, in which I am required to classify some handwritten digits. Assume that my input is a N*D where D is the number of features in each input sample and I need to classify each sample into "K" classes. I am using multiclass logisitic regression and in my training phase I am trying to find my weighted matrix(D*K) with gradient descent algorithm. I have read so many articles about this algorithm and in many of them it is suggested that we need to choose the weighted matrix at which our algorithm converges. I am assuming that here the word "converges" means where all the entries in our gradient matrix will become '0'. I have tried around 1000 iterations of this algorithm and with each iteration my error is going down but my gradient matrix is not becoming 'zero'. So can someone tell me why I am getting this behaviour or getting a perfect '0' for all of the elements in the gradient matrix is impossible?


title: Distance metric for survival functions?

text: Similar to this question, I would like a good metric for the distance between multiple ecdfs. However, in this case I have survival functions that are right-censored. **Question** Is there a metric that is commonly used with survival functions?   * If not, is there any metric that weights divergence at earlier values of $t$ more than later values? Early failures are much less desirable than later ones, so it seems natural to devise a metric with such a property.      * If not, does it make sense to devise one, and does anyone have any pointers for doing this? I have some intuition that cdfs might naturally penalize early failures, but I don't know enough about the subject to get from there to a metric.


title: How to predict match outcome in team game based on participating players

text: I have a typical scenario and wanted to know how to approach the problem to solve it. Let say the data set contains $N$ records and each record contains $10$ names and an output variable. The first $5$ names belong to Team $A$ and the next $5$ names belong to Team $B$. The output variable tells whether Team $A$ wins or Team $B$ wins. Which algorithm are used to solve this type of data set? I am trying to use Naive Bayes classification. Is there any other method better suited for these problems? My attempt: Calculated $P(A_{win}),P(B_{win}),P(N_i),P(N_i|A_{win}),P(N_i|B_{win})$   $$P(A_{win} | N_1N_2\cdots N_5) = \frac{P(N_1N_2\cdots N_5 | A_{win}) * P(A_{win})}{P(N_1N_2\cdots N_5)}$$ $$ = \frac{P(N_1 | A_{win}) * \cdots *P(N_5 | A_{win}) * P(A_{win})}{P(N_1)*\cdots *P(N_5)}$$ $$P(B_{win} | N_6N_7\cdots N_{10}) = \frac{P(N_6N_7\cdots N_{10} | B_{win}) * P(B_{win})}{P(N_6N_7\cdots N_{10})}$$ $$ = \frac{P(N_6 | B_{win}) * \cdots *P(N_{10} | B_{win}) * P(B_{win})}{P(N_{6})*\cdots *P(N_{10})}$$ If $P(A_{win} | N_1N_2\cdots N_5) > P(B_{win} | N_6N_7\cdots N_{10})$ Then $A$ wins Not sure whether this approach is right or wrong!


title: How to set the significance level in a hypothesis test correctly

text: I have difficulties understanding probabilities. Let's imagine I have a test based on time series regressions and my null hypothesis is that I will pass my next exam in statistics. The test is performed on a small sample size and I choose a significance level of 5%. Unfortunately, I have to reject the null. :( Luckily, I was educated by the best statisticians (in my city) and was told to be very conservative about doing such tests. Therefore, I decrease my alpha level to 1%, which in turn, decreases my rejection region and, hooray, now I fail to reject the null! I am going to pass! :) Now, I also have a silly friend. Although she is a bright statistician, she is not very socially able. In her ingenuity she developed a statistical test herself based on cross sectional analysis that uses a large sample size. However, her null hypothesis is that I (=me) fail my next exam. Her initial choice of a a 5% significance level rejects the null hypothesis. Yet, because she is a nuisance to me, she lowers her significance level to 1% too with the consequence that she cannot reject the null either. Now, I am going to fail my exams! So, according to my test at 1% significance level, I am going to pass. (I used a small sample size) According to her test at 1% significance level, I am going to fail. (She used a large sample size) I have four emerging questions: 1) Who is right or wrong here? 2) If my friend and me had exactly the same sample size, can we be more sure if I am going to pass or fail my exam? 3) Does knowing the exact p levels help me to know more about who is right? 4) Given that we don't know anything about the structure of this two asymmetric tests, is it better to be against one own desirable outcome. In other words, if I want to pass, should I pick the test with the null hypothesis of failing - having my conservative education here in mind? 5) Is there a sure way to determine which test is better? 6) Am I going to fail my exam?


title: When two variables are correlated, how can I draw two variables for SEM using AMOS?

text: I would like to use AMOS for SEM. According to the literature, either (1) worry is correlated with anxiety, or (2) worry is characterized by anxiety. Would you help me as to how I can draw one variable or the other, or can I not draw (1) or (2)?


title: What's the correct distribution for page reading time?

text: I want to simulate a user reading a book. The amount of time spent on a page will be random, but subject to a particular distribution, I just don't know which one. A normal distribution has the problem that it can produce negative values, which is clearly not right. What's the correct distribution for this kind of random variable?


title: Picking the right statistical test

text: Definite stats newbie, and I'm looking for a little bit of direction on which statistical test to use. Here's a quick summary: ![enter image description here](http://i.stack.imgur.com/OkPxc.jpg) I'm trying to determine if the cases are on more than one narcotic at a time with a .05 significance. Is this an easy as doing an OR (1304*3404)/(6095*133)?


title: Differentiating strength of relationships between categorical and continuous variables in SEM?

text: I'm currently planning an SEM study designed to investigate attitudes toward energy technologies. My past research has suggested that the persuasive effect of positive/negative information is minimal, and that differences in attitudes are better explained by stable personality traits. So my model looks something like this: ![SEM model](http://i.imgur.com/Heya4En.png) (strong and weak relations are my predictions) I've only dabbled with SEM in the past, so I was hoping to clear up this design. I intend to take measures of all the personality traits, and manipulate persuasion by giving participants one of two versions of a persuasive stimulus. So they'd either read something that was pro-fracking or anti-fracking (for instance), for various different technologies, as well as completing a bunch of personality scales, and then be tested on an output measure for their overall attitudes. This means the personality and attitude measures are continuous, and the persuasion (blue) measures are binary/categorical. My question: ### Is it possible to meaningfully compare the strength of relationships in an SEM model between two continuous variables, and a continuous and a binary variable? If not, is there an appropriate method for doing so? I'm aware there is a limitation in this design, in that the difference between persuasion levels is to some degree arbitrary and thus a true quantitative comparison between causes is impossible, but comparing the size of effects (or something similar) would be good enough for my purposes. Please let me know if this is a dumb question!


title: Calculating Hedge's g and its variance from t-statistic

text: I believe there is a way to calculate Hedge's g and its variance from a t-statistic. I have found the formulas to calculate Hedge's g from one-sample t, and from independent groups t. one sample: g = t/√N independent groups: g = t√((n1+n2)/n1n2) I can't find a formula anywhere for calculating the variance of g in this situation, or for calculating g from a paired-sample t-test. Any ideas?


title: Type of regression method to use

text: I have the marker data of 32 patients for eight different markers. WHat needs to be done here is to predict the type of marker which is suitable for the disease control. I used Disease control as the dependent variable and markers as the independent variable and did a multiple regression analysis. And from the results of which I did not see a linear trend in the data and the below figure gives an idea of how the data looks like and it is very clear that a model cannot be fit using linear methods. I would like to focus on the non-linear trends in the data. Will doing a support vector regression help in this case? Iam using R to do the regression. ![enter image description here](http://i.stack.imgur.com/GoAla.png)


title: Meaning of Weibull distribution for Mean Time to Repair?

text: I just am wondering, if the distribution is mean time to _failure_ (or maybe mean time between failure), and it is Weibull distributed, then its shape parameter $\beta$ takes on importance. If $\beta < 1$, there is high infant mortality. If $\beta = 1$, there is random failure. If $\beta > 1$, there is wear-out failure. But what if the variable under consideration is mean time to _repair_? Are there any categorical classifications according to parameters, like in MTTF/MTBF? Is there special significance or is the MTTF/MTBF just a special case?


title: Interpreting regression coefficents when the variables are in proportions

text: I have the following cross-sectional regression. per capita car ownership = 0.025*per capita college degrees + 0.012*availability of underground(dummy variable) - 0.287 I am having difficulty with interpreting the coefficients of both per capita car ownership and per capita college degrees. Since they are percentages, should I treat them like log-log level and interpret like 1% increase in per capita college degrees is likely to increase per capita car ownership by 0.025%?


title: How to calculate median absolute deviation in a cross validation scenario?

text: In a cross validation scenario "leave one out" I want to measure how the estimated continuous variable fits with the observed variable. I learned from Wikipedia that the Median absolute deviation (MAD) could by used. My question: How is the MAD to be calculated in this scenario? I have two ideas. The first is inspired from the definition of MAD where the center is the median of all deviations:   1. Set the deviation $D_i = E_i - O_i$ for each corresponding estimated and observed outcome    2. Calculate the median $M = \underset{i}{arg\;median}({D_i})$   3. Set $MAD = \underset{i}{arg\;median}({|M-D_i|})$ The second one seems more appropriated to me in the context of cross validation:   1. Set the deviation $D_i = E_i - O_i$ for each corresponding estimated and observed outcome    2. Set $MAD = \underset{i}{arg\;median}{(|D_i|})$. The last is the median of the absolute deviation between estimated and observed value, literally. However, it seems not to confirm to the definition given by Wikipedia. What is the best solution in my scenario? BTW: In the wikipedia article about the Mean absolute error I found an interesting comment about this issue: > The mean absolute error is a common measure of forecast error in time series > analysis, where the terms "mean absolute deviation" is sometimes used in > confusion with the more standard definition of mean absolute deviation. The > same confusion exists more generally.


title: Find the expected value of $(\bar X_n-p)^3$

text: > Let $X_1,X_2,\dots, X_n$ be a random sample from a Bernoulli distribution > with parameter $p$. Let $\bar X_n$ be the sample average given by $\bar > X_n=\frac{1}{n} (X_1+X_2+\dots+ X_n)$). Find the expected value of $(\bar > X_n-p)^3$. Trial: I know $\sum_{i=1}^n \sim \text{Bin}(n,p)$ but then how I calculate the given calculation. Please help.


title: mapping data with a spike to a heat map

text: I have the following data set that I need to display on the heat map:               [ 30, 15, 66, 7, 9999, 78, 42, 132 ]      So if I map the values to the color scale using a linear function I only see the spike while the rest of the values I cannot tell apart because the difference between them is insignificant in the scale of that spike. I tried a log function but it didn't get a good looking picture either. I don't care about accurate correspondence between the value and the color intensity. Is there a way to map the values to the color scale so that the difference between all values was visible? Can I mitigate that spike somehow? I am asking about established approaches or algorithms addressing problems like this.


title: Estimating the number of birds in a specific area

text: I want to count the birds in an area. What's the proper statistical method for it? What are the good references for this topic?


title: Reject if p-value = alpha?

text: I am creating the bootstrap to create a p-value. Suppose my statistic from the sample is A. Suppose for simplicity, that I do 10 bootstraps and I get A_1, A_2, ..., A_10. Suppose that A_i > A two times and my alpha is .2. Should I reject or fail to reject? I know that 10 bootstraps is nowhere near enough and that in practice I could just do more bootstraps until the p-value goes above or below. But I'm curious regarding this specific case. Also how does this answer change if A_i = A for some i. Is that the same as A_i > A or does it count as A_i < A ? Again I don't see this case considered because A is often taken to be continuous. But what about non-continuous test statistic? I've found this answer: Is a p-value of 0.04993 enough to reject null hypothesis? which suggests I hould reject if there is equality. But the particular case of equality is not explained. I have searched in elementary text books but most just say that p-value is continuous so equality won't happen. I've also found this answer: Do you reject the null hypothesis when $p < \alpha$ or $p \leq \alpha$?  but I don't have the book they mention and it is a little too theoretical for me. I guess my question is partly whether S_1 is {a| a > A} or {a | a >= A}? I think the above questions have the material to answer my question but I don't know how to implement the knowledge with the specific case of a bootstrap. My attempt at answer is that if A_i > A two times, we are saying Pr(A_i > A) = .2. We want probability of rejecting, given the null is true, to be .2. So we should reject. But as you can see I don't really understand things.


title: ANOVA + follow-up using repeated measures design with a modification

text: I want to compare performance of strategies on a group of tasks. I determine performance by running simulations, so I can independently run every strategy on each task (repeated measures):               strategy_A  task_1  result_sA_t1     strategy_A  task_2  result_sA_t2     strategy_A  task_3  result_sA_t3               strategy_B  task_1  result_sB_t1     strategy_B  task_2  result_sB_t2     strategy_B  task_3  result_sB_t3               strategy_C  task_1  result_sC_t1     strategy_C  task_2  result_sC_t2     strategy_C  task_3  result_sC_t3      There is a random component in the strategies, so I decided to modify the standard design and run every simulation 7 times:               strategy_A  task_1  result_sA_t1_1     strategy_A  task_1  result_sA_t1_2     .     .     .     strategy_A  task_1  result_sA_t1_7          strategy_A  task_2  result_sA_t2_1     .     .     .     strategy_A  task_2  result_sA_t2_7          strategy_A  task_3  result_sA_t3_1     .     .     .     strategy_A  task_3  result_sA_t3_7          (the same for strategy_B and C)      I have found a number of blog posts (post_1, post_2) describing how to perform ANOVA and follow-up Tukey's test in R, but they all assumed that there is only one datapoint for a strategy-task pair, while I have 7 of them. How can use R's ANOVA and Tukey's test to analyse my data?


title: Why might the mean of a bootstrapped distribution not equal the original summary statistic?

text: **Background:** I have n samples and their average. The mean of this empirical bootstrapped distribution seems quite different form the average of my original sample. My original average for the n samples is about the 62nd percentile on the empirical distribution. It seems to me that the means should approach one another as I take more bootstrapped samples. ### Questions **In general, why might the mean of a bootstrapped distribution not approach the original summary statistic as the sample size gets very large?**


title: Given a table defining the joint probabilities, how do I calculate certain parameters of the marginal distributions?

text: The number of items sold on any one day in the traditional shop is a random variable X and the corresponding number of items sold via the Internet is a random variable Y. The joint distribution of X and Y is described by the probability function p(x,y) below:                                Y     X    0       1       2       3     0    0.01    0.07    0.05    0.00     1    0.09    0.15    0.15    0.06     2    0.00    0.13    0.10    0.19      Find Px(x) the marginal distribution of X Find E(X), the expected number of items sold per day in the traditional shop Find V(X) the variance of the number of items sold per day in the shop Given E(Y) = 1.70 and V(Y) = 0.91 find the Co-variance between X and Y? Owing to reduced costs, items sold on the Internet attract a higher profit margin. The daily profit from sales of the item, P, is determined by the equation: P = 6X + 8Y - 5 Find E(P), the expected daily profit from the sales and also the standard deviation of P."


title: Why is logistic regression in R creating multiple coefficients for one variable?

text: Thank you so much for your input to my question. My variable is HPtsLag and is continuous (although integers, and <=18 - perhaps this could be the issue?). Anyway, here is my code and output. Hopefully this explains it better.               > mylog = glm(cbind(HW,NotHW) ~ HPtsLag, data = g, family = binomial)     > summary (mylog)      Call:               > glm(formula = cbind(HW, NotHW) ~ HPtsLag, family = binomial, data = g)          Deviance Residuals:          Min       1Q   Median       3Q      Max       -1.5898  -1.0787  -0.9431   1.2142   1.6259            Coefficients:                 Estimate Std. Error z value Pr(>|z|)         (Intercept)  -1.0116     0.3371  -3.001  0.00269 **      HPtsLag1      0.3998     0.4023   0.994  0.32031         HPtsLag10     0.9255     0.3527   2.624  0.00869 **      HPtsLag11     0.9694     0.3613   2.683  0.00729 **      HPtsLag12     1.0722     0.3659   2.931  0.00338 **      HPtsLag13     1.5224     0.3652   4.168 3.07e-05 ***     HPtsLag14     1.7693     0.4032   4.388 1.15e-05 ***     HPtsLag15     1.2658     0.4004   3.161  0.00157 **      HPtsLag16     1.8348     0.3919   4.681 2.85e-06 ***     HPtsLag18     1.9432     0.4700   4.135 3.55e-05 ***     HPtsLag2      0.4318     0.3883   1.112  0.26615         HPtsLag3      0.7807     0.3700   2.110  0.03488 *       HPtsLag4      0.5873     0.3565   1.648  0.09944 .       HPtsLag5      0.6881     0.3544   1.942  0.05219 .       HPtsLag6      0.6862     0.3545   1.936  0.05292 .       HPtsLag7      0.6956     0.3508   1.983  0.04739 *       HPtsLag8      0.7749     0.3532   2.194  0.02824 *       HPtsLag9      0.9588     0.3541   2.708  0.00677 **      ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1           (Dispersion parameter for binomial family taken to be 1)              Null deviance: 5432.7  on 3929  degrees of freedom     Residual deviance: 5314.6  on 3912  degrees of freedom     AIC: 5350.6          Number of Fisher Scoring iterations: 4      **Why is this happening?**


title: What is the best way to use a discrete variable with a lot of classes in linear regression?

text: I have a data set with a few discrete variables having a lot of classes(>100,000). This is too large to encode using a one-hot method. What is the best way to deal with such variables? Should I be using some other model? I have yes/no responses to a set of questions and am trying to predict features based on these. But each user answers only a very small random subset of questions, so in order to make a prediction I need to determine the probability of answering yes/no for the rest of the questions.


title: What is "gaussianity", and how do you perform gaussianity testing in macroeconomic time-series?

text: I have several questions regarding the usual gaussianity ( _broad normality_ ) assumptions in econometrics. Though people often check for normality (with apparently weak tests), I've seen just one example of "gaussianity" testing. > 1.Is the "finite variance" assumption the same as a gaussianity assumption? > Ie., is it the same to assume that the variable follows any distribution in > the family of Elliptically Symmetric distributions? 2.If gaussianity is a > requirement, how do you test for it? The one example I know of doing some informal testing for gaussianity is NN Taleb's _Errors, Robustness, and The Fourth Quadrant_. Here's SSRN PDF version http://maint.ssrn.com/?abstract_id=1343042 and here the technical part of the paper in a friendly html format http://www.fooledbyrandomness.com/EDGE/index.html. It that paper Taleb uses some measurements in a big number of financial time series trying to show that gaussianity is implausible. He does so by:   * Trying to see if the data is consistent with the central limit theorem seeing if kurtosis converges when increasing data aggregation.   * Trying to see if the data is consistent with either a gaussian decay in the conditional expectations of the variable or a power law (I think, page 8 of the PDF).   * Trying to see a non gaussian incidence of rare events. Finally, questions #3 and #4: > 3.Taleb performs his tests on data of a much higher frequency than what is > commonly found in macroeconomics. What would the appropriate tests be for > monthly data? 4.Are there other necessary conditions for the usual > econometric models besides finiteness of variance not typically tested for? Please bear in mind that I'm a graduate student taking a fairly basic time series course, this is just for intellectual curiosity. Thanks!


title: Use of further analysis on factors formed by principal component analysis in regression

text: I want to find out the relationship between 6 independent variable (4 categorical, 2 continuous) and 6 dependent variables (5 likert scale). As my data is categorical (likert scale) I thought of using logistic regression (i think ordinal), but I have to reduce my dependent variables for which I used principal component analysis (PCA) to reduce in two factors. But using PCA make my data type continuous as compared to original data that was categorical. My question is that Could I now run simple regression on the 2 factors, means is it logically correct to change the data type (from categorical to continuous) and using regression instead of logistic regression? OR Do you have any other solution?


title: How do I combine two outcome measures from two different questionnaires into one variable?

text: I have been asked by a journal editor to combine two separate measures of child disruptive behaviour into one variable. One meaure is parent rated and is from the 36-item Eyberg Child Behaviour Checklist. It is measured on a 7-point likert scale and gives a total score. The other measure is teacher- rated and is from the Total Difficulties score from the Strengths and Difficulties Questionnaire. It is based on a 3-point likert scale. They want me to combine these two measures into one variable, however I do not know how I should do this as they are separate measures from separate rating scales. I wondering about standardising these two variables but then did not know whether an average of these two standardised variables would suffice. Your help would be very appreciated.


title: Unique (?) idea for forecasting sales

text: I'm working on developing a model to predict total sales of a product. I have about a year and a half of bookings data, so I could do a standard time series analysis. However, I also have a lot of data about each 'opportunity' (potential sale) that was either closed or lost. 'Opportunities' are progressed along stages of a pipeline until they are closed or lost; they also have associated data about the prospective buyer, sales person, interaction history, industry, estimated size of bookings, etc. My goal is ultimately to predict total bookings, but I want to account for all of this information about the current 'opportunities' which are the true 'root cause' of bookings. One idea I have is to use two different models serially as follows:   1. Use historical 'opportunities' to build a model that predicts the bookings arising from an individual 'opportunity' (I'd probably use random forests or even plain old linear regression for this step).   2. Use the model from 1 to predict the estimated bookings of all 'opportunities' currently in the pipeline, then sum those estimates based on the month each 'opportunity' was created.    3. Use a time series model (possibly ARIMA?), using the 1.5 years of monthly historical time series data AND the predicted (using the model from 1) total bookings for all 'opportunities' created in that month. Granted there would be a lag in those opportunities converting to actual bookings, but the time series model should be able to deal with the lag. How does this sound? I've done a lot of reading on time series and predicting sales, and from what I can tell this is a somewhat unique approach. Therefore I'd really appreciate any feedback!


title: What is the distribution of changes in adjusted R squared when adding a random normal variable?

text: Suppose we have a model with $n$ data points and $p$ predictors. The model has an $R^2_{\text{adjusted}}=q$. What sort of distribution does the _change_ in $R^2_{\text{adjusted}}$ have when a random Gaussian predictor is added to the model? I am assuming that the expected value is zero, since $R^2_{\text{adjusted}}$ is supposed to adjust for predictors that have no explanatory power. But I can't find the answer anywhere, and I'm having trouble working it out myself. ...


title: How to perform meta-analysis comparing different proportional mortality ratios of subgroups?

text: I am working on a meta-analysis of observational studies (basically 39 retrospective papers on different surgical interventions for one condition). I've obtained different weighted proportions (random effects model) through MedCalc for both the full data set and the different subgroups. So far I have only found information on meta-analyses for randomized controlled trials, which I cannot apply to my data since I lack the "control group" in my retrospective papers.   * **What is the best statistic method and/or software to use for single-arm, uncontrolled, retrospective studies?**   * **How I can compare the different proportional mortality ratios of my subgroups?**


title: Uniform random variable distribution

text: This is a homework problem out of the book. It says > If $U$ is a uniform random variable on [0,1], what is the distribution of > the random variable $X = [nU]$, where [$t$] denotes the greatest integer > less than or equal to $t$? There is a second part that says > Do this for $n = 10$. True or false, and explain: “Random digit” is a good > name for the random variable $X$ I don't even know where to begin. All the book says about Uniform Distribution is that uniform density is $f(x) = 1, 0 \le x \le 1$. What does $t$ represent and how do I start this problem?


title: How to create histogram with "zoom-in" feature

text: I was impressed by the Reputation histogram on the SE sites allowing one to zoom in to any time interval. How was it created?


title: Animating the effect of changing kernel width in R

text: I have some data in R, stored in a list. Think               d <- c(1,2,3,4)       although this is not my data. If I then enter the command                plot(density(d, kernel="gaussian", width=1))      then I get the kernel probability density estimate, where the kernel is standard normal. If I replace 1 with other numbers, of course the picture changes. What I would like to do is create a video or animation in which each frame is such a plot, but the bandwidth of the kernel varies from frame to frame, thereby showing the effect of changing the bandwidth. How can I do this? (My apologies if this is not the right place to ask questions about R.)


title: Forecasting Amazon or Netflix demand

text: Suppose I want to predict Amazon or Netflix demand, using demand data over the past year. For example, I might want to forecast the number of sales in the Electronics category on Amazon, or the number of times someone wants to rent Titanic on Netflix. My dataset consists of daily demand per item over the past couple of months, along with item metadata (tags and categories), split by things like customer demographics (age group, gender, location, browser, job -- some of these might be unknown). To be concrete, let's suppose I want to forecast the number of times someone wants to rent a Comedy on Netflix, and I want to make this forecast at various levels (e.g., overall, by the state the customer lives in, by male/female, etc.). How would I go about this? My naive first thought is to form a time series at each level I care about (e.g., form a time series of comedy demand by all the males living in Florida), and build some kind of time series model on top of this (I guess an ARIMA model...?). But this seems wrong for a bunch of reasons (not only would I be building a ton of different models for all the different possible levels, but each level would be ignoring a lot of data from closely related levels). Any suggestions? Surprisingly, I couldn't find any papers related to this problem when Googling, but I might just be using the wrong search terms. (I learned a smidgen of time series analysis a couple years ago, but I was incredibly bad at it.) Also, I'm interested in both methods (what algorithms to use) and particular statistical libraries that might be useful (e.g., R packages or Python libraries).


title: Reference category and prediction

text: I am struggling a little bit at the moment with a question related to logistic regression. I have a model that predicts the occurrence of animal based on land cover with reference to forest. I am not grasping the concept of a reference class and struggle to extrapolate the model onto a new area. Any explanations or guidance towards papers, lecture notes etc would be highly appreciated.


title: How to decode NOAA TREND files?

text: How do I decode NOAA TREND files such as: ftp://tgftp.nws.noaa.gov/data/observations/metar/trend/KABQ.TREND I know how to decode METAR, SYNOP, SHIP, and BUOY files, but not these. [is there a better place to post this?]


title: Online vs offline learning?

text: What is the difference between offline and online learning? Is it just a matter of learning over the entire dataset (offline) vs. learning incrementally (one instance at a time)? What are examples of algorithms used in both?


title: Modeling of real-time streaming data?

text: I am interested in tools/techniques that can be used for analysis of streaming data in "real-time"*, where latency is an issue. The most common example of this is probably price data from a financial market, although it also occurs in other fields (e.g. finding trends on Twitter or in Google searches). In my experience, the most common software category for this is " **complex event processing** ". This includes commercial software such as Streambase and Aleri or open-source ones such as Esper or Telegraph (which was the basis for Truviso). Many existing models are not suited to this kind of analysis because they're too computationally expensive. Are any models** specifically designed to deal with real-time data? What tools can be used for this? _* By "real-time", I mean "analysis on data _as it is created_ ". So I do not mean "data that has a time-based relevance" (as in this talk by Hilary Mason). ** By "model", I mean a mathematical abstraction that describe the behavior of an object of study (e.g. in terms of random variables and their associated probability distributions), either for description or forecasting. This could be a machine learning or statistical model. _


title: Clustering a dataset to get the most abnormal data

text: I have several datasets in R+, each containing two training and test sets. For example the following dataset. I want to train a classifier by using training data such that by applying the test data, I get some reasonable number of points as anomaly so that I can analysis the related situations. The higher the value, the more abnormal it is. By reasonable number I mean it to be less than P% in each 100 points (each day 100 points are generated, most of them should be considered normal and I want to analysis about P of the most abnormal ones). I tried K-means with K=2. But as you see in the above link, anomaly cluster is selected by the outliers to be too high. So there would be no anomaly in test data.


title: why simulated gamma distributed data have negative kernel values?

text: I know that Gamma distribution does not allow 0 or negative values. I was doing some simulation and when I write this code in R               set.seed(1234)     Gamma<- rgamma(2032,shape=1.77, rate= 4.33)     hist(Gamma, prob=T,ylim=c(0,2), xlim=c(-1,2.5))     lines(density( Gamma), col='blue')      I obtain this ![histogram of simulated data and line of the kernel](http://i.stack.imgur.com/cJIaw.png) I can see that the kernel line strart before the axes origin. I'm sure it is correct but can someone explain me why?


title: F-adjusted Mean Residual Test in R

text: I'm running diagnostics on a logit model in `R` produced with `glm(formula = formula, data = data, family = "binomial")`. I'm looking at the Goodness of Fit and I've run a Hosmer-Lemmeshow Test via `hoslem.test()`, but even Hosmer prefers using a **F-adjusted Mean Residual Test** when a model fails a mis-specification test. Based on my results from Ramsey's RESET test, I believe that would be appropriate in this case. Is it possible to do this test in `R`? I'm aware that in `Stata` one could use `svylogitgof` but I can't seem to find an equivalent in `R` and without a "canned" function to run the test I'm not confident of how to carry it out.


title: Weighted spatial clustering

text: I am pretty new to clustering, so please be patient. I have a set of points, and each point has a weight. I need to group these points into N clusters (N is defined). I need these clusters to satisfy two conditions:   * The points of a cluster must be spatially connected.   * If a cluster has points with high weights, it must be smaller (less points). On the contrary, if the sum of the weights is small, it must have more points. I have read another post that did something very similar. It defined the distance between points, inserting the weight as a forth dimension. Then it defined several parameters to give more importance to the distance or to the weights. I cannot define these parameters (I think this would change for each example I try). Also, this other post did not recommend any clustering algorithm...and I don't know where to start. Thanks for the help! PS: By the way, it would help if the algorithm is very fast.


title: How do you deal with different distance based features?

text: If I have a model where the set of features where a cosign distance measure makes sense for some of the features, and a Euclidean distance measure makes sense for the others for example using a BOW feature set with additional features like number publications made by the author, etc., how do I pick which metric I should use or split the feature set to make more sense?


title: Naive Bayes classifier for predicting performance

text: I am trying to predict the future performance of customers on one of the websites I run. We provide practice tests leading up to a final exam, taken elsewhere. Based on the user's answer to multiple questions (only counting the first time they see the question), I am hoping to predict if they will pass their final exam. This is part of my dataset for past customer performance                   Set A       Set B     Q   Cor Tot Cor Tot     1   16  34  6   12     3   33  37  11  12     4   41  43  9   11     6   23  31  7   10     7   18  29  5   7     9   32  34  10  10     10  19  33  6   8     11  14  29  5   10     12  34  38  9   10     13  15  25  5   7      `Q` is a question identifier, Set A contains the customers who passed the final exam on their first try, Set B contains the customers who did not pass the final exam on their first try. `Cor` indicates how many customers answered that question correctly on their first try. `Tot` indicates how many customers have tried answering that question. In total, set A contains 620 customers, set B contains 162 customers. * * * Now, assume that our customer Alice whose performance we want to predict answers the questions as follows (`1` is correct, `0` is wrong):               Q   A     1   1     3   1     4   0     6   1     7   1     9   1     10  0     11  1     12  1      * * * I am assuming this fits well into a naive Bayes classifier: $p(Pass) = 620 / (162 + 620) \approx 0.7928388747 $ is the probability a random person (who has answered our survey) will pass the test on their first attempt. $p(Q_n)$ (answering question $n$ correctly) can then be calculated: $p(Q_1) = \frac{16+6}{34+12} \approx 0.4783$ We need to find $p(Pass|Q_n)$ (probability of passing the exam given answering question $n$ correctly): $p(Pass|Q_n) = \frac{p(Q_n|Pass) \times p(Pass)}{p(Q_n)}$ For question 1, this is: $p(Pass|Q_1) \approx \frac{\frac{16}{34} \times 0.7928}{0.4783} \approx 0.7801$ This is where it gets fuzzy for me (not being a mathematician nor statistician, and not having done statistics for quite some years): I believe I need to get the answer to the following: $p(Pass|\cap_{i=0}^n{Q_i}) = p(Pass|Q_1 \cap Q_3 \cap \neg{Q_4} \cap Q_6 \cap Q_7 \cap Q_9 \cap \neg{Q_{10}} \cap Q_{11} \cap Q_{12})$ The formula of which I can be expressed as: $\frac {\frac{\prod_{x=Q_i}{p(Pass|x)}}{p(Pass)}} {\frac{\prod_{x=Q_i}{p(Pass|x)}}{p(Pass)} + \frac{\prod_{x=Q_i}{p(\neg Pass|x)}}{p(\neg Pass)}}$ (I am not sure how to express this using math in the formula above, but the idea is to use $p(\neg Q_n) = 1-p(Q_n)$ whenever the user answers incorrectly). However, when calculating this, the number converges on 1 after just calculating a few terms. None of the customer data I've tried results in a number distinguisable from 1 after only 20 answers to questions have been taken into account. Am I doing something wrong (my assumption), or is my data actually showing that our logged data cannot predict future performance?


title: Reliability and validity of a standard survey

text: I had eleven people play five versions of a video game in single sessions, for a total of 55 sessions (11x5). I used the Task Evaluation Questionnaire from the Intrinsic Motivation Inventory to capture their experiences. Do I need to test the reliability and validity of this survey for the data that I collected? Someone said that I may have to and I don't know how it's done, or whether these 55 different sessions are enough. On the website the authors say: > The IMI items have often been modified slightly to fit specific activities. > Thus, for example, an item such as "I tried very hard to do well at this > activity" can be changed to "I tried very hard to do well on these puzzles" > or "...in learning this material" without effecting its reliability or > validity. This makes me think that I don't have to, but I don't know much about survey research. What are your thoughts? Here's a follow-up question: Even if you think that I don't need to, let's say I really want to compute reliability and validity of this survey for my experiment. How would I do that? Would the data that I already have (55 sessions) be enough? I'm asking because collecting new data is very inconvenient at the moment. Thank you so much.


title: Model comparison with AIC based on different sample size

text: Let's assume I have two models M1 and M2:               M1:  y ~ x1 + x2 + x3     M2:  y ~ x1 + x2 + x3 + x4      Since variable x4 has some missing values the sample size of M2 is lower than sample size of M1: n2 < n1.   Is it still admissible to use AIC to compare models based on different sample sizes?


title: Naming of dependent and independent variables in simple linear regression

text: I'm currently doing some self-study on Simple Linear Regression. I came across the terms "dependent variable" (eg. Sales volume) and "independent variable" (eg. Advertising budget). I am curious to learn **why** is it **called dependent and independent variable**. I know what these variables does,but just have no clue why they are called as such.


title: Joint prior distributions in WinBUGS

text: Suppose we have a hierarchical model summarised by the following: $y_{i} \sim N(\mu_{i}, \sigma^{2})$, for $i = 1, \ldots, n$; (For these purposes, assume $\sigma^{2}$ is known) where $\mu_{i} = \alpha + \beta x_{i}$ and each $x_{i}$ is a given predictor variable for $i = 1, \ldots, n$ Next, $\alpha \sim N(\mu_{\alpha}, \sigma_{\alpha}^{2})$ and $\beta \sim N(\mu_{\beta}, \sigma_{\beta}^{2})$ Finally, $p(\mu_{\alpha}, \mu_{\beta}, \sigma_{\alpha}^{2}, \sigma_{\beta}^{2}) \propto \frac{1}{\sigma^{2}_{\alpha}\sigma^{2}_{\beta}}$ How can this model be fitted in WinBUGS? I know that the last two levels are given by:               for (i in 1:N){      y[i] ~ dnorm(mu[i], sigma.squared)      mu[i] <- alpha + beta*x[i]     }     alpha ~ dnorm(mu.alpha, sigma.squared.alpha)     beta ~ dnorm(mu.beta, sigma.squared.beta)      However, I'm not sure, how the joint prior for $\mu_{\alpha}, \mu_{\beta}, \sigma_{\alpha}^{2}, \sigma_{\beta}^{2}$ should be incorporated into the model. In particular, how is the hyper-prior distribution represented in the WinBUGS code, how can I achieve values for $\mu_{\alpha}$ and $\mu_{\beta}$ from this distribution and how should $\sigma_{\alpha}^{2}$ and $\sigma_{\beta}^{2}$ be chosen?


title: Accuracy of neural network

text: How to calculate accuracy of a neural network? is there any functions or formulas? knowing that I use neural net work for prediction, with 10 neurons in the input layer, and 2 Hidden layers and 1 neuron in the output layer. for error minimization, I use MSE. Thanks


title: How to check the assumption of equal error variance might be violated?

text: ![enter image description here](http://i.stack.imgur.com/Kr7yr.png) I want use this box plot to check it. Thank you


title: scale estimate for positive and negative data

text: I have 2 datasets (nonpaired), P1 & P2, and I want to know 'scale parameter' of one to the other, aka P1=a*P2. As both datasets are highly nonnormal, I am performing a Mann-Whitney Test on the data. I take the log of the data, do the MW test, get the Hodges-Lehmann shift estimate (HL) and then take the antilog of the HL shift estimate to get the scale estimate. I do the same thing with confidence intervals. This all seems to work fine, until I deal w/ datasets which are positive & negative, since the log of a negative number is imaginary, and then everything needless to say explodes... What method can I use to get the estimate for positive & negative data?


title: Are linearly dependent contrasts permitted when performing planned comparisons in within-subjects ANOVA?

text: * Here it says that planned comparisons should be linearly independent.    * Here (page 13) and elsewhere I see people giving examples of linearly dependent contrasts.  How should this conflicting advice be reconciled?


title: Algorithm for calculating the top K movers (gainers and looser) in a time series dataset

text: I've a timeseries dataset:               t1 -> { k1 -> 10, k2 -> 20, k3 -> 21, k4 -> 100, k5 -> 50 }     t2 -> { k1 -> 11, k2 -> 22, k3 -> 5,             k5 -> 200, k6 -> 10, k7 -> 5}     t3 -> { k1 -> 10, k2 -> 19, k3 -> 1,  k4 -> 50,  k5 -> 240, k6 -> 12, k7 -> 120 }     t4 -> { k1 -> 13, k2 -> 25,        ,  k4 -> 70,  k5 -> 220,          }      .....      .....      `t1, t2, t3 ....` are timestamps that are X time units apart (e.g., 1 minute, 10 minute, 30 minute and 1 hour). At a particular instant I can show the top K entities (e.g., at time = `t1 k4, k5` and `k3` are the top 3). However now I want to show the top movers (gainer and looser over time). For example, `k1` and `k2` have hardly changed in the dataset above. However, `k5` and `k7` have changed significantly. Is there is a standard method/algorithm for calculating the gainers and losers for such a dataset? Thanks.


title: How to test whether variance explained by first factor of PCA differs across repeated measures conditions?

text: ### Context: I have a study where six numeric variables are measured in each of two repeated measures experimental conditions (n = 200). Lets call the conditions $A$ and $B$ and the variables $A_1, A_2,..., A_6$ and $B_1, B_2,..., B_6$. Theoretically, I expect that in condition $B$ more variance in the variables should be explained by the first factor of a principal components analysis (PCA). Typical values would be:   * First factor of PCA on $A_1, ..., A_6$ accounts for 30% of variance   * First factor of PCA on $B_1, ..., B_6$ accounts for 40% of variance. ### Questions:   * How can I test whether this difference is statistically significant?   * How could this be implemented in R?


title: Feature importance

text: The extremely randomized trees classifier (scikitlearn) provides a (multivariate) feature importance measurement Ensemble methods/feature importance evaluation. For each feature, the classifier produces a statistical measurement (and the corresponding standard deviation) for how important the feature was for predicting the target variable. The basic use of this information is to create a "feature ranking" among the features (from high importance value to low). The question that I have is if I can use this information to make a conclusion that my features are weak or strong for my training data? Has the size of the "importance value" a meaning in itself? For instance, the highest ranking feature has an importance value of 0.0494 with a standard deviation of 0.024. Can I make the conclusion that the features are weak because the "importance values" are very small?


title: Random walk or not?

text: I'm trying to understand whether the observed time series can be described as a random walk or not. When I check autocorrelations of the differences, none of the autocorrelation bars for the difference series exceeds the blue dashed thresholds. However, I was not sure whether it was enough, so I've tried the augmented Dickey-Fuller test on the differences. The p-value turned out to be high, so we could not reject the null-hypothesis of stationarity. While on its own, this does not necessarily mean that the difference series is not stationary, it is still confusing. Moreover, my intuition says that there is an increasing trend, but I'm not sure how to describe it. Data:               Start = 1998      End = 2011      Frequency = 1      59 334 333 402 450 461 452 468 461 463 508 573 639 567


title: Ordinal Probit model in plain English

text: Suppose we have a model looking at the association between sodium intake $X$ and levels of body fat $Y$. So $Y$ is an ordinal variable that can take the integer values from $\{0,1,2 \}$. It is ordered according to increasing levels of body fat (e.g. "no bodyfat increase", "very little bodyfat increase", and "a lot of bodyfat increase"). What exactly does an ordinal probit model do? I plotted the data. So is an ordinal probit model trying to fit a line through the data points?


title: If using interval data (independent variables) and grades (outcome or dependent variable), what type of analysis would one use?

text: First of all, my background in statistics is a bit shaky these days due to a trauma to the brain. I am considering a study that examines at least three independent variables (e.g., creativity, locus of control, and assertiveness) on course grades (only grades were obtained not percentages . . . so this would be a ranking) in a college setting. The independent variables can be pushed into three different categories if needed (interval, ordinal, or nominal). Thus, I can use an interval score for creativity, for example. I can switch this to low versus high creativity as well (ordinal or nominal). Most would probably assume high creativity is better than low creativity (ranked), but some might think it is just nominal data. I am thinking that ordinal regression is the best fit. Am I off base here?


title: How to start an analysis of keywords from a bibliography and detect correlations?

text: (I'm a bit outside my comfort zone, so apologies if this is badly worded, or off-topic) I have a bibliographic database, containign details of about 1200 different papers, books, web sites etc, all with various details, including keywords and an abstract. I want to somehow analyse this database and produce some graphics showing the correlations between different keywords. (like "drug" is often present with either "pharmacology" or "assay"). Ideally this would be in R, but general advise would also be welcome. (I've seen this question/answer which piqued my interest, and this heatmap graphic also seem related) My database could be in bibtex, or could be converted to plain text.


title: Unbiased estimation of covariance matrix for multiply censored data

text: Chemical analyses of environmental samples are often censored below at reporting limits or various detection/quantitation limits. The latter can vary, usually in proportion to the values of other variables. For example, a sample with a high concentration of one compound might need to be diluted for analysis, resulting in proportional inflation of the censoring limits for all other compounds analyzed at the same time in that sample. As another example, sometimes the presence of a compound can alter the response of the test to other compounds (a "matrix interference"); when this is detected by the laboratory, it will inflate its reporting limits accordingly. I am seeking a practical way to estimate the entire variance-covariance matrix for such datasets, especially when many of the compounds experience more than 50% censoring, which is often the case. A conventional distributional model is that the logarithms of the (true) concentrations are multinormally distributed, and this appears to fit well in practice, so a solution for this situation would be useful. (By "practical" I mean a method that can reliably be coded in at least one generally available software environment like R, Python, SAS, etc., in a way that executes quickly enough to support iterative recalculations such as occur in multiple imputation, and which is reasonably stable [which is why I am reluctant to explore a BUGS implementation, although Bayesian solutions in general are welcome].) Many thanks in advance for your thoughts on this matter.


title: Using bootstrap for glm coefficients variance estimation (in R)

text: I am fitting a GLM model (in R), and would like to get an estimation of the variability of the coefficients estimated by the model. If I understand it correctly the method to use in such a case is bootstraping (not, cross validation). Am I correct that an easy way to do this is by using the boot command from the boot package, then output the coefficients at each simulation, and at the end calculate their var? Or is there something I might be missing?


title: How to calculate percent deviance explained on square-root-transformed dependent variable?

text: I have a multiple linear regression model with a square-root transformed dependent variable. I want to report % deviance explained for each parameter. I assume I need to back-transform in some way, but I am not sure how. Or is it okay to report % deviance explained on the original (non-transformed) model? I am using R to calculate model deviance, if it matters. I appreciate any insight on this issue.


title: Correlation vs measure of independence

text: So I have some data of which I'm looking at two different metrics, the first being "how sick a person is (higher = sicker)", and the second being "the presence of adverse events", and I've run 20 experiments:               table=       Columns 1 through 11               1     0     1     1     0     0     1     1     2     0     2          0     0     1     1     1     1     0     0     1     0     0            Columns 12 through 20               2     1     2     2     2     2     2     0     1          0     0     1     0     0     0     1     0     0      The correlation coefficient between these two data sets is -0.0066, which to me implies there is a fair amount of independence. I guess the question is now, is there any way to expose that certainty. I understand that the t-statistic can be calculated from that, and equals -0.0281, which suggests a p-value (dof = 2) of 0.980134 (two-tailed) or 0.490067 (one tailed). Now I may be stating the obvious, but does the high p-value imply a high probability of independence?


title: What is the difference between the F-test and the Wald / Likelihood Ratio / Score tests of linear models?

text: In statistical inference, it's common that investigators will report the results from Wald tests for simple univariate inference or likelihood ratio tests for nested models. However, in regression modeling, under standard assumptions about the normality of errors, the F-test is exact and equally powerful in finite samples relative to the other tests. It's commonly included in statistical output, yet it's not favored for inference. Can someone explain why the F-test is not used?


title: What does it mean to correlate residuals in SEM?

text: I have been reading a paper by Cole, Ciesla, and Steiger, which argues in many cases allowing residuals to correlate is justified. However, I am not entirely sure what it means for residuals to correlate. A good definition and example would be much appreciated. Cole, D. A., Ciesla, J. A., & Steiger, J. H. (2007). The insidious effects of failing to include design-driven correlated residuals in latent-variable covariance structure analysis. Psychological Methods, 12, 381–398.


title: How to fit random "slopes" in gamm model from mgcv?

text: I have been struggling to figure this out for awhile, and I haven't had much luck with find this on the internets. I want to fit a gamm model using the gamm function from the mgcv package in R. As near as I can tell the random terms you put into the "random" argument to the function apply to the parametric coefficients in the model, and hence would cause something similar to a random intercept model. And, if you have "time" as an independent variable, I suspect that fitting "time" as a spline and then putting a random "time" slope in the random argument is really appropriate. Though I'm definitely not an expert at this... What I want to do is include something analogous to a random "slope" from a mixed model. Is there a way to include a separate spline term for each subject? I've found that one can sort of do what I want by using the "factor smooth" basis to fit a separate spline within each subject. But then I'm left with a separate spline for each subject when what I really want in the end is a marginal effect. Just a quick toy example using data from _nlme_.               library(mgcv)     data(Orthodont)      Without really paying attention to whether this model represents the best analysis or whatever, we use the gamm function—               mod1 = gamm(distance ~ s(age, Subject, bs="fs", k=4), data=Orthodont)     summary(mod1$gam)         plot(mod1$gam)      In the summary and plot we see a ton of fits (one for each subject). But I don't really care so much what subject "M01" is doing. I really want to know what the average (possibly curvy) relationship is between age and distance. Something along the lines of "level=0" from lme. I'm sure there's a way, and I just don't know the theory well enough to see it. Thanks!


title: Multidimensional similarity analysis - graph similarity?

text: I study several chemicals that their effects on different _reference_ surfaces is measured and plotted individually (on a semi-log plot of effect vs concentration of chemical). Is there a way to find out that chemical A and C are quite similar while A and B are 25% similar using **all** datapoints on the graphs, and not one representative 50%-effect point (${E}_{50}$)? Currently the way usually done is to calculate a representative ${E}_{50}$ effect for each plot, and calculate a _Pearson coefficient_ for each two chemicals. However, some may not have ${E}_{50}$ representative points and hence the analysis fails. Schematics:   ![http://cl.ly/image/3g0w361t3L17](http://i.stack.imgur.com/4Fe9L.png) To clarify a bit, each chemical has profile I, II, III and IV, and these profiles should be compared 1 to 1, as in the response of Chem A on surface I should be similar to Chem C on surface I, and so on; comparing Chem A on II with Chem C on III is irrelevant.


title: How to deal with a variable that ranges from 0 to 1 and the distribution has two spikes at these values with normal-like distribution in the middle.

text: I have the following setting (& would like to pick a model/or transformation that can help): dv is normally distributed continuous variable, all IVs are continuous, but one of them is the above mentioned variable which has values between 0 and 1, having two spikes at 0 and 1. In between there is a normal- like distribution (though slightly skewed and quite flat). What can I do with this variable? Ln(Y) or Ln(Y+c) or other transformation (tried inverse trig, inverse, sqrt)? Is there a nice model? Non-parametric? Dummies? Multiple tests? Help appreciated, a beginner in statistics


title: Distribution of product of sums of a set

text: Given a set $A$ of $n=kl$ positive real numbers, let $a_{ij}$ be a random number removed from the set $A$ (without replacement) for $i=1 \dots k$ and $j=1 \dots l$. What is the distribution and expected value of the following expression? $$ \prod_{i=1}^{k} \sum_{j=1}^{l} a_{ij} $$ Note: Using a set of 30 numbers for $l=3$ and $k=10$, I generated 10000 samples and computed the above formula for it. The following figure shows the distribution of the samples. ![Distribution](http://i.stack.imgur.com/LPJzO.png) My question is that: how can I find the above distribution and expected value analytically? **Update** To makes the things clear, I past the code I used to generate the histogram here. This is a Mathematica code. The code generates a set of 30 random numbers between 0 and 1 (variable "nums"). Then it does the following 10000 times to generate the numbers for the smooth histogram: Shuffle the numbers in "nums" and assign it to "rands"; split the numbers in "rand" into 10 groups of 3 numbers; compute the sum of each 3 numbers; multiply the 10 summations to have the result.               SmoothHistogram@Module[{nums = RandomReal[1, {3*10}], rands},        Table[          rands = RandomSample@nums;           Times @@ Total /@ Table[rands[[{i, i + 1, i + 2}]], {i, 1, Length[nums], 3}],           {10000}       ]     ]


title: Choosing fold size for highly Imbalanced dataset + nested CV + svm

text: I am trying to classify a dataset with ~1000 points. 90/10 is the class ratio - super imbalanced. Here are the following steps I did:   1. Use 20 relevant features from previous knowledge   2. Remove highly correlated features   3. Perform backwards feature selection (use all features, take one out)   4. Estimate which feature-set is the best and repeat (3) until it doesn't get any better To evaluate (3) accurately I do nested CV as follows: a. Split dataset into the two classes (say 0 and 1) and split each class into $X$ folds and merge folds together (stratification). b. Take $X-1$ folds for training and one for testing. Use training data to split again into $Y$ folds (again train and test) and perform SVM to find optimal parameters c. Once parameter found, use the test data to estimate accuracy. To estimate accuracy I used the F2 score to account for the imbalanced dataset. After each outer fold has completed, I end up with $X$ F2 scores that I average to get my final score to be able to estimate (4). **Here is the problem:** If I do (3) and (4) with multiple iterations I have around ~10% variability in my average F2 score. So it makes it hard to say which feature set is the best. I think this is because of the randomly chosen folds in (a) and (b) Now here are my questions:   1. What do you think is the best number of folds for this particular dataset? I have tried with 5 for outer and 5 inner. But maybe I should decrease the number of folds in the outer CV since I have a few data points in the inner CV?! Maybe the best thing to do is to try different combinations and see what is best and stick to it?    2. Alternatively, I thought to do iterative nested CV (around 10-50 times) and to take the average of that to be able to choose the best feature set. Do you think this is OK to do?   3. Is overall the approach that I do for this classification legit? Thoughts and comments are very much appreciated.


title: Correct enunciation of $ \, \sum \big( {Y_{i}}-{ \hat{Y}_{i}} \big)^2$

text: How can I enunciate this notation correctly, could you show me that, please? $$ \sum \big( {Y_{i}}-{ \hat{Y}_{i}} \big)^2 $$ Thank you very much.


title: Repeated-measures ANOVA looking at interactions for a BACI design in R

text: Here are my data:               Time    Treatment    Ephemeroptera    Trichoptera    Plecoptera     Before  Control                 98             76            32     After   Control                 91             54            26     Before  Impact                  24             30            12     After   Impact                  29             21            45      Input (example):               Exp = read.csv("fast.csv", header = T)     attach(Exp)     names(Exp)     m1 = aov(Ephemeroptera ~ Time)     summary(m1)     #It gives me output with p-value, df etc. (which is expected)     m2 = aov(Ephemeroptera ~ Treatment)     summary(m2)     #It gives me output with p-value, df etc. (which is expected)     m3 = aov(Ephemeroptera ~ Time*Treatment)     summary(m3)     #It gives me the interaction output (Time:Treatment), but it doesn't provide a p-value.      I am not on my home computer, so I can't include what I got for the output right now. However, I will update it in a little bit (when I get home). Until then, is there any reason why it does not give me any p-values for the output when I try the repeated measures, but it provides it for the regular anovas? Should I try lme instead?


title: Plotting multiple binhex with the same z levels

text: I have meteorological records for a point, these include temperature and solar irradiation. I want to plot them against another variable, that we shall call Rating, and see how the data are distributed. This is what I'm doing:               d1 <- ggplot(data = mydata, aes(Temperature, Rating, fill = ..density..)) + stat_binhex(na.rm = TRUE) + opts(aspect.ratio = 1)#, legend.position = "none")      d2 <- ggplot(data = mydata, aes(Solar.Irrad, Rating, fill = ..density..)) + stat_binhex(na.rm = TRUE) + opts(aspect.ratio = 1)#, legend.position = "none")      I get both things on the same window by using grid.arrange from package gridExtra:               grid.arrange(d1,d2, nrow=1)      ![Two graphs together, different z scale](http://i.stack.imgur.com/dPHEk.png) This produces the image shown. Now, my problem is that I would really like that both graphs shared their z scale: the legend should be the same and the color scheme should be homogeneous through different graphs. Is this possible? I'm totally lost here, do anyone know of a way to do this?


title: How do I identify which parameters are correlated to binary output?

text: I'm trying to program an SVM in Python to categorize proteins as "Go" or "No- Go". I have a list of about 30 proteins, each with ~ 100 columns of structure- related parameters and 1 column of "True" or "False". I will use this as my training set. I don't want to write an SVM that uses all 100 columns of data. How do I identify which of the columns impact the output? Things I've looked at, albeit briefly: correlation, linear regression, multiple regression, nonparametric regression ..but all of these seem overly complicated because they're intended to predict non-discrete dependent values (e.g. average height or dollar-cost) instead of a binary result (True | False). Although I appreciate any guidance, I'm really just looking for the correct vocabulary so that I can research my problem without flailing in the dark. Thank you~


title: Should you ever standardise binary variables?

text: I have a data set with a set of features. Some of them are binary $(1=$ active or fired, $0=$ inactive or dormant), and the rest are real valued, e.g. $4564.342$. I want to feed this data to a machine learning algorithm, so I $z$-score all the real-valued features. I get them between ranges $3$ and $-2$ approximately. Now the binary values are also $z$-scored, therefore the zeros become $-0.222$ and the ones become $0.5555$. **Does standardising binary variables like this make sense?**


title: Is it accurate to say that all the Null Hypothesis states is the absence of a significant difference between sets of data?

text: H0 is commonly understood to signify the absence of a treatment effect or difference between two groups. Doesn't this understanding ignore the fact that sample data (being a sample) can never fully accurately reflect the nature of the phenomena being studied? Doesn't the existence of things like Type II errors require acceptance that statistics don't always reflect reality? In short: Can H0 be considered a satement about numbers and data, not necessarily about the real world?


title: Curve smoothness - local adjacency

text: I am looking for statistical measures of curve smoothness. Time-series values {(1, 0.5), (4, -0.6), (200, 1.0)} where (time-unit, value) is linearly interpolated from one to the next. The smoothest curve would keep all values the same. I am looking for curves that are smooth locally but have motion over longer spans. Similarly I am looking for highly jagged curves. What kinds of measures can I use to examine these properties?


title: Obtaining Standard Error of Weighted Averages using Bootstrapping?

text: My problem is finding a way to estimate the standard error of a flow-weighted mean concentration. The FWMC is computed by summing the years flow * concentration measurements and dividing by the sum of flow measurements (17 measures of paired concentration and flow for the year). This FWMC number (in mg/L) is then multiplied by the known total volume of water in the stream to arrive at an estimate of the total mass of pollution. I have yet to find a method which accounts for the dependence of mass on flow. Is there a way to do this in R via bootstrapping? Data:               Phosphorus (ug/L)   Flow (L/sec)    Phosphorus *Flow (ug/sec)        44.5    424.752699  18890.13722      39.6    311.4853126 12324.55805      41.7    87.78222446 3664.848179      29.9    226.5347728 6768.125039      43.1    254.8516194 10994.0542       51.6    192.5545569 9928.871618      91.6    53.80200854 4930.741027      265.4   11.32673864 3005.662573      85.8    155.7426563 13363.84022      66.6    175.5644489 11699.2181       41.6    226.5347728 9417.440145      32.2    283.168466  9116.450189      36.8    150.079287  5525.724244      40.5    158.574341  6421.823144      29.4    368.1190058 10811.05885 FWMC     sum 3080.87291  136862.5528 44.42330365


title: Regression when dependent variable is the difference betwen two proportions

text: Here is some sample data:               structure(list(     y.t2 = c(NaN, 0.05, 0.02, 0.02, 0.02, 0.04, 0.06, 0.05, 0.07, 0.1),      n.t2 = c(0, 231, 228, 219, 210, 175, 250, 255, 270, 257),      y.t3 = c(0.07, 0.11, 0.2, 0.19, 0.17, 0.12, 0.18, 0.23, 0.18, NaN),      n.t3 = c(226, 224, 223, 208, 206, 224, 228, 246, 233, 0),      x = c(-6, 0, 2.7, -0.1, -2.4, -2.5, -1.5, -0.4, 0.3, -0.3),      diff = c(NaN, 0.06, 0.18, 0.17, 0.15, 0.08, 0.12, 0.18, 0.11, NaN)),      .Names = c("y.t2", "n.t2", "y.t3", "n.t3", "x", "diff"),      row.names = c(NA, -10L), class = "data.frame")      `y` was measured at times `t2` and `t3`. I used `glm` like:               summary(glm(y.t2~x,binomial,my.df,n.t2))     summary(glm(y.t3~x,binomial,my.df,n.t3))      to test the effect of `x` on `y` from two time points, `t2` and `t3`. But if I want to test the effect of `x` on the difference, as in, `y.t3-y.t2`, how do I go about it? Though in my case, I always expect a positive difference (to use `glm` with logit link), in theory, it can be negative as well, ranging between -1 and 1. Do I do a linear regression then, or what method would be appropriate in this case? I want to do `diff~x`. I am also wondering about the weights argument. Some subjects dropped out and new ones joined between time `t2` and time `t3`; the number of cases varies between the two instances.


title: $p$ value for a randomized test

text: For a randomized test, how do we compute the $p$-value? For example, suppose the test is of the form: $\phi(X)=1$ if $X > c$, $\phi(X)=p$ when $X=c$ and $\phi(X)=0$ otherwise. How do we compute the $p$ value?


title: Lagrange-Multiplier test for Serial Correlation

text: When testing whether $\epsilon_t$ is serially correlated in: $$y_t = \alpha + \beta_1x_{1t} + \ . . . \ + \beta_kx_{kt} + \epsilon_t $$ Can we simply test: $H_0: p = 0$ in the following: $$y_t = \alpha + \beta_1x_{1t} + \ . . . \ + \beta_kx_{kt} + p\hat{\epsilon}_{t-1} + \epsilon_t $$ All the reading I've found tests within the context of: $$\hat{\epsilon}_{t} = \alpha + \beta_1x_{1t} + \ . . . \ + \beta_kx_{kt} + p\hat{\epsilon}_{t-1} + u_t $$ Any help is much appreciated!


title: Which type neural networks for time series classification

text: I would like to use neural networks to classification of time series ( I have some Patterns and I want to adjust input time series to an appropriate class) -ist it possible to do this job with neural network? which type of neural network should I use?


title: How does Standard Deviation (Error) change with sample size change in this scenario. Explanation needed for a nonprofessional

text: I have this question that I want figured out. A person's Blood pressure was taken 4 times,the mean of these 4 observations came out to be say 120mm of Hg And the SD was 2.5. Now we have taken 4 more observations and added their values in too. Question is: 1. What will happen to SD of BP? & 2. With what percentage it should change? 30% 10% or 5%? What I did was used standard error for means formula (s/n-underroot) and SE came out to be 1.25 with four observations and 0.883 with 8 samples. AM I doing it right? I know the SD is supposed to shrink with increasing sample size but by what value? Can anyone please solve this for me with the context in mind that I am not a statistics student. I am in medicine (which is increasingly becoming aligned with statistics).


title: dose anybody know usg fastSOM packages in R?

text: I recently know the fastSOM package from google. So I download the package. But I’m beginner in R. And I really want to studying spillover measures because I’m interested in volatility spillover. But It’s quite hard for me coding the spillover index. I read the manual. and manual said               # generate randomly positive definite matrix Sigma of dimension N     N <- 10     Sigma <- crossprod(matrix(rnorm(N*N),nrow=N))     # generate randomly coefficient matrices     H <- 10     A <- array(rnorm(N*N*H),dim=c(N,N,H))     # calculate estimates of the average, minimal,     # and maximal spillover index and gives the corresponding ordering     # of the model variables     soi_avg_est(Sigma, A)      I think It's data genearating method. .I got the paper from "http://www.ssc.upenn.edu/~fdiebold/papers/paper75/DY2final.pdf" and I convert the data csv file. and I load the real data from csv file.               rm (list = ls(all=TRUE))         data<-read.csv("Diebold and Yilmaz.csv", header=TRUE)     returns<-data[1:829,2:20]       but my question is how I can apply the FastSOM packages to the real data? and How I can plot the spillover using 200-week rolling windows in paper on 9page? please let me know how to I use your FastSOM package accurately. If you do so, I'd really appreciate that.