title: How does one prove that a separating hyper-plane exists for a linearly separable pattern?

text: How does one prove that a separating hyper-plane that can be represented as a linear combination of the training samples exists for a linearly separable pattern? Although, it looks pretty obvious physically, I was interested to know if it could be proved mathematically.


title: Correlation structures in Linear mixed models

text: The linear mixed model I want to fit to my data shows some major correlations. Z.transformation already reduced the correlations but not as much as I had hoped. I calculated the variance inflation factors and they seem a little high but not off the charts (between 1.3 and 3) Here is the correlation structure of one of my models as printed by summary(lme):                                        (Intr) orgns_ exp.ty systmm     organism_typeheterotroph -0.832                          exp.typelab              -0.853  0.552                   systemmarine             -0.172  0.281 -0.078            log(exact.duration)      -0.895  0.768  0.639 -0.019      The random factor (random intercept) for my models is studyID which perhaps might explain high correlations for the Intercepts (or am I wrong there, already?) However, the rest of the correlations are also pretty high. I wanted to implement a correlation function to the model but I'm not sure how to do that and which one to choose. According to Pinheiro and Bates "Mixed-Effects Models in S-and S-Plus" there are several correlation structures, but I think only the "corSymm" function would work for me, as I have no temporal or spatial structure in my Data. However, I'm not sure how to implement it. What should be my "value"-argument? Is it simply the correlation matrix that I get via summary(lme) (see above)? I hope I'm not on the completely wrong path here! Please help me out!


title: Adjustment to Sensitivity & Specificity

text: Good Day All, I am trying to measure sensitivity and specificity of a new test in the face of an 'imperfect' gold standard. It turns out that the literature has known Sensitivity and specificity of my 'imperfect' gold standard and as I understand it, it is possible to adjust Sensitivity and specificity based on this. I found this website showing the formulas (Correction of Sen/Spec) but I am concerned about two things:   1. Is the the appropriate set of formulas to correct Sen/Spec ?   2. I tried to work through theexample using the data from Table 1 and I did not understand something. If I use the same values in Table 1 a=760 b=190 c=40 and d=1010) and make the sensitivity of the Gold Standard=60% and the specificity of the Gold Standard = 1 then the corrected values for Sensitivity=95% and the Specificity Adjusted=148% ! I can further play around with the Gold Standard Sensitivity/Specificity and I can make the same matrix get a **negative** adjusted specificity by changing gold standard sensitivity to 20% Am I doing something wrong here ?


title: Do zero inflated continuous covariates cause "problems" in binary logistic regression?

text: I am trying to do a logistic regression to look at the relationship between the number of cigarettes smoked by subjects in a sample (0-60 per day) and a yes/no outcome. As a lot of people in the sample are non-smokers, there are lot of zeros for the continuous covariate. I have a few questions about this:   * Is this correctly described as a zero-inflated continuous covariate?    * Will this cause "problems" in the logistic regression e.g. affect the validity of the coefficients etc.    * Should a stratified be carried out on smokers and non-smokers instead? Thanks


title: Distribution of a sample from a normal population

text: If I were to collect ONE sample of size 20 from a normal population, how justified would I be in claiming that the sample is normally distributed? I'm getting a little confused since by the Central Limit Theorem, the sample size has to be _sufficiently large_ and 20 is a relatively small number. On the other hand, if we have assumed the population to be normal than surely we are right to assume any sample from this population is also normal, despite its size.      The population in question consists of various times for a particle to travel from A to B. I have plotted the 20 samples as a histogram in R and they do not seem strongly normal. However, there are some negative values in the sample and once these are omitted, the plot more closely resembles a normal distribution. Am I right to omit the negative values if it makes the plot more normal, considering the population is normally distributed?


title: Forecasting time-series ahead by multiple time horizons

text: Suppose that I have daily data on the population of a small village, given by $Y(t)$, as well as daily data on various factors that are relevant to the size of the population in the future, given by vector $X(t)$. These explanatory variables include untransformed variables as well as features engineered to be informative over long horizons (e.g. one of the variables captures the number of deaths over the last 30 days). I have collected this data for 8 years. My objective is to forecast $Y(t)$ ahead by 1,2,3,...,365 days. I expect long- run forecasts to be different to short-run forecasts. If a holiday season is coming up I might expect a downwards spike in a few months time (people visiting the city), but if someone is on their deathbed then I will expect a downwards spike in a few days. Since the population is sufficiently small that $\Delta Y(t+k)$ is typically in $\\{-2,-1,0,1,2\\}$ for the forecasting horizon under question, I will use a multiple categorical response variable classification model that will assign probabilities to the various class labels being observed. My question centers on the specific considerations I need to make when constructing forecasts of the change from $Y(t)$ to $Y(t+k)$ where $k$ is large (e.g. 100 days). Basically there will be the most hideous autocorrelation structure in $\Delta Y(t+k)$ over these time scales. If someone dies on day $2$, they are also dead on day $3, 4, ..., k$, meaning a string of $k$ or so $\Delta Y(t+k)$ will contain this same information. These queries result:   * What are some ways of dealing with this immense autocorrelation structure in my response. Is it even a problem?   * Are there alternative methodologies to the ones I've proposed for forecasting these horizons (aside from typical machine learning methods such as random forests which I'm already working with).   * Any other handy advice.


title: Interpretation of quantile regression when high quantile estimates are lower

text: Suppose I estimate a multivariable quantile regression $Q_Y(\tau | X) = \alpha(\tau) + \beta(\tau)X + \epsilon(\tau)$. Note that $X$ is a vector of independent variables. Suppose I then 'plug in' my values of $X$ that I observe, and find that $\hat{Q}_Y(0.6|X) < \hat{Q}_Y(0.4|X)$.   * How do I interpret QR in this instance? What are the consequences of this on the validity of the estimates?   * Do I throw the whole model away for every future observation of $X$, or only throw it away for that particular observation; justified that it was an unlikely realisation from the joint density of $X$ that doesn't reflect the underlying generating process.


title: Parameters of a Statistical Distribution

text: Any statistical distribution is described in terms of shape, scale and location parameters. But what do these parameters mean, geometrically, statistically and for a layman with minimum statistical knowledge? I have explored wikipedia and still, this doubt continues to exist.


title: Motivation for statistical distributions

text: As statisticians, we come across many distributions under the banners "discrete","continuous" and "univariate","multivariate".But can anyone, offer me a good reason behind the existence and motivation for so many distributions. How do we get them? and what can a layman understand from it? What is the logic behind the existence of distributions?


title: Discriminant Analysis and Ordinal Outcomes

text: Suppose we have a continuous independent variable $X$ and a dependent ordinal variable $Y$. Is it possible to perform a linear discriminant analysis on the data? Or would both variables need to be continuous? In other words, would we have to convert the ordinal variable to a real number?


title: Regression methods

text: What is the fundamental difference between:   1. Linear regression    2. Non linear regression    3. Parametric regression, and    4. Non-parametric regression? When should we use each type? How do we know what to choose? What kind of data are required? What are the assumptions unique to each? At times, if you go through papers you get to see a combination of the names above. * * * Well, the ideas presented above have led me to the following conclusions: 1) Linear Regression : Regression methods associated with a linear model, linear with regard to the parameters of interest 2) Non-Linear Regression : Regression methods associated with a non-linear model, non linear with regard to the parameters of interest. 3) Parametric Regression: Regression methods associated with a linear model/non-linear model (accordingly called as Linear Parametric / Non-linear Parametric), but the basic assumptions of regression including those associated with errors have to hold truth. 4) Non-Parametric Regression: Regression methods associated with a linear model/non-linear model (accordingly called as Linear Non-Parametric / Non- linear Non-Parametric), but the basic assumptions of regression including those associated with errors are not true. Am I right ? Is there an error or misleading idea here? Please respond.


title: Should I use t-test on highly skewed data ? Scientific proof, please?

text: I have samples from a highly skewed (looking like an exponential distribution) dataset about users' participation (e.g.: number of posts), that have different sizes (but not less than 200) and I want to compare their mean. For that, I'm using two-sample unpaired t-tests(and t-tests with the Welch’s factor, when the samples had different variances). As I have heard that, for really large samples, it doesn't matter that the sample are not normal distributed. Someone, reviewing what I've done, said that the tests I am using were not suitable for my data. They suggested to log-transform my samples before using the t-tests. I am a beginner, so it sounds really confusing to me to answer my research questions with "log of participation metric". Are they wrong? Am I wrong? If they are wrong, is there a book or scientific paper which I could cite/show them? If I am wrong, which test should I use?


title: AUC vs error rate for classification

text: I'm trying to build a recommendation system, and have a bunch of (item,item_features,liked) triplets, where liked is binary. Most items are not liked. So I'm running a logistic regression with glmnet of the form `liked ~ item_features` This yields an AUC of around 0.75 (it doesn't vary much with the regularization parameter). However, the error rate (also doesn't vary much) is only a tiny,tiny bit better than what you would get if you just always predicted "don't like." What is the best way (or any way, really!) to think about the value or lack thereof this recommender?


title: Minimum sample size to achieve a specific confidence level and interval

text: I am building software components that are going to be used by multiple users simultaneously. At the current point in time, these components are **known bad**. That is, if I simulate multiple users simultaneously interacting with each component **long enough** I know the component **will fail**. I cannot determine that a failure happened until the end of the simulation. The result of the simulation gives me two things:   1. A success or failure indicator, and   2. The actual number of times an error occurred (although I don't know when in the simulation the error actually happened) I also have no idea what kind of distribution the errors have because changing the application to record the failures at the time that they happened would likely change the distribution itself. My goal is to determine the smallest sample size I can use to achieve a target confidence level CL at a target confidence interval CI so that I can automatically have these components tested the necessary number of iterations. I'll then fix the components but leave the automated component tests to give me some confidence that the components aren't broken in the future. How can I determine the smallest number of iterations (samples) the simulator needs to run for in order to reproduce a failure at confidence level CL% given a confidence interval of CI%? Or, if I can't equate the number of iterations to samples, what approach might I take to achieve the above goal? If it helps, I can run the simulator any number of times at any number of iterations in order to have a sufficient sample size. As I do not know enough statistics to know exactly what questions to ask or what research to do, let me explain my current plan:   1. Run the simulator for X iterations   2. If the simulator does not fail, repeat step 1 with a larger X until the simulation fails.   3. (As the simulation failed at X iterations) Run the simulator 1000 times with X iterations to determine what I suspect would be something akin to the confidence level. (i.e., if it failed 950 of the 1000 times I would assume a confidence level of 95%.)   4. If the confidence level is not sufficiently high, repeat starting at step 1 one with a larger X (assuming I'll get a better confidence level). I doubt the above is mathematically sound and would like to fix that. It feels like the unknown distribution makes it impossible to determine the likelihood of a failure per some number of iterations. It also feels like I could use the actual number of times an error occurred within a simulation to my advantage - perhaps by calculating variance over repeated simulations. Data:               # iterations      # failed simulations / 1000 simulations     2^n (0 <= n <= 6) 0     128               662     256               837     512               916     1024              974     2048              734     4096              779     8192              891     16384             953     32768             968     65536             991     131072            999     2^n (n >= 18)     1000      As you can see, at 2048 iterations the simulation failed significantly fewer times than at 1024 iterations, which to me suggests some large variance. I'm fairly capable with general mathematics, but quite uninformed on the statistics front, so gearing answers in that direction would help.


title: How to interpret the results of the R pnorm and analogous functions?

text: The help for the pnorm function states: It says that pnorm gives the "distribution function", but it seems that it gives the quantile, for example, `pnorm(q = 0, 0, 1)` returns `0.5`, which suggests that `q=0` refers to the 50th quantile of a N(0,1). I understand what the "normal probability density function" is, but not why `pnorm` is called a 'distribution function'. The R help says that the `q` argument is a "vector of quantiles", but it appears in practice that `q` represents an observed value. What I want to know is, if I observe '2', what does `pnorm(2)` say about my assumption that it came from a N(0,1) distribution?


title: Competing risk data

text: I want to simulate survival time for two competitive events using `R`. Consider that there are no censored data, and the survival time for each event has exponential distribution with rate 0.25 and 0.1 respectively. Can anyone help me please?


title: What test should be used to compare several regression lines?

text: I want to determine if there are significant differences between movement and temperature of three groups. I have four regression lines and I want to know if there are significant differences in the slopes. What test do you use to do this? I had read about the ANCOVA, but recently someone suggested I not use it.


title: To what extent can statistics improve patient's treatment?

text: I was requested by a friend's mother to analyze her data from her past treatment. Since she got leukemia two years ago, she had been instutionlized for various periods and underwent various treatments. So I can download a data chart with time points and measurement after medicine A, B,C,D, E, etc. However the measurements were not on a level enable me to comparing them with one another via a simple graph, but more like heart rate and blood pressure which must be somewhat related, however having a different mean and standard deviation. With 5-6 medicine at hand and 2 years data available to me, I could not really analyze the data properly as I am a mathematican without proper training in statistics. But now I want to ask about this in general, because the friend had died from the leukemia, and I wish I could have been making some difference, despite the fact that there is absolutely no cure for her type. The question I want to ask out of this personal experience is: to what extent can we help (real) patients via statistical methods? How can we manage to extract the hidden relation behind a not so large discrete set of data (300-600 data points, say) and have reasonable confidence this might mean something? Suppose we had all the programming power available and money issue is not really a concern, is there any real hope we can garner personal data to help a patient on an indivudal level? I was not a statistican, and I tried linear regression between the variables, which failed miserably as there seemed to be no obvious relationship between them. So I want to ask a professional's viewpoint on this. (I had read at least 100+ leukemia papers to try find something that might help either biologically or via mathematical biology like population dynamics, but I failed) Update: I forgot to write that metamed (a medical start-up company) does offer individual tailored medical analysis, seems mostly based on patients' genetic data. They charge an extremely high price. However I do not know if their service is reliable. In my personal case, my friends' family decided against seeking their help. I do not really know if this would make any difference, for in her case all the therapeutic (experimental or well-established) methods has been exhausted. Since this thread may be of interest to others, I decided to include this 25c at here.


title: Can I test the effect of Block in a split plot design in R?

text: I have been confused with the problem for a very long time, and hope that somebody here can help me out~ I have a experiment installed in split-plot design, with 3 temperal groups as blocks, and 2 factors (each have 3 fixed levels) combinations in each of the block, and I have 4 replicates in each treatments. I used to build the model like this:               model<-aov(Var~Block+A*B+Error(Block/A/B),data)      where A and B are fixed factors of my focus. It seems that R will calculate the Block as an error term, where only DF, SS and MS were reported, but not F-ratio and p value. My question is: is it make sense for me to estimate the effect of block(i.e. different time period) in R? Acutally, the 4 replication were sampled in four continuous days to develop a temporal sequence for another analysis. I would like to know how could I test the sampling effect, is it another block, or could be treated as a fixed factor nested within the treatment? Thanks in advance~~ Marco


title: Measure of closeness

text: Given a list of numbers, is it possible to find out (or in other words, is there a statistical measure to tells the) the closeness of the numbers (do note that i am not talking about correlation - this would be for 2 sequences - something like correlation between height and weight). I am looking for something like a closeness coefficient for a given series of numbers, so given a series [0,10,20,30,40] - the 'closeness coefficient' gives me the spread of the numbers. It would also be nice if the 'closeness coefficient' depicts the 'density' of the numbers - but if this is a different computable statistical measure, then it shouldnt be a problem.


title: What logistic regression is best to use?

text: My project is about whether early language scores (on a reading task) can predict later scores on a social attribution task. I have to conduct a logistic regression for my project but I am stuck on which one is appropriate. This is because I have conducted a binary logistic and multinomial logistic regression but I always get warnings and I do not know what to do. Any help will be much appreciated. It's just that my supervisor specifically instructed me to use a logistic regression, sorry I forgot to add that there are three groups involved autism, language impaired and typically developing kids and I need to look at whether langauge scores of the autistic kids and language impaired kids are also a signifcant predictor of scoring low on a social attribution task in later life. Please can you help with this??


title: How to perform power analysis using simulation in Stata

text: I often hear from statistical experts (including on this site) that simulation is the preferred method to calculate power for a clinical trial (rather than using traditional sample size calculation formulae). I use Stata, and I was wondering if anyone can show me how to do this **in detail**? For example, two scenarios: 1) **Continuous variable:** A new drug given intraoperatively is hypothesized to result in a reduction in the mean troponin release from 2.2 to 1.8 (SD of both 2.0) after cardiac surgery, compared to placebo. 2) **Proportions:** A sexual educational intervention is hypothesized to result in a reduction in the incidence of HIV transmission amongst teenagers from 20% to 15%. Assuming a power of 80% and an alpha of 5%, in Stata, I would normally type the following: **1)**               sampsi 2.2 1.8, sd(2) power(0.8)          Estimated sample size for two-sample comparison of means          Test Ho: m1 = m2, where m1 is the mean in population 1                         and m2 is the mean in population 2     Assumptions:               alpha =   0.0500  (two-sided)          power =   0.8000             m1 =      2.2             m2 =      1.8            sd1 =        2            sd2 =        2          n2/n1 =     1.00          Estimated required sample sizes:             n1 =      393             n2 =      393      **and for 2) I would type:**                sampsi 0.2 0.15, power(0.8)          Estimated sample size for two-sample comparison of proportions          Test Ho: p1 = p2, where p1 is the proportion in population 1                         and p2 is the proportion in population 2     Assumptions:               alpha =   0.0500  (two-sided)          power =   0.8000             p1 =   0.2000             p2 =   0.1500          n2/n1 =   1.00          Estimated required sample sizes:                  n1 =      945             n2 =      945      Can anyone please show me how the simulation technique might look in Stata for these two examples? Many thanks!


title: Outlier detection in short time series with two seasonalities

text: I have short daily time series (less than 4 years) representing sales and exhibiting two seasonalities (weekly and yearly) and I am seeking to identify outliers (not only data reporting errors but also particular events impacting the sales). Because of the presence of a trend and seasonalities, I think detrending and deseasonalising data beforehand is necessary; so I was thinking of using decomposition models based on moving averages to get the irregular component and then use statistical tests (Z score methods, use of interquartile range...) or distance/density based algorithms but the results will be closely related to the quality of the decomposition (that depends itself on the number of outliers) and I am not even sure it will work. Or I was thinking of using the method exposed by Tsay to detect the different types of outliers (additive, innovational, level shifts...) using regARIMA models with dummy variables for festivals, trends and seasonalities (I hope I will have enough data for that). I have only free software (R) at my disposal. Which would be the best approach to adopt ? I would be vey grateful for any help.


title: Text categorization/classification for small scale text

text: I'm looking into a way to classify/categorize sentences into pre-defined categories (around 10-15). Yes, indeed sentences, not articles or paragraphs. Given the average length of articles are not too long (2-10 normal pages) and number of articles (tens-hundreds) are relatively small, it is quite a small scale problem. However, accuracy is much more important. Because I am quite new in this field, I started by looking into some introductory papers and few generic open source projects (e.g. WEKA, GATE and LingPipe ). However, what I found so far are pieces too hard to put together to fit my purpose. What specific algorithm/software/resource do you recommend on this problem? Thanks in advance. * * * I did find a tool called TagHelper which is quite suitable for few aspects of my purpose and am still exploring it. But I still expect more insights and suggestions. Thanks!


title: Binomial Distribution & t-Test

text: Starting with a Binomial Distribution with parameters $n=1000, p=0.5$ and measured successes of 300, I would like to test whether there is a significant difference between success and failure. The obvious solution (using R):               n1 <- 300     n2 <- 700     p <- 0.5     binom.test(n1, n1+n2, p, alternative='two.sided')      I also want to use the similarity of the Binomial and Normal Distribution for "large" numbers of observations. A trivial solution may be this:               t.test(c(rep(0, n1), rep(1, n2)), mu=p, alternative='two.sided')      Properties of a Normal Distribution based on a Binomial Distribution can be calculated directly:               mu <- (n1+n2)*p     sig2 <- p*(1-p)*(n1+n2)      Therefore it should be possible to simply apply a one-sample t-test. After some trial-and-error I got this solution:               t <- (n2-mu)/sqrt(sig2)     p.value <- 2*abs(1-pt(abs(t), n1+n2-1))      Luckily the results are rather similar. I do not understand why the t-test stated for example in Wikipedia, where an additional $\sqrt{n_1+n_2}$ is used, does not produce the right result:               t.wrong <- sqrt(n1+n2)*(n2-mu)/sqrt(sig2)      Why do I have to omit this part of the tests formula?


title: How to analyse Multiple Imputation data with SPSS?

text: I have to work on a dataset treated with the Multiple Imputation method to handle missing data. In fact, I have 5 different variant of the same dataset, with missing data replaced by probable values. How can I analyses those datasets? Should I merge them? Thank you for your time. Best, cml.


title: Any alternatives for plotting (long) time series?

text: I'm looking for ways to visualize large time-series other then in traditional 2-axis line charts. More specifically, I hope to find a way that intuitively reveals patterns in the data such as the spiral approach found in http://ieg.ifs.tuwien.ac.at/~aigner/teaching/ws06/infovis_ue/papers/spiralgraph_weber01visualizing.pdf In my case, since I'm working with sensor data to track ones movements, days are a suitable interval. Another benefit of using spirals is that it doesn't require a linear increase in pixel space. Other ideas: Overlaying data from multiple days on the same 24-hour x-axis or displaying the days individually, similar to a calendar. Is there a better way? Any tips?


title: Feature selection using caret + repeatedcv

text: I am using `caret` and `repeatedcv` with repeats for feature selection. That is,               rfeControl(functions = svmFuncs, method = "repeatedcv", number = 10, repeats = 5,                  rerank = TRUE, returnResamp = "all", saveDetails = FALSE, verbose = TRUE)      I am quite confused about the way `rfeControl` splits the input data using repetition. In general, if I am not mistaken, the most unbiased way of assessing the performance of the model is to:   1. iteratively create 2 subsets (test and training set)   2. do the validation to the training set (i.e. cross validation) and select the most significant predictors    3. assess the performance with the unknown test set In case of `rfeControl` with repeated-cv and repetition, the repetition is applied from (1) or during the validation process (2)?


title: Using sequential observations to perform online prediction

text: I'm trying to perform predictions from a sequence of events. My problem is this: **Data collection:** Suppose you can continuously observe a person sitting in a library. You take note of every time that the person checks his smartphone, goes to the restroom, or is reading. At some point after several hours, the person then gets up and leaves. Assume that you can do this for several days to get a lot of data. **Problems to be solved:** Now, suppose I can perform observations only at sparse discrete points in time (e.g. once every 10 or 20 minutes). I would like to perform online (in real- time, with a short window) the following two computations: Given a sequence of sparse online observations of either (a) using smartphone, (b) going to restroom, or (c) reading, I would like to know the probability that the user will next be reading for more than t minutes. For example, if the user has a habit of checking his smartphone and then starts reading uninterrupted for 20 minutes, then if I make an online observation that he is indeed checking his smartphone, then I would say that the probability he will read for more than 10 minutes is, say, 0.90. I would as few needed observations as possible. If I want to determine when the user will get up and leave within the next t minutes with some threshold probability, how would I schedule the minimal number of needed observations? I have been reading up on Markov chains, HMMs, and random walks, but my problem seems to be a bit different from what I've read. Any help would be appreciated.


title: 3-4-5 Rule How to partition the sets?

text: In Data Mining course, we are taking 3-4-5 Rule to segments the data uniformly. I'm trying to understand these lines below, and how they are linked to the graph below too. > If an interval covers 3, 6, 7 or 9 distinct values at the most significant > digit, partition the range into 3 equi-width intervals > > If it covers 2, 4, or 8 distinct values at the most significant digit, > partition the range into 4 intervals > > If it covers 1, 5, or 10 distinct values at the most significant digit, > partition the range into 5 intervals Why did we partitioned step 3 into 3 intervals and step 4 to 4 intervals? The graph: ![3-4-5 Rule Example](http://i.stack.imgur.com/iFF7P.png) Would really appreciate it if someone could help explaining my question above. Thanks!


title: Mean of means by a factor versus global mean

text: I would pretty much appreciate if someone could explain the discrepancy detailed below. Let's do a theoretical test first.               # make some test data with three levels and a vector of random numbers     test <- data.frame(ind = gl(3, 10), val = runif(30))     # calculate mean by "ind" (stored to a variable for pedagogic purposes     ind.mean <- with(test, tapply(X = val, INDEX = ind, FUN = mean))     # calculating mean of means by "ind" and a global mean yields same results - FINE     mean(ind.mean)     mean(test$val)      Here's for some tricky data.               rld <- structure(list(date = c("2011", "2011", "2011", "2011", "2011",      "2011", "2011", "2011", "2011", "2010", "2010", "2010", "2010",      "2010", "2010", "2010", "2010", "2010", "2010", "2010", "2010",      "2010", "2010", "2010", "2010", "2010", "2010", "2010", "2010",      "2010", "2010", "2010", "2010", "2010", "2010", "2009", "2009",      "2009", "2009", "2009", "2009", "2009", "2009", "2009", "2009",      "2009"), pm = c(1, 0.80952380952381, 0.75, 1, 0.866666666666667,      0.8, 1, 0.8, 0.833333333333333, 0.666666666666667, 0.666666666666667,      1, 1, 1, 1, 1, 1, 1, 1, 0.666666666666667, 0.75, 0.75, 1, 1,      0.833333333333333, 0.833333333333333, 0.714285714285714, 0.833333333333333,      0.818181818181818, 0.8, 0.666666666666667, 1, 0.5, 1, 1, 0.166666666666667,      0.166666666666667, 1, 1, 0.8, 0.5, 0.5, 0.25, 0.333333333333333,      0, 0.909090909090909)), .Names = c("date", "pm"), class = "data.frame", row.names = c(NA,      -46L))          > head(rld)       date        pm     1 2011 1.0000000     2 2011 0.8095238     3 2011 0.7500000     4 2011 1.0000000     5 2011 0.8666667     6 2011 0.8000000      One would expect the results to be identical (sensu above test case), alas...               > mean(rld$pm)     [1] 0.7822699     > mean(tapply(X = rld$pm, INDEX = rld$date, FUN = mean))     [1] 0.7500214      _EDIT_ Chase nailed it. Here's proof:               test3 <- data.frame(ind = rep(1:3, times = c(5, 15, 10)), val = runif(30))     > mean(test3$val)     [1] 0.4972464     > mean(tapply(X = test3$val, INDEX = test3$ind, FUN = mean))     [1] 0.468399


title: Testing regime change in SAS

text: I have two samples, A and B. I run the same model for A and B separately to obtain the estimates. I now want to test whether the parameters for models in A are the same with the parameters in B using SAS. Please advise.


title: What is the relation between two IIN mean zero random variables?

text: I have trouble proving the following fact in my econometrics homework. The lecturer said that I should merely look at my statistics books, but I cannot seem to find it anywhere! Thus, sorry if it is (too) ignorant a question. Suppose that random variables $\varepsilon_{1t}$, $\varepsilon_{2t} \sim IIN(0,\Sigma)$ (i.e. identically independently normally distributed with a vector of means equal to $0$ and a variance-covariance matrix $\Sigma$). How can I then show that $\varepsilon_{1t}=\lambda\varepsilon_{2t}+u_t$, where $\lambda = \frac {\sigma_{12}} {\sigma_{22}}$ and $Var(u_t)=\sigma_{11}-\frac{\sigma_{12}^2}{\sigma_{22}}$ and $u_t$ is a disturbance term? ($\sigma_{ij}$ denotes the corresponding element of the variance-covariance matrix). All help is greatly appreciated. :) * * * Clarification: I just wanted to add that this question comes from a time- series context. Thus, IIN means that the $\varepsilon$'s are independent over time (i.e. no autocorrelation) and that the distribution does not change. However, there is contemporaneous correlation between the $\varepsilon$'s as they come from a bivariate distribution.


title: Using standardized regression coefficients to measure relative contribution

text: I know I can calculate partial or semi-partial correlations to measure the relative contribution of multiple independent variables to the variation in a dependent variable. However, if I only have access to standardized regression coefficients, how can I use these to measure the relative contribution of each independent variable? Would it be sufficient to square the regression coefficients, and then claim that this accounts for their share of the variation in the independent variable? Ideally I'd like to get to the point where I can claim that "IV X explains Z% of the variation in IV Y".


title: When is a sample proportion p hat instead of x bar

text: I just started my first statistics class and am not majoring in statistics so sorry if this sounds like a beginner question and also sorry if my language is incorrect. (feel free to correct me.) I have been learning about creating sample distributions of phat and also sample distributions of xbar. I was wondering if you can tell the difference between the two just by looking at a mean, standard deviation and sample size. I have two examples from my class one is a sample distribution of phat and the other a sample distribution of xbar ## First example using the sample distribution of xbar Aamco Heating and Cooling, Inc., advertises that any customer buying an air conditioner during the first 16 days of July will receive a 25 percent discount if the average high temperature for this 16 day period is more than 5 degrees above normal. Daily high temperatures in July are normally distributed with a mean of 84 degrees and a standard deviation of 8 degrees. If we consider the first 16 days of July to be a random sample, what are the expected value, standard deviation, and shape of the sampling distribution of the sample mean? (don't answer this question it's just here to show the question in context.) ## And now the second using the sample distribution of phat Assume that 30% of all business students at a university invest in the stock market. We randomly pick 500 students Show the sampling distribution of phat, the sample proportion of business students at this university who invest in the stock market. (Yet again no need to do this just giving context.) So yet again I'm just asking if there is a way to tell if I need to use the equations for xbar or for phat when given a mean, standard deviation, and sample size and asked to give a sampling distribution. (And yes I know the second example says give the sampling distribution of p-hat, but I want to know if there is a way to tell if it didn't say that.) Thanks and sorry again if this is a bad question.


title: Logistic growth inflection point

text: I have a logistic growth curve as follows: $y = \frac{1}{(1 + ae^{-bx})}$, where x is the independent measure (x-axis) and a and b are paramaters. The inflection point of this equation is when y = 0.5. Given that information, I need to find the x-value at y = 0.5. But this equation isn't easy to solve. I'm using the statistical language R to find these fits. So, given the parameters a and b, how can I find the x-value at y = 0.5 (and again, I can't do things like derivatives because I am using a programming language). By the way, I did try to solve for x, but it's quite ugly and I'm not sure how to do it...


title: If I have a table of conditional probability values can I find any joint distribution value?

text: If my table is $(KV)^2$ where $K$ is the number of states and $V$ is the number of variables. Thus, I have the conditional probability for any 2 variables. Is it possible to find the joint distribution for any of the variables?


title: What are differences between covariate and covariance ?

text: The terms covariate and covariance appear to mean one and same thing. Is covariance equal to covariate?


title: How to determine the best relationship (linear, log, etc.) between input predictor variable(s) and output variable for multiple linear regression?

text: I am trying to determine the most accurate relationship between two variables (each predictor versus the output eventually). I want to know if the relationship is linear, or log-linear, or log-log, or some other form. What is the systematic approach to determining the underlying relationship that isn't full-out obvious and sometimes seens non-existant. In some regression models I have seen, the author has had 10 predictors, and then changed the whole function from linear to log and ended up doubling the predictability (R^2 from ~40% to 80%). I'm fully aware that R^2 or adjusted R^2 isn't the best for determining the relationship but it is just for example. Thanks in advance!


title: How to test reliability of an index when the index is composed of distinct facets?

text: I am measuring a concept (such as 'friendship') that is multidimensional in nature. To capture the multidimensional nature of the concept, I have measured the concept through several aspects, each of which focus on one part (or dimension) of the concept. Each aspect has several Likert scale items, which I have added together to arrive at sub scores for the aspect. I have then added all the sub scores to arrive at the overall score for the concept. For example, if the total of the Likert scale items for Aspect A is 6, Aspect B is 3 and Aspect C is 4, then the overall score for the concept (which is made up of A, B and C) is 13. In this way, the overall score is a single number, which is a unidimensional index. My focus of interest is the overall score (not the sub scores for each aspect). How do I test the reliability of my index? Would Cronbach's alpha be appropriate in this case?


title: Is R's trimmed means function biased?

text: I've been trying to understand how R's trimmed mean function works. I suspect it might be biased, but would like to get feedback here before I file a bug- report (if this is an inappropriate forum for such a question please let me know). Consider the following example (sorted for clarity):               > x <- c(1, 2, 3, 45, 55, 56, 57, 58, 403, 900)     > length(x)     [1] 10     > percent.to.trim <- 0.25      One would expect that the trimmed mean would be based on `length(x) - 2 * length(x) * percent.to.trim = 5` observations. If this is the case, there are three possible methods of calculation:   1. To remove the two smallest observations and the three largest observations. In this case, `mean(x[3:7]) = 43.2`.   2. To remove the three smallest observations and the two largest observations. In this case, `mean(x[4:8]) = 54.2`.   3. To remove the two smallest observations, the two largest observations and to average the third smallest and third largest observation (this is to be preferred for obvious reasons). In this case `mean(c(x[4:7], mean(c(x[3], x[8])))) = 48.7`. However, R gives the following result:               > mean(x, trim=percent.to.trim)     [1] 45.66667      This must result from the trimmed mean being based on `length(x) - 2 * floor(length(x) * percent.to.trim) = 6` observations, thus removing only the first two and last two observations:               > mean(x[3:8])     [1] 45.66667      Is this a bug? I'm using _R version 2.13.1 (2011-07-08), svn rev 56322_.


title: When is the autocorrelation function of a stationary process decreasing/nonincreasing? Markovian?

text: When is the autocorrelation function of a stationary process strictly decreasing or nonincreasing? Can being Markovian make it true? When is the autocorrelation function of a stationary process (strictly) increasing? Thanks!


title: Understanding Latent Dirichlet Allocation Inference

text: I'm reading the wikipedia page about how Latent Dirichlet Allocation assigns a topic distribution to a document after the model's been learnt (see this link). I'm very confused by this part of it: > Let $n_{j,r}^i$ be the number of word tokens in the $j^{th}$ document with > the same word symbol (the $r^{th}$ word in the vocabulary) assigned to the > $i^{th}$ topic. So, $n_{j,r}^i$ is three dimensional. If any of the three > dimensions is not limited to a specific value, we use a parenthesized point > $(\cdot)$ to denote. For example, $n_{j,(\cdot)}^i$ denotes the number of > word tokens in the $j^{th}$ document assigned to the $i^{th}$ topic. Could anyone explain this in simpler terms? Thank you!


title: Formula for probability of getting consecutive tails in X coin flips

text: I need to know the chance that I will get consecutive tails (i.e., 2 tails in a row on successive coin flips) at some point in $X$ number of coin tosses. For example, if I toss the coin twice, I know that I have a 25% chance. What I am really looking for is a formula to solve this question.


title: Using squashing functions to impose finite bounds on data

text: I have data that essentially correspond to Euclidean distances in some space. More specifically, I have dimensionality-reduced user data (quantified by their behavior), and a distance matrix in this dimensionality-reduced space from user to user. I want to come up with a 'Similarity Percentage', to determine that, say, User A is 85% similar to User B. I then want to analyze the distribution of 'Similarity Percentage's for a particular user. If I pass these Euclidean distances through some sort of squashing function like the sigmoid, could I then model these squashed distances as a distribution on [0,1]? Basically what I'm asking is, what information would I lose (if any) by transforming my data through the sigmoid function?


title: Efficient parametrization of the covariance matrix with some covariances constrained to zero

text: I'm trying to estimate the unknown 8x8 covariance matrix X in R using the maximum likehood, but I have problems of figuring out the efficient way of parametrization of X when some of the covariances in X are constrained to zero. When there's no constraints, I have used a cholesky factorization, ie. I have parameters which correspond to the lower triangular matrix L, and I get X by X<-t(L)%*%L which is a proper positive definite matrix, but how to do it now that I want some of the cells of X to be constrained to zero? In my case, X is constrained in a way that the upper left and lower right 4x4 matrices can be anything (but of course having the properties of proper covariance matrices), and the off-diagonal 4x4 matrix is a diagonal matrix, ie. the the first variable x1 correlates with variables x2,x3,x4 and x5, variable x2 correlates with x1,x3,x4 and x6 etc. Thanks.


title: How can I determine which of two complementary hypotheses should be the null?

text: How can you determine the direction of the test by looking at a pair of hypotheses? How can you tell which direction (or no direction) to make the hypothesis by looking at the problem statement (research question)?


title: Machine learning book with code examples

text: I am studying Machine Learning and implementing ML algorithms with Matlab. I follow Ethem Alpaydin, Duda and Hart, Bishop and Mitchell's books. However none of them exactly give the pseudo codes and algorithm based examples and code snippets as usual. Are there any books which follow topics with code examples with Matlab or Java or Python?


title: Analyze proportions

text: I have a dataset containing multiple proportions that add up to 1. I am interested in the change of these proportions along a gradient (see below for example data).               gradient <- 1:99     A1 <- gradient * 0.005     A2 <- gradient * 0.004     A3 <- 1 - (A1 + A2)          df <- data.frame(gradient = gradient,                      A1 = A1,                      A2 = A2,                      A3 = A3)          require(ggplot2)     require(reshape2)     dfm <- melt(df, id = "gradient")     ggplot(dfm, aes(x = gradient, y = value, fill = variable)) +       geom_area()      ![enter image description here](http://i.stack.imgur.com/aBbHc.png) **Additional information:** It need not be necessarily linear, I did this just for easiness of the example. The original counts from which these proportions are calculated are also available. The real dataset contains more variable adding up to 1 (eg. B1, B2 & B3, C1 to C4, etc) - so a hint for a multivariate solution is would be also helpful... But for now I'll stick on the univariate side of statistics. **Question:** How can one analyze such kind of data? I´ve read a little bit around, and perhaps a multinomial model or a glm is suited? - If I run 3 (or 2) glms, how can I incorporate the constraint that the predicted values sum up to 1? I dont want to only plot such kind of data, I also want to do a deeper regression like analysis. I preferably want to use R - how can i do this in R?


title: Why dummy variables rather than one "factor" variable in modelling?

text: I was speaking to a statistician recently who suggested that using dummy variables rather than one variable with several levels reduced the constraints on models, particular reducing the assumption of linearity. I didn't understand the explanation and was wondering if someone could make it clear?


title: linear mixed models differences between groups

text: I should test if the differences between 5 groups are statistically significant. I should use linear mixed models with restricted maximum likelihood. Group and time should be my fixed factors and the value of variable x the dependent variable. Can you suggest me a tutorial or examples? if using Matlab it would be cool! Thanks


title: Convergence time of a Markov chain

text: We know that a regular Markov chains converges to a unique matrix. The convergence time maybe finite or infinite. My interest is in the case where the convergence time is finite. How can we accurately determine this time or in other words the number of transitions for convergence? I am interested to go through the relevant materials needed to determine different methods that maybe out there to calculate this. If anyone could suggest a good reference to start off with I would be delighted. Any assistance will be welcomed.


title: Would a sorted class harm a 10-split cross validation?

text: I am using the Java API of Weka to apply a Naive Bayes classification to an .arff file I've created. The (@data part) file has the following format:               0 0 0 0 1 0 ... 0     1 0 0 0 0 1 ... 0     .     .     .     0 0 1 0 0 1 ... 3      .     .     .     0 0 0 0 1 0 ... 5      Where each number belongs to [0,1], except the last one which is the class [0,5]. Considering that I am using a 10-split cross validation, would it be a mistake to have my data in a sorted by class form? Would that lead in taking the test data from the last class only for example? I am getting the following confusion matrix, which is obviously wrong:               0.0 |0.0 |0.0 |0.0  |0.0  |0.0|       0.0 |0.0 |0.0 |0.0  |0.0  |0.0|          0.0 |0.0 |0.0 |0.0  |0.0  |0.0|        0.0 |0.0 |0.0 |0.0  |0.0  |0.0|       42.0|14.0|15.0|114.0|233.0|7.0|       71.0|16.0|30.0|241.0|86.0 |7.0|      Any ideas why the first 4 classes are only zeroes? My .arff file has examples (not so evenly distributed) from all 6 classes. EDIT: I shuffled my data and now I am getting a much more rational result.               27.0|2.0|8.0 |24.0 |27.0 |2.0|     4.0 |2.0|0.0 |2.0  |1.0  |0.0|     6.0 |3.0|15.0|16.0 |19.0 |3.0|     29.0|4.0|13.0|326.0|87.0 |33.0|     20.0|5.0|7.0 |37.0 |110.0|6.0|     5.0 |0.0|1.0 |17.0 |8.0  |7.0|      I am using the code I found here. Is there anything wrong with it?


title: Bias of the maximum likelihood estimator of an exponential distribution

text: The maximum likelihood estimator of an exponential distribution $f(x, \lambda) = \lambda e^{-\lambda x}$ is $\lambda_{MLE} = \frac {n} {\sum x_i}$; I know how to derive that by find the derivative of the log likelihood and setting equal to zero. I then read in an online article that "Unfortunately this estimator is clearly biased since $<\sum_i x_i>$ is indeed $1/\lambda$ but $<1/\sum_i x_i > \neq \lambda$." Why does $<\sum_i x_i> = 1/\lambda$? If I am correct in deducing the $< >$ operator means expected value, then I thought $E(x_i) = 1/\lambda$ - that is, the expected value of one such $x_i$, is $1/\lambda$, not the sum of all $x_i$'s. And can someone explain the second of the statement and how these two statements demonstrate the MLE is biased?


title: Bivariate normal probabilities

text: Let $(X,Y)\sim N(\mu_x=1,\mu_y=1,\sigma^2_x=4,\sigma^2_y=1,\rho=1/2)$. Compute $P(X+2Y\leq 4)$. How do you compute probabilities of a bivariate normal? For a regular normal distribution I remember we had to use tables or software because there is no close form solution for the normal CDF.


title: Problem with singular covariance matrices when doing Gaussian process regression

text: I'm working with Gaussian process regression. Currently I start testing different covariance functions and compositions to see what type of data they could describe best. I made an own implementation in Java. **My problem:** Most of the covariance functions I use result in a singular covariance matrix which is not invertible.   1. Shouldn't the proposed covariance functions/estimators produce only invertible matrices?   2. Are there methods or hints for regularizing the matrices? Or can that be done by using other values or ranges as inputs? May the introduction of error terms would help as well? Most problems I get with integer $x$ inputs to the Brownian motion covariance function $k(x,x') = \min(x,x')$. When I am using this the matrix it is always singular.


title: Non-parametric discriminant analysis in R

text: I want to use Discriminant Analysis between two non normal populations in R. Can anybody tell me the name of the R function to do so? Could also anybody tell me how accurate my results will be if I violate the normality assumption?


title: How to translate the results from lm() to an equation?

text: We can use `lm()` to predict a value, but we still need the equation of the result formula in some cases. For example, add the equation to plots.


title: Get overlap between two normally distributed variables

text: I have three variables ($X_G$,$X_B$, and $X_R$), each of which is normally distributed. Is there a way to show the overlap between each of the pairs using the mean and the standard deviation? An independent t-test shows that the three groups are very much significantly different, which is good, but I want a way to show that one pair is _most_ different (while two other pairs are _equally different_ ) To show what I mean graphically I've plotted the distributions below; ![enter image description here](http://i.stack.imgur.com/xyuE2.jpg) i.e. red and green are very different, while blue and green are _less_ different. Is there a good way to quantify this difference in overlap? The t-statistic for red and blue is 16.6789, while the t-statistic for green and red is 38.9686. I do find this kind of odd, given how much overlap red and blue seem to have though... What I'd like is to be able to directly calculate the information presented in the nomogram here. However, I guess I'd thought there would be standard implementations in matlab/R to do this, but couldn't find any.


title: Why is my manipulation check not correlated with my outcome even though the manipulation produce a significant effect?

text: I have a stats conundrum I have an MSc thesis in which two groups were manipulated to either include others in their group, or manipulated to exclude others (c1 inclusive and c2 exclusive). My dependent variable is how much income tax the participant would be willing to pay for the other group, with the prediction from the literature that when primed inclusively, people would pay a higher amount of tax for them (we pay for our own). A one-way ANOVA did find a significant difference between the groups. Also, there is a significant difference (in the same direction) between groups in how much they identify with the group they need to pay for (superordinate identity, which is supposed to reflect the manipulation). At this point I got excited. However, there is no significant correlation between superordinate identity and my DV (how much income tax willing to pay). I’m really not sure what this means? Does this mean my manipulation isn't effective? I took measures for other things which may be important to the DV. Many of these were strongly significantly correlated with the DV. For example, attitude towards paying taxes (makes sense), attitudes towards the other group (makes sense), how much of a threat the other group is (results didn't make much sense in context). However, there was no significant difference in these measurements between the two groups, showing that the manipulation had no significant impact on these factors. I attempted a multiple regression to see which predictors best predicted the DV and these other strongly correlated predictors were included in the model, but not my predictor of interest (superordinate identity). Now I really don’t know what this means.   * How is there only a significant difference between groups on the DV and the scale which reflects the manipulation (superordinate identity) but no significant differences between groups on the other measurements?   * Additionally, why is my superordinate identity scale (reflecting the manipulation) finding a significant difference between groups, but is not correlated with the DV or included in a MR model to predict the DV?  I spent a restless night with it going round in my head, over and over, but still cannot understand what has happened. I'm petrified that my whole project might be a waste of time.


title: Identifying a trend in binomial experiments

text: I have a survey running each month that's answered each time by a random set of people. I want to investigate the trend in a particular set of questions to see whether opinions are changing over time. This is a fairly standard survey research question but as far as I know, the usual approach is just to sig test each successive pair of results. But that doesn't incorporate the long term movement. Is there a better way of treating the data, other than multiple binomial Sig tests, to identify change over time.


title: Canonical correlation analysis on a MICE data set

text: I am looking to do a canonical correlations analysis (CCA) in R, using the `CCA` package, on a multiply imputed dataset (obtained from the mice package). I know that the `mice` package allows you to pool the complete data sets and do linear models and general linear models using formulas, e.g.:               mice.data <- mice(data, m = 5)     mice.lm <- with(data = mice.data, exp = lm(y ~ x + ...))     pooled.mice.lm <- pool(mice.lm)      but I don't think it has the ability to do that for CCA. The reason is that the `pool` function requires a variance-covariance matrix, which CCA does not compute. **Question:**   1. What is the best way to pool my data sets/results? Is it valid to do a CCA on each complete data set separately, then average the canonical variables for each imputation to obtain a final set of canonical variables?    2. Or is there some way to do this all at once with a long-format `complete()` dataset that contains separate .id and .imp variables? E.g. for Question 1: Vars 2-5 are the predictor set, and vars 6-15 are the criterion set; the merged data that includes both are imputed together to produce mice.data.               mice.data <- mice(data, m = 5)     mice.data.list <- vector("list", 5)     for (i in 1:5) {       m.data <- complete(mice.data, i)       predictors <- m.data[, 2:5]       criterions <- m.data[, 6:15]       mice.data.list[[i]] <- cc(predictors, criterions)     }      So I end up with a list of canonical correlations on each complete imputed dataset, which I then have to figure out how to average/pool together. Alternatively, for Question 2, I'd have a single data frame with a separate .imp variable specifying the number of imputation for that dataset, but I'd still have to figure out how to pool those results together, potentially with a clever aggregation? Perhaps this isn't possible, actually.


title: Are decision trees almost always binary trees?

text: Nearly every decision tree example I've come across happens to be a binary tree. Is this pretty much universal? Do most of the standard algorithms (C4.5, CART, etc.) only support binary trees? From what I gather, CHAID is not limited to binary trees, but that seems to be an exception. A two-way split followed by another two-way split on one of the children is not the same thing as a single three-way split. This might be an academic point, but I'm trying to make sure I understand the most common use-cases.


title: Expected Value and Variance of Estimation of Slope Parameter $\beta_1$ in Simple Linear Regression

text: I am reading a text, "Probability and Statistics" by Devore. I am looking at 2 items on page 740: the expected value and variance of the estimation of $\beta_1$, which is the slope parameter in the linear regression $Y_i = \beta_0 + \beta_1 X_i + \epsilon_i$. $\epsilon_i$ is a Gaussian($\mu = 0, variance=\sigma^2$) random variable and the $\epsilon_i$ are independent. The estimate of $\beta_1$ can be expressed as: $\hat{\beta_1} = \frac{\sum (x_i - \bar{x}) (Y_i - \bar{Y})}{\sum(x_i-\bar{x})^2} = \frac{\sum (x_i - \bar{x})Y_i}{S_{xx}}$, where $S_{xx} = \sum (x_i - \bar{x})^2$. So, my question is: how do I derive $E(\hat{\beta_1})$ and $Var(\hat{\beta_1})$? The book has already given the results: $E(\hat{\beta_1}) = \beta_1$ and $Var(\hat{\beta_1}) = \frac{\sigma^2}{S_xx}$. My work in the derivation: $E\left(\frac{\sum (x_i - \bar{x})Y_i}{S_{xx}}\right) = E\left(\frac{\sum (x_i - \bar{x})(\beta_0 + \beta_1 x_i + \epsilon)}{S_{xx}}\right) = E\left(\frac{\sum (x_i - \bar{x})\beta_1 x_i}{S_{xx}}\right)$, since $\sum(x_i - \bar{x})c = 0$ and $E(c\epsilon) = 0$. But I am stuck. Also, $Var\left(\frac{\sum (x_i - \bar{x})Y_i}{S_{xx}}\right) = Var\left(\frac{\sum (x_i - \bar{x})(\beta_0 + \beta_1 x_i + \epsilon)}{S_{xx}}\right) = Var\left(\frac{\sum (x_i - \bar{x})\epsilon}{S_{xx}}\right) = Var\left(\frac{\sum (x_i - \bar{x})}{S_{xx}}\right) \sigma^2$. But I am stuck. Any help would be appreciated. Thanks.


title: How to use Gaussian mixture model for multivariate pattern classification

text: I am new to statistical pattern recognition and trying to learn.To begin with I am trying to work with two class problems and trying to classify motion activities as mentioned in the paper "Object Trajectory-Based Activity Classification and Recognition using Hidden Markov Models". In this paper, the authors have used GMM for estimating the PDF for each motion pattern class. But I have several doubts and shall be grateful for the following answers   1. If there are 5 experimental data with 4 feature vectors(4 columns of data) of length 1000 samples/rows for each of the two classes(say,running and walking) then is the pdf from Gaussian mixture model obtained for each of the 5 experimental data or only for each class? If it is for each class, then how is it done? I really do not know and please pardon if this sounds too trivial.   2. Is it always necessary to find the pdf of the data before classification task in general? Does the pdf control in deciding which model based approach to choose for classification purpose?    3. Are there any such model based approaches for classification?   4. As far as my understanding goes, GMM is a clustering algorithm like k-means. So, are there other ways other than using Hidden Markov Model (I want to avoid HMM due to further complexities in understanding it) for classification with GMM?   5. Can GMM be combined with regression models like AR,MA,ARMA for model based classification? If so, then pointers to resources which explain this shall be helpful.   6. Is there a Matlab implementation of multivariate GMM for the said application of classification? Thank you in advance.


title: Refining a linear regression model for condominium prices

text: I'm hoping someone here is able to help me refine a linear regression model I'm working on at work. I am in no way a statistician, but I guess I have the most experience (basic stats course and decently capable with excel) in my office. I've been tasked with creating a model that would help predict condo prices (dependent variable) in a particular city. I've collected data from the Multiple Listing Service for use as my independent variables. The data I am collecting is from condos that have sold or are currently active within the last 6 months, and that are between 0 and 2 years old. The data is also limited to 4 storey wood-frame construction within a particular city. The independent variables I have used are: Square footage, top floor (dummy variable), corner unit (dummy), unit type (1 bed, 2 bed etc.), exposure (dummy, direction it faces), material spec (quality of finishings). I have since dropped exposure from the equation because it wasn't statistically significant (t stat was was around .3 - .4). All of the other coefficients have a t Stat over 2, however two of them are confusing me. The top floor and corner unit coefficients have a negative relationship when logically they should have a positive one. In my experience, top floor and corner units hold a premium over lower level and inside units. Does anyone have any idea why this could be? I have around 40 samples so far, would expanding my data set to include more samples help fix this? Also, I understand real estate prices can be a tricky thing to model because of subjective variables that can't really be accounted for. Anyways, any help would be appreciated as I am trying to learn about regression as I work on this! Sincerely, Rob


title: Estimating the distribution of a very large population of known size and unknown variance

text: I would like to estimate the distribution of a very large population of known size but unknown mean and variance. I cannot assume anything about the shape of the underlying distribution. However, I am certain that the values in the population are non-negative, non- zero integers. I believe that the distribution is naturally lumped in some way discretely rather than being continuously distributed over the entire range. I cannot sample the entire population but I would like to estimate the probability density function. I would also like some level of assurance of the correctness of the estimated distribution. What is the best way to go about this?


title: Relationship between average and characteristic function of a Gaussian process

text: I'm having trouble understanding an equality given in a book ("Speckle Phenomena in Optics" by Joseph Goodman p.145) for a zero mean, stationary Gaussian process: $\overline{\exp(i [\phi(x_1)-\phi(x_2)])}=\exp(-\sigma^2[1-\mu(x_1-x_2)])$ where $\sigma^2$ is the variance of the process $\phi$ and $\mu$ is the normalized autocorrelation of the process. Could anyone derive this equality for me or give me a reference that explains it? The author says: "from the relationship of the average to characteristic functions we have ..." Thanks!


title: Is it possible for a multinomial sample to be a single number?

text: I'm reading the Latent Dirichlet Allocation paper trying to understand it. However I got stuck at the very first part! When they sampled from a multinomial distribution and considered the result to be a single number! My understanding if the multinomial distribution is to return a vector where the number of its elements are the same as the number of probability vector parameter plugged into the multinomial. I'm not sure if I misunderstood the notation. **EDIT:** Consider Q is a probability vector of size K. Z must be a scalar from 1 to K. But I don't understand how is that achieved from the notation of Z ~ multi(Q) Here is the link to the paper (page 4): http://www.cs.princeton.edu/~blei/papers/BleiNgJordan2003.pdf Here is a picture: ![enter image description here](http://i.stack.imgur.com/Zk3DV.png)


title: Two questions on limitations of linear embedding techniques

text: Here's a quote from an article I read which is about non-linear embedding techniques: > This family of non-linear embedding techniques appeared as an alternative to > their linear counterparts which suffer of severe limitation when dealing > with real-world data: (i) they assume the data lie in an Euclidean space and > (ii) they may fail when the number of samples is too small. I don't understand the reasoning behind the two limitations mentioned above, why they fail when the number of samples is too small and why is the other limitation.


title: Cluster analysis, what to do with different scales?

text: I want to identify different groups of respondents out of up to five variables of the European Values Study 2008. At first I took 4 questions for cluster analysis all on a scale from 1 to 10. However, after performing a Two-Step cluster analysis the silhouette measure was close to zero and thus indicating a poor solution. Therefor I'd like to take some other variables. I was thinking about:   * how often discuss politics with friends [scale from 1(frequently) to 3(never)]   * describe your state of health these days [scale from 1(very good) to 5(very poor)]   * most people try to take advantage of you or try to be fair [scale from 1(most people would try to take advantage of me) to 5(most people would try to be fair)]   * Why are there people in this country who live in need? Which one reason do you consider to be most important? (four options: because they are unlucky; because of laziness and lack of willpower; because of injustice in our society; it’s an inevitable part of modern progress) My knowledge of cluster analysis is based on Marija J. Norušis PASW Statistics 18 Statistical Procedures Companion and whatever I can find on the internet. Unfortunately, I'm not able to find answers to these questions:   1. Should I recode the answers so that they are all on the same scale?   2. Is there anything I should do with the 4-option question? Thank you


title: How to smear a histogram

text: I was asked to perform a Gaussian smearing on the bins of an histogram. What does this mean?


title: Non-parametric test

text: I was given the following exercise: The Results obtained by 10 students in a test are the following: 72 95 79 83 93 80 91 74 70 86 Test the hypothesis that the mean score is 75.Use two test:one parametric and another non-parametric. My doubt is for the parametric test, should I make a t-student test? Because I don't have a big sample to use the normal distribution...


title: what's the advantages of bayesian version of linear regression, logistic regression etc

text: For many regression/classification algorithms, we have the bayesian version of it. Like bayesian linear regression, bayesian logistic regression, bayesian neuron network. I do not fully understand the math in them, but what are its advantages compared with the original algorithm? Is is of great practical use?


title: Independence test for values between 0 and 1

text: I want to compare a set of frequencies and determine the p-value of their dependence. One of them is the null distribution of frequencies and the other is the distribution that I want to test. The data look like this:               nucleotide   background    selected     1   0.1489113   0.06074766     2   0.1428619   0.04205607     3   0.1189465   0.63084112     4   0.1209048   0.05140187      5   0.1218093   0.07476636     6   0.1282073   0.04205607     7   0.2183589   0.09813084      I cannot use the $\chi^2$ test because the numbers are less than 1. I've tried to take the $\log_2$ of the ratio, but I end up with some negative numbers so the $\chi^2$ is a no go again. How can I conduct a test of independence with these values and compute a p-value? EDIT, additional informations: basically i'm working on DNA sequences. i have a background pool of sequences that act like a null distribution and from this background i have selected a number of sequences for a particular biological function. now, i wanted to compare the distribution of mutations in a small sequence of 7 nucleotides in the pool of selected sequences versus the background. To achieve this, i've taken all the instances where this sequence is present with one mutation in both pools and determined the frequency of mutation for each nucleotide (e.g. number of mutations at a given nucleotide / total number of mutations over all nucleotides), that is how i end up with those frequencies. now my aim was to show that the pattern of mutation is indeed much different in the selected pool with respect to the background (63% of the mutations in the selected pool happen in the 3rd nucleotide, and the other nucleotides undergo visibly less mutations than in the background) to provide further information, the selected column describes the frequency of 214 mutations over the 7 nucleotides while the background column describes the frequency of 120508 mutations over 7 nucleotides.


title: Dependent Bernoulli trials

text: The probability of a sequence of n independent Bernoulli trials can be easily expressed as $$p(x_1,...,x_n|p_1,...,p_n)=\prod_{i=1}^np_i^{x_i}(1-p_i)^{1-x_i}$$ but what if the trials are not independent? How would one express the probability to capture the dependence?


title: Co-efficient of correlation weighted method for spatial interpolation

text: Teegavarapu and Chandramouli (2005) has mentioned Coefficient of correlation method for spatial interpolation of moisture data that calculates the coefficients between a point and its neighbors and uses it for interpolation. R has many spatial packages across many levels (sp, geoR, gstat, automap). I have been able to find many methods including geostatistical methods implemented in R especially for cross validation. Does anyone know a package that implements above method and especially its cross validation version?


title: How to analyze ordered data type with past missing observations?

text: I am having trouble understanding what method to apply for the analysis of the following type of data:               id  period  grade   GLP NAB ROE ROA OSS ESS PMA P30     1   1   NA  1790835 769 0.3795033   0.1171393   1.66014 1.221868    0.3976413   0.007312334     1   2   NA  2597351 1143    0.5896583   0.188034    1.947474    1.748581    0.4865142   0.01906484     1   3   15  3359203 1343    0.474556    0.1559151   1.814949    1.712795    0.4490202   0.02821528     2   3   16  12271110    15960   0.05    0.01    1.04    1.04    0.04    0.02     2   1   NA  5392109 9290    -0.02   -0.01   0.96    0.7 -0.04   0.02     2   2   NA  11128069    1253    -0.03332067 -0.005665337    0.9908606   0.9871047   -0.00922374 0.01818852     3   2   NA  820903  2225    0.06835087  0.004860066 1.012239    0.9919102   0.01209093  0.072     ...     9   2   NA  652028  364 0.027   0.007   1.027   0.982   0.026   0.008     10  2   NA  31731236    251352  0.07033455  0.01930156  1.11506 1.069495    0.1031871   0.01566419     10  3   20  33016098    247747  0.0750367   0.02032998  1.124112    1.086046    0.1104093   0.01443777     10  1   NA  25151189    217818  0.03529102  0.01133722  1.059813    1.052836    0.0564377   0.02853068      They refer to credit ratings computed over three periods. The dependent variable is the grade (therefore ordered) and it does also depend on periods 1 and 2, but it not assigned until period 3, and therefore is missing. I thought to use ordered GEEs with a logit link, but in that case I would have either to put in the grade vector 0 or the period 3 values for each subject to analyze everything. Do you have any idea how to maintain the past observation without 'falsifying' the data? Help would be much appreciated!


title: What is deviance in lassoglm

text: I am trying to fit a lasso penalized logistic regression model to a certain data. I am using lassoglm for that in matlab. I use the following function [B,FitInfo] = lassoglm(X,Y,'binomial','Lambda',0.01,'CV',10); Using 10 fold cross validation, it will use different samples every time to fit the model. However, I didn't get what deviance means in this case. Lets say for the first run of cross validation, I have k non zero features. For next run, I have a model with m non zero features and so on. What does deviance then measure and how the plot is generated like using that lassoplot. Suggestion? Actually, I didn't get this figure. What does it specify? ![enter image description here](http://i.stack.imgur.com/DAKzv.png)


title: Calculating chance of event over time from sample data

text: I am doing the following experiment: I have slots which I monitor, the slot can either be occupied / free. The study is per slot, so we are discussing one slot here. Every now and then I receive the following indications for example: `s(2)=free`, which means that once a spot was free (after it was occupied) it remained free for at least 2 minutes. I collect all these data (the signals `s(2)`, etc.) into tables. Here is another example: `s(4)=occupy`, which means that this slot was free at some point and then 4 minutes after it was occupied. I want to answer the question: **what is the chance that a spot will remain free for X minutes?** In an attempt to answer this, I divided it into intervals (1-2 minutes, 2-3 minutes, ... , up to 10 minutes, which is the max). I then placed each sample into an interval with its value (free / occupy). For example, for the 3-4 minute interval, I have 120 free, and 60 occupy. So for this interval i(3)=120/120+60=0.66 = 66% that it is free in this interval. I know what i(1)=.. i(2)=.. etc., are as well. **How do I take into account the other interval data that I have?** Clearly the chances that the spot remains free in the next 3 minutes is not 66%, but is also based on other interval results.


title: Count data from unequal time frames

text: I have count data for events measured within each of multiple individuals. However, the amount of time over which each individual was observed varies. Normally, if the counts were obtained over equal amounts of time, I'd do something like:               fit = lmer(         data = my_data         , formula = count ~ (1|individual)         , family = poisson     )      However, I'm not sure how to handle things when the amount of time is unequal. If I simply divide each individual's count by the amount of time the were observed, this yields a counts-per-minute measure, but the poisson family does not like non-integer values in the dv. Any suggestions?


title: Significance test on the difference of Spearman's correlation coefficient

text: (Thanks a lot for the quick responses! I did a poor job of asking the question, so let me retry.) I do not know how to find out whether or not the difference between two Spearman's correlations is statistically significant. I would like to know how to find it out. The reason I wanted to find out is that in the following paper: Wikipedia- based Semantic Interpretation for Natural Language Processing, by Gabrilovich and Markovitch ( _Journal of Artificial Intelligence Research_ 34 (2009) 443-498). In Table 2 (p. 457), the authors are showing that their method (ESA-Wikipedia) achieves a higher and statistically significant Spearman's correlation than other methods, and I would like to do the same to show that my method is better than previous methods for some problem. I do not know how they calculated statistical significance, and I would like to know. The author of the paper did state that Spearman's rank correlation was treated as Pearson's correlation. I am not sure if that is the right way to do it. I have two Spearman's correlations and I would like to know ifthe difference between them is statistically significant or not. I am aware that web sites, such as http://faculty.vassar.edu/lowry/rdiff.html, provide online calculator for obtaining the difference between two Pearson's correlations. I am unable to find a similar online calculator for the difference between two Spearman's correlations. ## A solution from the link provided by Peter Flom NOTE: The procedures only support the Spearman's correlations that are under 0.6.   1. Let $z_A$ = the Fisher transform of the observed correlation of set $A$, $z_B$ = the Fisher transform of the observed correlation of set $B$.   2. For $i = 1,\dots,n$, let $y_{A_i} = nz_A- (n - 1)z_{A'i}$, where $z_{A'i}$ is the Fisher transform of set $A$ of the one-left-out correlation obtained by deleting $(x_i,y_i)$, re-ranking, and re-computing the correlation. (Each $z_{A'i}$ is based on $n-1$ pairs; each deletion is temporary, for that i only, not permanent.) Repeat for set $B$.   3. $\bar y_A = \sum y_{A_i}/n$ is the jackknifed Fisher transform. Repeat for set $B$.   4. $v_{\bar y_A} = \sum (y_{A_i}-\bar y_A)^2 /(n(n-1))$ is the variance of $\bar y_A$. Repeat for set $B$.   5. Use a heteroscedastic (Welch-Satterthwaite) $t$-test to compare the two jackknifed estimates: $$ t = \frac{\bar y_A - \bar y_B}{\sqrt{v_{\bar y_A} + v_{\bar y_B}}},\quad \text{df}=\frac{(v_{\bar y_A} + v_{\bar y_B})^2}{\frac{v_{\bar y_A}^2}{n_A-1}+\frac{v_{\bar y_B}^2}{n_B-1}}$$ where $n_A$ and $n_B$ are the number of samples of set $A$ and $B$ respectively. ## Before first edit I have got a human-rated set of ranking (HUMAN-RANKING), a set of ranking generated by the presently used, popular method (PRESENT-RANKING), and finally a set of ranking generated by my purposed method (MY-RANKING). I calculated the Spearman's correlation between HUMAN-RANKING and PRESENT- RANKING. Let me call this: HUMAN-PRESENT-SPEARMAN. I then found out the Spearman's correlation between HUMAN-RANKING and MY- RANKING. Let me call this: HUMAN-MY-SPEARMAN. How can I find out if the difference between HUMAN-MY-SPEARMAN and HUMAN- PRESENT-SPEARMAN is statistically significant?


title: Computing confidence intervals for count data

text: I have a representative sample token on area of 60 mq (squared meter) of a territory of 54077 mq. This sample contains the number of little plants that there are for each mq. The sample is defined in R as:               s=c(13,7,10,4,28,0,10,0,0,0,0,0,0,0,0,0,6,         0,0,0,0,0,0,0,4,0,0,0,4,0,0,0,1,2,2,0,         2,3,3,3,1,3,12,33,1,31,0,1,21,0,3,1,8,         0,1,1,6,0,2,0)      The sum of `s` is 227. To compute the CI of `s` I used a `t.test` (also it is not a normal distribution: it is not that my problem).               t.test(s)     t = 3.9606, df = 59, p-value = 0.0002039     mean of s =  3.783333 - 95 % confidence interval:  1.871905 - 5.694762      My question is: Can I assume that the CI of number of plant in the entire territory is between $1.871905\times 227\times (54077/60) - 5.694762\times 227\times (54077/60)$? I think NO because the count is very simplified but I hope YES.


title: How to find the expectation of $X_1X_2$ given $X_1$ and $X_2$ are i.i.d. normal?

text: $X_1, X_2 \sim \text{i.i.d.} N(\mu, \sigma^2/n)$. Any ideas?


title: Use a computer and generate 50 random samples, each of size n=25 , from a normal probability distribution with µ=130 and σ=10.

text: Use a computer and generate $50$ random samples, each of size $n=25$, from a normal probability distribution with $\mu=130$ and $\sigma=10$.   1. Calculate the 95% confidence interval based on each sample mean.   2. What proportion of these confidence intervals contains $\mu=130$?   3. Explain what the proportion found in part (2) represents.


title: Categorising continous data in logistic regression

text: If a predictor in a logistic regression is not discriminating between 'goods' and 'bads' for most of the range and is discriminating only at higher or lower values (monotonic), should the variable be used as continuous or as a dummy indicating the range in which it is predictive?


title: Proxy variables versus instrumental variables

text: Very short question. What exactly is the difference between an instrumental variable and a proxy variable when building a regression model?


title: Validity of a statistical procedure to calculate an odds ratio for a Fisher's exact test?

text: I am trying to determine whether this is a valid statistical procedure. I have ~1000 samples, and each sample has around 1-500 data points. Each data point consists of two fractions (value1 and value2) that range from 0 to 1. I am trying to perform a fisher's exact test with the two fractions (I have the numerator and denominator for each fraction in a 2x2 table) to test for their equality. However, in examining the data, I found that, in general, the fractions are not equal, but rather value2 is frequently greater than value1. In order to correct for this, I want to modify my Fisher's exact test by changing the odds ratio to reflect this modified null hypothesis. In my setup, this odds ratio should not change within a sample. I think the modified odds ratio should be the expectation for the ratio of value2/value1 within each sample. One way to obtain this ratio is to simply perform a regression of value2 on value1 (for each sample). I perform this regression through the origin because data points with value1=0 must also have value2=0. Then, using the slope of this regression, I would have a new value for the expected ratio (value2/value1) for each sample, and I could use this as the odds ratio in my fisher's exact test. (Note that some samples have only 1-5 data points, so regression of value2 on value1 is problematic, but I can estimate the slope of the regression by finding similar samples). My question is, would this procedure be statistically valid? It seems like this may be a case of overfitting. Also, I am not sure whether it is reasonable to use the data so much before performing my statistical test. However, I think that this is essentially a test for outliers, so it seems valid to use the entire data set.


title: Analysing  relation between changes in two time series

text: In relation to the question below I have uploaded the detrended plot and the differenced plot to the following links (I tried to add the images to the post but I got a 'new user' error msg). If anybody knows a way to make a statistical connection between the two without doing regression analysis (because of the nonstationary red line) I'd be grateful to hear. http://imageshack.us/photo/my-images/27/differenced.jpg/ http://imageshack.us/photo/my-images/827/detrend.jpg/ * * * I'm trying to work out how to regress one set of time series data against another. I have attached a graph of two lines: ![enter image description here](http://i.stack.imgur.com/jd5Lr.jpg) The blue line (8 yrs of monthly data) has a blip just at the time when the red line (5 yrs of semi-annl data) rapidly increases. As the red line falls the blue line returns to its original trend. Does anyone know of a standard method to show whether or not the red line 'caused' or 'is associated with' the blue line along the lines of 'Step1: A followed by Step 2: B...etc'? I have a working knowledge of regression of cross-sectional data but, after 2 weeks of trying to work this out by googling and trial-and-error, I'm not all that much further. I'm using excel. In terms of what I have tried...   1. I found out that both data was autocorrelated and that it had a unit root, using an online Dickey-Fuller test. I also tested the residuals of a regression of each against time using a D-W test and that showed autocorrelation too.   2. I detrended the blue line by plotting a graph of the residuals from a regression over time. However, the detrended line (the residuals) still visually had a trend which seemed odd. Also it had unit root according to a D-F test.   3. I detrended the blue line just using data to the start of the red line and it showed no trend   4. I detrended the blue line just for the period the red line exists and it still showed a trend at the begining of the red line and then returned to the trend when the red line starting falling. Inbetween it leapt up along with the red line but lagged. I would be grateful to hear any comments.


title: When to use longitudinal (panel) weights vs cross-section weights in complex surveys

text: I'm currently working with a longitudinal dataset, the Kauffman Firm Survey. The survey tracks about 5000 firms starting from 2004 - 2009. Firms die out over the years. It has both cross-sectional weights and longitudinal weights. I've checked out Lumley's Complex Surveys using R, but the section on repeated samples does not provide guidance on the use of different weights. I also have Levy and Lemshow's Sampling of Populations but its section on weighting says longitudinal weights is outside the scope of the book. The most explicit advice I've found on the matter is here: http://help.pop.psu.edu/help-by- statistical- method/weighting/Introduction%20to%20survey%20weights%20pri%20version.ppt , slide 23. From this power point, I would assume the following use cases: a) Let’s say I want to get the mean revenue from years 1,2,3, but only for firms that survived through year 3, then I would use the longitudinal weight. b) However, if I want to get the mean revenue for all firms from years 1,2,3 (including those that died out), I would use the cross section baseline weight for year 1 – NOT the longitudinal weights for that year, which is what I would have presumed needed to be used. c) Lastly, if I only wanted to look at the mean revenue of firms from year 3 (not including years 1 and 2), I would use cross-section weight for year 3. Questions:   1\. Are these statements correct? 2\. What are some good resources for the correct use of longitudinal (panel) weights?


title: Identifying largest proportion in a set based on a common set of companies

text: To run a certain activity, a population of companies use a finite set of inputs. There are 8 different inputs, but each company may not use all of them. For instance, company 1 uses inputs A, B and D, whereas company 2 uses inputs B, D, E and F I have conducted a survey to ask 30 different companies about which inputs they use, and I would like to statistically evaluate the most used input. What statistical test do you recommend for this?


title: Computing marginals on a graphical model in Python

text: I am looking for libraries available from Python to compute marginals on an undirected graphical model (i.e. a random field) with loops. Some algorithms for this could be LBP (loopy belief propagation) or GBP (generalized belief propagation), although I don't need to use a specific algorithm. I checked SciKit learn but they don't seem to have much for graphical models.


title: Finding the degrees of freedom for a confidence interval for the difference of two means

text: This is more a question to make sure I have a good understanding.   1. Assuming that the variance of each sample is similar i.e., homogeneity of variance and $n_1$ and $n_2$ are the same. $$\text{df} = (n_1-1) + (n_2-1)$$ also the $$\text{MSE} = \frac{s_1^2 + s_2^2}{2}$$    2. Assuming that the variance of each sample is similar ie Homogeneity of Variance but $n_1\neq n_2$ $$s_{x_1-x_2} = \sqrt{\frac{2\left(s_1^2+s_2^2\right)/ \left(n_1+n_2-2\right)}{1/n_1+1/n_2}}$$ $$\text{df}= (n_1-1) + (n_2-1)$$   3. Also, if you can assume that the two samples have the same variance then you can pool them $$s_p = \sqrt{\frac{\left(n_1-1\right)s_1^2 + \left(n_2-1\right)s_2^2}{n_1+n_2-2}}$$ with $\text{df} = n_1+n_2-2$ but have also read not to use this method   4. I have also read that to be more exact to use $$\text{df} = \frac{\left(\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}\right)^2}{\frac{1}{n_1-1}\left(\frac{s_1^2}{n_1}\right)^2+\frac{1}{n_2-1}\left(\frac{s_2^2}{n_2}\right)^2} $$ Works well when both $n_1$ and $n_2$ are both larger than 5.    5. I have just read about Welch-Satterthwaite estimate,used when $\sigma_1 \neq \sigma_2$ $$\text{df} = \frac{\left(\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}\right)^2}{\frac{s_1^4}{n_1^2\left(n_1-1\right)}+\frac{s_2^4}{n_2^2\left(n_2-1\right)}} $$ To be honest, I don't know which one to use. I think using 4 is good, but not sure which test it goes with. Also, why is pooling bad (and if it's so bad why put it in the book)? I am studying for the AP statistics exam.


title: Confusion related to likelihood

text: I was reading this paper http://stat.wharton.upenn.edu/~lzhao/papers/MyPublication/MultiExpert_ICML_2009.pdf However, I have some confusion related to maximum likelihood derivation in this paper We have the parameters of the model $\theta = \\{w,\alpha,\beta\\}$ The maximum likelihood is given by $P(D|\theta) = \prod_{i=1}^{i=N}P(y_i^{1},...y_i^{R}|x_i,\theta)$ where $y_i^1,...y_i^R$ are the noisy labels predicted by my R predictors. Then the paper says conditioning on the true label, $y_i$ and also using the assumption that $\alpha^j$ and $\beta^j$ do not depend on the instance $x_i$ the likelihood can be written as $P(D|\theta) = \prod_{i=1}^{i=N}P[y_i^1...y_i^R|y_i=1,\alpha].P[y_i=1,x_i,w] + P[y_i^1...y_i^R|y_i=0,\beta].P[y_i=0,x_i,w]$ I didn't get how this equation was derived. Any insights will be helpful.


title: Difference between two groups of unequal age

text: This is probably a very simple question! I am trying to compare two groups on a balance evaluation, a normal group and one with a neuro deficit. However, these two groups are significantly different in age. Since age is significantly correlated to the balance score, is there a way to compare my two groups while taking into account the effect of age? Thanks!


title: What prior distributions could/should be used for the variance in a hierarchical bayesisan model when the mean variance is of interest?

text: In his widely cited paper _Prior distributions for variance parameters in hierarchical models_ (916 citation so far on Google Scholar) Gelman proposes that good non-informative prior distributions for the variance in a hierarchical Bayesian model are the uniform distribution and the half t distribution. If I understand things right this works well when it is the location parameter (e.g. the mean) is of the main interest. Sometimes the variance parameter is of the main interest however, for example when analyzing human response data from timing tasks mean timing variability is often the measure of interest. In those cases it is not clear to me how variability could be modeled hierarchical with, for example, uniform distributions, as I after the analysis want to get the credibility of the mean variance both on the participant level and on the group level. In order to do that I need to use prior distributions that can be parametrized with the mean variance or similar, right? My question is then: **What distribution's are recommended when building a hierarchical Bayesian model when the variance of the data is of the main interest?** I know that the gamma distribution can be reparametrized to be specified by mean and SD. For example, the hierarchical model below is from Kruschke's book Doing Bayesian Data Analysis. But Gelman outlines some problems with the gamma distribution in his article and I would be grateful for suggestions of alternatives, preferably alternatives that are not to difficult to get working in BUGS/JAGS. ![enter image description here](http://i.stack.imgur.com/ZerKf.png)


title: Regression with an unknown dependent variable

text: I want to know if there is any literature about the following regression problem: $$ Y=X\beta +\epsilon$$ where $Y$ is unknown. But, i know $X$ and the OLS estimator of $\beta$ $$ \hat{\beta}=(X^\top X)^{-1}X^\top Y$$ because i know the value of $X^\top Y$ In fact i have an inverse problem of quadratic programming where i know the solution and i want the regression problem associated to this soluiton. Thanks


title: Data set for forecasting

text: I am looking for a data set which can be used for ARIMA or any forecasting models. The data should be such that, over a period of time the range of inputs change i.e. the band of input data changes significantly. This should lead to incorrect forecasting. I should see the error in forecasting and should retrain the model. I do not want to simulate and i am looking for some real world example.


title: Multicollinearity question

text: I have got a dataset with N=51. The dataset comprises two groups (n=26, and n=25). I have one outcome variable and one covariate (both continues variables). I am interested in if there is an effect of group on outcome measure. I used a general linear model to estimate the beta of group. In the GLM, I included the covariate. However, the covariate is significantly correlated with the dichotomous variable group (Pearson r=-.527, p(2-tailed)<.001), and the covariate is also marginally correlated with the outcome measure (Pearson r=-.269, p(2-tailed)=.056). Because of this multicollinearity, I think I cannot interpret the effect of group. The beta value in the model without the covariate is relatively large and significant; in the model with the covariate, there is no effect of group (actually, beta group is very close to zero). As far as I know, there is no good way to get rid of this multicollinearity (please correct me if I am wrong). The next best thing that came to mind is estimating the effect of the covariate **within** the two groups. If the covariate indeed is a confounder, the estimates should be significant and similar in both groups, right? I calculated the correlations of the covariate and the outcome measure for both groups, and got the following results: Group A (n=26): Pearson r=-.047, p(2-tailed)=.819 Group B (n=25): Pearson r=-.354, p(2-tailed)=.082 Does this mean I can disregard the effect of the covariate, and go with the model that does not include the covariate? Thanks!


title: What is the right h0?

text: the problem is : A manufacture claims that a new brand of air-conditioning unit uses only 6.5 kilowatts of electricity per day,A consumer agency believe the true figure is higher and runs a test on a sample of size 50.if the sample mean is 7.0 kilowatts with a standard deviation of 1.4,should the manufacturer's claim be rejected at a significance level of 5% ?Of 1%? the answer is :Assuming an SRS we have: H0:mean=6.5   HA:mean>6.5   (omitted the following process) here ,H0:mean=6.5 in the book,i don't think it is right,it should be:   H0:mean<=6.5   HA:mean>6.5 am i right?


title: Interpreting Cluster Heat Maps From R

text: So I have been looking at how to plot high dimensional clustered data, and one of the options that come up is a heat map. Although there are many webpages that provides code on how to create one, there is very little information about how to interpret a heat map. So I was wondering how would I interpret the heat map below? ![heat map and dendrogram](http://i.stack.imgur.com/ToYNX.png)


title: Hypothesis testing for non-linearity

text: I have $n_X$ observations of variable X, $n_Y$ of variable Y, and $n_Z$ of variable Z. I'd like to test the hypothesis that the true mean of $X$ is equal to the sum of the means of Y and Z. $$H_0: \mu_X - (\mu_Y+\mu_Z) = 0$$ **Initial thoughts** I can use the sample means to define an estimator $\hat{\gamma} = \bar{y}_X - (\bar{y}_Y+\bar{y}_Z)$. Can I estimate the variance using $$\hat{\sigma}^2(\hat{\gamma}) = \sigma^2 (1/n_1+1/n_2+1/n_3)?$$ If so, is the test statistic $\hat{\gamma}/\sqrt{\hat{\sigma}^2(\hat{\gamma})}$ t-distributed?