title: Maximum Margin of Error

text: Here's a stats question I'm having trouble with: Suppose the researchers plan to do a follow-up study in the future to once again estimate π = the true proportion of the population that gets help for pain associated with joint pain or stiffness from acupuncture (In the study, 31 of 32 of the respondents reported that they got help from the acupuncture treatment). They want the margin of error to be at most ± 3%. How large of a sample size will they need to ensure that the margin of error would not exceed this amount? Can anyone point me in the right direction for this problem? Would I just set the margin of error equal to .03 and do: .03 = 1.96 sqrt(.96875(1 - 0.96875))/n) and do solve for n? Am I on the rigth track?


title: predicting using multiple variables

text: I have time series analysis for a disease, as well as climate data over the same time period which includes rainfall and temperature. I have these data for eight distinct locations (latitude and longitude points). I want to see if there is any correlation between seasonal patterns of the disease and climate, and see if climate can predict disease prevalence. I am unsure how to test this. What I have done so far is this - from the time series, I can estimate the timing of the average annual peak, and have used multivariate regression in R:               model = lm(disease_peak ~ latitude, rainfall_peak, temp_peak, data=mydata)     summary(model)      Are there any other/better ways of doing this analysis with time series data? (I have the matlab stats toolbox)


title: Recruiting participants for study

text: I am planning to recruit participants in a college by advertising through posters, fliers, and the College Facebook. Is this method of recruitment considered clustered sampling, which is a type of probability sampling? I am afraid that if I make use of a convenience sampling, it will not be considered "probability sampling" and hence I will not be able to apply some statistical tests like regression analysis, factor analysis or ANOVA. Any thoughts would be appreciated. Thank you


title: How to characterize a problem of standardizing product descriptions

text: I'm looking for some advice for where to start on this problem. Let's say I have sales transaction data from a number of different retailers that all sell the same products. Even though they are selling the same products they all identify them a little bit differently. For example for one particular product one retailer calls it "Kellogs Corn Flakes 20oz." and another calls it "Kel CF, 20". And there are other representations for the 100's of other retailers. The problem to solve is how to map each of the different products to a standard set of product descriptions so that the data can be aggregated. Assume that I have some group of retailers already mapped manually. I've been going over in my head where to start with this. Is it a search problem where I consider the search query is the retailer's representation and the standard description is the "document" to find? Or is it a classification problem in that I'm trying to classify each description into a standard product description category? Or can named entity recognition play into it somehow. Any advice you could provide to get me started would be much appreciated. I've been looking at tools like Apache Lucene, Solr and OpenNLP but it's just not clear to me how to characterize the problem.


title: When does a logistic regression model have a unique solution?

text: Mathematically speaking, for which data does a logistic regression model have a unique solution?


title: k-means implementation with custom distance matrix in input

text: Can anyone point me out a k-means implementation (it would be better if in matlab) that can take the distance matrix in input? The standard matlab implementation needs the observation matrix in input and it is not possible to custom change the similarity measure. Thanks.


title: Generalization of Brownian motion to $\alpha$-stable distributions

text: Brownian motion is constructed as a limit of the sum i.i.d. Gaussian increments. Can one use a non-Gaussian $\alpha$-stable distribution (e.g. the Cauchy distribution) instead, and still construct a process? Would the scale parameter of such process evolve according to the formula $c_t = t^{1/\alpha}$?


title: Correlating various diagnoses with amount of weight gained

text: I have a processed dataset where patients took a medication and gained a varying amount of weight. In addition, any ICD9 diagnoses the patients have received while being monitored are recorded. I'd like to see if certain diagnoses are correlated with amount of weight gained. What would be a good procedure for testing this? Of note, each patient has several diagnoses: there are about 1,200 unique diagnoses in total. Currently, I have calculated the average wt. gain along with the standard deviation, then grouped the patients by standard deviation, say:               (not actual #'s)     Total Patients: 420     Total in Group 1 (>2 Sd weight Loss): 10     Total in Group 2 (>1 Sd weight Loss: 50     Total in Group 3 (<1Sd weight gain or loss): 300     Total in Group 4 (>1 sd weight Gain): 50     Total in Group 5 (>2 sd weight Gain): 10      I was thinking of looking at how many patients have x,y,z (etc.) diagnoses in each group. Say Type II DM:               Total: 200 Pts Have it     In Group 1: 4/10 Pts have it     Group2: 15/50 Pts have it     etc...      With this method, would simply computing a Pearson correlation on the absolute count of Diagnoses vs. wt change group be appropriate? Additionally, if by the time I get to Group 2 or Group 4 0 patients carry a particular diagnoses, how will this impact running the analysis this way? Appreciate any pointers.


title: k-fold CV of forecasting financial time series -- is performance on last fold more relevant?

text: I am working on an ANN-based forecasting model for a financial time series. I'm using 5-fold cross-validation and the average performance is so so. Performance on the last fold (the iteration where the last segment is omitted from training and used for validation) is better than average. Is this a coincidence / data-dependent, or is validation performance on the last fold usually better? (presumably because training with all preceding data is more related to the subsequent data in time series) This feels a bit like a weird question, but I'm hoping for some responses anyway. Thanks in advance :)


title: What techniques are used for empirical, stochastic simulation of a time series?

text: Suppose you have recorded a set of paths in the $y,t$ plane, with $y = f(t)$, $f$ is a stochastic function (i.e. there is a noise term), and $t$ might be time or some other monotonic increasing independent variable. Suppose further that you know the underlying process is random, meaning the same set of initial conditions (even, "hidden variables") do not produce the same path. Do not assume that changes in $y$ are independent, or even stationary (though, stationary but dependent changes are an important special case). What techniques are available to simulate an incomplete curve into the future?   What I mean by "incomplete" curve is: your set of historical curves are _each_ over some range of interest, $T=[0,t_{max}]$. You are given a curve that represents data $y_1(t)$ recorded up until some intermediate value of $t' \in T$.


title: How to interpret the coefficient when Y is at levels and RHS is first difference?

text: To fix the idea, suppose C is consumption in dollars and Y is income in dollars. This is a time series model. Suppose for some strange reason I need to estimate the model $Y = a + b\cdot \Delta X$ where $\Delta X$ is the first difference of X. How do I interpret this $b$ coefficient? If both sides are FDs, we can just interpret it as at levels since the model comes from the levels (and time trend). But what if the LHS is levels and the RHS is FDs as in the above regression? Thanks!


title: Bayesian calculation with compound independent factors

text: I have simple model for some medical test characteristics calculation.   * $T$=test   * $D$=Disease    * $D+$= Disease present   * $T+$= Test positive (yes\no) $P(D+)$ is very small (0.000001).   The problem is I don't know what the probability of Test being positive in the general population. This is a very new test and we don't know the exact profile of this very expensive test yet. Test result can be affected by many factors. Given this problem definition, how can I incorporate these many factors that may change the test result (cause a false positive or a false positive) into my general basic Bayesian model below? My model is $P(T+|D+) = P(D+|T+)P(D+) / P(D+)$ Thank you


title: What is the difference between graph clustering and community detection methods?

text: Basically, the goal of graph clustering and community detection methods are to compute clusters. Is there any difference between them? Many thanks for any advice.


title: Re-estimate classification model with biased data

text: Let's say that I score (with, for example, a logit model) a group of 10,000 customers according to their potential of buying a product, and that I decide to contact the top 1,000 with a special offer. I use the customers previous behavior as my training data. This far this is just a normal classification problem. However, my question concerns my long-term plan: how should I incorporate the results from this campaign into my next scoring? Let's say that I want to score customers and send out new offers once every few months. The feedback from the latest round (= the knowledge of who picked up on the offer and who didn't) is valuable information in order to tune the model. But at the same time, the sample from which this new information is derived is not a representative sample of the whole customer database (I only get feedback from the 1,000 with the highest scores). If I train a new model based purely on the latest round I will quickly move towards a model that completely neglects the lower scoring customers. Yet, if I only used the original data (and updating the variables linked to the 1,000 customers picked for the campaign) I feel that I don't put enough weight on the new information AND this also introduces a bias: as only the 1,000 picked customers receive updated values, it is more likely that they may (or may not) me picked in the next round. For example, if a central variable in my model is `recency` (eg. the shorter period of time since the last campaign, the more likely the customer is to buy again), the recently picked would have an unfair advantage over other, latent potential in the customer database. What options and solutions are usually implemented in these situations? let's say that this scoring process needs to be repeated dozens of times.


title: Mathematical description of problem where sample is asked about three most liked brands?

text: I have a population of _branded products_ lets say diary food and want to know which brand is the most liked. I ask 1001 people to tell me their _three most liked brands_. What is the mathematical description of the problem?


title: How can you derive the sample size to obtain a particular type II error rate?

text: I often see that the sample size for a $z$-test to achieve a particular type II error rate $\beta$ (at a given significance level, $\alpha$) is: $$n = \frac{(Z_{1-\alpha/2}+Z_{1-\beta})^2\sigma^2}{\delta^2}$$ How do they reach that conclusion/determine that?


title: How can I increase the accuracy of predicting the occurrence of event using logistic regression?

text: I will work on climate data to predict if a disease it will happen or not on a crop. During search I found logistic regression is best choice for my research, but it has several problems led to reduce the prediction. I'm new to prediction analysis and I don't know how to increase my prediction accuracy. There are several selection methods used with it. Which one is best to select data to fit the model. Which type of regression help me in this research?


title: Particle filter (sequential Monte Carlo) for a non-Gaussian hierarchical model

text: I have the following, which I am attempting to model with a particle filter. \begin{align*} y_{i,t}&\sim\mathrm{Poisson}\left(\lambda_{i,j,t}\right)\\\ y_{j,t}&\sim\mathrm{Poisson}\left(\mu_{i,j,t}\right) \end{align*} \begin{align*} \lambda_{i,j,t}&=\delta_1\exp\left(\alpha_{i,t}-\alpha_{j,t}\right)\\\ \mu_{i,j,t}&=\delta_2\exp\left(\alpha_{j,t}-\alpha_{i,t}\right) \end{align*} \begin{align*} \alpha_{i,t}&\sim\mathrm{N}\left(\alpha_{i,t-1},\sigma_{\alpha}^2\right)\\\ \alpha_{j,t}&\sim\mathrm{N}\left(\alpha_{j,t-1},\sigma_{\alpha}^2\right)\\\ \end{align*} Where: t is a time index, $t=1,\ldots,5320$; $i,j\in\left\lbrace 1,\ldots,40\right\rbrace$, $i\ne j$ (so 40 states altogether but only 2 update at each time $t$); the $\alpha$ states are unobserved; and the $y$'s are observed for every $t$. For simplicity, assume the values of $\delta_1$, $\delta_2$, and $\sigma_{\alpha}^2$ are both known and constant. My idea was to have a set of particles for each state. Then update them (the 2 sets involved at time $t$) with Gaussian noise, using the equations for the states. Given these values, the lambdas can be calculated and then the conditional distributions for $y_{i,t}$ and $y_{j,t}$, given the states, are known. However, on calculating the importance weights from the conditional distributions $y_{i,t}|\alpha_{i,t},\alpha_{j,t}$ and $y_{j,t}|\alpha_{j,t},\alpha_{i,t}$, you get two sets of importance weights from \begin{align*} y_{i,t}&|(\alpha_{i,t}-\alpha_{j,t})\\\ y_{j,t}&|(\alpha_{j,t}-\alpha_{i,t}). \end{align*} My question is how then to resample particles for $\alpha_{i,t}$ and $\alpha_{j,t}$, or possibly how can I do this differently to overcome the problem. I am fairly new to sequential Monte Carlo methods, so any other advice or suggestions on tackling the problem would be appreciated.


title: Model selection criterion produces non-normal residuals

text: I was wondering if I can use Akaike or Schwarz criterion even when the residuals that I get from the model when I run the regression are not normal. Is there any normality assumption with these criteria or can I use them all the time regardless?


title: Estimating a time frame for interventions or judging the extent to which events become "determined" as time goes on

text: In certain arenas, it's valuable to be able to intervene early on to prevent problems from getting worse, because after a certain point there's not much you can do. Two examples might be public health or education. Say we have a bunch of information about students -- their demographic info, their past transcripts, their grades in a course as it progressed, the ultimate outcome (A, B, C, D, F) etc. -- and we're interested to know, in some sense, at what point their outcome is "fixed". (I'm actually interested in knowing at what point changes in their behavior or teacher interventions aren't going to change anything, but I don't have data from experiments with interventions done at different points in time or anything like that.) My current thinking is that I ought to build a series of predictive models based on information that was available on the first day, fifth day, tenth day, etc. Then I would evaluate the accuracy of the probability distributions over the possible grades that each of these models spits out for each student (perhaps with the Brier score) and plot this vs. the date of the information which the model takes into account. Then I could plot on the same graph how the average variance of the probability distributions changes as time goes on. This would give me an idea of how certain we can be of a student's outcome as time goes on and perhaps reveal a point after which we could guess that interventions would no longer be effective. Does this seem reasonable? Can you think of better ways to do this?


title: Dense Data Compression (Delta Encoding?)

text: Let $X $ be a dense integer set such that its elements are closely knit in value. For instance:               1 1 1 1 2 2 2 2 2 3 3 5 5 6 6 6 6 6 7 7 7 7 8 8 8 8 8 8 ...      I am looking to compress this data set in whatever means possible, currently resorting to delta encoding. Through delta encoding, I am able to store the above array with only bits of data rather than full integers, as I'm only taking the difference from one element to the next. However, as you'll notice, the above set lacks a 4. Although the data is fairly dense, there are few values (`<0.1%`) that are not found. I have resorted to inserting these elements like so:               1 1 1 1 2 2 2 2 2 3 3 4 5 5 6 6 6 6 6 7 7 7 7 8 8 8 8 8 ...      in order to create a delta-encoded set:               1 0 0 0 1 0 0 0 0 1 0 1 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 ...      This results in some fuzzy data, leading to minimal but questionably negligible side-effects to the original set. My question is, are there other methods for normalizing dense data and compressing it to reduce the space complexity revolved around it?


title: How can I convert a Gauss-Markov process to i.i.d. Gaussian process?

text: I am wondering is there any straight forward approach to convert a Gauss- Markov process, i.e., a First order autoregressive process with i.i.d. Gaussian input, with the covariance matrix $K=Toeplitz(1, \rho, \rho^2, \ldots, \rho^{N-1})$ where $0 \leq \rho < 1$ is the normalized correlation coefficient, to i.i.d. Gaussian process. As I know, I can apply eigenvalue-eigenvector decomposition (KLT) to make the process uncorrelated and independent (since the process is jointly Gaussian), but the converted process is not identically distributed due to the variety of variances (eigenvalues). Thanks a lot in advance, Farzad


title: Dimensional reduction with synonomy and polysemy

text: I was wondering something for a computer science project I am doing. Documents usually have too many distinct words to process on a computer efficiently. One solution is to map all words to a single synonym. For instance, 'giant' and 'enormous' and 'huge' would all map to big. I also want to account for polysemy, where words can have more than one meaning. If I were to convert all words to their synonyms while accounting for all possible double,triple,etc. meanings, do you think that the number of distinct words in a document would increase or decrease? (Note: assume that all of the words have already been stemmed as well). In addition, anyone have any good API suggestions for this possibly? (preferably Java) In order to address some issues from the comments below: My database of documents consists of mainly forums from the web and links that are fetched through the forums (A web crawler was used to crawl about 10 million sites so far with many more to come). And maybe this might be a better question to ask: For anyone that has tried different methods of dimensional reduction, did you find mapping words to synonyms useful, and if you did, then how did you account for polysemy?


title: Online, scalable statistical methods

text: This was inspired by Efficient online linear regression, which I found very interesting. Are there any texts or resources devoted to large-scale statistical computing, by which computing with datasets too large to fit in main memory, and perhaps too varied to effectively subsample. For example, is it possible to fit mixed effects models in an online fashion? Has anyone looked into the effects of replacing the standard 2nd order optimization techniques for MLE with 1st order, SGD-type techniques?


title: Compute $\mathrm{Cov}(\sum_{i=1}^NX_i,\sum_{i=1}^NY_i)$

text: Let $X_1,X_2,\dots$ be i.i.d. Bernoulli random variables with parameter $\frac{1}{4}$. Let $Y_1,Y_2, \dots $ be another sequence of i.i.d. Bernoulli random variables with parameter $\frac{3}{4}$. And let $N$ be a geometric random variable with parameter $\frac{1}{2}$ (i.e., $\mathrm{P}(N = k) =\frac{1}{2^k},\, \forall\ k=1,2,\dots$). Assume the $X_i$’s, $Y_j$ ’s and $N$ are all independent. Compute $$\mathrm{Cov}(\sum_{i=1}^NX_i,\sum_{i=1}^NY_i).$$ Here $N$ is also an r.v. so I think I need to use conditional covariance. How can I solve this? Where to start? I am stuck in the first step?


title: How to aggregate and merge these two data.frames?

text:
Here is the structure of my data.frames:               > str(c)     'data.frame':   21633 obs. of  20 variables:      $ Trade        : num  7e+12 7e+12 7e+12 7e+12 7e+12 ...
          $ New          : Factor w/ 2 levels "N","Y": 1 1 1 1 1 1 1 1 1 1 ...      $ Amended      : Factor w/ 2 levels "N","Y": 1 1 1 1 1 1 1 1 1 1 ...
          $ Unwound      : Factor w/ 2 levels "N","Y": 1 1 1 1 1 1 1 1 1 1 ...      $ Fixed        : Factor w/ 2 levels "N","Y": 1 1 1 1 1 1 1 1 1 1 ...
          $ FX.Fixed     : Factor w/ 2 levels "N","Y": 1 1 1 1 1 1 1 1 1 1 ...      $ Branch       : Factor w/ 1 level "nil": 1 1 1 1 1 1 1 1 1 1 ...
          $ Currency     : Factor w/ 8 levels "AUD","EUR","USD",..: 1 1 1 1 1 1 1 3 1 1 ...      $ Product      : Factor w/ 4 levels "A","B","C",..: 2 2 2 2 2 2 2 2 2 2 ...
          $ Today.NPV    : num  444 222 0 0 -77777 ...      $ Today.Unreal : num  13.4 5555.54 0 0 6666.36 ...
          $ Today.Real   : num  0 0 0 0 0 0 0 0 0 0 ...      $ MTD.Real     : num  0 0 0 0 0 0 0 0 0 0 ...
          $ MTD.Unreal   : num  222 -333 0 0 -444 ...      $ New.Deals.P.L: num  0 0 0 0 0 0 0 0 0 0 ...
          $ Amend.P.L    : num  0 0 0 0 0 0 0 0 0 0 ...      $ Unwind.P.L   : num  0 0 0 0 0 0 0 0 0 0 ...
          $ Fixing.P.L   : num  0 0 0 0 0 0 0 0 0 0 ...      $ Carry.P.L    : num  16.6 81.9 0 0 -319.3 ...
          $ FX.Fixing.P.L: num  0 0 0 0 0 0 0 0 0 0 ...     > str(predict)     'data.frame':   735755 obs. of  7 variables:      $ X.Financial.Object.  : num  7e+12 7e+12 7e+12 7e+12 7e+12 ...
          $ X.Scaling.Currency.  : Factor w/ 7 levels "AUD","CAD","EUR",..: 4 4 4 4 4 4 4 4 4 4 ...      $ X.Maturity..date..   : Factor w/ 108 levels "12Aug2011","12Aug2014",..: 22 98 95 88 85 ...
          $ X.Rate.Change.       : num  0 0 0 0 0 0 0 0 0 0 ...      $ X.Env.COB.Date.      : int  20110815 20110815 20110815 20110815 20110815  ...
          $ X.Predict.           : num  74.36 -3.84 16.77 4.66 11.88 ...      $ X.Reporting.Currency.: Factor w/ 7 levels "AUD","CAD","EUR",..: 4 4 4 4 4 4 4 4 4 4 ...      **Goal 1:** aggregate the first data frame 'c' to sum all numeric values by Currency. I do not need text fields other than the currency field I tried to use               > aggregate(x=c, by=list("Currency"), FUN="sum")     Error in FUN(X[[1L]], ...) : arguments must have same length      It is apparently incorrect. **Goal 2:** I would like to merge the data.frame c with predict using this relationship: "c.Trade == predict.Financial.Object" I tried:               > cc = merge(c, predict, by=c("Trade","Financial.Object"))     Error in fix.by(by.x, x) : 'by' must specify valid column(s)      but it does not work. One further question: why the field names in data frame 'predict' have the dot in them? What is the proper way to refer to them?


title: How to test for significant differences in means across  four within subject conditions where dependent variable is a proportion?

text: I have a dataset with four within-subject conditions. Participants gave binary answers to 8 questions in each condition, and these were averaged to form a proportion for each condition. I would like to compare the mean proportions of people choosing a particular answer between conditions. For example 80% chose a particular answer in the first condition, 60% in the second 40% in the third 20% in the fourth. Here's a graph of the results: ![plot](http://i.stack.imgur.com/qW5Od.png) Mean probability of guessing RED drops as no. GREEN increases. I want to see whether the drop is significant.   * How can I somehow summarize whether this drop is significant?


title: How to sample randomly from a population?

text: What do you think about the following method to perform a random sampling:   1. Generate as many floats (between 0 and 1) as individuals using a quantum random generator (each value appears only one time).   2. Assign floats to individuals.   3. Sort individuals according to the order of the floats.   4. Select individuals as needed. Since the generator has good quality, is it risky to use this method?


title: Differences between cross validation and bootstrapping to estimate the prediction error

text: I would like your thoughts about the differences between cross validation and bootstrapping to estimate the prediction error. Does one work better for small dataset sizes or large datasets?


title: Finding If A Replacement Manufacturing Test System Fails More Units

text: I'm currently writing software that replaces an existing legacy system, which tests units against different engineering specifications. Since I'm currently verifying everything works, and I was able to pull all data for all parts historically tested in the legacy system into my new database, I decided to run all units against their respective specifications (~780,000 units). I took the results from the tests, and formatted them in a way for the engineering team to overview. The engineers became skeptical of my version of the software, claiming my system is failing too many units. From my limited testing, it's seemed as though the new and old software have been passing the same number of units. I cannot do the same analysis with the legacy software, running it against all units to get an overall failure percentage. However, I can run small sample reports on my own time in the new and old system and compare the results for the same units. If I am able to collect data on X units between each system, and gather the percentages that fail on the new and old system, what statistical test can I use to prove the data does not significantly differ, or does significantly differ from system to system? I have experience with basic statistics classes, and I remember learning a similar problem, but I'm completely drawing a blank from my courses on what methodology I should be using for this type of sampling.


title: Normalizing Vs. Scaling

text: Are the concepts of normalizing and scaling of data in conflict with each other? I am adding weights to my features, I have tried normalizing the weights and it didn't make any difference in the outcome. I have also scaled my input data. and got positive results. however, I have heard from other sources, fellow students- not so reliable, that I should **scale the weights**. So I am rather stumped, I don't know if it is something really simple and I am having a mental block or it it is something more complex that I am not understanding.


title: Subsample bootstrapping

text: I have been working on the uncertainty associated with a quantity calculated from a Monte Carlo project. Normally I would use the bootstrap method by resampling with replacement, for a couple of technical reasons that is not particularly easy here. It was suggested that I just break up my MC data set and perform the experiment with these subsets and find the uncertainty that way. I have in the past come across references to bootstrapping with only a subset of the original dataset. Can someone point me to a tutorial on this or explain briefly how it is different to bootstrapping with replacement and just setting the number of samples to a fraction of the total size. I would be particularly interested in a method that meant that $n$ could be different for each subsample, this would make my analysis much more simple.


title: should weights be scaled too?

text: I am using supervised learning algorithms (specificly SVM) on my data. I know that scaling was needed for my input data. however as I am also adding weights (using pairwise comparison), I am not sure if my weights need to be scaled as well. I don't mean normalizing the weights.


title: How do I quantify a discrepancy which only exists in a subset of data?

text: I was looking at two different correlations (`BU`-`OEU` and `BC`-`OEC`) and found them **both to be significantly negative** (see **Figures 1 and 2** ). However, eyeballing the data for the low end of `BU` and `BC` (`> 0.1`) it would appear that there is a **much larger spread of scores for`OEU` than for `OEC`** (ignoring the outlier in `OEC`; see **Figures 3 and 4** ). **Is there a way of quantifying this?**   I was thinking perhaps kurtosis + standard deviation, but am unsure. **Figure 1** ![Figure 1](http://i.stack.imgur.com/4PajF.png) **Figure 2** ![Figure 2](http://i.stack.imgur.com/Tq4OF.png) **Figure 3** ![Figure 3](http://i.stack.imgur.com/bzQmL.png) **Figure 4** ![Figure 4](http://i.stack.imgur.com/NNtAX.png)


title: Support vector machines and Granger causality

text: I was wondering if Granger causality would be an efficient tool for searching for relevant input data for an SVM system. For example if I want to forecast SP 500 returns, I could put in my input data the libor rate, the USD/EUR exchange rate, the FTSE index, the Eurostoxx 50 index... Would it be relevant and meaningful to investigate, before adding those data to the SVM input features, if there is Granger causality beetween the SP index returns and the USD/EUR exchange rate and so on? Is Granger causality a adapted tool for SVM input data preselection?


title: How do I compare the quasi likelihood of two multi-level generalized estimating equation models?

text: With the log-likelihood chi-square statistics I can compare two linear mixed models (Maximum Likelihood) and see which one is the better one. But the GEE gives Quasi Likelihood under Independence Model Criterion (QIC) and I don't see the degrees of freedom so I am unsure how to statistically test two models against each other and select the one with the best fit.


title: How to fix a coefficient in an ordinal logistic regression without proportional odds assumption in R?

text: I want to do an ordinal logistic regression in R without the proportionality odds assumption. I know this can be done directly using `vglm()` function in `R` by setting `parallel=FALSE`. But my problem is how to fix a particular set of coefficients in this regression setup? For example, say the dependent variable $Y$ is discrete and ordinal and can take values $Y = 1$, $2$, or $3$. If the regressors are $X_{1}$ and $X_{2}$, then the regression equations are $$ \begin{aligned} {\rm logit} \big( P(Y \leq 1) \big) &= \alpha_{1} + \beta_{11}X_{1} + \beta_{12}X_{2} \\\ {\rm logit}\big(P(Y \leq 2) \big) &= \alpha_{2} + \beta_{21}X_{1} + \beta_{22}X_{2} \end{aligned} $$ I want to set $\beta_{11}$ and $\beta_{22}$ to $1$. Please let me know how can I achieve this. Also if `R` can't do this, could you also please let me know if I can achieve this in any other statistical software?


title: Can we simply compare the predicted percentages of the outcome between studies?

text: I used the multinomial logistic regression to predict the percentages of students who voted 'acceptable', 'uncertain', and 'unacceptable' to natural ventilation use in three observed classrooms during cool and hot seasons. The sets of significant IVs of the cool and hot season cases were different; for example, the effect of window size (i.e. small, medium, large) on the acceptance of natural ventilation was significant for cool season case but not for hot season case. Question 1: how can I compare the percentages of 'acceptable' votes of the two seasons? Can I look at the differences in the predicted percentages between two seasons directly? Question 2: how can I compare the percentages of 'acceptable' votes of students in the rooms with different window size? Thank you. Please forgive me for my bad English as I am not a native English speaker.


title: Supervised classifier for events with missing data

text: I'm working on an experiment in which we try to sort events into three categories based on roughly 20 inputs using a feed-forward neural network (trained via backpropagation). Unfortunately, many variables which offer good discrimination become meaningless in some situations. So far, the procedure has been to set meaningless variables to some placeholder value which lies outside the normal range for that variable. But that's roughly half the inputs in something like half the events. The classification isn't as strong as we'd like, so the options I've come up with are:   1. Split the events based on whether the unreliable variables are present, train several neural nets on the sub-samples. The classified samples would then be merged.    2. Switch to some other classifier (boosted decision trees are very popular in my field) which handles missing information more naturally.  Is one of these approaches obviously superior? Is there some other obvious option I'm missing?


title: Radial visualization for Probability Density Functions

text: I saw the following representation in a paper that I was reading. Can anyone can shed any light on how it was developed? this is the Paper \- page 34 ![Radial representation of Probability Density Functions](http://i.stack.imgur.com/CiCPw.jpg)


title: Standard error of a count

text: I have a dataset of incident cases by season of a rare disease. For example, say there were 180 cases in the spring, 90 in the summer, 45 in the fall, and 210 in the winter. I'm struggling with whether it is appropriate to attach standard errors to these numbers. The research goals are inferential in the sense that we are looking for a seasonal pattern in disease incidence that might recur in the future. Thus, it feels intuitively like it should be possible to attach a measure of uncertainty to the totals. However, I'm not sure how one would compute a standard error in this case since we are dealing with simple counts rather than, e.g., means or proportions. Finally, would the answer depend on whether the data represent the population of cases (every case that has ever occurred) or a random sample? If I am not mistaken, it generally does not make sense to present standard errors with population statistics, since there is no inference.


title: How to treat variable with limit in correlation/regression?

text: I am dealing with climate data and e.g. precipitation can never be lower than 0, relative humidity can never be higher than 100, etc. So when e.g. plotting precipitation against temperature a lot of points sit on the x (and/or y) axis. Can I ignore that in my analysis or will e.g. correlation coefficients not be reliable in that case?


title: Statistics for benchmarking code

text: My application has a factory class which has modular elements registered to it. A run method is called on the factory class which subsequently executes the run methods on the modules. I am using PEAR PHP's Profiler to measure the speed of execution for each module in microseconds. So far I have been storing the module runtimes and the deltas of each runtime in my database. This has produced somewhat interesting statistics when I graph the runtime (I can see spikes, etc). However, I don't have much experience with statistics, what kind of analysis would prove valuable for looking at code performance? My intention is to catch naively written modules before deployment. Thanks!


title: Estimating the quantiles of a latent variable

text: I am trying to estimate the quantiles of the function $f(x_i)$ in the equation: $$ y_{it} = \alpha_i + f(x1_{it}, x2_{it}) + \epsilon_{it} $$ My current, probably naive, approach is to run the linear regression $y_{it} = \alpha_i + X_i\beta+ \epsilon_{it}$, calculate the predicted value $X_i\beta$ and its empirical CDF. I am then bootstrapping to estimate the confidence intervals of the various quantiles. I also considered using a hierarchical model and assuming that that $f(x1_it, x2_it)$ follows a specific distribution but this seemed like almost assuming away the problem. Any suggestions appreciated. Also, as this is my first post, suggestions on how to better frame the question also appreciated.


title: Does a distance have to be a "metric" for an hierarchical clustering to be valid on it?

text: Let us say that we define a distance, which is not a metric, between N items. Based on this distance we then use an Agglomerative hierarchical clustering. Can we use each of the known algorithm (single/maximum/avaerage linkage etc), to get meaningful results? Or put differently, what is the problem with using them if the distance is not a metric?


title: How to display my data in a visually pleasing way?

text: I have some funky data that I'd like to put into a final presentation form and I was hoping to get some creative ideas on how to do it. My data compares measurements of different lengths, so for example I have 52 week long samples of 6 different chemicals (times several duplicates), 26 2 week long samples, 12 month long samples, 4 quarterly samples, 2 semiannuals and an annual sample. I have compared the accuracy of each duration with one another for a total of 15 comparisons (week vs 2 week, week vs month, etc) to find how much one deviates from another on average by computing the % bias, where negative bias indicates that the longer of the 2 samples under-reports and positive bias indicates that the shorter of the 2 samples under-reported. So I have an average % bias and a 95% confidence interval for each of 6 chemicals for each duration comparison. My confidence intervals are of varying widths because I obviously have many fewer data points for comparisons involving annual samples. An example is given below:               Chem1          week v biweek    week v month    week v quarter    week v semiannual    ....     low95    -10               -12              -15               -17     mean      0                  2                2                -1     high95    10                14               17                15      How might I pack this into one nice looking graphic (or color coded table, or something) that can convey the average bias for each chem and the accompanying confidence interval (maybe highlighting those with greater power?) and also for each duration comparison (ideally maintaining some visual representation of relative time lengths). I would be content having 6 graphics (one for each chemical), but I don't think I want 15 (one for each comparison). I was thinking that a color ramp would help ad an extra dimension to it, but I don't really know how to implement that in R. Any creative solutions? Help on the color-ramping? Please keep in mind that I'm working in R and I'd say my R skills are about intermediate. Thanks a ton!


title: Combining Box Plot with frequency count

text: Are there graphing techniques (certain kinds of plots) that are similar to the Edward Tufte's box plots - shown below - but that consider the distribution of the data points? ![Edward Tufte style box plots](http://i.stack.imgur.com/ZC2SJ.jpg) But instead of a black line traveling vertically, I want a colored line (say red) where the intensity of the redness changes in relation to the frequency of data points at the location. What I am picturing exactly is a vertical red line that does from 1% to 99%, with horizontal grey lines at 25% and and 75%, a black line at the median, with the color intensity of the vertical red line changing with respect to the number of times each data point along the vertical location was measured. Any ideas? Thanks, Tom


title: How to interpret coefficients in a Poisson regression?

text: How can I interpret the main effects (coefficients for dummy-coded factor) in a Poisson regression? Assume the following example:               treatment     <- factor(rep(c(1, 2), c(43, 41)),                              levels = c(1, 2),                             labels = c("placebo", "treated"))     improved      <- factor(rep(c(1, 2, 3, 1, 2, 3), c(29, 7, 7, 13, 7, 21)),                             levels = c(1, 2, 3),                             labels = c("none", "some", "marked"))         numberofdrugs <- rpois(84, 10) + 1         healthvalue   <- rpois(84, 5)        y             <- data.frame(healthvalue, numberofdrugs, treatment, improved)     test          <- glm(healthvalue~numberofdrugs+treatment+improved, y, family=poisson)     summary(test)      The output is:               Coefficients:                      Estimate Std. Error z value Pr(>|z|)         (Intercept)       1.88955    0.19243   9.819   <2e-16 ***     numberofdrugs    -0.02303    0.01624  -1.418    0.156         treatmenttreated -0.01271    0.10861  -0.117    0.907   MAIN EFFECT       improvedsome     -0.13541    0.14674  -0.923    0.356   MAIN EFFECT      improvedmarke    -0.10839    0.12212  -0.888    0.375   MAIN EFFECT      ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1       I know that the incident rate for `numberofdrugs` is `exp(-0.023)=0.977`. But how do I interpret the main effects for the dummy variables?


title: intepreting Negative coefficients of Poisson model

text: i have fitted a Poisson regression to my claim frequency data. my predictor is make of vehicle. I have obtained the following result:               Coefficients:                   Estimate  Std. Error  z value Pr(>|z|)       (Intercept)  -19.99774  1138.82118  -0.018  0.98599       make2         -0.30873    0.20550  -1.502   0.13302      make3         -0.39177    0.21129  -1.854   0.06372      make4         -0.38375    0.13388  -2.866   0.00415 **      How do I interpret the negative coefficients? Thanks


title: Interpretation for a fixed effect model

text: firm            float   %9.0g                 firm ID     year            float   %9.0g                 year, 82-88     va              float   %9.0g                 value added in millions of lire     labor           float   %9.0g                 permanent employees at year end     capital         float   %9.0g                 fixed assets at historical cost in                                                     millions of lire     memb            float   %9.0g                 proportion of permanent employees who                                                     are members of the worker                                                     cooperative     bonus           float   %9.0g                 average distributed profits per                                                     worker in millions of lire     own             float   %9.0g                 average capital stake per                                                     worker-member in millions of lire     pindex          float   %9.0g                 price index for goods and services                                                     with 1985=1     kpindex         float   %9.0g                 price index for capital goods with                                                     1985=1          gen RVA = va/pindex     gen ROWN = own/pindex     gen RBONUS = bonus/pindex               COOP = a dummy variable for a worker cooperative.  (Use FIRM to create COOP.)     lnQ = the natural logarithm of output (real value added -- RVA)     lnL = the natural logarithm of labor     lnK = the natural logarithm of the real capital stock (RCAPITAL)                 . xtreg lnQ lnL lnK memb ROWN RBONUS t83 t84 t85 t86 t87 t88, i(firm) fe          Fixed-effects (within) regression               Number of obs      =       410     Group variable: firm                            Number of groups   =        87          R-sq:  within  = 0.2226                         Obs per group: min =         1            between = 0.6663                                        avg =       4.7            overall = 0.7219                                        max =         7                                                          F(11,312)          =      8.12     corr(u_i, Xb)  = 0.1447                         Prob > F           =    0.0000          ------------------------------------------------------------------------------              lnQ |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]     -------------+----------------------------------------------------------------              lnL |   .5887753    .088453     6.66   0.000     .4147355    .7628151              lnK |   .1322499   .0940503     1.41   0.161    -.0528031    .3173028             memb |   .3400343   .3263662     1.04   0.298    -.3021226    .9821913             ROWN |   .0114904   .0370171     0.31   0.756    -.0613443    .0843251           RBONUS |   .0508802   .0873018     0.58   0.560    -.1208946     .222655              t83 |  -.0421102   .0544063    -0.77   0.440    -.1491598    .0649395              t84 |  -.1505105   .0542093    -2.78   0.006    -.2571725   -.0438485              t85 |  -.1355591   .0541061    -2.51   0.013    -.2420181   -.0291002              t86 |  -.0414257   .0563853    -0.73   0.463    -.1523692    .0695178              t87 |   .0662506   .0587879     1.13   0.261    -.0494202    .1819215              t88 |   .0377116   .0654083     0.58   0.565    -.0909855    .1664087            _cons |   5.023615   .8488226     5.92   0.000     3.353475    6.693755     -------------+----------------------------------------------------------------          sigma_u |  .44866063          sigma_e |  .25416569              rho |  .75704764   (fraction of variance due to u_i)     ------------------------------------------------------------------------------     F test that all u_i=0:     F(86, 312) =     4.32             Prob > F = 0.0000      What features of the 87 firms in the sample might be captured by the firm effects? (Recall that the firm effects are not reported by STATA.) What might the fixed year effects (the coefficient on the YEAR dummy variables) be capturing?


title: Forecasting call volumes over short intervals using R

text: I am trying to do a basic forecast of call volumes using the forecast library for R. I am not having too much trouble forecasting on a daily or monthly interval, however when I try to forecast on an hourly or or quarter hourly basis I am having problems. The variation between points is so small as to be non existent. The data given are made up of aggregates for each period going back up to 24 months (I've tried shorter periods with no success). The points are only between 8am and 8pm, meaning that there are 12 points per day for hourly data and 48 points per day for quarter hourly. I believe the issue lies either with cycle_length or possibly needing to try a different smoothing algorithm, unfortunately my experience with statistics is so small as to be negligible. My code is as follows:               data <- read.csv(input_csv, header=FALSE)          ts_data <- ts(data, frequency = cycle_length)          ts_forecast <- forecast(ts_data, h = forecast_periods)          write.csv(ts_forecast, output_csv, row.names=FALSE)      Edit: As requested, a chart of some of the data. The full data set is 24 months long (I've experimented with using all or part of it). Obviously this is impractical to put up, so here is a randomly selected 7 day snapshot. Quarter hourly intervals 12 hours per day, meaning 48 points per day. ![Chart of call data](http://i.stack.imgur.com/WFqZo.png)


title: Does there exist a name for comparing individual grades to a set of grades?

text: I'm a complete newbie to statistics (although I find it really interesting!), and I have taken the task of distributing feedback to speakers of a conference I'm co-organizing. Each speaker was given a grade on the scale 1-5 from the participants, and we combine feedback from all participants into a mean score, for instance 3.56. We can then order the speakers by mean score. In addition to giving the speakers their mean score, we also want to give them a clue about how they did compared to the other speakers. To avoid discouraging the speakers who did the worst (we want people to try again!), we came up with giving back a more fuzzy metric. We want to divide the speakers into four groups: 10% best/20%/30%/40% worst. Does there exist a name for this? **EDIT** : I'll try an example. If I have ten talks with scores of { 1.2, 1.3, 2.1, 2.4, 2.7, 3.0, 3.2, 4.1, 4.2, 4.5}, I would divide them up like this: 10% best: 4.5   20% "next best": 4.2, 4.1   30% "next worst": 3.2, 3.0, 2.7   40% worst: 2.4, 2.1, 1.3, 1.2 What I want to know is if there is a name for these ranges.


title: How can one test the assumptions of a zero-inflated negative binomial model in R?

text: I have fitted a zero-inflated model with a random effect using a negative binomial distribution in R, using the function glmmadmb. This is due to a large number of zeros and over dispersion. For a standard poisson regression I know that one must test collinearity, leverage and stability of coefficients (DF Beta). I can do all this in R using various functions for a poisson regression, but none exist for glmmadmb that I can find. I wondered if anyone knew a way to test these? Thanks


title: Controlling covariates in linear regression in R

text: I have made these two models:               (model1 <- summary(lm(mpg ~ drat + wt + cyl, mtcars)))          Call:     lm(formula = mpg ~ drat + wt + cyl, data = mtcars)          Residuals:         Min      1Q  Median      3Q     Max      -4.2944 -1.5576 -0.4667  1.5678  6.1014           Coefficients:                 Estimate Std. Error t value Pr(>|t|)         (Intercept)  39.7677     6.8729   5.786 3.26e-06 ***     drat         -0.0162     1.3231  -0.012 0.990317         wt           -3.1947     0.8293  -3.852 0.000624 ***     cyl          -1.5096     0.4464  -3.382 0.002142 **      ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1          Residual standard error: 2.613 on 28 degrees of freedom     Multiple R-squared:  0.8302,    Adjusted R-squared:  0.812      F-statistic: 45.64 on 3 and 28 DF,  p-value: 6.569e-11               (model2 <- summary(lm(mpg ~  wt + cyl + drat, mtcars)))          Call:     lm(formula = mpg ~ wt + cyl + drat, data = mtcars)          Residuals:         Min      1Q  Median      3Q     Max      -4.2944 -1.5576 -0.4667  1.5678  6.1014           Coefficients:                 Estimate Std. Error t value Pr(>|t|)         (Intercept)  39.7677     6.8729   5.786 3.26e-06 ***     wt           -3.1947     0.8293  -3.852 0.000624 ***     cyl          -1.5096     0.4464  -3.382 0.002142 **      drat         -0.0162     1.3231  -0.012 0.990317         ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1          Residual standard error: 2.613 on 28 degrees of freedom     Multiple R-squared:  0.8302,    Adjusted R-squared:  0.812      F-statistic: 45.64 on 3 and 28 DF,  p-value: 6.569e-11      My understanding is that R uses "sequential" partitioning for the variance in `mpg`. So in `model1`, `drat` should be unadjusted, `wt` should be adjusted for `drat` and `cyl` should be adjusted for `drat` and `wt`. In `model2`, `wt` should be unadjusted, `cyl` should be adjusted for `wt` and `drat` should be adjusted for `wt` and `cyl`. However, the coefficients in each model appear to be exactly the same, suggesting coefficients are not being adjusted at all. Are the coefficients not being adjusted at all?


title: Rand index calculation

text: I'm trying to figure out how to calculate the Rand Index of a cluster algorithm, but I'm stuck at the point how to calculate the true and false negatives. At the moment I'm using the example from the book An Introduction into Information Retrieval (Manning, Raghavan & Schütze, 2009). At page 359 they talk about how to calculate the Rand index. For this example they use three clusters and the clusters contains the following objects.   1. a a a a a b   2. a b b b b c   3. a a c c c I replace the object (orginal signs to letters, but the idea and count stay the same). I'll give the exact words from the book in order to see what they are talking about: > We first compute TP +FP. The three clusters contain 6, 6, and 5 points, > respectively, so the total number of “positives” or pairs of documents that > are in the same cluster is: TP + FP = ${6 \choose 2}$ + ${6 \choose 2}$ + ${5 \choose 2}$ = 15 + 15+ 10 = 40 > Of these, the a pairs in cluster 1, the b pairs in cluster 2, the c pairs in > cluster 3, and the a pair in cluster 3 are true positives: TP = ${5 \choose 2}$ + ${4 \choose 2}$ + ${3 \choose 2}$ + ${2 \choose 2}$ = 10 + 6 + 3 + 1 = 20 > Thus, FP = 40 − 20 = 20. Till here there calculations are clear, and if I take other examples I get the same results, but when I want to calculate the false negative and true negative Manning et al. state the following: > FN and TN are computed similarly, resulting in the following contingency > table: The contingency table looks as follows:               +--------+--------+     | TP: 20 | FN: 24 |     +--------+--------+     | FP: 20 | TN: 72 |     +--------+--------+      The sentence: "FN and TN are computed similarly" is not clear to my and I don't understand which numbers I need to calculate the TN and FN. I can calculate the right side of the table by doing the following: TP + FP + FN + TN = ${n \choose 2}$ = ${17 \choose 2}$ = 136 Source: http://en.wikipedia.org/wiki/Rand_index Thus, FN + TN = 136 - TP + FP = 136 - 40 = 96, but this doesn't really help my in figuring out how to calculate the variables separately. Especially when the authors say: "FN and TN are computed similarly". I don't see how. Also when I look at other examples they calculate each cell of the contingency table by looking at each pair. For example: http://www.otlet- institute.org/wikics/Clustering_Problems.html#toc-Subsection-4.1 My first question, based on the example of Manning et al (2009), is it possible to calculate the TN and FN if you only know the TP & NP? And if so, how does the similar calculation looks like based of the given example?


title: Why the trees generated via bagging are identically distributed?

text: I have problem in intuitive understanding of following arguement: "The trees generated via bagging are identically distributed, thus the expectation of the average of a set of trees is the same as the Expectation of a single tree" 1) What does really distribution of tree mean? 2) Why, when there are generated via bagging are ID?


What is the difference between vector error correction model and UVECM? Which one is better to use for a 30 years time series data (to study long term and short term relationship) which has some missing values?


title: Weighting time series coefficients using model's likelihood

text: I have a question regarding to time series forecasting. In particular I've been working with a Bayesian approach, but I think the question is independent from that. I have several time series which are very stable in time, except on specific dates that they have sudden changes. The problem is that if I use a forecasting technique that looks at the past to predict the future, such as ARIMA, the days after the sudden changes have high impact on the forecast. Thus, to give a simple example, suppose I'm predicting $x_{t+1} = \sum \beta_j x_j, j<t+1$, I would like to add another weight witch accounts for the probability of $x_j$, something like $x_{t+1} = \sum f(x_j)\beta_j x_j, j<t+1$ where $f(x_j)$ is proportional to $P(x_j)$. Thus, a sudden change has low probability and should not contribute to the prediction. Does anyone know how to deal with these kind of problems? I'm trying to implement this in a Bayesian model, but I'm now sure how I should do it. Anyway, thanks Pablo


title: Non-Measure Theoretic Argument for Var(X) = 0 iff X is constant (X continuous RV)

text: I am studying out of DeGroot and Schervish trying to carefully understand the math of prob/stats. In ch 4.3 on variance, they state the theorem that given X a RV whose mean and var exist, then Var(X) = 0 iff X is a constant. The theorem itself is simple to understand: I am trying to get a firm grasp on how to prove it rigorously for the case where X is a continuous RV. The book gives a proof, which is not really self consistent with their treatment of expectations. I am thinking it is possible to prove this result (w/o the use of measure theory) via a tool they introduce earlier in which they use the fact that if $A_n$ are a collection of nested sets, then the limit of the probability that an even is in their intersection is the limit of the probability of $A_n$. (So in ths case, one could consider the collection of closed nested intervals $A_n$ = [c-$\frac{1}{n}$,c+$\frac{1}{n}$]. If one assume the pdf to be continuous then it seems you should be able to use this fact to argue that the limit of the expected values will be equal to the true expected value. I haven't covered limits of estimators though so even though I think intuitively this makes sense, I don't actually know if this method is rigorously correct. Does this sound like a proper way to proceed? TIA, JZ


title: Estimating Sample sizes for Null Hypothesis Testing

text: Wikipedia has the following table for computing sample sizes:                                                           ![enter image description here](http://i.stack.imgur.com/YFl1b.png) The table was generated for the following targets:   * Significance level of 0.05.   * Statistical power   * Cohen's d (i.e. the effect size) How does one compute this table? For a given input (desired $\alpha$, power and Cohen's d), how does one compute the minimum sample size? Is there an algebraic relationship between these quantities? Is this an inverse problem? Also, I presume that the above assumes that both groups have the sample size $n=n_1=n_2$. Is there a similar relationship (or equation) when $n_1$ and $n_2$ are allowed to be different?


title: Ratio between two densities

text: I would like to obtain the ratio between two distribution densities and I wonder whether there is some well-founded and established approach to do that. Let's say that I have two samples for the random variable $X$ coming from two populations, $P_1$ and $P_2$, that are likely to be distributed differently across the valid range of $X$. Then I'll like to at least approximate the function $f(X)$ of the ratio between the densities in both populations across the range of $X$. $$f(x)=\frac{Density(x|P_1)}{Density(x|P_2)}$$ What I am currently doing in R is to estimate the density of each sample using kernel density estimation (`density()` function). Then I can approximate $f(X)$ for any given value by interpolating the closest point estimates returned by `density()` on each sample and dividing the resulting values. Although this is satisfactory to some extent I wonder whether there is a better way. Specially, if there is an existing implementation in R. For example, I am afraid that the ratio could be extremely variable in regions where one or both distributions have very low density and I wonder whether there is a method to also obtain some estimate of uncertainty.


title: interpreting chi square value for validating random numbers

text: I am trying to use the chi-square test for identifying random data. I have 256 categories, i.e. 255 degrees of freedom and count the occurrence of byte- values (0-255). As suggested by Knuth, the observed sequence of bytes is random, if the chi-square value is between the chi-square-values of p=5% and p=95% (I read these value from the table). the test works but I am trying to understand why :) I have no idea about statistics, this is my problem right now. I am trying to understand what the p-value actually is and its relation to alpha. It is said that the null-hypothesis is rejected if the computed chi-statistic is above the chi-square value depending on alpha. Knuth has a table where all chi-square-values are listed depending on the p-value. can anybody explain to me why? Why does he say that the calculated chi-square-statistic MUST be between p=5% and p=95% if data is random? Thank you guys very much for any answer!


title: PyMC Robust Linear Regression with Measured Uncertainties

text: I posted this same question on Stack Overflow. I use least squares regression of data with measured errors in both x and y and use the reduced chi-square (mean square weighted deviation: mswd) as a measure of the fit. However, some of the assumptions for using reduced chi-squared likely are not met and I'd like to move towards an mcmc/bayesian approach using PyMC. I've searched the web but can't exactly seem to find what I'm looking for, most examples assume the data uncertainty is Gaussian, but here I have measured uncertainties in x and y. It seems like I should be able to do this in PyMC2 or PyMC3 with glm. Here's a typical dataset plotted up: ![enter image description here](http://i.stack.imgur.com/un9Rr.png) And the data to go with it:               # Data in Columns, Observations in Rows     # Measured values x versus y,      # Measured standard deviations sx and sy.                 x          sx         y           sy     0.3779397 0.001889699 0.5130084 2.748546e-05     0.3659092 0.001829546 0.5129624 2.721838e-05     0.3430834 0.001715417 0.5129023 2.720073e-05     0.4121606 0.002060803 0.5130235 2.755231e-05     0.3075815 0.001537908 0.5128739 2.776967e-05     0.3794471 0.001897236 0.5129950 2.842079e-05     0.1447394 0.000723697 0.5126784 2.816200e-05      I'm looking for any examples and references where people have done this. Thanks in advance.


title: Using R, can I feed a set of inputs in to autofit a distribution and derive a cumulative distribution function given any value?

text: I have some "polluted" distributions (i.e. distributions that are based on multiple variables quantified to a arbitrary % value). Right now we are just using normdist (i.e normal cumulative distribution function) to derive %'s for each value. Is there a better way to account for distributions that might be skewed or not normal using r? If so, could someone offer the commands? Thanks in advance. On the r-project homepage, I see a non normal distribution drawn and what looks like a cumulative density function applied to a set of values, possibly to find it's area to derive a % from the range. http://www.r-project.org/


title: Optimal Kernel Size for Lanczos resampling

text: The kernel for Lanczos Resampling is defined by $$K(u) = \frac{a\text{sin}(\pi u)\text{sin}(\pi u/a)}{\pi^2u^2}.$$ How does one go about finding a value of $a$ to minimize the mean squared error, $\text{MSE} = b^2 + \sigma^2$? Is there at least a scaling relationship with size of the data set, $n$, as there is with Gaussian Kernels? Note: I don't know if will matter, but lets assume a strictly positive distribution.


title: How to run a Simes test?

text: I'm looking to set up a Simes test in R. I'm trying to show that a Simes test is conservative with when a positive correlation exists. I know how correct the p-values using the p.adjust function, but I am looking to set up a simulation under the following conditions: 1.) Multivariate $\mathcal{N}$(0,1) distribution. 2.) $\alpha$ = .05 3.) number of k tests = 2, 20, and 100 4.) $\rho$ = 0, 0.5, and 0.9 I have this code as a reference to use:               library(MASS)     alpha <- .05     m <- 20     n.sim <- 10000     n.Rejection <- rep(0,n.sim)     rho<- 0     mu <- rep(0,m)     Sigma <- matrix(rho,m,m)     diag(Sigma) <- 1     for (i in 1:n.sim)     {     z <- rnorm(m)     z <- mvrnorm(1,mu,Sigma)     p <- pnorm(z)     n.Rejection[i] <- length(p[p<=alpha])     }


title: Statistical test to tell whether two samples are pulled from the same population?

text: Let's say I have two samples. If I want to tell whether they are pulled from different populations, I can run a t-test. But let's say I want to test whether the samples are from the same population. How does one do this? That is, how do I calculate the statistical probability that these two samples were pulled from the same population?


title: Should normalization completely weed out correlation?

text: I have two variables: ordering & length. The former measures the ordering of a sequence (i.e. all permutations of A-B-C), and the former is the length of the sequence (i.e. A-B-C has a length of 3). These are highly correlated, and I want to normalize the ordering measure by length. I was expecting this normalization to completely eradicate the correlation - but it doesn't. How can this be the case?               id      order   length  order/length     X1      4           3   1.333333333     X33     2           1   2     X566    44          6   7.333333333     X681    4           2   2     X682    46          6   7.666666667     X80     2           1   2          correlation before normalization: 0.958     correlation AFTER normalization: 0.610      The correlation has been reduced, but the variables are still highly correlated. My ambition was to partial out the component of "order" that is separate from "length", but it doesn't seem like I'm achieving that here. How can I do this? Where has my thinking gone wrong?


title: How to standardize data

text: I have the test scores of two groups, say A, and B. And the former consists of 186 individuals whereas the latter only has 100. The test scores range from 1 to 12, and because group A has more people, it obviously has a total higher score than group B. My question is how can I standardize the test scores so that I can compare them? I thought about dividing group A's scores by 186 and group B's by 100...would that work? Edit: The scores are divided into 7 groups, including agitation, irritability, sleep, etc. Here is an example of one.                   >table(status, sleep_scores)         status   0   1   2   3   4   6   8   9          0     170   3   3   7   0   1   1   1          1      82   5   3   6   3   1   0   0      Status is a binary variable, 0 meaning that the participant does not have the disease, and 1 otherwise. So in the table above, 170 of the normal controls (aka people who do not have the disease) score a 0 on the test (the lower the score, the better the performance). 3 people score a 1, where as in the disease group, 5 people scored 1. So I have 7 sets of these scores, and they are pretty similar to one another (most people score 0's, only a few score > 0). So I would like to compare which group is doing worse? And in which category are they doing worse in? Sleep, agitation, etc?


title: Confusion related to calculation of likelihood

text: I was reading this paper related to Learning from multiple annotator using Gaussian processes. The idea is if we don't have the actual ground truth of a certain data, but only the labels from some noisy experts, then how can we learn a model from this data from multiple noisy experts and predict on future data Basically, what the guys have done is taken the weighted average of the labels from individual predictors, based upon their precision, to get a single label for each point. Then they have used GP to assume that the data points belong to a Gaussian distribution. However, I have this question/confusion when they have derived the likelihood of $p(Y)$ where $Y$ is the set of labels given by the noise experts. I have added the screenshot as well ![enter image description here](http://i.stack.imgur.com/6vbLU.png) I know the one who can answer this question has to go through the paper and all, but I have already spend significant time trying to figure out how this $-\log p(Y)$ turned out like that but couldn't. I would really appreciate if someone could give me some pointers


title: Defintion for model diversity?

text: Two models are diverse if they make prediction errors on different instances. I know there are different measures to quantify diversity, however, I'm looking for formal conceptual definition of what we are trying to measure. Any suggestions?


title: decision tree : categorical vs numeric explanatory variables

text: I found similar topic but it dealt with the question whether it's better to discretize numerical dependent variable or leave it as it is. I'd like to find out what are your experiences with training decision trees using continous explanatory variables. Is it better to convert them to categorical/ordinal values? If so what methods can be used : should we take quantiles of their distributions?


title: Relationship between number of training set and classification performance

text: Are there any research/paper on the relationship between the number of documents for training and the classification performance using support vector machine?


title: Does a sample version of the one-sided Chebyshev inequality exist?

text: I am interested in the following one-sided Chebyshev inequality: $$ \mathbb P(X - \mathbb E (X) \geq t) \leq \frac{\mathrm{Var}(X)}{\mathrm{Var}(X) + t^2} \,. $$ Basically, if you know the population mean and variance, you can calculate the upper bound on the probability of observing a certain value. (That was my understanding at least.) However, I would like to use the sample mean and sample variance instead of the actual population mean and variance. I am guessing that since this would introduce more uncertainty, the upper bound would increase. Is there an inequality analogous to the above, but that uses the sample mean and variance? **Edit** : The "sample" analog of the Chebyshev Inequality (not one sided), has been worked out. The Wikipedia page has some details. However, I am not sure how it would translate to the one sided case I have above.


title: Non-parametric regression

text: I am conducting a mulitple first order regression analysis of genetic data. The vectors of y-values do not all follow a normal distribution, therefore I need to implement a non-parametric regression using ranks. Is the `lm()` function in R suitable for this, i.e.,               lin.reg <- lm(Y~X*Z)      where Y, X and Z are vectors of ordinal categorical variables? I am interested in the p-value assigned to the coefficient of the interaction term in the first order model. The `lm()` function obtains this from a t-test, i.e., is the interaction coefficient significantly different from zero. Is the automatic implementation of a t-test to determine this p-value appropriate when the regression model is carried out on data as described? Thanks. **EDIT** Sample data for clarity:               Y <- c(4, 1, 2, 3) # A vector of ranks     X <- c(0, 2, 1, 1) # A vector of genotypes (0 = aa, 1 = ab, 2 = bb)     Z <- c(2, 2, 1, 0)


title: ROC plot for continuous data in R

text: I am currently estimating a bunch of ARMA models, and using them to predict subsets of my data. In order to evaluate their predictive accuracy I would like to make some ROC plots, however since all of my variables are continuous, I wonder how this could be done in R. Best, Thomas P.S: I have looked at the ROCR package, but this seems to only work for dichotomous variables.


title: Orthogonal sampling, latin hypercubes and low discrepancy sequences

text: What is the difference between each of them? this wiki page - http://en.wikipedia.org/wiki/Latin_hypercube_sampling says that: "In Orthogonal Sampling, the sample space is divided into equally probable subspaces", which to me sounds like a property of low discrepancy sequence.


title: Conditional probability - signal extraction

text: Suppose I have $Z=X+Y$, $X$,$Y$ normally distributed. How do I calculate: $Pr(X>X'|Z,Z')$ when I have two realizations, $X$,$X'$?


title: What distribution has the maximum entropy for a known mean absolute deviation?

text: I was reading the discussion on Hacker News about the use of the standard deviation as opposed to other metrics such as the mean absolute deviation. So, if we were to follow the principle of maximum entropy, what kind of distribution would we use with if we only knew the distribution's mean and the mean absolute deviation? Or does it make more sense to use the median and the mean absolute deviation from the median? I found a paper Maximum Entropy Principle with General Deviation Measures by Grechuk, Molyboha and Zabarankin which appears to have the information I am curious about, but it is taking me a while to decipher it.


title: How to fix one coefficient and fit others using regression

text: I would like to manually fix a certain coefficient, say $\beta_1=1.0$, then fit coefficients to all other predictors, while keeping $\beta_1=1.0$ in the model. How can I achieve this using R? I'd particularly like to work with LASSO (`glmnet`) if possible. Alternatively, how can I restrict this coefficient to a specific range, say $0.5\le\beta_1\le1.0$?


title: Please help me interpret these Wilcoxon signed-rank test results

text: I have 4 sets of data points and for each I want to see how much these values vary from zero. I used the tool for calculating the results of the test on this page but I'm having trouble understanding the results. The results for my 4 sets of data points are:               W+ = 3,       W- = 12,      N = 5,   p <= 0.3125     for 101 data points     W+ = 12,      W- = 3,       N = 5,   p <= 0.3125     for 101 data points     W+ = 451,     W- = 1089,    N = 55,  p <= 0.007618   for 297 data points     W+ = 950.50,  W- = 427.50,  N = 52,  p <= 0.01746    for 297 data points      Please could you tell me which values I should be looking at and what the mean in terms of the above results? Do the low p values mean that the variance from zero is not significant? Thanks.


title: Online site to easily generate stats on census data

text: If I wanted to answer the following question, is there any website which will allow me to do it? (or has it already been done?) What percentage of people with Computer Science Degrees aged 30-40 are Married? What percentage of people with other degrees aged 30-40 are Married? Surely somewhere online there is a site where I can put criteria like that in and generate basic results?


title: What's the sense and the benefit of auto-encoder algorithm?

text: I have been studying **auto-encoders** in this period. If I understood correctly, an auto-encoder is a neural network where the input layer is identical to the output layer. So, the neural network tries to predict the output using the input as golden standard. I read a bunch of documents about this, and also watched some video clips, but I cannot understand this concept... **What is the usefulness of this algorithm? What are the benefits** of trying to reconstruct some output elements, making them as equal as possible to the input elements? **Why should one use all this machinery to get to the same starting point?** I know this question may seem weird...but I really really have no idea of how to solve this.


title: Checking that values are piecewise uniform

text: I have a set of values and I wish to check if they are piecewise uniform. I hope I'm using the correct terms, but I'll explain what I mean. Consider the following values - 100,105,100,103,98. We can "see" they are close to each other. We can define this uniformity by normalizing the Standard deviation. Now let's look at the following values - 100,105,100,103,98,198,200,203,205,100,105,100,103,98 Standard deviation will not work here, but it's clearly uniform in each segment. To check the uniformity here we can check how many of the sample deviate by no more than d% from their predecessor. This seems to work but (and here's the question) - is there a better approach? I don't even know how to call this test to try and search for articles. This doesn't handle outliers very well. I can use a moving average or a moving median, and again - is there something better? Thanks.


title: Is the R-squared for this model respectable?

text: I have a model where the political position (left-right) of the coalition in power, among other things, explains stock returns. All data is monthly. See the results below. The question is, would you consider the ~28% R-squared high enough? The financial data is all returns, so I believe it should be stationary. Any other comments on what you see? ![enter image description here](http://i.stack.imgur.com/wFAMM.png)


title: How to conduct am A/B test for a feature which cannot be accessed by every visitor?

text: I am prepping for an interview and here is one of the questions I have come across: `How would you conduct an A/B test on an opt-in feature?` I think you cannot perform a "fair" test with an opt-in option. People viewing the A version of the site will come from the whole distribution of users whereas people viewing B will only come from the set who have opted in. This seems to violate the conditions for a controlled experiment. What is your take on this?


title: Pearson Correlation with multiple observations per each level of independent variable

text: I am giving different doses of a drug to mice and measuring a quantitative response. I want to determine if there is a linear relationship between the dose of the drug and the response. Pearson correlation seems to be able to do that. But I am testing multiple mice per each dose, so I have several observations of the dependent variable per each level of the independent variable. And I am interested in the correlation between the dose and the true population mean of the response in each dose. I would like to know the r and p values (in standard Pearson terminology), i.e. how strong the relationship is and how statistically significant it is. So I need something like Pearson correlation on repeated measures, but I wasn't able to find any application of that (although I thought it is a pretty typical situation and common analysis). Could you please suggest how to do such an analysis? Thank you.


title: How to do a generalized linear model with multiple dependent variables in R?

text: I have six dependent variables (count data) and several independent variables, I see that in a MMR the script goes like this:               my.model <- lm(cbind(DV1,DV2,DV3,DV4,DV5,DV6) ~ IV1 + IV2 + ... + IVn)      But, since my data are counts, I want to use a generalized linear model and I tried this:               my.model <- glm(cbind(DV1,DV2,DV3,DV4,DV5,DV6) ~ IV1 + IV2 + ... + IVn, family="poisson")      And appears this error message:               Error in glm.fit(x = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,  :        (subscript) logical subscript too long`      Can anyone explain me this error message or a way to solve my problem?


title: What's the probability that as I roll dice I'll see a sum of $7$ on them before I see a sum of $8$?

text: This question is from DEGROOT's PROBABILITY and STATISTICS. Problem Suppose that two dice are to be rolled repeatedly and the sum $T$ of the two numbers is to be observed for each roll. We shall determine the probability $p$ that the value $T =7$ will be observed before the value $T =8$ is observed. Solution The desired probability $p$ could be calculated directly as follows: We could assume that the sample space $S$ contains all sequences of outcomes that terminate as soon as either the sum $T = 7$ or the sum $T = 8$ is obtained. Then we could find the sum of the probabilities of all the sequences that terminate when the value $T = 7$ is obtained. However,there is a simpler approach in this example. We can consider the simple experiment in which two dice are rolled. If we repeat the experiment until either the sum $T = 7$ or the sum $T = 8$ is obtained, the effect is to restrict the outcome of the experiment to one of these two values. Hence, the problem can be restated as follows: Given that the outcome of the experiment is either $T = 7$ or $T = 8$, determine the probability $p$ that the outcome is actually $T = 7$. If we let $A$ be the event that $T = 7$ and let $B$ be the event that the value of $T$ is either $7$ or $8$, then $A ∩ B = A$ and $$ p = Pr(A|B) = \frac{Pr(A ∩ B)}{Pr(B)} =\frac{Pr(A)}{Pr(B)} $$ But $Pr(A) = 6/36$ and $Pr(B) = (6/36) + (5/36) = 11/36$. Hence, $p = 6/11$. Now, my doubts are   1. Why does the author say  > We could assume that the sample space $S$ contains all sequences of outcomes > that terminate as soon as either the sum $T = 7$ or the sum $T = 8$ is > obtained. Then we could find the sum of the probabilities of all the > sequences that terminate when the value $T = 7$ is obtained. >   2. How can we go from lengthy sequences of outcomes that terminate as soon > as either the sum $T = 7$ or the sum $T = 8$ is obtained to just the outcome > of the experiment for which either $T = 7$ or $T = 8$ ? >


title: Confusing Results in a Cross-Correlogram

text: My hypothesis was that Consumer Sentiment (how people feel about my market) influenced the DowJones, I ran some basic VAR and Granger causality tests and found that the exact opposite is true, that the DowJones influences Consumer Sentiment at a 99% confidence interval (just some background). Regardless, I was hoping that someone could help me explain whats going on, or at least come up with a theory as to why this looks so different from the basic Cross-correlogram without the first difference. If you want me to attach any other models just let me know. "Cross-Correlogram of Consumer Sentiment and first-difference DowJones Index (monthly average)" _What I want to interpret_ )![](http://i.stack.imgur.com/WwpPu.png) "Cross-Correlogram of Consumer Sentiment and DowJones Index (monthly average)" ![](http://i.stack.imgur.com/UK2zG.png) Thanks in advance!


title: Comparing results from linear quantile regression with mean regression

text: Say we fitted quantile regression models with a set of quantiles between 0 and 1 and a linear regression (i.e. mean regression) to a same dataset with the same set of covariates. In terms of the estimate for regression coefficient for a specific covariate, say x, we will get nine estimates from the quantile regression corresponding to each of the nine quantiles and only one estimate from the mean regression. If we plot those estimates in the same plot, with y-axis represents the values of the estimate and x-aixs for the quantiles, we should get a horizontal line for the estimate from mean regression and a zigged line connecting the estimates from quantile regression. My question is if these two lines must cross at some point (i.e. some quantile) in every case or is it possible for them not to cross with each other for all possible quantiles? Is the conclusion consistent in the setting of linear quantile mixed model and linear mixed model? edit @05/13: sample plot inserted ![the quantile regression \(solid\) doesn't not cross with mean regression line \(dotted\)](http://i.stack.imgur.com/Q8KbO.png) Thanks.


What does "ordinate" in "posterior ordinate" mean? For example, > The starting point in this computation is a decomposition of the **posterior > ordinate** into **marginal and conditional ordinates** followed by a Monte > Carlo estimate ... I only know the meaning of "ordinate" wrt abscissa.


title: Model fit is High but Ramsey RESET Test suggests omitted variables. What to do?

text: I'm trying to figure out what next steps to take. I created a model and ran OLS on a very large sample of data (over 400000 observations) and got an R-squared value of 0.80. So the model fit seems really good. However, I ran a Ramsey RESET test and its test statistic strongly suggested that there were omitted variables. I'm not sure what do do next. Do I just throw away the model, saying that the estimates are biased. Do I keep adding terms until the RESET test no longer suggests omitted variables? The model is intended to be predictive within and out of the sample and given the model fit to the observed data, is it still truly a good fit, despite there being omitted variables? Thanks!


title: Testing proportional hazards assumption

text: Can you use the log rank test or the Wilcoxon test to test the proportional hazards assumption in the Cox model?


title: Weighted sum of Dirichlets is Dirichlet

text: I am studying Sethuraman's paper on the Dirichlet process and having difficulty showing a lemma. He states: Let $\boldsymbol\gamma=(\gamma_1,...\gamma_k)$ and $\gamma=\sum_j \gamma_j$ and let $\beta_j=\frac{\gamma_j}{\gamma}$, $j=1,2,...,k$. Then, $\sum_j \beta_j\mathcal{D}_{\boldsymbol\gamma+e_j}=\mathcal{D}_{\boldsymbol\gamma}$ In this lemma $\mathcal{D}_{\boldsymbol\gamma}$ is the Dirichlet distribution with $\boldsymbol\gamma$ as the parameters and $e_j$ is a vector of 0s with a 1 in the jth position. Sethuraman says proofs of this lemma are found in many standard text books, including Wilks (1962). I am trying to prove this lemma. I have not found it in any textbooks and cannot find a copy of the Wilks book. I have tried it many ways. Notably, by expressing the Dirichlets as gammas and for k=2.


title: Regarding the feature generation method with SVM-based classification method

text: When using SVM to build classifier for a collection of documents, we can use term occurrence, term frequency or even TF/IDF. I would like to know what are the main disadvantages of using term occurrence as the vector generation method? I once heard that this vector generation method can cause support vector machine to fail. How to explain this?


title: what is the way of dealing with textual valued feature vectors for classification task?

text: I aim to work on twitter data for sentiment analysis but I am curios for the way of dealing such a huge number of textual features (words). Is using the Bag-Of-Words approach is the best? However I've learnt that using all the words on tweets is better approach so how the state of art methods dealing with such a textual classification problem?


title: Exponential regression: calculating p-value and F significance

text: I have a set of independent data and dependent data $(X,Y)$, where I would like to do an exponential regression to obtain its p-value and significant $F$ (already obtained $R^2$ and also the coefficients through mathematical calculation). Often an exponential data, $y=be^{mx}$ will be converted first to a linear data, $\ln y = mx + \ln b $. Then a linear regression will done on the converted data, obtaining its p-value etc. Assume we use a statistical tool such as Excel's Analysis ToolPak: Data Analysis : Regression, it will produce a result such as below, ![enter image description here](http://i.stack.imgur.com/pCOPq.png) I believe the p-value and Significant $F$ value above is representing the converted linear data and not the original exponential data. Questions:   1. What is the approach/steps used by Excel to get the p-value and Significant F value for the converted linear data as shown in the statistic output in the image above? It is not clear in their help page or website.   2. Can the p-value and Significant F could be mathematically calculated for **exponential regression** without using a statistical tool? Can you assist to point me to the right link if this has been answered before. Spent a week on the internet to study this but unable to find the right answer.


title: Why do the results change when using Anova

text: So I ran an a ANOVA with two Independent variables (IV) and one Dependent variable (DV). One IV is categorical the other is continuous. DV is also continuous. However when I run the data again through SPSS, after removing the continuous IV from the analysis, the main effect for my earlier categorical IV disappears and as such I am left with a non-significant main effect for my categorical IV. Why is this happening?


title: Law of total probability on conditional

text: I often saw a formula, used mostly to integrate on the parameter space like: $$ p(x|y) = \int p(x|\theta) p(\theta|y) d\theta $$ where $\theta$ is the parameter. I am confused and I hope to explain me how is that. I know that $p(x) = \int p(x,\theta)d\theta = \int p(x|\theta)p(\theta)d\theta $. If I use a similar idea which in my mind means "a marginal distribution is obtain by integrating a join distribution over one variable", but with a conditional, I would expect to have: $$ p(x|y) = \int p(x,\theta|y)d\theta = \int p(x|\theta,y)p(\theta|y)d\theta$$ which obviously not not look the same. Any help with that? Later edit: The first formula was not correct, the integration is over $\theta$. Thanks


title: Help clarify the implication of linearity in an Ordinary Least Squares (OLS) Regression

text: > **If the dependent variable is linearly related to the independent > variables, there should be no systematic relationship between the residuals > and the fitted values**. In other words, the model should capture all the > systematic variance present in the data, leaving nothing but random noise. I don't understand the implication in bold. I don't see how the conclusion (no systematic relationship between the residuals and the fitted values) follows from the hypothesis (the dependent variable is linearly related to the independent variables). Can someone please explain it in the context of a simple linear regression?


title: Maximum likelihood estimation and the n-th order statistic

text: Let $X_1, ..., X_n$ be a sample of independent, identically distributed random variables, with density $$ f_{\theta}(x)=e^{ (\theta -x)}$$. $x \ge \theta$, otherwise $f_\theta = 0$ The question is: Determine the maximum likelihood estimator $\hat{\theta}_n$ of $\theta$. I don't understand this question. What exactly does $\hat{\theta}_n$ mean? Wikipedia says something about the nth order statistic: > In statistics, the kth order statistic of a statistical sample is equal to > its kth-smallest value. Together with rank statistics, order statistics are > among the most fundamental tools in non-parametric statistics and inference. I tried: $$ L(\theta)=\prod_{i=1}^ne^{ (\theta -x_i)} = e^{(n\theta - \sum_{i=1}^n{x_i})} $$ What's next?


title: How should I handle a left censored predictor variable in multiple regression?

text: I have a dataset (N=350) for which I would like to regress a neuropsychological test score (continuous) on age, education, symptom severity (continuous), and diagnosis (binary). Symptom severity is censored: symptom severity score was only generated if a participant passed a screening item. It appears that a tobit model would work for predicting symptom severity as the dependent variable, but is there a defensible method to include my left- censored variable as a predictor?


title: Multiple definitions of AdaBoost

text: The description of AdaBoost in Kevin Murphy's Machine Learning book (shown in a snapshot below) differs from the one in Wikipedia. I am trying to relate both definitions. Step by step, my questions are:   1. What exactly is $\text{err}_m$ (step 4) supposed to capture below? Is this equivalent to the $\epsilon_t$ in Wikipedia's definition?   2. Why isn't there a stopping rule in Kevin Murphy's method but there is one in Wikipedia's definition?   3. There seems to be a typo in the parenthesis in the denominator, just in case - is it supposed to say the following?: $\text{err}_m = \frac{\sum_{i=1}^N w_{i,m} \mathbb{I}(\hat{y} \neq \phi(\mathbf{x_i)}}{\sum_{i=1}^N w_{i,m}} $   4. Most importantly, Wikipedia provides the following criteria for **choosing the weak learner** and for stopping: > $h_{t} = \underset{h_{t} \in \mathcal{H}}{\operatorname{argmax}} \; > \left\vert 0.5 - \epsilon_{t}\right\vert$ where $\epsilon_{t} = > \sum_{i=1}^{m} D_{t}(i)I(y_i \ne h_{t}(x_{i}))$ > > $If \left\vert 0.5 - \epsilon_{t}\right\vert \leq \beta$, where $\beta$ is a > previously chosen threshold, then stop. while Kevin's book defines the full algorithm as follows, and I don't see those two steps above in it: ![enter image description here](http://i.stack.imgur.com/YejfD.png)


title: How does neural network recognise images?

text: I am trying to learn how Neural Network works on image recognition. I have seen some examples and become even more confused. In the example of letter recognition of a 20x20 image, the values of each pixel become the input layer. So 400 neurons. Then a hidden layer of neurons and 26 output neurons. Then train the network, and then it works, not perfect. What confused me about the Neural Network is, how it learns about what's in an image. You don't need to do thresholding,or segmentation, or measurement, somehow the network learns to compare images and recognize. It is like magic to me now. Where to start to learn neural network?


title: Is this test answer good enough to show Granger Causality?

text: I used an inbuilt Matlab function to check for Granger Causality between two time series P and T that have a correlation coefficient of 0.6. The function is : https://www.mathworks.com/matlabcentral/fileexchange/25467-granger-causality- test/content/granger_cause.m On running it, I got the following answer:               >> [fs,cv] = granger_cause(P, T,0.05,2)          fs =            1.0281e+003               cv =              2.9966      where FS is the F-Statistic and cv is the critical value from the F-Distribution. Does this show causality? Actually... what does it show?


title: Fama and French Three Factors: A time series analysis

text: Is there any literature on whether the three factors in the Fama–French three- factor model follow any kind of time series models, such as multivariate ARMA?


title: Which kernel function for Watson Nadaraya classifier?

text: I am trying to implement a Watson Nadaraya classifier. There is one thing I didn't understand from the equation: $${F}(x)=\frac{\sum_{i=1}^n K_h(x-X_i) Y_i}{\sum_{i=1}^nK_h(x-X_i)}$$ What should I use for the kernel K? I have a 2-dimensional dataset which has 1000 samples (each sample is like this: `[-0.10984628, 5.53485135]`). What confuses me is, based on my data, the input of the kernel function will be something like this:               K([-0.62978309,  0.10464536])      And what I understand, it'll produce some number instead of an array, therefore I can go ahead and calculate F(x) which will also be a number. Then I'll check whether it is > or <= than zero. But I couldn't find any kernel that produces a number. So confused. Edit: I tried to implement my classifier based on the comments, but I got a very low accuracy. I appreciate if someone notices what's wrong with it.               def gauss(x):             return (1.0 / np.sqrt(2 * np.pi)) * np.exp(- 0.5 * x**2)          def transform(X, h):             A = []             for i in X:                     A.append(stats.norm.pdf(i[0],0,h)*stats.norm.pdf(i[1],0,h))             return A                   N = 100         # pre-assign some mean and variance         mean1 = (0,9)         mean2 = (0,5)         cov = [[0.3,0.7],[0.7,0.3]]              # generate a dataset         dataset1 = np.random.multivariate_normal(mean1,cov,N)         dataset2 = np.random.multivariate_normal(mean2,cov,N)         X = np.vstack((dataset1, dataset2))              # pre-assign labels         Y1 = [1]*N         Y2 = [-1]*N         Y = Y1 + Y2         # assing a width         h = 0.5              #now, transform the data         X2 = transform(X, h)              j = 0         predicted = []              for i in X2:                 # apply the equation                 fx = sum((gauss(i-X2))*Y)/float(np.sum(gauss(i-X2)))                 # if fx>0, it belongs to class 1                 if fx >0:                         predicted.append(1)                 else:                         predicted.append(-1)                 j = j+1


title: Best test to assess statistical difference between two groups on a set of 5-point questionnaire items

text: I have a set of data which contains answers to a questionnaire (questions about satisfaction rated 1 to 5) and some demographic information (sex, age). I can divide the data in two groups and I want to see if there is any statistical difference between those groups. I tried the Wilcoxon test, but I have many ties, so I'm not sure this is the test that I need. What would be the best test in this case?


title: What statistical model can be used to find how following variables relate?

text: I want to find the relationship between a disease incidence in a region and a number of other environmental factors such as temperature, elevation etc. I have tables containing this data for a particular country. The data files are raster files of the country map with each pixel having certain numeric value for the particular parameter. Example [126, 540, 359...] say disease cases or [23.34, 19.02...] for temperature etc. I have read about **multiple regression** , **Pearson Chi square test** , **t-tests** etc. But I didn't find them suitable: Multiple regression because what I read it assumes a linear relationship between the variables. Chi square test applies to categorical data. t-tests for small datasets. I've also read about **EOF analysis** (Empirical Orthogonal Function) but it requires time series data. **CCA** (Canonical Correlation Analysis) but it finds the relationship between two sets of multiple variables. **Which statistical method can I use to find the relationship between these variables?** I want to use a method that gives a relationship, say, more is the elevation, more is the number of disease incidence etc. Without assuming anything before hand like, "there'd be a linear relationship between the variables, so using linear relationship model." _Also, is some method I mentioned appropriate but my understanding is not correct?_ I don't necessarily need to find relationship one to many, it'd be fine to use one on one methods. On a side note, I'm working with GIS data and would use Python or R modules to find and plot the relationship. Also, the arrays don't have same size. Thanks.


title: Testing for validity of a formative measurement model

text: I have developed a formative model which tests the relationship between two composites. Obviously, I have to ascetain the validity of this model. I am told that Confirmatory Factor Analysis (CFA) is the way to go on it.   1. How do I perform CPA in SPSS?   2. How do I interpret the results?


title: How can I calculate a sample size for a ranked list of items across a population?

text: I have a population - in this case they are phone calls into a call center. Each call can be one of approximately 200 'Problems'. (an example of a Problem is 'I cannot connect to the internet'). I would like to predict the top 10 most common problems across the entire population based on analysis of a sample set of phone calls. How many phone calls do I need to categorize in order to generate this list with different confidence levels and population sizes? Ideally I would like to find a formula which allows me to derive sample size from population size, problem set size and confidence level. Disclosure: \- I am a programmer, not a mathematician - I hope I have asked this question in the correct manner! \- I have done reading prior to asking this question, most content I can find is about estimating p(x) in a population, my problem has more dimensions. Sorry if this is a duplicate and I am just not smart enough to realize!