title: Generative vs. discriminative

text: I know that generative means "based on $P(x,y)$" and discriminative means "based on $P(y|x)$," but I'm confused on several points:   * Wikipedia (+ many other hits on the web) classify things like SVMs and decision trees as being discriminative. But these don't even have probabilistic interpretations. What does discriminative mean here? Has discriminative just come to mean anything that isn't generative?   * NB is generative because it captures $P(x|y)$ and $P(y)$, and thus you have $P(x,y)$ (as well as $P(y|x)$). Isn't it trivial to make, say, logistic regression (the poster boy of discriminative models) "generative" by simply computing $P(x)$ in a similar fashion (same independence assumption as NB, such that $P(x) = P(x_0) P(x_1) ... P(x_d)$, where the MLE for $P(x_i)$ are just frequencies)?   * I know that discriminative models tend to outperform generative ones. What's the practical use of working with generative models? Being able to generate/simulate data is cited, but when does this come up? I personally only have experience with regression, classification, collab. filtering over structured data, so are the uses irrelevant to me here? The "missing data" argument ($P(x_i|y)$ for missing $x_i$) seems to only give you an edge with training data (when you actually know $y$ and don't need to marginalize over $P(y)$ to get the relatively dumb $P(x_i)$ which you could've estimated directly anyway), and even then imputation is much more flexible (can predict based not just on $y$ but other $x_i$'s as well).   * What's with the completely contradictory quotes from Wikipedia? "generative models are typically more flexible than discriminative models in expressing dependencies in complex learning tasks" vs. "discriminative models can generally express more complex relationships between the observed and target variables" Related question that got me thinking about this.


title: How do I do a Monte Carlo simulation?

text: Which software is better? R, Excel or others. Would appreciate recommendations of good resource websites/books that you found useful.


title: Machine learning task with feedback loop

text: What are the available options if I want to perform a scoring task on a set of observations that: a) have a set of variables connected to them and, b) each round I get new information about the success of the latest round. What I mean is, I start with a normal scoring task: Using the available training data I want to rank each observation in a new dataset. This part is straightforward and can naturally be performed by a range of different methods. However, the nature of problem is that I get feedback from this ranking when see how observations react. Specifically, when a number of observations are picked from those with the highest score, the feedback consists of these observation either reacting positively or negatively to them being picked. This information can in turn be used for the next round, which is performing a new scoring and picking the next observations with the highest scores. What mathematical methods can be used to process this feedback information? How should/could the feedback be used in conjunction with the original (background) information? What R packages are there for these types of on-line machine learning problems?


title: Using regression forest for a factor variable

text: I want to predict if a customer is interested in a new product and I use the randomForest package for that. Target variable : factor (Yes or No) _so I use the randomForest for classification_ :               randomForest(x=train,y=labels_train,xtest=test, ytest=labels_test,  ntree=100)      Variables :   * the city (A,B,C)   * Gender : Male / Female   * Age classes : 18-25, 25-59 ,60 and + **Problem :** only 40% of the residents of the city A are interested (same proportion for both gender and all age classes). In this case, ALL the tree of the forest assign "Not interested" for the residents of this city. According to the forest these residents have a probability of 0% to be interested by the product. **Proposition :** Using regression instead of classification. In this way, each tree will declare 40%, and the vote is replace by an average wich give a probability of 40% Could you confirm it's correct to do that when the goal is not really to classify but to have a probability of interest, in order to only contact people who have a probability higher than 30% for example.


title: Is it reasonable for a classifier to obtain a high AUC and a low MCC? Or the opposite?

text: Let's say I have 2 models: 1) High Matthew's correlation coefficient (MCC) score, low area under the curve (AUC) 2) Low MCC, high AUC When I say high and low, I mean relatively to the other model. I'm not quite sure which model is "better" and how to interpret this difference between the 2 models. Also to clarify, these 2 models both return probability estimates. MCC is determined at a threshold of 0.5.


title: Is this a valid markov chain?

text: ![Markov Chain](http://i.stack.imgur.com/alLc5.jpg) I have the Markov chain presented in the image. It hast 4 states S=(1,2,3,4). The transitional probabilities are all $\frac{1}{2}$ and the direction from one state to another is given by the arrows. Is there an error with the original problem? We have 3 arrows from state 2, so $p_{24}=1$, but $p_{21}=\frac{1}{2}$. Can you confirm that this is not possible?


title: Dummy variable trap issues

text: I am running a large OLS regression where all the independent variables (around 400) are dummy variables. If all are included, there is perfect multicollinearity (the dummy variable trap), so I have to omit one of the variables before running the regression. My first question is, which variable should be omitted? I have read that it is better to omit a variable that is present in many of the observations rather than one that is present in only a few (e.g. if almost all observations are "male" or "female" and just a few are "unknown", omit either "male" or "female"). Is this justified? After running the regression with a variable omitted, I am able to estimate the coefficient value of the omitted variable because I know that the overall mean of all my independent variables should be 0. So I use this fact to shift the coefficient values for all the included variables, and get an estimate for the omitted variable. My next question is whether there is some similar technique that can be used to estimate the standard error for the coefficient value of the omitted variable. As it is I have to re-run the regression omitting a different variable (and including the variable I had omitted in the first regression) in order to acquire a standard error estimate for the coefficient of the originally omitted variable. Finally, I notice that the coefficient estimates I get (after re-centering around zero) vary slightly depending on which variable is omitted. In theory, would it be better to run several regressions, each omitting a different variable, and then average the coefficient estimates from all the regressions?


title: How to add horizontal lines to ggplot2 boxplot?

text: I have a boxplot output in R using ggplot2:               p <- ggplot(data, aes(y = age, x = group))     p <- p + geom_boxplot()     p <- p + scale_x_discrete(name= "Group",)     p <- p + scale_y_continuous(name= "Age")     p      ![ggplot2 boxplots](http://i.stack.imgur.com/fxI8B.png) I need to add horisontal lines like on common boxplot (and to change vertical line style if possible):               boxplot(age~group,data=data,names=c('1','2'),ylab="Age", xlab="Group")      ![common boxplots](http://i.stack.imgur.com/hvP44.png) **How I could do this using ggplot2?**


title: Sample size required to determine which of a set of advertisements has the highest click through rate

text: I am a software designer by trade and I am working on a project for a client, and I would like to make sure that my analysis is statistically sound. Consider the following: **We have _n_ advertisements (n < 10), and we simply want to know which ad performs the best.** Our ad server will randomly serve one of these ads. Success is if the a user clicks on the ad -- our server keeps track of that. Given: Confidence Interval: 95% Question: What is the estimated sample size? (How many total ads must we serve), Why? (remember i am a dummy) Thanks


title: AIC/BIC: how many parameters does a permutation count for?

text: Let's say I have a model selection problem and I am trying to use AIC or BIC to evaluate the models. This is straightforward for models that have some number $k$ of real-valued parameters. However, what if one of our models (for example, the Mallows model) has a permutation, plus some real-valued parameters instead of just real-valued parameters? I can still maximize the likelihood over the model parameters, for example obtaining a permutation $\pi$ and a parameter $p$. However, how many parameters does $\pi$ count toward for computing AIC/BIC?


title: Stationary matrix given a transition matrix

text: I am given the following transition matrix $$P= \pmatrix{ 1-\alpha & \alpha \\\ \beta & 1-\beta}, \ \alpha,\beta \in (0,1)$$ with the states $S=\\{1,2\\}$. I want to determine the stationary distribution $\pi$ of the Markov chain determined by a starting distribution and the transition matrix $P$. We have the system: $\pi(1)=(1-\alpha)\cdot \pi(1) + \beta\cdot\pi(2)$ $\pi(2)=\alpha\cdot \pi(1) + (1-\beta)\cdot\pi(2)$ When I substitute for $\pi(2)$ from the first equation in the second, I get $0=0$. However, the answer should be: $$\pi(1)=\frac{\beta}{\alpha+\beta},\pi(2)=\frac{\alpha}{\alpha+\beta}$$. What am I doing wrong? Thank you for your time.


title: Intuitive explanation of Gauss-Newton regression

text: I read from a textbook that Gauss-Newton regression is also called 'artificial regression'. Please give me an example, how does it work? And what's the relation with Newton's method? Thank you.


title: compare strength algorithms across different data sets

text: Reflecting on this paper. The use of a statistical significance test is required in order to compare the algorithms across different data sets. In that paper and other papers, the number of data sets are always bigger than 5. I only have access to two data sets at the moment and I was wondering if it would be useful to use a statistical significance test to compare the strength of the different algorithms across the two data sets?


title: What is the reasoning behind the formulae for the different standard errors of measurement?

text: McManus (2012) discusses the three standard errors of measurement and provides the following definitions. $SE_{meas} = SD \sqrt{(1 - reliability)}$   The standard error of measurement is an estimate of the validity of the actual scores likely to be obtained given a candidate’s true score (which is not and cannot be known). $SE_{est} = SD \sqrt{(reliability × (1 - reliability))}$   The standard error of estimation is an estimate of the variability of a candidate’s true score, given their actual score. $SE_{pred} = SD \sqrt{(1 - reliability^2)}$   The standard error of prediction is an estimate of the variability of a candidate’s actual score when taking a test a second time, given their performance on the first test. One thing I find mystifying is SEpred reaches the point of maximum divergence from SEest at around .6 reliability. Here's image from a little Excel chart I made that illustrates this phenomenon. ![enter image description here](http://i.stack.imgur.com/vIiUo.png) I can follow the definitions but the article doesn't really explain why the calculations should be different in the different contexts. Is anyone able to provide an explanation? McManus, I. C. (2012). The misinterpretation of the standard error of measurement in medical education: A primer on the problems, pitfalls and peculiarities of the three different standard errors of measurement. _Medical Teacher_ , 34, 569-576.


title: One huge cluster + small ones with vector-space model + cosine distance

text: I'm trying to cluster _meaningfully_ a set of objects characterized by a vector space (bag-of-words) model. Each of those 5000 objects has 1-8 features ("words") from a set of 5500 possible. I used a vector space model ($A_i = 1$ if feature $i$ is present) and cosine distance as a dissimilarity measure, $d (A, B) = \sqrt{2 - 2 cos (A, B)}$ where $$cos (A, B) = \frac{\sum{A_i B_i}}{\sqrt{\sum A_i^2 \sum B_i^2}}$$ No matter whether I apply R's `pam` or `hclust` / `agnes` (with `cutree (k = K0)`) to the dissimilarity matrix, I seem to get one big, degenerate cluster (thousands of members) and several small ones, unless I crank up the number of clusters to many hundreds (10% the number of objects or so). I think one problem is that most objects have no features in common and thus sit at the maximal distance. What can I try? **Update** : I've summarized the number of neighbors sitting at non-maximal distance ($\sqrt{2}$) and I got this (meaning one object has 13% of all objects at less than $\sqrt{2}$ distance, while the median object has 0.5% of all objects at less than $\sqrt{2}$):                    Min.   1st Qu.    Median      Mean   3rd Qu.      Max.       0.0003766 0.0015070 0.0054610 0.0163400 0.0222200 0.1298000


title: Chi-squared test with scipy: what's the difference between chi2_contingency and chisquare?

text: I'd like to run a chi-squared test in Python with scipy. I've created code to do this, but I don't know if what I'm doing is right, because the scipy docs are quite sparse. Background first: I have two groups of users. My null hypothesis is that there is no significant difference in whether people in either group are more likely to use desktop, mobile, or tablet. These are the _observed_ frequencies in the two groups:               [[u'desktop', 14452], [u'mobile', 4073], [u'tablet', 4287]]     [[u'desktop', 30864], [u'mobile', 11439], [u'tablet', 9887]]      Here is my code using `scipy.stats.chi2_contingency`:               obs = np.array([14452, 4073, 4287], [30864, 11439, 9887])     chi2, p, dof, expected = stats.chi2_contingency(obs)     print p      This gives me a p-value of `2.02258737401e-38`, which clearly is significant. My question is: does this code look valid? In particular, I'm not sure whether I should be using scipy.stats.chi2_contingency or scipy.stats.chisquare, given the data I have.


title: Covariance matrix estimation in presence of missing data

text: I want to estimate a covariance matrix from data with some missing values. Ideally I'd like an R package but python could be ok. R has some built in ways of doing this. You can use               cov.mat=cov(X,use='pairwise')      Or the same using cor (correlation). The trouble is that if you do this with cov, the matrix will not be guaranteed to be positive definite. If you do cov2cor(cor.mat), you will find correlation coefficients outside of [-1,1]. Using pairwise with cor seems to handle this. Then I could use the diagonal variances to go from cor.mat to cov.mat. Still, this is probably not optimal. There appears to be a few packages that claim to do this (mvnmle, rsem) but neither appear to work. rsem fails to run for me. mvnmle can only handle up to 50 variables. I need to handle roughly 1500 variables. Would like it to run in a few seconds. Anyone know of a good package for this?


title: In simple linear regression, where does the formula for the variance of the residuals come from?

text: According to a text that I'm using, the formula for the variance of the $i^{th}$ residual is given by: $\sigma^2\left ( 1-\frac{1}{n}-\frac{(x_{i}-\overline{x})^2}{S_{xx}} \right )$ I find this hard to believe since the $i^{th}$ residual is the difference between the $i^{th}$ observed value and the $i^{th}$ fitted value; if one were to compute the variance of the difference, at the very least I would expect some "pluses" in the resulting expression. Any help in understanding the derivation would be appreciated.


title: Calculate Cross Correlation of two matrices of the 'Values Vs. Time' representation

text: I have 2 acceleration vectors, each represented by a matrix with its first column corresponding to the magnitude of acceleration and second column corresponding to the time (in ms) They both represent the same data, but one sensor is started a little later than the other, so I'm trying to remove the time lag using correlation. How do I find the correlation between these 2 matrices? I tried to do `xcorr2` but it doesn't seem right and doing `xcorr` would probably calculate correlation with only the values into consideration, but I also want to account for the time. The idea is that I should finally get a correlation vector wrt time and then when I calculate the peak of the correlation vector, its index would correspond to the time lag. I'm really confused here, will an FFT help? How would you suggest I go about it?


title: What are the software limitations in all possible subsets selection in regression?

text: If I have a dependent variable and $N$ predictor variables and wanted my stats software to examine all the possible models, there would be $2^N$ possible resulting equations. I am curious to find out what the limitations are with regard to $N$ for major/popular statistic software since as $N$ gets large there is a combinatorial explosion. I've poked around the various web pages for packages but not been able to find this information. I would suspect a value of 10 - 20 for $N$? If anyone knows (and has links) I would be grateful for this information. Aside from R, Minitab, I can think of these packages SAS, SPPS, Stata, Matlab, Excel(?), any other packages I should consider?


title: Correlating two Likert type items

text: I have done a survey using a questionnaire that involves Likert type questions. The answers range from 1 - Strongly agree to 5 - Strongly disagree. In this questionnaire I have included some control questions. First I would ask something like "Would you use functionality X." Followed later by something like: "I would prefer a system without functionality X." Now I would like to see if people that answer positively to question 1 answer negatively to question 2. I have been unable to find a clear answer on what to use, whether it is Spearman's p, logistic regression, polychoric correlation, I have no idea. In short my question is: Which measure do I use to find a correlation between two Likert type questions and how do I calculate that measure? Thanks in advance!


title: Suggestions for improving a probability and statistics cheat sheet

text: ### Context: In an effort to structure the center pieces that I have came across in probability theory and statics, I created a reference document focussing on the mathematical essentials (available here). By sharing this document, I hope to give statistics students a comprehensive summary of the core material taught in graduate courses about these topics. While mainly intended as a teaching resource, folks might also find it helpful as a personal reference, e.g., to look up distribution relationships or illustrations of common PDFs. I also maintain a page with updates and fixes. Feedback is always much appreciated. ### Question:   * What are your favorite statistics cheat sheets, references, or cookbooks that I could use for inspiration?   * What helped you to structure your knowledge in this domain?    * In the long term, my plan is to enrich this document (or create a separate one) with R examples to bridge the gap between theory and practice. Would you deem this a valuable extension?


title: What practical implications/interpretations are there of a kurtotic distribution?

text: I'm familiar with what the 2nd moment (variance) indicates as well as what the 3rd moment (skewness) indicates. I know that on a histogram the 4th moment (kurtosis) indicates the "peeked-ness" of the data. My question asks what are the practical implications/interpretations of a kurtotic distribution. I'm asking this because I haven't found a case yet where I thought the 4th moment was theoretically interesting and interpretable. I understand that such interpretations/implications are likely to be dataset specific, so I am looking for descriptions of example datasets where the kurtosis of the distribution was theoretically interesting and interpretable.


title: Separating effects of foods and of food components in nutritional epidemiology

text: I have data on average long-term intake of common foods of a medium/large cohort (n=500+). It was assessed semi-quantitatively (that is, interval- censored). From these about 100 food items and their rough intake frequency I have, using a food database, calculated each cohort member's average intake of about 150 food components as weighted sums of the food intakes. Now I'd like to relate time-to-event data on my disease of interest in this cohort to dietary behaviour. The problem is that since the components are weighted sums of the foods, some foods and some components are highly correlated. Often, the components are more significantly associated with my outcome, but I'm uncertain whether this really reflects their health impact. The foods can be expected to have been assessed with quite some error, probably inducing some regression dilution bias, which should be alleviated in the components as they are kind of an average. Is there something that can be done about this from a statistical point of view?


title: Two-tailed test

text: In one-tailed test , we give our decision at $\alpha$ level of significant. But in two-tailed test , why do we give our decision at $2\alpha$ level of significant? Why do we not give the decision of two-tailed test at $\alpha$ level of significant? For example , The Durbin-Watson test for autocorrelation ![enter image description here](http://i.stack.imgur.com/6bx4H.png) The decision rule for two-tailed test against alternative $\rho\ne0$ is: If $d<d_L$ or $4-d<d_L$, conclude $d$ is significant, reject $H_0$ at level $2\alpha$ . Why at level $2\alpha$ ? Why not at level $\alpha$ or $\alpha/2$?


title: Notation: Deterministic Variable, Random Variable, Realization of Random Variable, Function

text: Is there a standard notation for distinguishing between the following:   1. Random Variables   2. Realizations of Random Variables   3. Deterministic Variables (not random)   4. Functions I am familiar with using capital letters for random variables, X, and lowercase letters for realizations of those variables or alternatively Greek and Roman letters. However, how can I then distinguish between a realization of a random variable and a deterministic variable? I am trying to avoid reinventing the wheel.


title: Are there any issues with the naive bayes classifier when the number of features are greater than the number of observations?

text: Say I have $f$ features and $n$ observations. If $f>>n$ are there any inherent issues that arise if a naive bayes classifier is used on such a data set? To be more concrete, let's say $f=200$ and $n=100$. My argument is that there is no issue with the high number of features compared to the number of observations because the naive bayes assumption is that all features are independent. So for each feature, the classifier will use the associated $n$ or $100$ observations to learn the distribution of that feature, for different values of the predictor. Most common distributions have only a few parameters associated with them (e.g. the normal distribution has $2$ associated parameters), so as long as $n$ is large enough to learn those parameters, that should be enough. However, I may be wrong, and would appreciate any insight into potential issues that can arise for $f>>n$ for naive bayes classifiers. Thanks in advance!


title: Polynomial regression analysis with response surface analysis for pre and post cortisol data and EI scores

text: I have done a study looking at the role of Emotional Intelligence in helping an individual regulate stress when they are abused by their boss. I took salivary cortisol samples before and after an abusive intervention. My supervisor is hell bent on me analyzing the data using polynomial regression (as an alternative to difference scores) and displaying the results in one of those response surface pattern 3D graphs. I have looked at Shanock's (2010) paper, (Polynomial Regression with Response Surface Analysis) where her DV is Z (Affective Commitment), and the two predictors are X (Perceived Org Support) & Y (Perceived Supervisor Support). Where my DV is the difference between the pre and post cortisol scores and my moderator is EI. So I am not sure if I can copy that SPSS code. Can anyone help? Would be most appreciated! Cheers


title: Omnibus test in generalized linear model in SPSS

text: I am testing a few covariates in generalized linear model in SPSS. When the omnibus test comes out as non-significant, does that mean the model is not significant even if the my covariates came out as significant on the "test of model effects"? Thanks.


title: Dimension Reduction using PCA and Random Forests

text: I an using scikit-learn as a toolset. I have 1K features as candidates and am trying to reduce the feature set as I believe the majority is noise (but am not sure). I wanted to somehow automate this using PCA and Random Forests. My end result would be a designated feature set. Any suggestions? I know random forests provides feature_importances that can be matched to feature_names. I know PCA can looks at features in terms of percentile that account for varience. Before I dive in the shark tank, any advice would be greatly appreciated. Thanks, Chris


title: Meaning of restricted prior

text: I encountered the usage of a "restricted prior". Could someone explain what distribution this refers to, and what is the meaning?


title: In MCMC, how is burn-in time chosen?

text: In MCMC, how is burn-in time chosen? In other words, how long do you need to wait before you think the Markov chain has reached its limiting distribution? Thanks!


title: Is k-th difference method suitable for making the data stationary?

text: I have a huge data set which is non-stationary. (Checked non-stationarity with unit root test). I'm wondering how can I make my data stationary. I need stationary series because I want do some statistical analysis later on (calculate skewness, kurtosis, std dev, ...). I read that the most easy to use (and understand) method of making data stationary is _kth difference method_ , described here: http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc442.htm. So, lets say, my measurements are `Z = {z(1), z(2), ..., z(n)}` and are non stationary. Now, using the _kth difference method_ I can get a new set, Y, where `y(i) = z(i) - z(i-1)`, and `Y = {y(1), y(2), ..., y(n-1)}.` I'm wondering if such differencing the original data wont pollute my results? When I tried to calculate some statistical values on this differenced data I got different results than with a `raw` data. Could such differenced results give reliable statistical values?


title: Combine several softmax output probabilities

text: I would like to combine the outputs of five neural networks, each with a softmax output layer of three classes each. A typical, example output is shown below:- ![enter image description here](http://i.stack.imgur.com/Xuqf0.png) where Figure 1 is the output of model 1, Figure 2 of model 2 etc. and the y-axis shows the output values and the x-axis is the time stamp of the (financial) time series. The blue line represents a "buy," the red line is a "sell" and the yellow line is a "do nothing." Since the sum of the outputs of the softmax layers sum to one, the value for each "signal" can be considered to be a probability, and so my question is: how can I combine these five separate, probabilistic outputs into one "global" probability output? This is perhaps not straight forward as the five outputs are not independent because   1. Each model takes as its input very similar features calculated from the same moving window on the underlying time series, although in the NN training the model assumptions and targets differ   2. There is obviously (and unsurprisingly) correlation between the different model outputs   3. There is auto-correlation within each model and, due to 2 above, this auto-correlation is correlated across models. I had thought of taking a simple sum across the models and dividing by 5 for each distinct output (blue, red or yellow) but on a gut level that just doesn't seem to be the right thing to do. Another useful set of information I have available is the output of a sixth neural net whose softmax output layer has five classes, where each class output is the "probability" that attaches to each of the above five NN models being the "real" model, given the input features to this sixth NN. This suggests to me that I should be looking at something like maximum likelihood or Bayesian updating, but I'm not sure where I should go from here.


title: Is a log transformation of predictors a suitable way of dealing with multicollinearity in multiple regression?

text: Suppose two independent variables in the linear regression initially have very high correlation of 0.95. This introduces severe multicollinearity into the model (as indicated by very high variance inflation factors). Can one take natural logarithm of each of them (this decreases correlation between them to 0.75), and use them in the same regression? VIFs do not indicate multicollinearity issues then. Is it a reasonable approach?


title: How to get the median of Z=f(x,y)?

text: Defining $Z=f(x,y)$ in interval $[q,1][q,1]$,$x,y$ are all i.i.d. uniform.   I know that, if $f(x,y)$ is non-increasing or non-decreasing for both $x$ and $y$, the median of $z$ is $f((1-q)/2,(1-q)/2)$. But if $f(x,y)$ is non- increasing or non-decreasing for only $x$, and it is not non-increasing or non-decreasing for $y$. How can I get the median of $Z$? The figure of $Z$ is as follows: ![enter image description here](http://i.stack.imgur.com/skWgim.png)


title: Confidence level for clusters

text: I have a sample, set of outcomes of some random variable. I divide it into "clusters", using some determinate approach. One of the clusters considered to be "correct", usually it is the one that has the most number of outcomes in it, but not always. I want to mark each cluster with some "confidence level", based on the number of outcomes it has. The formulae should depend on the number of outcomes in each cluster totally, and in the current one. Say, if the outcome numbers, for each cluster sequentially, are: `220, 31, 28, 21`, then the first cluster should have very high confidence level, something close to 100%, but not exactly 100%. But if the distribution is `113, 110, 106, 103`, then all the confidences should be almost equal - and _not_ even close to 100% - since there are several similar clusters, we cannot mark each of them as "probably correct". I cannot use just relative proportion to the maximum, because it will cause highest cluster to have always exactly 100% confidence level, and it shouldn't be. Is there some approach, or formulae, that gives such estimations? Thank you.


title: Generalization of a hazard model

text: I am looking for a statistical technique similar to a estimating a hazard model. Suppose a person chooses between two actions, buy or sell. Given data on trading decisions, once a person has bought, say, a stock, I can calculate the time that elapses until they sell. Using a hazard model I can model this hazard as a function of other variables. This allows me to answer questions about what predicts a person's decision to sell. But this ignores the fact that sometimes instead of selling a person buys more stock. This is the _opposite_ of selling, and moves the person further away from "failure" (ie, selling). I can think of no analogy in epidemiology, which perhaps explains why I haven't been able find a method that is suitable for this situation. Any suggestions for techniques that may be useful here would be greatly appreciated. Thanks!


title: Is this explanation of the Box-Jenkins approach correct?

text: Can someone please tell me what they think of my explanation of the Box- Jenkins approach to forecasting time series? Do you have anything to add (in particular to my explanation as to the intuition behind stationarity and the spurious regression problem)? "I use the Box-Jenkins approach to develop a model for $Y_t$. This approach involves the following steps:   1. Calculate ACF of raw $Y_t$. If $Y_t$ is non-stationary, you should difference the data or take logarithms.    2. Observe ACF structure of the transformed variable. Compare the observed structure of the ACF to the theoretical properties of different ARMA processes. Use this information to propose a model(s). If the ACF displays seasonal patterns, this should be modelled.    3. Estimate your model and check the residuals for autocorrelation. If the model is correctly specified, the residuals should not be autocorrelated, i.e., they should be white noise. If the residuals are autocorrelated, then you should go back to step 2. You may be asking, why don’t we model $Y_t$ as it is? Well, for two reasons: Firstly, ARMA models are not going to be useful if the series is non- stationary. To understand this, you should understand what non-stationarity is. Intuitively, it means the behaviour of the series changes over time. More technically, it means the mean, variance, and covariance structure change with time. For example, if the mean of the series is increasing with time, then the mean value during the sample is probably not going to be a very good predictor of the future value of the series. Of course, this depends how quickly it is changing and how far in the future you want to forecast. Therefore, it should be clear, intuitively, that modelling non-stationary data, although entirely possible, is not the best idea. Secondly, we don’t want to fall into any spurious regression traps, i.e., we don’t want to go around claiming $Y_t$ is causally related to $X_t$ simply because a regression of $Y_t$ on $X_t$ results in a statistically significant slope estimate. Because time series variables are (usually) trending, you should expect a statistically significant slope coefficient even if there is no causal relationship. Therefore, it is not a great idea to model these variables as they are. In other words, you are probably less likely to avoid this spurious regression problem if you model changes in $Y_t$ , or percentage changes, than the series themselves. There is mathematics behind this too, but the intuition should suffice."


title: Deriving risk estimates using forecasting confidence limits and out of sample hold-out cases

text: I was hoping for some advice. I use SAS for automatic forecasting (I have a large number of forecasts to complete in a limited timeframe). As part of the forecast output from SAS, I get a mid-point (median or mean), and an upper and lower confidence limit for each forecast. This is determined at a pre-specified level (i.e. 95%). Evidently, the confidence limits are derived statistically. I appreciate that within the upper and lower limits there is a range of potential forecasting results that could occur. Based on my interpretation, values close to the upper and lower limits of the forecasts will be less likely because they depart a fair way from the mean/median and are close to the tails in terms of the distribution of possible forecasted values. In a number of potential forecasting scenarios I face each month, I have a target the business needs to achieve by the end of the next financial year (june 30, 2012). I also have a forecast of the likely future value + upper and lower forecast limits for end of year (june 30, 2012) derived using moderate to long term historical performance. I need to quantify the probability of reaching a target given a forecasted result, using the point where the target is located relative to the mean/median and also the upper and lower forecast limits. For example, if the series has a mean forecast of 50, a LCL of 0 and UCL of 100 and the target is 75, I need to quantify the probability this target will be met at June 30, 2012. It seems to me I can use the fact the target of 75 falls on the 75th percentile relative to the forecast upper and lower forecast limits.   * Is it fair to say that there is a 25% chance of hitting the target under an assumed uniform distribution?    * Or is it more appropriate to assume the forecast series within the upper and lower limits is normally distributed with a greater concentration of values in the CDF closer to the mean?   * Or is this model dependent (i.e. different for arima and esm models) Also, as a rule of thumb how many holdouts do people use for   1. a series with 12 months (I am using 10% or 1)   2. a series with 24 months (I am using 10% or 2)


title: Multivariate linear bayesian regression in matlab with normal-gamma assumption for data

text: 1. for the general case : data is normal-gamma (mean normal and sd is gamma) and I want to estimate the $b$ and $a$ distribution ( they assumed to be normal) in $y=bx+a$ using Bayesian regression. I know I need assumption for my priors but I think I have to be OK with most simplistic case. not very good in reading statistics literature to derive the code from heavy literature.   2. {trying to be more clear} piece of code is helpful as well as pseudo-code of how to do it. Here I explain the case I did for single parameter as an example. there are 2 ways actually by conditioning on ro/p (precision) or unconditional. Keep these pre-calculation: `Sxx<-sum((x-xbar)^2) , Syy<-sum((y-ybar)^2) , Sxy<-sum((x-xbar)(y-ybar)) , See<-sum( ( y-ybar-b(x-xbar))^2)` } Posterior distribution for $\rho$ is Gamma with shape $(n-2)/2$ and scale $2/S_{ee}$. Conditional on $\rho$, $\eta$ and $\beta$ are independent and normally distributed, $\eta$ has mean $y$ and precision $n\rho$, $\beta$ has mean $b$ and precision $S_{xx}\rho$. Uncoditional: Marginal posterior distribution for $\eta$ and $\beta$. $\eta$ has nonstandard t distribution with: Center $\bar y$, spread $(n(n-2)/S_{ee})-1/2$, degrees of freedom $n-2$. $\beta$ has nonstandard t distribution with: Center $\bar y$, spread $(S_{xx}(n-2)/S_{ee})-1/2$, degrees of freedom $n-2$. $\eta$ and $\beta$ are uncorrelated but not independent when we marginalize out $\rho$. I dont know how this could be for MultiVariate Case. I know I can use Winbugs but I need to write it in Matlab.


title: Predicting the significance of a dataset for predicting labels

text: I have a dataset with some samples and labels for each sample. I can cluster the samples into like lets say 10 clusters. I want to know how much significant this dataset is in predicting the labels. I was thinking of doing some kind of hypergeometric test for each cluster and give the cluster a score. Then the average score of the dataset will be the average of the scores obtained by the clusters. The idea is that the dataset will have special grouping that can define the labels. Any suggestions guys? What kind of metric/significance testing, I can use to find out the significance of a cluster?


title: Forecasting when the dependent and independent variables are differenced

text: Consider the following model for $Y_t$: $\Delta$log($Y_t)$ = $\beta_0$ + $\beta_1$$\Delta$$log(X_t)$ + $u_t$ where $u_t$ ~ IID Normal(0,$\sigma^2$). I want a forecast for $Y_{T+1}$. Thus, I need a mathematical formula for $E(Y_{T+1}|\Omega_T)$. After some algebra we have: $log(Y_{T+1}) = log(Y_T) + \beta_0 + \beta_1 log(X_{T+1}) - \beta_1 log(X_T) + u_{T+1}$ Taking exponentionals of both sides, $Y_{T+1} = Y_T e^{\beta_0} {X^{\beta_1}_{T+1}} {X^{-\beta_1}_T} e^{u_{T+1}}$ Then take conditional expectations of both sides with respect to the time $T$ information set: $E(Y_{T+1}|\Omega_T)$ = $Y_T e^{\beta_0}X_T^{-\beta_1}E({X^{-\beta_1}_{T+1}}e^{u_{T+1}}|\Omega_T)$. Where do I go from here? What assumptions, if any, do I need to make to chug out a formula for $E(Y_{T+1}|\Omega_T)$. FYI I am currently using EViews to produce some forecasts. I just want everything to be mathematically sound. Any suggestions regarding how I should produce a forecast for $Y_{T+1}$? I should note that I have forecasts for $X_t$, $t >T$. Thank you. Christian


title: Which kernel method gives the best probability outputs?

text: Recently I have used Platt's scaling of SVM-outputs to estimate probabilities of default-events. More direct alternatives seem to be "Kernel logistic Regression" (KLR) and the related "Import Vector Machine". Can anyone tell which kernel method giving probability-outputs is currently state of the art? Does an R-implementation of KLR exist? Thank you very much for your help!


title: Variance of a function of one random variable

text: Lets say we have random variable $X$ with known variance and mean. The question is: what is the variance of $f(X)$ for some given function f. The only general method that I'm aware of is the delta method, but it gives only aproximation. Now I'm interested in $f(x)=\sqrt{x}$, but it'd be also nice to know some general methods. **Edit 29.12.2010**   I've done some calculations using Taylor series, but I'm not sure whether they are correct, so I'd be glad if someone could **confirm** them. First we need to approximate $E[f(X)]$   $E[f(X)] \approx E[f(\mu)+f'(\mu)(X-\mu)+\frac{1}{2}\cdot f''(\mu)(X-\mu)^2]=f(\mu)+\frac{1}{2}\cdot f''(\mu)\cdot Var[X]$ Now we can approximate $D^2 [f(X)]$   $E[(f(X)-E[f(X)])^2] \approx E[(f(\mu)+f'(\mu)(X-\mu)+\frac{1}{2}\cdot f''(\mu)(X-\mu)^2 -E[f(X)])^2]$ Using the approximation of $E[f(X)]$ we know that $f(\mu)-Ef(x) \approx -\frac{1}{2}\cdot f''(\mu)\cdot Var[X]$ Using this we get:   $D^2[f(X)] \approx \frac{1}{4}\cdot f''(\mu)^2\cdot Var[X]^2-\frac{1}{2}\cdot f''(\mu)^2\cdot Var[X]^2 + f'(\mu)^2\cdot Var[X]+\frac{1}{4}f''(\mu)^2\cdot E[(X-\mu)^4] +\frac{1}{2}f'(\mu)f''(\mu)E[(X-\mu)^3]$   $D^2 [f(X)] \approx \frac{1}{4}\cdot f''(\mu)^2 \cdot [D^4 X-(D^2 X)^2]+f'(\mu)\cdot D^2 X +\frac{1}{2}f'(\mu)f''(\mu)D^3 X$


title: How to tell if the slope of a line is 0 or there is just no relationship?

text: I am attempting to examine the change in slope between a predictor and response over two years. In year 1, it is definitely positive. (Linear regression, the 95% CI of the slope doesn't overlap 0). In year 2, the point estimate of the slope is close to 0 (0.002) and the CI overlaps 0. This is what I would expect if the slope was, well, actually 0. And given that any test of the slope will suggest that I cannot reject that it is 0 - great! Although, yes, I know, that doesn't confirm that the slope is 0. Not falling into that trap. But, with a slope value very close to 0, and a CI that overlaps 0...it all seems like the slope has declined. However, I now have a colleague who thinks I cannot say that the value of the slope declined between year 1 and year 2. Rather, they think that I can only say that there is no relationship between the predictor and response in year 2. Am I correct in interpreting my results that the slope of the year 2 relationship is indeed ~0? Or is there a problem with my interpretation?


title: Excepted conditional density and conditional expectation

text: Apparently one can obtain a regression analysis as $$g(x)=\frac{\int yf(y,x)dy}{f(x)}$$ where $$f(x)=\int f(y,x)dy$$ is the marginal density of $X_i$. In effect, I believe, the above expression calculates the expected value of the conditional density $f(y|x)$. I am confused, is the above expression a generalization of the conditional expectation $E[y|x]$, or are they the same?


title: Difference between different kinds of entropy

text: The following concepts are baffling and would be obliged for a constructive explanation. (Q1) What is the conceptual difference between (a) Kolmogorov-Sinai entropy, (b) Shannon entropy, (c) Source entropy (d) topological entropy and (e) Boltzmann entropy? (Q2) What are the relationships between different entropies? (Q3) Are they interchangeable and mean the same? (Q4) Where can I find a good lucid explanation about Kolmogorov entropy and why it is needed and its significance? (Q5) Can Boltzmann entropy be equal to Kolmogorov entropy?


title: When would least squares be a bad idea?

text: If I have a regression model: $$ Y = X\beta + \varepsilon $$ where $\mathbb{V}[\varepsilon] = Id \in \mathcal{R} ^{n \times n}$ and $\mathbb{E}[\varepsilon]=(0, \ldots , 0)$, when would using $\beta_{\text{OLS}}$, the ordinary least squares estimator of $\beta$, be a poor choice for an estimator? I am trying to figure out an example were least squares works poorly. So I am looking for a distribution of the errors that satisfies previous hypothesis, but yields bad results. If the family of the distribution would be determined by mean and variance that would be great. If not, it's OK too. I know that "bad results" is a little vague, but I think the idea is understandable. Just to avoid confusions, I know least squares are not optimal, and that there are better estimators like ridge regression. But that's not what I am aiming at. I want an example were least squares would be unnatural. I can imagine things like, the error vector $\epsilon$ lives in a non-convex region of $\mathbb{R}^n$, but I'm not sure about that. Edit 1: As an idea to help an answer (which I can't figure how to take further). $\beta_{\text{OLS}}$ is BLUE. So it might help to think about when a linear unbiased estimator would not be a good idea. Edit 2: As Brian pointed out, if $XX'$ is bad conditioned, then $\beta_{\text{OLS}}$ is a bad idea because variance is too big, and Ridge Regression should be used instead. I'm more interested is in knowing what distribution should $\varepsilon$ in order to make least squares work bad. $\beta_{\text{OLS}} \sim \beta+(X'X)^{-1}X'\varepsilon$ Is there a distribution with zero mean and identity variance matrix for $\varepsilon$ that makes this estimator not efficient?


title: Using Naive Bayes to calculate the probability of user presence based on the presence of her belongings

text: Alice, Bob, and Charlie work for a company and each of them has a cell phone and a car. They drive to work and they have their cell phones with them when they show up at work. The company has installed some sensors to detect the presence of the cars and cell phones. Sometimes our users use the public transportation to work or they may forget their cell phones at home. At the same time, they may leave their cell phone behind at work or use public transportation to go home. We interviewed the users for their habits and they say with a certain probability they are where their belongings are. I would call these probabilities P(user | belonging) which are shown in the parentheses in front of each item. Alice – iPhone (99%), Honda (90%) Bob – Blackberry (98%), Ford (95%) Charlie – Samsung (99%), Jeep (98%) Alice is an early bird and 99% of the time, she is at work by 9:00 AM (P(Alice)=0.99). Bob usually works between 9:00 AM to 5:00 PM but he is occasionally late. We can say 80% of the time he is in the office by 9:00 AM (P(Bob)=0.8). Charlie is fashionably late and only 10 percent of the time he is in the office by 9:00 AM (P(Charlie)=0.10). The sensors have created a database of historical information about presence of each item over time. So by 9:00 AM, the chance of each item being at the office is: P(iPhone)=98% - P(Honda)=91% P(Blackberry)=82% - P(Ford)=75% P(Samsung)=9% - P(Jeep)=10% So we want to know if Charlie’s car and cell phone are both detected by the sensors at 9:00 AM, what the probability of Charlie being at the office is. Based on Naïve Bayes: P(Charlie|Samsung,Jeep) = (P(Charlie) * P(Samsung|Charlie)*P(Jeep|Charlie))/(((P(Charlie) * (P(Samsung|Charlie) * P(Jeep|Charlie))+ (P(~Charlie) * (P(Samsung|~Charlie)*P(Jeep|~Charlie))) We can calculate: P(Samsung|Charlie) = P(Charlie|Samsung)*P(Samsung)/P(Charlie) = 0.99 * 0.09/0.1= 0.891 P(Samsung|~Charlie) = (1-P(Charlie|Samsung))*P(Samsung)/P(~Charlie) = 0.01 * 0.09/0.9 = 0.001 P(Jeep|Charlie) = P(Charlie|Jeep)*P(Jeep)/P(Charlie) = 0.98 * 0.10 /0.1 = 0.98 P(Jeep|~Charlie) = (1- P(Charlie|Jeep))*P(Jeep)/P(~Charlie) = 0.02 * 0.10/0.9 = 0.002 P(Charlie|Samsung,Jeep) = 0.1 *(0.891*0.98) /(0.1*(0.891*0.98))+(0.9*(0.001*0.002))) = 0.087318/(0.087318+0.0000018) = 0.999 Now, I have 2 questions: 1- Is this math correct? 2- If the prior probability of Jeep goes from 0.1 to 0.11(or higher), P(Jeep|Charlie) becomes 1.078(or higher). What is wrong now?


title: price prediction

text: In the project I am working we have a couple of items with different prices. Each item has description but it is possible to find items with similar description. I would like to know how can I estimate a price of a new item? I was thinking to match items based on their descriptions but it is not always useful as it is too likely two items with high similarity in description have different prices.


title: Measuring parameter sensitivity and variability (standard-error) in k-fold cross-validation

text: I mainly use k-fold cross-validation for parameter tuning and model selection for prediction problems. Now, is there a standard or if not a less-known way to measure the sensitivity of the parameters over the cross-validation error, and to quantify it? Also, is it standard to look at the variability of the cross-validated error across folds as well? I know that the question around the variability has a lot of caveats, but what is a good way to do this, say if I was reporting cross-validation results and the cv errors were either close by/ were relatively changing based on the size of the scaled dataset- then how can I report the variability apart from the cv errors itself? Note: Assume that I have just two parameters to tune, and I use a discrete grid of the two parameter combinations


title: Separate time series data into two trends

text:
I have a time series data-set that contains two separate trends(one trends has relatively lower values/ the other has higher values).If you plot it in excel, it will be very clear to see those two trends. My question is: how can I separate those two different trends? I want to separate those two trends based on the value of the data over time. Thus, I come up with two different time series that has two different trends(those trends could be nonlinear) Is there anyone knows how to deal with this problem? The data is as following: the first column is the data the second column is the time stamp the data is collected.               TT  EST
     741 1/5/2012 15:30
     662 1/5/2012 15:31
     722 1/5/2012 15:32
     743 1/5/2012 15:32
     688 1/5/2012 15:32
     776 1/5/2012 15:34
     735 1/5/2012 15:34
     797 1/5/2012 15:37
     667 1/5/2012 15:37
     843 1/5/2012 15:37
     1013    1/5/2012 15:37
     688 1/5/2012 15:37
     672 1/5/2012 15:37
     784 1/5/2012 15:37
     799 1/5/2012 15:38
     759 1/5/2012 15:38
     823 1/5/2012 15:39
     659 1/5/2012 15:39
     814 1/5/2012 15:40
     784 1/5/2012 15:40
     760 1/5/2012 15:41
     843 1/5/2012 15:41
     806 1/5/2012 15:41
     792 1/5/2012 15:42
     794 1/5/2012 15:42
     618 1/5/2012 15:42
     843 1/5/2012 15:43
     723 1/5/2012 15:43
     893 1/5/2012 15:44
     736 1/5/2012 15:44
     914 1/5/2012 15:44
     819 1/5/2012 15:45
     674 1/5/2012 15:45
     692 1/5/2012 15:46
     850 1/5/2012 15:47
     735 1/5/2012 15:49
     707 1/5/2012 15:49
     891 1/5/2012 15:49
     852 1/5/2012 15:49
     741 1/5/2012 15:51
     767 1/5/2012 15:51
     748 1/5/2012 15:51
     886 1/5/2012 15:52
     884 1/5/2012 15:53
     915 1/5/2012 15:53
     773 1/5/2012 15:53
     874 1/5/2012 15:53
     682 1/5/2012 15:53
     682 1/5/2012 15:53
     687 1/5/2012 15:53
     1125    1/5/2012 15:53
     940 1/5/2012 15:54
     899 1/5/2012 15:54
     886 1/5/2012 15:55
     723 1/5/2012 15:55
     891 1/5/2012 15:55
     896 1/5/2012 15:55
     923 1/5/2012 15:56
     936 1/5/2012 15:56
     884 1/5/2012 15:56
     926 1/5/2012 15:56
     921 1/5/2012 15:57
     912 1/5/2012 15:57
     878 1/5/2012 15:58
     880 1/5/2012 15:59
     860 1/5/2012 15:59
     885 1/5/2012 15:59
     786 1/5/2012 15:59
     894 1/5/2012 16:01
     735 1/5/2012 16:02
     740 1/5/2012 16:02
     1109    1/5/2012 16:03
     706 1/5/2012 16:03
     701 1/5/2012 16:03
     895 1/5/2012 16:03
     831 1/5/2012 16:03
     818 1/5/2012 16:03
     829 1/5/2012 16:04
     698 1/5/2012 16:04
     829 1/5/2012 16:05
     845 1/5/2012 16:05
     830 1/5/2012 16:06
     718 1/5/2012 16:06
     802 1/5/2012 16:06
     830 1/5/2012 16:07
     720 1/5/2012 16:07
     840 1/5/2012 16:08
     793 1/5/2012 16:08
     1237    1/5/2012 16:08
     721 1/5/2012 16:09
     721 1/5/2012 16:09
     789 1/5/2012 16:09
     800 1/5/2012 16:09
     795 1/5/2012 16:09
     800 1/5/2012 16:10
     661 1/5/2012 16:11
     828 1/5/2012 16:13
     1399    1/5/2012 16:14
     786 1/5/2012 16:14
     801 1/5/2012 16:14
     835 1/5/2012 16:15
     767 1/5/2012 16:16
     843 1/5/2012 16:16
     779 1/5/2012 16:17
     695 1/5/2012 16:17
     700 1/5/2012 16:17
     785 1/5/2012 16:18
     1450    1/5/2012 16:18
     770 1/5/2012 16:18
     735 1/5/2012 16:18
     779 1/5/2012 16:19
     789 1/5/2012 16:20
     788 1/5/2012 16:20
     852 1/5/2012 16:20
     793 1/5/2012 16:20
     894 1/5/2012 16:21
     685 1/5/2012 16:21
     792 1/5/2012 16:21
     837 1/5/2012 16:23
     824 1/5/2012 16:24
     700 1/5/2012 16:25
     831 1/5/2012 16:25
     730 1/5/2012 16:26
     826 1/5/2012 16:27
     720 1/5/2012 16:27
     844 1/5/2012 16:28
     875 1/5/2012 16:28
     737 1/5/2012 16:29
     737 1/5/2012 16:30
     859 1/5/2012 16:30
     847 1/5/2012 16:31
     741 1/5/2012 16:31
     911 1/5/2012 16:33
     934 1/5/2012 16:34
     812 1/5/2012 16:36
     937 1/5/2012 16:37
     925 1/5/2012 16:37
     949 1/5/2012 16:39
     934 1/5/2012 16:39
     976 1/5/2012 16:40
     1016    1/5/2012 16:41
     947 1/5/2012 16:41
     863 1/5/2012 16:41
     869 1/5/2012 16:41
     989 1/5/2012 16:41
     948 1/5/2012 16:44
     727 1/5/2012 16:44
     985 1/5/2012 16:45
     974 1/5/2012 16:45
     715 1/5/2012 16:45
     979 1/5/2012 16:45
     998 1/5/2012 16:46
     750 1/5/2012 16:46
     1078    1/5/2012 16:47
     746 1/5/2012 16:48
     1017    1/5/2012 16:48
     976 1/5/2012 16:48
     973 1/5/2012 16:49
     1069    1/5/2012 16:50
     869 1/5/2012 16:50
     906 1/5/2012 16:51
     865 1/5/2012 16:51
     1011    1/5/2012 16:52
     727 1/5/2012 16:52
     960 1/5/2012 16:52
     1013    1/5/2012 16:54
     1024    1/5/2012 16:54
     769 1/5/2012 16:56
     1068    1/5/2012 16:57
     1067    1/5/2012 16:57
     758 1/5/2012 16:57
     768 1/5/2012 16:57
     1087    1/5/2012 16:57
     1064    1/5/2012 16:57
     1158    1/5/2012 16:59
     1332    1/5/2012 17:00
     1024    1/5/2012 17:00
     1025    1/5/2012 17:00
     1073    1/5/2012 17:01
     1078    1/5/2012 17:01
     819 1/5/2012 17:01
     1098    1/5/2012 17:02
     1038    1/5/2012 17:03
     783 1/5/2012 17:04
     1076    1/5/2012 17:05
     761 1/5/2012 17:06
     1049    1/5/2012 17:06
     754 1/5/2012 17:07
     780 1/5/2012 17:07
     780 1/5/2012 17:07
     795 1/5/2012 17:08
     778 1/5/2012 17:08
     772 1/5/2012 17:08
     1077    1/5/2012 17:09
     789 1/5/2012 17:09
     1077    1/5/2012 17:10
     1076    1/5/2012 17:10
     1083    1/5/2012 17:12
     1099    1/5/2012 17:12
     1638    1/5/2012 17:15
     1117    1/5/2012 17:15
     1104    1/5/2012 17:15
     765 1/5/2012 17:16
     749 1/5/2012 17:16
     801 1/5/2012 17:16
     1073    1/5/2012 17:16
     1068    1/5/2012 17:16
     1099    1/5/2012 17:17
     1152    1/5/2012 17:19
     797 1/5/2012 17:19
     782 1/5/2012 17:19
     1121    1/5/2012 17:20
     838 1/5/2012 17:20
     1107    1/5/2012 17:21
     1204    1/5/2012 17:21
     1122    1/5/2012 17:22
     1133    1/5/2012 17:22
     1246    1/5/2012 17:22
     1174    1/5/2012 17:22
     823 1/5/2012 17:22
     812 1/5/2012 17:22
     1373    1/5/2012 17:23
     815 1/5/2012 17:24
     1729    1/5/2012 17:24
     842 1/5/2012 17:25
     842 1/5/2012 17:25
     1257    1/5/2012 17:25
     1230    1/5/2012 17:26
     1286    1/5/2012 17:27
     1243    1/5/2012 17:27
     1019    1/5/2012 17:27
     1382    1/5/2012 17:29
     1861    1/5/2012 17:30
     825 1/5/2012 17:30
     1356    1/5/2012 17:33
     867 1/5/2012 17:33
     850 1/5/2012 17:33
     1314    1/5/2012 17:34
     1345    1/5/2012 17:34
     1471    1/5/2012 17:34
     1661    1/5/2012 17:35
     1325    1/5/2012 17:36
     1023    1/5/2012 17:38
     940 1/5/2012 17:38
     1348    1/5/2012 17:38
     1396    1/5/2012 17:41
     1374    1/5/2012 17:41
     1484    1/5/2012 17:42
     1485    1/5/2012 17:42
     956 1/5/2012 17:42
     956 1/5/2012 17:42
     945 1/5/2012 17:42
     980 1/5/2012 17:44
     1407    1/5/2012 17:45
     959 1/5/2012 17:46
     1428    1/5/2012 17:47
     2085    1/5/2012 17:47
     1398    1/5/2012 17:48
     1805    1/5/2012 17:48
     1572    1/5/2012 17:48
     1365    1/5/2012 17:49
     1474    1/5/2012 17:50
     1567    1/5/2012 17:50
     1382    1/5/2012 17:50
     1382    1/5/2012 17:50
     1757    1/5/2012 17:51
     1390    1/5/2012 17:51
     951 1/5/2012 17:51
     1479    1/5/2012 17:52
     1400    1/5/2012 17:54
     1491    1/5/2012 17:54
     1426    1/5/2012 17:55
     2074    1/5/2012 17:56
     913 1/5/2012 17:56
     1449    1/5/2012 17:56
     1439    1/5/2012 17:57
     920 1/5/2012 17:57
     1501    1/5/2012 17:57
     1385    1/5/2012 17:57
     1522    1/5/2012 17:57
     936 1/5/2012 17:58
     1011    1/5/2012 17:58
     916 1/5/2012 17:58
     1423    1/5/2012 17:59
     1505    1/5/2012 17:59
     908 1/5/2012 17:59
     1505    1/5/2012 17:59
     907 1/5/2012 18:00
     1415    1/5/2012 18:00
     1226    1/5/2012 18:02
     1464    1/5/2012 18:02
     872 1/5/2012 18:04
     874 1/5/2012 18:04
     1544    1/5/2012 18:04
     1424    1/5/2012 18:06
     1432    1/5/2012 18:07
     1447    1/5/2012 18:07
     967 1/5/2012 18:08
     1544    1/5/2012 18:08
     1442    1/5/2012 18:08
     1527    1/5/2012 18:08
     852 1/5/2012 18:11
     1546    1/5/2012 18:12
     1535    1/5/2012 18:12
     1512    1/5/2012 18:12
     1517    1/5/2012 18:12
     1522    1/5/2012 18:13
     856 1/5/2012 18:13
     1504    1/5/2012 18:15
     1488    1/5/2012 18:15
     1567    1/5/2012 18:15
     965 1/5/2012 18:15
     964 1/5/2012 18:17
     944 1/5/2012 18:17
     1314    1/5/2012 18:17
     1502    1/5/2012 18:17
     920 1/5/2012 18:18
     1397    1/5/2012 18:18
     1448    1/5/2012 18:20
     1342    1/5/2012 18:21
     1323    1/5/2012 18:23
     1359    1/5/2012 18:25
     1350    1/5/2012 18:25
     888 1/5/2012 18:26
     1357    1/5/2012 18:26
     1335    1/5/2012 18:27
     1514    1/5/2012 18:27
     1555    1/5/2012 18:27


title: A little confusion on Profile likelihood

text: As we all know, profile likelihood is an effective method for the estimation of conditional parametric model. But I still don't know exactly why it works. Profile likelihood was thoroughly studied by Severini and Wong (1992). By profile likelihood, can we obtain an unbiased, asymptotically efficient estimator for the parameter? Can we get a unbiased estimator for the nonparametric part?


title: Repeated measures tests without "before and after" measurements

text: When reading about repeated measures ANOVA, until now 99% of examples are always with "before" and "after". But if I understood correctly the point of repeated measures ANOVA is not necessarily to estimate differences _in time_ but rather when one measure may be dependent on another, correct? Can you mention examples of repeated measures ANOVA (or any other test for that matter, like paired t test) which do NOT involve measuring something before and after? Ideally, an example where time is not a factor to consider...


title: How to measure multichannel distances between "event" sequences?

text: In TraMineR, seqdistmc is used to measure multichannel distances between "state" sequences. I am wondering if there is a function to measure multichannel distances between "event" sequences.


title: Cox PH model selection and validation

text: I am trying to analyze my data using survival CoX PH in SPSS v.19 and also attempting to make different prediction models (without and with a biomarker of interest). I am a clinician (not a biostatistician) and this is for a paper that I need for completing my Ph.D. For my model selection I used the following method. First, I identified the univariate predictors for the outcome, and then I used all of them in a Cox backward stepwise analysis to obtain my final model (I have only 25 events). Now, it was suggested to use some a priori variables (like for example age or sex - variables that weren't associated with the outcome in my univariate analysis and as such not included in the final model) and retain the variables only if they lead to an improved model fit (as assessed by LLR). My first questions are:   1. Is it statistically correct to use an a priori defined model and then add my statistically significant variables, comparing the LLR of the different models (using the hierarchical Cox analysis)?    2. If this is correct, how do I initially select a model to which I then add another variable?   3. How do I solve the problem of overfitting? Do I have to use bootstrapping for my final model?  I will also try to compare the final model with a model that includes my biomarker.   1. How do I save the predicted probabilities for each model in order to then compare them (I need them to perform C statistic and NRI analysis)? In SPSS in simple logistic regression there is this option. Is there any macro that could do this?   2. Is is correct to use the predictive probabilities from the simple logistic regression?


title: Cluster with distance threshold in R

text: I'd like to get clusters with a maximum inner distance threshold. Now I use `hc <- hclust(d)` and `cutree(hc, numofclasses)`. But I would like to use something like in python:               >>> cl = HierarchicalClustering(data, lambda x,y: abs(x-y))     >>> cl.getlevel(10)     # get clusters of items closer than 10      How can I get this in R?


title: How can IRT-Models be understood in GLM/ SEM Framework? (Predict Learning with added Paradata-Covariates)

text: I'll be working with data from an intelligent tutor system similar to one studied in the KDD-Cup 2010 on student performance prediction and plan to use IRT models to infer item and ability parameters. Other covariates such as 'number of attempts' or 'time spent on the problem' are readily available. I am interested in **incorporating these kind of covariates ('paradata') in the estimation of the latent traits**. Do you consider this a meaningful project, or do you expect that not much can be gained by including these additional variables? Are you aware of similar projects? What would be a good place to _learn how to fit IRT models in the more general framework of GLMs/ SEMs?_ I have been pointed toward publications from Sophia Rabe-Hesketh and others, but **ideally** I would be interested in any **(online) resources or demos that are specific for a learning context and IRT**. I currently have an alright understanding of IRT models, but don't have much experience in fitting them yet. So recommendations on R packages to work with in this context would also be appreciated. (I am aware of the psychometric task view on CRAN.)


title: Possible extensions to the default diagnostic plots for lm (in R and in general)?

text: I started digging a bit into the plot.lm function, this function gives six plots for lm, they are:   1. a plot of residuals against fitted values   2. a Scale-Location plot of sqrt(| residuals |) against fitted values   3. a Normal Q-Q plot, a plot of Cook's distances versus row labels   4. a plot of residuals against leverages   5. a plot of Cook's distances against leverage/(1-leverage) And I am wondering **what other common/useful extensions of current plots exists for linear models, and how can they be done in R? (links to articles of packages are also welcomed)** So the boxcox function (from {MASS}) is an example of another useful diagnostic plot (and such an answer would be great), however, I am more curious about variations/extensions on **existing** default diagnostic plots for lm in R (although general other remarks on the topic are always welcomed). Here are some simple examples of what I mean:               #Some example code for all of us to refer to     set.seed(2542)     x1 <- rnorm(100)     x2 <- runif(100, -2,2)     eps <- rnorm(100,0,2)     y <- 1 + 2*x1 + 3*x2 + eps     y[1:4] <- 14 # adding some contaminated points     fit <- lm(y~x1+x2)          #plot(y~x1+x2)     #summary(fit)      ### To plot the residuals vs each of the potential x               plot(resid(fit)~x1); abline (h = 0)     plot(resid(fit)~x2); abline (h = 0)     # plot(resid(fit)~x1+x2) # you can also use this, but then you wouldn't be able to use the abline on any plot but the last one      ### To add the the 0-1 line (how is this line called in English?!) to the qqplot so to see how much the qqline deviates from it               plot(fit, which = 2); abline(0,1, col = "green")      ### To plot the qq-plot using externally studentized residuals               # plot(fit, which = 2); abline(0,1, col = "green") # The next command is just like this one     qqnorm(rstandard(fit), ylim = c(-2.2,4.2)); qqline(rstudent(fit), lty = 2) ;abline(0,1, col = "green")     qqnorm(rstudent(fit), ylim = c(-2.2,4.2)); qqline(rstudent(fit), lty = 2) ;abline(0,1, col = "green")     # We can note how the "bad" points are more extreme when using the rstudent


title: Which statistical classification algorithm can predict true/false for a sequence of inputs?

text: Given a sequence of inputs, I need to determine whether this sequence has a certain desired property. The property can only be true or false, that is, there are only two possible classes that a sequence can belong to. The exact relationship between the sequence and the property is unclear, but I believe it is very consistent and should lend itself to statistical classification. I have a large number of cases to train the classifier on, although it might be slightly noisy, in the sense that there's a slight probability that a sequence is assigned the wrong class in this training set. Example training data:               Sequence 1: (7 5 21 3 3) -> true     Sequence 2: (21 7 5 1) -> true     Sequence 3: (12 21 7 5 11 1) -> false     Sequence 4: (21 5 7 1) -> false     ...      In rough terms, the property is determined by the set of values in the sequence (e.g. the presence of an "11" means that the property will almost certainly be false), as well as the _order_ of the values (e.g. "21 7 5" significantly increases the chance that the property is true). After training, I should be able to give the classifier a previously unseen sequence, like `(1 21 7 5 3)`, and it should output its confidence that the property is true. Is there a well-known algorithm for training a classifier with this kind of inputs/outputs? I have considered the naive Bayesian classifier (which is not really adaptable to the fact that the order matters, at least not without severely breaking the assumption that the inputs are independent). I've also investigated the hidden Markov model approach, which appears to be inapplicable because only a single output is available, instead of one output per input. What did I miss?


title: Is there a GLS estimator that has lower variance than OLS for sum of parameters in linear model under Gauss-Markov conditions?

text: I have a model $$Y=\beta_0 + \beta_1 x_1 + \beta_2x_2 +\epsilon$$ I would like the minimum variance unbiased estimate of $\gamma=\beta_1 + \beta_2$. Assuming the Gauss Markov conditions hold, but $x_1$ and $x_2$ are correlated, is there a more efficient way to estimate $\gamma$ than running OLS and adding the estimates of $\beta_1$ and $\beta_2$? Given that var($\hat{\theta}$)=var($\hat{\beta_1}$)+var($\hat{\beta_2}$)+2cov($\hat{\beta_1},\hat{\beta_2}$) I'm specifically wondering if there is a GLS estimator that reduces cov($\hat{\beta_1},\hat{\beta_2}$) faster than it increases the variance of the individual estimates.


title: Combining machine learning models

text: I'm kind of new to datamining/machine learning/etc. and have been reading about a couple ways to combine multiple models and runs of the same model to improve predictions. My impression from reading a couple papers (which are often interesting and great on theory and greek letters but short on code and actual examples) is that it's supposed to go like this: I take a model (`knn`, `RF`, etc) and get a list of classifiers between 0 and 1. My question is how to do combine each of these lists of classifiers? Do I run the same models on my training set so that the number of columns going into the final model are the same or is there some other trick? It would be great if any suggestions/examples included R code. NOTE: This is for a data set w/ 100k lines in the training set and 70k in the test set and 10 columns.


title: Sophisticated models for classifying short pieces of texts

text: I have about 30000 book names assigned to 6 categories, and I want to build scalable and accurate classifiers. So far I have only been able to use Naive Baye's and LibLINEAR classifiers and they both give me an (almost) identical precision and recall values of 0.8 and 0.7 after 10 fold CV. I am wondering if I would be able to do better if I were to use more complex models . The problem is that the time complexity of the sophisticated models seems to increase super-linearly with the number of training instances. SVM (SMO implementation from WEKA), for example, has been running for the past 3hrs already on this data, whereas the Naive Baye's and LibLINEAR finished in about 15mins and 40mins respectively. I am trying to build a general framework for short texts classification (twitter, text messages etc.), and so will be running many experiments over varied data sets. I require techniques that scale and work well (don't we all :-)). Any suggestions? Another question is with regard to dimension reduction. When I pre-process my text, I apply stemming, stopword removal and convert the text to tf-idf vector representation. Dimension reduction techniques (Info gain, in particular) again seems to be taking an inordinately long time. Any scalable way to do feature selection? Would pruning by tf-idf scores an acceptable approach? Edit 1: By "Info Gain", I meant Information Gain . And currently I am not doing any feature selection.


title: Variable selection for increasing accuracy

text: I know that there are various posts regarding variable selection but I am asking something particular. With respect to the question that I posted today in the following link: Low accuracy in out of time validation If you had a look at the above link, you have seen that my problem is low detection rate in out of time data (ie, low true positives) though I had a very good accuracy in out of sample (80.5%). Please comment on the thoughts below that I have for this problem. Since I need to have a model which has reasonably good accuracy with the past as well as future data would the following things be of any use to me?   1. Trying and selecting those variables which are shock resistant to time variation in data (not really sure whether there are such variables but trying to think intuitively that model is as good as its data and variables)-- what would this variable look like?   2. I had done profiling of both sample and out of time validation data; should I consider dropping the variables which have high variation or difference in distribution or statistics (in case of continuous variables). Agree, it might decrease my model accuracy from 80 to 70 (may be) but, I guess, it would help me in keeping only those variables which are more shock proof to the seismic waves of time -- please suggest. All in all, I want suggestions on which variables to keep so as to maintain my initial accuracy. I dont mind initial accuracy of 65% detection and out of time accuracy of 50% finally but a drop from 80 to 35 is a worry.


title: $L_1$ or $L_.5$ metrics for clustering?

text: Does anyone use the $L_1$ or $L_.5$ metrics for clustering, rather than $L_2$ ?   Aggarwal et al., On the surprising behavior of distance metrics in high dimensional space said (in 2001) that > $L_1$ is consistently more preferable then the Euclidean distance metric > $L_2$ for high dimensional data mining applications and claimed that $L_.5$ or $L_.1$ can be better yet. Reasons for using $L_1$ or $L_.5$ could be theoretical or experimental, e.g. sensitivity to outliers / Kaban's papers, or programs run on real or synthetic data (reproducible please). An example or a picture would help my layman's intuition. This question is a follow-up to Bob Durrant's answer to When-is-nearest- neighbor-meaningful-today. As he says, the choice of $p$ will be both data and application dependent; nonetheless, reports of real experience would be useful. * * * Notes added Tuesday 7 June: I stumbled across "Statistical data analysis based on the L1-norm and related methods", Dodge ed., 2002, 454p, isbn 3764369205 -- dozens of conference papers. Can anyone analyze distance concentration for i.i.d. exponential features ? One reason for exponentials is that $|exp - exp| \sim exp$; another (non- expert) is that it's the max-entropy distribution $\ge$ 0; a third is that some real data sets, in particular SIFTs, look roughly exponential.


title: Bayesian two-factor ANOVA

text: I am interested in fitting a Bayesian Two Factor ANOVA in BUGS or by utilizing some R package. Unfortunately I am having a hard time finding resources on this topic. Any suggestions? Even an article describing the approach would be helpful.


title: Density function question

text: Suppose $x_{1}, x_{2} \dots x_{N}$ are gaussian RVs with variance $S$ and mean $1$. What is the density function of $$\frac{ |\sum_{n=1}^{N}x_{n}|^{2}}{\sum_{n=1}^{N}|x_{n}|^{2}}\text{?}$$


title: Sample size and power

text: I came across a paper mentioning sample size calculation for 10% shift, 20% shift and 50% shift. I understand the SSC concept but what exactly is 10% shift and 20% shift.? So lets say for example if have a mean difference of 8787 , and SD of 102 , how do i calculate Sapmple size for this at 10% shift, 20% shift and 50% shift. ? Thanks!


title: Summary of residuals in R

text: ![\[Here is an image for details!\]\(https://onlinecourses.science.psu.edu/stat504/sites/onlinecourses.science.psu.edu.stat504/files/lesson07/crab_r_output_02.gif\)](http://i.stack.imgur.com/gMBYg.gif) Disregarding "Deviance" in the image, the output of multiple regression analysis in R looks pretty much like this. As far as I understand, residuals are errors. Do the 5 value summary refer to residuals as errors of estimate, or are these different? Or is my understanding of residuals mistaken?


title: Cluster similarity percentages with inverted Y-axis in R

text: I'd like to ask a question here that I've also asked on Biostar (stackexchange) and someone there forwarded me to this website. I was wondering how I could perform a Bray Curtis similarity clustering in R in which I show the similarity percentages on an inverted Y-axis and all tree nodes ending at 100% as I've shown in a dendogram: ![enter image description here](http://i.stack.imgur.com/rkwcT.png) At the moment I create my plot in the following way (using S17 Bray Curtis dissimilarity measure, which just scales regular Bray Curtis to 0-100%):               library(vegan)     mat = 'some matrix'     d = (1 - vegdist(mat, method="bray")) * 100      h = hclust(d)     plot(h)      Inverting the Y-axis (with `ylim=c(100,80)`) doesn't work. How can I create a dendogram as shown above from a distance matrix? Thanks for any help / advice! Original question can be found on the Biostar website here


title: Why efficiency matters?

text: Suppose we are trying to estimate the quantity $\theta$ and we have that the estimator $\hat\theta_n$. Suppose it is efficient, i.e. is variance is the smallest among certain class of other possible estimators of $\theta$, say that this class is a class of unbiased estimators. Efficient estimators are naturally desired, since they are the "best" in some sense. But what do we lose when we use estimator which is not efficient? Suppose we have two estimators which are asymptotically normal, then we can say that the confidence interval of the efficient estimator is narrower than the one of non-efficient one. But surely there is a better explanation than such hand-waving? Is there some quantification of what is lost? My question was motivated by this quote by Ch. Sims found here: > Frequentist inference could be approached in the same way: Deﬁne your model, > derive fully efﬁcient estimators, pay no attention to anything else Note that I would not like to ignite another frequentist vs Bayesian war, I just feel that there might be some deep result I am not familiar with.


title: Equivalent of Google spreadsheet's arrayformula in Excel

text: **Input**               1   0     7   77     6   66      **Expected Output**               7   66      **Solution in Google Spreadsheet**               = arrayformula(if(B2:C2,B3:C3,B4:C4))      How to do that in Excel? Please, add label Spreadsheet -- the if clause is the same with Excel and Google Spreadsheet.


title: Measuring entropy/ information/ patterns of a 2d binary matrix

text: I want to measure the entropy/ information density/ pattern-likeness of a two- dimensional binary matrix. Let me show some pictures for clarification: This display should have a rather high entropy: A) ![enter image description here](http://i.stack.imgur.com/Fxy5j.jpg) This should have medium entropy: B) ![enter image description here](http://i.stack.imgur.com/ptCTR.jpg) These pictures, finally, should all have near-zero-entropy: C) ![enter image description here](http://i.stack.imgur.com/E3wZ1.jpg) D) ![enter image description here](http://i.stack.imgur.com/SzXTx.jpg) E) ![enter image description here](http://i.stack.imgur.com/Ew4OE.jpg) Is there some index that captures the entropy, resp. the "pattern-likeness" of these displays? Of course, each algorithm (e.g., compression algorithms; or the rotation algorithm proposed by ttnphns) is sensitive to other features of the display. I am looking for an algorithm that tries to capture following properties:   * Rotational and axial symmetry   * The amount of clustering   * Repetitions Maybe more complicated, the algorith could be sensitive to properties of the psychological "Gestalt principle", in particular:   * The law of proximity: ![law of proximity](http://i.stack.imgur.com/J9Nqw.png)   * The law of symmetry: Symmetrical images are perceived collectively, even in spite of distance:![symmetry](http://i.stack.imgur.com/fgCm6.jpg) Displays with these properties should get assigned a "low entropy value"; displays with rather random / unstructured points should get assigned a "high entropy value". I am aware that most probably no single algorithm will capture all of these features; therefore suggestions for algorithms which address only some or even only a single feature are highly welcome as well. **In particular, I am looking for _concrete, existing algorithms_ or for _specific, implementable ideas_ (and I will award the bounty according to these criteria).**


title: Copula generation (Gaussian, t and Gumbel) with the help of correlation matrix using R

text: I have a set of data of 2 variates. I have generated correlation matrix between the variates. Using `copula` package of R, I computed t-copula using correlation matrix. I used the following technique for that:   1. With the help of correlation matrix for example let it be `corr_mat`, I computed parameter vector, lets say `param_vect`.   2. Then I computed `tcopula` object, let it be `t_object`, using `tCopula(param_vect, dim =..., dispstr="un", df =...)`   3. Lastly I simulated 1000 sample from the copula using `rcopula(1000,t_object)`. The above procedure works fine for t-copula. Can someone please suggest me amethod to simulate samples from the Gaussian and Gumbell copulas? I tried with the `gumbelcopula` function, but it fails with errors. Could anyone one suggest the method to simulate copula using correlation matrix?


title: Representing confidence ratings on choices as a variable or as part of the choice alternative in Mixed Logit Simulation

text: I am estimating a mixed logit model with hierarchical Bayes procedures to deal with my categorical data. I am wondering if I'm representing the data correctly. The data comes from experiments where participants are shown constructed images and need to identify if the image is constructed from information of type A or type B. After each choice the participant makes, they are additionally asked to rate how confident they are that they have answered correctly using a 7 point likert scale. Each subject makes this classification of type A or type B for a series of images and rates their confidence each time. In my model, I encode the choice as having 2 alternatives: either the image was correctly identified or it was not. Previously, I thought perhaps I should encode each choice as having 14 alternatives ranging from strongly confident correct answer to strongly confident wrong answer. (Or that I should do it as a nested choice as mentioned in section 6.3 of this: elsa.berkeley.edu/choice2/ch6.pdf) However, I now have doubts because what I'm really interested in is what variables are relevant to a subject's ability to make a correct identification of the image type. Since the Bayesian procedures allow me to estimate covariance among all random coefficients, is it perhaps more reasonable to encode the likert rating not as part of the choice alternative but as one of the (random coefficient) variables? Is that even a valid approach? Should I revert to my 14 level approach? Is there something else I should be considering? Thanks very much for your help.


title: Nonhomogeneous Poisson and Heavy tail inter arrival time distribution

text: What is the relationship between a Nonhomogeneous Poisson process and a process that has heavy tail distribution for its inter arrival times? Any pointer to a resource that can shed some light on this question would be hugely appreciated


title: How to reduce number of items on a multi-item scale where each item requires ranking four response options

text: ### Context I have a survey of 16 questions, each with four possible responses. The purpose of the survey is to measure the respondent's propensity towards four categories (which we will denote A, B, C, D). Each of the four responses per question are representative of an aspect of the four categories, A, B, C, D. The respondent rank orders each of the four responses (we will denote the first response by "4", the second by "3", etc). To score the categories, we add the responses up based on the coding above. There are 16 x (4 + 3 + 2 + 1) = 160 total points. The sums for each category are computed, and the maximum score is deemed the respondent's dominant category. Therefore each survey looks like the following (in CSV format)               question_num, A, B, C, D     1, 4, 3, 1, 2     2, 3, 4, 1, 2     3, 3, 4, 2, 1     4, 4, 3, 1, 2     5, 4, 3, 1, 2     6, 4, 3, 2, 1     7, 4, 3, 1, 2     .     .     .     16, 3, 4, 1, 2     sums, 64, 48, 24, 24      I have about 325 surveys completed. ### Aim I want to remove possible redundant items in the survey so I can reduce the burden on future respondents. ### Questions   * My first strategy was to do a multi-logistic regression with the response as the dominant category (described above). Is this a good idea?   * Would PCA be helpful?   * Are there any other strategies for identifying redundant items?


title: Data Analysis Work-- Is there Any Freelance Opportunity?

text: Just wonder, is there any data analysis/ statistic/ data mining work that are available on freelance basis? This could be subjective and argumentative, which is why I put it as CW.


title: What is the difference between distribution free statistics/methods and non-parametric statistics?

text: From Wikipedia > The first meaning of non-parametric covers techniques that do not rely on > data belonging to any particular distribution. These include, among others: > >   * distribution free methods, which do not rely on assumptions that the > data are drawn from a given probability distribution. As such it is the > opposite of parametric statistics. It includes non-parametric statistical > models, inference and statistical tests. >   * non-parametric statistics (in the sense of a statistic over data, which > is defined to be a function on a sample that has no dependency on a > parameter), whose interpretation does not depend on the population fitting > any parametrized distributions. Statistics based on the ranks of > observations are one example of such statistics and these play a central > role in many non-parametric approaches. > I can't see the difference between the two cases: distribution free methods, and non-parametric statistics. Do they both not assume the data coming from some distribution? How do they differ? Thanks and regards!


title: Analogues of sensitivity and specificity for continuous outcomes

text: How can I calculate the sensitivity and specificity (or analogous measures) of a continuous diagnostic test in predicting a continuous outcome (e.g., blood pressure) without dichotomizing the outcome? Any ideas? It appears that researchers have done this using mixed effects modeling (see link below), but I'm not familiar with their use of the technique: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3026390/ By the way, I'm most familiar with R, so it would be ideal for the implementation you suggest to be accompanied with an R function (but it's okay if not). Thank you in advance for any suggestions!


title: How to estimate Poisson process using R? (Or: how to use NHPoisson package?)

text: I have a database of events (i.e. a variable of dates) and associated covariates. The events are generated by the non-stationary Poisson process with parameter being an unknown (but possibly linear) function of some covariates. I think the NHPoisson package exists just for this purpose; but after 15 hours of unsuccessful research I am still nowhere near getting to know how to use it. Heck, I even tried reading both referenced books: Coles, S. (2001). An introduction to statistical modelling of extreme values. Springer. Casella, G. and Berger, R.L., (2002). Statistical inference. Brooks/Cole. The one single example in the documentation of fitPP.fun doesn't seem to fit my setup; I don't have extreme values! I just have bare events. Can anyone please help me with a simple example of fitting a Poisson process with parameter $\lambda$ with single covariate $X$, and assumption, that $\lambda = \lambda_0 + \alpha \cdot X$? I'm interested in estimation of $\lambda_0$ and $\alpha$. I provide a two-column data set with times of events (let say, measured in seconds after some arbitrary time $t_0$) and another column with values of the covariate $X$?


title: How to deal with correlated fixed effects in a mixed-effects model?

text: I am trying to predict (binary) memory for pictures based on two continuous fixed effects: memorability and clutter. Using a mixed effects model, I found that both effects predict memory and including both in a model is better than either on their own. However, memorability and clutter are highly correlated. 1) How do I interpret my result (that including both is better), given that they are correlated? Can they still be independent predictors, yet be correlated? 2) Is this a problem for the model in general? How do I deal with correlated fixed effects? * * * In R:               m_mem <- lmer(memory ~ memorability + (1|subject), data=memDat, family='binomial')     m_clut <- lmer(memory ~ clutter + (1|subject), data=memDat, family='binomial')     m_mem_clut <- lmer(memory ~ memorability+clutter + (1|subject), data=memDat, family='binomial')      `anova(m_mem_clut,m_mem)` and `anova(m_mem_clut,m_clut)` are significant. [Edit: R `dput` can be found here ]


title: Options for aggregating dispersion data

text: I have a bunch of web page response time data, and I'm recording dispersion stats--both variance and quartile--on an hourly basis. Are there good ways to roll this up into larger time scales (e.g., daily, weekly), as opposed to having to reprocess the original dataset? I understand that in the initial hourly rollup I'm losing some information, so I may have to have estimated values for the daily, weekly etc rollups, and that's fine as long as the estimate is reasonably close. One thought that I have is that in the hourly rollups I can model the distribution of data, and then aggregate the models themselves for the daily/weekly/etc rollups. I haven't actually done the work yet of determining appropriate models for that hourly data, but conceptually this seems like a plausible direction. (Just as an aside, are there distributions that are generally accepted as being reasonable models for web page response times?) Another idea would be to have some pre-defined bins--e.g., [0, 0.5), [0.5, 1.0), [1.0, 2.0), etc., and store counts. Of course I could have more bins for better resolution. Are there other approaches worth exploring? I'm not necessarily looking for theoretically perfect solutions here. Basically I'm putting together an operational dashboard for a bunch of web applications, and I need a practical way to allow operational staff to view dispersion statistics at different timescales.


title: Understanding and interpreting consistency of OLS

text: Many econometrics textbooks (e.g. Wooldridge, "Econometric analysis...") simply write something similar to: "If the population model is $y = xB + u$ and (1) $\text{Cov}(X,U) = 0$; (2) $X'X$ is full rank, then OLS consistently estimate parameters $B$". I've worked out the math behind consistency but I'm a bit lost in interpreting the meaning. We're talking about consistent estimation, but estimation of what? Please tell me if following points are correct: (1) If we have random sample $X,Y$ and $X'X$ is invertible, then we can always define Best Linear Predictor of $y$ given $x$. And then OLS always consistently estimate coefficients of Best Linear Predictor (because in BLP we have $\text{Cov}(u,x)=0$ from definition). Bottom line: we can always interpret OLS estimates as coefficients of BLP (2) The only question is whether BLP corresponds to conditional expectation $\text{E}(y|x)$. If it does (for which we need $\text{E}(u|x) = 0$), then we can interpret OLS estimates as partial effects. What is wrong with this or what am I missing? I don't get the point of stating the assumption of $\text{Cov}(u,x) = 0$, if this assumption is by definition fulfilled in case of Best Linear Predictor. And if we want to give structural interpretation to BLP coefficients (partial effects) we need stronger assumption of $\text{E}(u|x) = 0$ anyway. Is there any other interpretation we may have with zero covariance?


title: Deliberately fitting a model without intercept

text: Is there a situation in which the mean of a Y variable is not 0 (e.g. not standardized), but we would still fit a regression model without intercept? It would yield a worse fitting model, so is there a scenario in which you would omit adding an intercept with a good reason?


title: Panel data issue in gravity model

text: I am using panel data set and want to know which model I should use, fixed effect or random effect. On which basis is this selection done? When I asked my teacher he told me that model is selected on the base of Hausman specification test . I am using the panel data for measuring the bilateral trade between two countries and to find out the effect of distance I am using panel data se otherwise I was using time series only. But only time series was not able to find out the impact of distance on bilateral trade between two countries..? I have read all the chapter of panel data of Gujarati book and still confused about model selection. I was thinking to run OLS on all the panel data set ignoring the cross sectional and time series...by doing this I will be able to get the impact of distance on bilateral trade also about other variables..


title: Log Transformation in Growth Mixture Model (Mplus user)

text: I am using parallel process growth mixture modeling to estimate bidirectional longitudinal dynamics between pain (0-100 theoretical range of values) and sleep measures (0-20 theoretical range of values). Conceptually speaking if I log transform both of these raw measures and then regress their latent variables (intercept and slope) on each other, should I interpret that model estimates as percentage change in x that is linked to percentage change in y. In other words, should I refer to this relationship as to elastic? Oleg