title: Sample size calculation for a single proportion Without Any Comparisons (descriptive only): How to determine the margin of error in the formula?

text: I want to calculate the sample size for a single proportion (the prevalence of a disease). I know the formula and all: n = (z/ **m** )^2 * P(1-P), where z is 1.96 (for 95% confidence level) and P stands for the prevalence of the disease in the pilot study. In the above formula, **m** is supposed to be the margin of error (I have seen some sources call it **d** too). Anyways I have heard that **m** (or **d** ) should be about 0.1 of the P estimated by the pilot study... However, I am seeing sources in which the author changes **m** at will. For example, this one. I just wonder if this rule is valid (that **d** should be 0.1 of P)? Or are there any other rules for estimating **m** (or **d** )?


title: Missing data in 3 dimensional Panel: Are comparisons across different samples valid?

text: I have a panel of 8 sectors across 24 OECD countries. Im estimating aggregate equations across all sectors, and 8 separate equations for each sector. Part of the analysis is to make comparisons across sectors. A few countries have all or about half of the observations missing for a single sector. This means that e.g. for one sector a whole country is missing. Are comparisons across sector equations and therefore slightly different samples still valid? I would have guessed yes, as long as the nature of the missing observations or country-sectors in unsystematic. However, it seems to me that in general practice, countries which have a missing sector would be dropped.


title: Bivariate Gaussian distribution test in R

text: Regarding the assumption that $X$ and $Y$ need to be normally jointly distributed, when applying Pearson's correlation test...   what is a good test (function and package) in R to accomplish this task? In this case, will low "p-values" represent that $X$ and $Y$ have Gaussian bivariate distribution, or the opposite? CV has already a similar question, but it relies specifically on Matlab solutions. Although this is a simple question, I think it is a good repository for CV.


title: What are distances between variables making a covariance matrix?

text: I have a $n \times n$ covariance matrix and want to partition variables into $k$ clusters using hierarchical clustering (for example, to sort a covariance matrix). Is there a typical distance function between variables (i.e. between columns/rows of the square covariance matrix)? Or if there are more, is there a good reference on the topic?


title: Accuracy score for comparison of two probability grids

text: I would like to calculate an accuracy score for two grids. The cells of the grid have a probability of being occupied (1 = occupied, 0 = free). One grid is supposed to be the reference grid (blue) and I want to measure how similar the second grid (red) is. For easier understanding I illustrated the problem. ![enter image description here](http://i.stack.imgur.com/8me7O.png) Am I on the right track with the Kullback-Leibler-Divergence? Or is there something better? I already stated that question yesterday but in retrospective I think I might have formulated it too specificly.


title: How to convert this equation into an equation where I can use regression to estimate coefficients?

text: Can someone please help me with this? I need to do a regression in order to estimate the alpha's in this following equation, but I can't seem to get this equation to be in a format that i could use regression with.Ii was just looking to simplify this equation so that I can get something in the form of Y = constant + alpha0 * something + alpha1 * something .. etc so that the regression coefficients will be my alphas. Any advice would be greatly appreciated, thanks a lot! D(T) stands for the balance at time T, and R(T) stands for the interest rate at time T. D(T) = D(0) * exp[alpha0 * T + alpha 1 * T^2 / 2 + Alpha 3 * (R(T) - R(0))] * ((1 / df(T)) ^ Alpha3 where df(T) = exp(summation from i to N of - R(TI) * dTI) (so df(T) is basically a constant number)


title: Build a path probability tree for journeys through a website

text: I'm currently doing analysis on a website which requires that I create a decision tree diagram showing the likely route that people take whenever they arrive on the website. I am dealing with a `data.frame` which shows the paths of all customers to the site, starting from the homepage. For example, a customer could take the following path:               Homepage - pg 1     Kitchen Items page - pg 2     Pots and Pans page - pg 3      so this customer would have a 3 page journey. What I want to try to do in R is combine all customers paths and thereby assign a probability to a customer following a certain path on the site. For example, if I were to examine all paths I could find that 34% of people who arrive on the Homepage go onto the 'Kitchen items page'. Does R have this facility? I have looked up different methods through the rpart and partykit packages but they didn't seem to be of any help. Any steer in the right direction for this is very much appreciated!


title: How to design a contrast matrix with combined levels for a categorical variable?

text: I wish to make a contrast matrix in the case of a linear model. I have one factor with three levels: `T`, `N` and `A`. I want the contrasts `T` vs `N`, `T` vs `A` and `T` vs (`N` and `A` combined). It is the last contrast with combined levels that cause troubles. The "correct" result I can get by redefining `N` and `A` to a new combined level `L` and then make a one column contrast matrix `T = 1`, `L = -1`. But how to do it when this level is split into two? The `T = 2`, `N = -1`, `A = -1` column does not give the result of the above one col contrast. I'm implementing it in R. The number of observations are T=30, N=14, A=15


title: When are order statistics not sufficient?

text: Lehmann and Casella (http://www.springerlink.com/content/978-0-387-98502-2) state it as an example (Example 1.6.10) that for arbitrary continuous distributions, the order statistics are always sufficient. Is this a theorem, or are pathological examples known to exist? Are there necessary conditions for the order statistics to constitute a set of sufficient statistics?


title: Analysis of categorical unmatched case-control design

text: My data look like this               trait   nobs.ctrls  nobs.cases     A            0          0     B            8          1     C           24          0     D           116         0     E            41         2     F           6          11      I have a lot more controls (`ctrls`) than cases (1000 vs. 50). The numbers mean that, for example, trait C is seen 24 times in the 1000 controls, but not seen at all in the cases (`nobs.cases`). What's the best way to test data of this type for significance? Do I need to control for N? The traits are independent. 24 times doesn't necessarily mean that it occurs in 24 of the 1000. I've done a sort of pooled analysis so I actually don't know how many people have the trait i.e. more than one may be in one person. I'm looking at SNPs in sequencing data across genes, with each "trait" really a gene.


title: Publicly available petabyte-scale data set?

text: Does anybody know of any data sets that are publicly available and huge (i.e., in the petabyte ballpark). Ideally, the data set would be relevant to machine learning analyses. Maybe something from the physics, chemistry, natural- language-processing domains? I guess you could say the Web itself is such a data set, but I'm looking for something a little more structured than that. Thanks in advance.


title: Proportional odds model for repeated measures ordinal outcomes

text: I'm looking to analyze data coming from a study for reduction in dependence on an addictive substance. We're looking at 4 subjective measures of clinical dependence called clinical global impression (CGI) scores. They have a domain of at most 7 possible outcomes. We're interested in looking at continuous differences before and after treatment. We're also possibly interested in a more robust categorical approach, since the distance between categorical scores may be egregiously different from a constant difference in latent addiction levels. We would like to use the proportional odds model (GLM with cumulative logit link) to do this. In order to measure changes from baseline, should we use the difference in categorical scores as the outcome or adjust for the baseline scores? I'm assuming we include time as a factor. Then to determine treatment efficacy, do we adjust for the interaction of time and treatment and measure the ratio of odds ratios for cumulative decrease in CGI scores? Lastly, how does one report the results from a proportional odds model interaction term? As I stated it is my best known approach, however, I would be curious to know if there is a more accurate interpretation. Information on these types of models is found in Categorical Data Analysis, by Alan Agresti ch. 5.


title: How do I compute the transition probability function?

text: I have to reproduce the algorithm of a scientific article and have chosen one that uses Reinforcement Learning. However, I don't understand how to compute the transition probability function ($f$) used in the iteration: ![Q-value iteration](http://i.stack.imgur.com/4H6tG.png) The article says: > In order to run the Q-value iteration algorithm (Table 2), the transition > probability function f was first computed. My question is: how was it computed?


title: What to choose from BIC/AIC/ridge/elastic net?

text: I have the following regression problem I have about 60 independent variables; some of them have a high correlation with others. I have around 3 million observations (1) - My main goal is out-of-sample-prediction, so my main question is: which regularization method should I use in this case? Some more questions (assumptions I have, probably a little confused) (2) - Ridge regression, while not completely removing coefficients, would keep those coefficients low that lasso/elastic net/BIC would completely remove; is that correct? (If it doesn't, would that be a problem?) (3) - If I wanted to use AIC/BIC in this case, I would have to test all possible combinations of the 60 independant variables? (4) - Would it make sense to start with AIC/BIC, then later do ridge regression with the remaining independant variables? (I guess ridge regression _after_ AIC/BIC might make sense because some of the independant variables correlate with others?) Thanks


title: Understanding p-value

text: I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification. Here is the definition of p-value from Wikipedia: > The p-value is the probability of obtaining a test statistic at least as > extreme as the one that was actually observed, assuming that the null > hypothesis is true. (http://en.wikipedia.org/wiki/P-value) My **first question** pertains to the expression "at least as extreme as the one that was actually observed." My understanding of the logic underlying the use of p-value is the following: If the p-value is small, it's unlikely that the observation occurred assuming the null hypothesis and we may need an alternative hypothesis to explain the observation. If the p-value is not so small, it is likely that the observation occurred only assuming the null hypothesis and the alternative hypothesis is not necessary to explain the observation. So if someone wants to insist on a hypothesis he/she has to show that the p-value of the null hypothesis is very small. With this view in mind, my understanding of the ambiguous expression is that p-value is $\min[P(X<x),P(x<X)]$, if the PDF of the statistic is unimodal, where $X$ is the test statistic and $x$ is its value obtained from the observation. Is this right? If it is right, is it still applicable to use the bimodal PDF of the statistic? If two peaks of the PDF are separated well and the observed value is somewhere in the low probability density region between the two peaks, which interval does the p-value give the probability of? The **second question** is about another definition of p-value from Wolfram MathWorld: > The probability that a variate would assume a value greater than or equal to > the observed value strictly by chance. > (http://mathworld.wolfram.com/P-Value.html) I understood that the phrase "strictly by chance" should be interpreted as "assuming a null hypothesis". Is that right? The **third question** regards the use of "null hypothesis". Let's assume that someone wants to insist that a coin is fair. He expresses the hypothesis as that relative frequency of heads is 0.5. Then the null hypothesis is "relative frequency of heads is not 0.5." In this case, whereas calculating the p-value of the null hypothesis is difficult, the calculation is easy for the alternative hypothesis. Of course the problem can be resolved by interchanging the role of the two hypotheses. My question is that rejection or acceptance based directly on the p-value of the original alternative hypothesis (without introducing the null hypothesis) is whether it is OK or not. If it is not OK, what is usual workaround for such difficulties when calculating the p-value of a null hypothesis? * * *       **I posted a newquestion that is more clarified based on the discussion in this thread.**


title: What kind of Diagram to visualize Results from Software Usability test?

text: I know how to use R but I am kind of stuck on the statistical part here: I need to visualize the results from a software usability test where the total number of clicks of each tester for every one of the three tasks they had to complete were recorded. The total time needed for each of the three tasks was also recorded. I want only to visualize the results from 10 testers, so I was thinking about using a bar plot, but I can't think of any plot where I can put the click count as well as the time needed into one plot that makes sense. I have the data in two .csv files (if that matters). Also do you think I should be aware of any caveats? such as should I normalize something? Any help would be appreciated! Martin


title: Is there a "hello, world" for statistical graphics?

text: In computer programming, there is a classic first program for learning/teaching a new language or system, called "hello, world". http://en.wikipedia.org/wiki/Hello_world_program Is there a classic first data visualization for using a graphing package? If so, what is it? And if not, what would good candidates be?


title: Correlation with different vector length - same time period

text: I have a data set containing equity price information on a tick by tick basis. I would like to calculate the correlation in R, which isn't a hassle. But the vectors are of different length due to one equity being traded more than the other. I can aggregate on a minute basis, but this also leaves the vectors of inequal length due to trades not occuring each minute for each equity. I guess it boils down to a statisical question. Can I calculate the correlation with different vector length given they sample the same time period? I thought about inputting the previous price in the empty time minute slots to achieve equal vector length, but perhaps there is a different approach? I am using R. Thanks * * * Thank you for your answers. I have looked into the proposed solutions, and I think an upsampling / downsampling will be best in this case. Is there a way to code the following in R? My hunch is an if loop that will check the corresponding data frame if the time is present if not then it will grabthe previous price and input a new price for the missing time. This is what I would like to produce: Two seperate colums, price and time, for each equity. If the time is present for equity X but not equity Y then the previous price should put into the vector. As per the following example:               X           Y        price   time        price   time     10  540     20  540     11  541     21  541     12  542     22  543     13  544     23  544     14  545     24  545          price   time        price   time     10  540     20  540     11  541     21  541     12  542     21  542     12  543     22  543     13  544     23  544     14  545     24  545


title: Assumption for valid hypothesis testing of the OLS estimators in the small samples

text: Please support me solve this question: In a simple regression model y = b0 + b1*x + u we have the five main assumptions 1 linearity in parameters 2 random sampling 3 zero conditional mean 4 variation in x 5 homoscedasticity IN ADDITION TO the 5 assumptions, what is the additional assumption for valid hypothesis testing of OLS estimators in the small samples?


title: How to integrate users' profile information into a recommender system

text: I need to build a recommendation system with lots of information about users (age, sex, location, income etc), but very sparse information about users' prefernces (i.e 1-2 products consumed out of 100 for each user). I have about 50 thousand records and think that using users' profiles may prove useful. My first reaction as an economist would be using conditional multinomial logit. According to an insightful post here, it may not be a good idea. Can anybody prompt me what kind of the recommendation algorithm should I use? (R package is of particular interest)


title: How to detect interactions in a random forest model?

text: When I fit a regression tree model in R, I can somehow look at the plotted tree to figure out the interactions of the predictors. However, when I fit a random forest model, I cannot plot a tree because there are too many of them. I can somehow find the importance order of the predictors (using varImpPlot function), but I do not know how to detect the interactions from the model. By the way, the importance plot only gives the importance order of the predictors, how can I figure out exactly how many of them should be included for further investigation. Thank you.


title: Selecting threshold for generalized Pareto distribution in R

text: I'm using the `POT` Package in R for fitting a Generalized Pareto distribution to my data. For choosing an approximate threshold I'm using the `tcplot()` and `mrlplot()` the following way(`x` is the data containing vector)               tcplot(x,u.range=c(0,quantile(x,probs=0.995)))     mrlplot(x,u.range=c(0,quantile(x,probs=0.995)),col=c("green","black","green"),nt=200 )      But I get a bit confused sometimes as what value I'll choose as threshold. Particularly as mentioned in this site that > interpretation of these plots often requires a good deal of subjective > judgement For example what would be the threshold given the following plot? ![A parameter stability and MRL plot](http://i.stack.imgur.com/f3oZh.png)


title: Why is sample standard deviation a biased estimator of $\sigma$?

text: According to the Wikipedia article on unbiased estimation of standard deviation the sample SD $$s = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (x_i - \overline{x})^2}$$ is a biased estimator of the SD of the population. It states that $E(\sqrt{s^2}) \neq \sqrt{E(s^2)}$. NB. Random variables are independent and each $x_{i} \sim N(\mu,\sigma^{2})$ My question is two-fold:   * What is the proof of the biasedness?   * How does one compute the expectation of the sample standard deviation  My knowledge of maths/stats is only intermediate.


title: Interpretation of a Durbin-Watson test?

text: I have calculated a Durbin-Watson test and got as far as $$\eqalign{ d&=2.207551844, \\\ dL&= 1.6164, \\\ dU&= 1.7896. }$$ I want to test $$H_0 \gt 0,\ H_1 \le 0.$$ However, I do not really know if I can reject $H_0$. What would happen if $H_0 \lt 0, H_1 \ge 0$? Please give me a hint on the interpretation of such a test!


title: Change point analysis

text: Could someone please explain change point to me. I'm using the package in R, and I don't really understand what the different methods mean, the pros and cons of each, and I especially do not understand the penalty value. When you increase the penalty value, what does that mean and what does it do? I have done a good amount of research online but I just keep finding the cran R and quick R sites, which are good, but the way they say it just isn't cutting it for me. Thanks so much.


title: How to determine best cutoff point and its confidence interval using ROC curve in R?

text: I have the data of a test that could be used to distinguish normal and tumor cells. According to ROC curve it looks good for this purpose (area under curve is 0.9): ![ROC curve](http://i.stack.imgur.com/oijy4.png) **My questions are:**   1. How to determine cutoff point for this test and its confidence interval where readings should be judged as ambiguous?   2. What is the best way to visualize this (using `ggplot2`)? Graph is rendered using `ROCR` and `ggplot2` packages:               #install.packages("ggplot2","ROCR","verification") #if not installed yet     library("ggplot2")     library("ROCR")     library("verification")     d <-read.csv2("data.csv", sep=";")     pred <- with(d,prediction(x,test))     perf <- performance(pred,"tpr", "fpr")     auc <-performance(pred, measure = "auc")@y.values[[1]]     rd <- data.frame(x=perf@x.values[[1]],y=perf@y.values[[1]])     p <- ggplot(rd,aes(x=x,y=y)) + geom_path(size=1)     p <- p + geom_segment(aes(x=0,y=0,xend=1,yend=1),colour="black",linetype= 2)     p <- p + geom_text(aes(x=1, y= 0, hjust=1, vjust=0, label=paste(sep = "", "AUC = ",round(auc,3) )),colour="black",size=4)     p <- p + scale_x_continuous(name= "False positive rate")     p <- p + scale_y_continuous(name= "True positive rate")     p <- p + opts(                 axis.text.x = theme_text(size = 10),                 axis.text.y = theme_text(size = 10),                 axis.title.x = theme_text(size = 12,face = "italic"),                 axis.title.y = theme_text(size = 12,face = "italic",angle=90),                 legend.position = "none",                 legend.title = theme_blank(),                 panel.background = theme_blank(),                 panel.grid.minor = theme_blank(),                  panel.grid.major = theme_line(colour='grey'),                 plot.background = theme_blank()                 )     p      data.csv contains the following data:               x;group;order;test     56;Tumor;1;1     55;Tumor;1;1     52;Tumor;1;1     60;Tumor;1;1     54;Tumor;1;1     43;Tumor;1;1     52;Tumor;1;1     57;Tumor;1;1     50;Tumor;1;1     34;Tumor;1;1     24;Normal;2;0     34;Normal;2;0     22;Normal;2;0     32;Normal;2;0     25;Normal;2;0     23;Normal;2;0     23;Normal;2;0     19;Normal;2;0     56;Normal;2;0     44;Normal;2;0


title: Re-expression units conversion

text: I was arguing with my Stats teacher today in class because we were stuck on the topic of re-expression. She decided to give me extra credit if I could prove to her, using the internet, something related to unit conversions during re-expression. When you perform a power transformation with a power of $-1/2$ or $-1$, you're supposed to change the units and the value to negative (ie. for a power of $-1$ you might get $-32.8$ -gallon/mile). Are you supposed to change it back to $32.8$ gallon/mile?


title: Why is the Dirichlet distribution the prior for the multinomial distribution?

text: In LDA topic model algorithm, I saw this assumption. But I don't know why chose Dirichlet distribution? I don't know if we can use Uniform distribution over Multinational as a pair?


title: Experiment involving darts, how should I measure the response?

text: I'm conducting a project where I determine which factors (3 in total) contribute to the accuracy in which one would hit a dartboard. Each factor has two levels, low and high. For example, one of the factors is distance and the lowest distance (closest to the dart board) is 2 meters while the highest distance is 4 meters. Another factor I'm using is the left and right hand to throw. My problem is, how do I measure the response? should I indicate on the dart board two areas where one is considered a bulls-eye (success) and the other a failure? Or should I measure the distance in centimeters between the bulls-eye and the dart to determine which dart has come closest? which is the appropriate method?


title: Pearson Correlation Question

text: I am a beginner on calculating correlations and I would like to ask possibly a very simple question. If I have two variables that I want to perform correlation on, and those variable are of unequal size, does this affect my correlation result? Thanks


title: Are there methods for automatically detecting features of a curve?

text: I have a raw discrete data(curves). I need to find methods for detecting features of each curve. Some example features: 1) Stable growing   2) Fast growing   3) Stable falling   4) Fast falling   5) And so on Here is good example of such curves type(russian): ![enter image description here](http://i.stack.imgur.com/OWefy.png) How can i do it?


title: How to deal with imperfect multicollinearity

text: I am estimating a model for an assignment, and found out that there is a 0.9 correlation between two of the independent variables. So If I am not wrong I should omit one variable and redo the regression. However, it is asked to me, to estimate the log-log model and then the correlation between variables drop from 0.9 to 0.5. so in the new model, should I still omit one of the variables? Also, which should be the cut-off in order to decide where I should omit a variable? Is 0.6 correlation high enough to give problems to the model?


title: Inter-rater statistic for skewed rankings

text: I have several sets of 10 raters which I want to compare. Each rater can cast only _Yes_ or _No_ vote, however this decision is skewed and the _Yes_ votes make only about 10% of all votes (and this is expected, i.e. the such proportion is objectively true). Which of the inter-rater agreement statistics would be suitable in this case?


title: Poisson regression vs. log-count least-squares regression?

text: A Poisson regression is a GLM with a log-link function. An alternative way to model non-normally distributed count data is to preprocess by taking the log (or rather, log(1+count) to handle 0's). If you do a least-squares regression on log-count responses, is that related to a Poisson regression? Can it handle similar phenomena?


title: How can I compare the following mutual information values?

text: How can I compare the following mutual information values ? I'm just wondering what's the most appropriate way to display them in my report table. I'm computing them with this formula ![http://d.pr/chkK](http://i.stack.imgur.com/g1i2L.png) where e and c are clusters and the intersection is the number of elements in common. For each couple e and c I have a I value (mutual information). Successively I average over all e belonging to the same category (not shown in the formula) and I end up with a table like:               cat1 0.0123     cat2 0.0012     cat3 0.0009     cat4 0.0100     ...      The mutual dependency values are usually very low (around 0.01), because n (total amount of documents in the collection) is very high. Should I use another measure, or... what do you suggest ? thanks


title: How to calculate stock volatility in %?

text: How can I calculate the stock volatility in **percentage**? Do i have to use sd() function without any other calculation ? Thanks


title: Sliding window validation for time series

text: I have a broad question about sliding window validation. Specifically, I am looking at using Rapid Miner to predict future values of a financial series using "lagged" values of that series and other covariates. I have been experimenting with the windowing operator in this software and lagging the values to prepare for modeling. What I am confused about, and suspect this is a general process, not just something centric to Rapid Miner and thus I ask it here, is the sliding window training/evaluation process.   1. Does anyone have sources to recommend for learning about sliding window processes for building data mining models on time series?   2. Specifically when building a model, I _think_ I understand that k instances are used to train a model (e.g. SVM) and the performance of this model is determined by predicting the next m records. Then, the window is slid forward some amount and the next k records are used for training and the evaluation is done on the subsequent m records. This continues until the end of the data.  Is my understanding correct? How is a final model built for use on future data? Is it always re-trained on the last k records and these last k records would only be used to create the final model?


title: What count-data models to choose besides negative binomial model when overdispersion occurs?

text: Assume that you have a Poisson model with overdispersion. Besides negative binomial models, what are other appropriate count-data modeling regression techniques?


title: Recursive partitioning using median (instead of mean)

text: As I am only familiar with the basics regarding decision trees I would like to ask, with the risk of stating silly question: Is it possible to perform recursive partitioning with the group median as the response/objective? For example, in stead of `R`'s `rpart()` using means, could a similar tree be created with medians? I want to do this because the continuous dependent variable I want to examine has a series of outliers that clearly affect the mean values (especially when the number of observations in each node gets smaller). Am I on the right track, or should I be using other kind of methods? Would preprocessing the data be another alternative (perhaps "capping" the values at a upper limit)? Thanks


title: How to proceed with applying concordance correlation coefficient on stock market data?

text: I am trying to find out if there is a relation between 5 different stock markets rates over a period of 5 years. I was advised by my supervisor to use the Concordance Correlation Coefficient which unfortunately I have not studied and I am unsure how to proceed. I was also advised to use Timeseries on the data. At the moment I am studying the theory from a book but would someone be able to advise me of how should proceed, I will be grateful.


title: Tangency portfolio in R

text: I am trying to solve for an efficient portfolio in R. How do I translate my constraints for a tangency point for 2 risky asset portfolio, and a given risk free rate to R solve.QP function? So basically I have the following equations:               w = weight of the first risky asset     R1 = mean return of the first risky asset     R2 = mean return of the second risky asset     sd1 = sdev of first risky asset     sd2 = sdev of second risky asset     corr = correlation between two risky assets     rf = risk free rate     Return of portfolio, R = R2*(1-w)+R1*w     Standard Dev of portfolio, SD = sqrt((sd1*w)^2+(sd2*(1-w))^2+2*w*(1-w)*corr*sd1*sd2)          Now I need to maximize R-rf while minimizing SD (that is maximize my sharpe).      Let sigma be covariance matrix. So my function to minimize is W^T*sigma*W where W is     the weights vector. Now simulataneously I need to maximize the excess return (R-rf)     and W^T*1=1. I don't know how to express that in the constraints function.      I am confused how to express these constraints as expected by http://pbil.univ-lyon1.fr/library/quadprog/html/solve.QP.html . If you could also point me to a solved derivation of the final formula, that would be helpful as well as I am unable to get to the final formula.


title: What kind of results are there about prior knowledge?

text: Consider the following two problems:   1. We want to do linear regression, where we know that the true model is actually linear. In the general case, linear regression includes an offset. However, we know, using prior knowledge, that the true model actually has offset 0. So clearly, it is better to fit the model while constraining that offset is 0.   2. We want to estimate a distribution $p(z)$ and find its mode. We estimate $p$ using, let's say, density estimator, to be $\tilde{p}$. Now, again, we know some properties about the mode, such that, for some function $F(z_{mode}) = 0$. Then, (perhaps?) it is better to find the mode of $\tilde{p}(z)$ while constraining $F(z)$ to be $0$, instead of just finding the mode of $\tilde{p}$ without this constraint. My question is: are there any known results about incorporating this kind of prior knowledge (if it is true)? A result that shows that our solution is better when using the prior knowledge, or so on? I hope it is not too vague. I am interested in reading more about this. Thanks.


title: How to begin reading about data mining?

text: I'm a novice who is going to start reading about data mining. I have basic knowledge of AI and statistics. Since many say that machine learning also plays an important role in data mining, is it necessary to read about machine learning before I could go on with data mining?


title: Working with Likert scales in SPSS

text: I performed a survey using a Likert 1 to 5 scale (totally agree/agree/neutral/ disagree/totally disagree) on 12 questions which are split into 3 statements which the respondent places a value of between 1 to 5 dependent on how much they agree or disagree - there are 36 statements in total. Respondents:               Group 1                          Group 2     architects UK   140 replied      Architects US   100 replied     engineers UK    140 replied      Engineers US    100 replied     Contractors Uk  140 replied      Contractors US  100 replied      The data is in Excel. ### Questions:   1. How do I input them in SPSS?   2. How do I work out the frequency of replies for each recipient?   3. How do I work out frequency of replies i.e agrees/disagrees etc for each group?   4. How can I rank each individual question (12 of them)? Remember, there are 3 individual statements to each question.   5. How do I compare UK architects to US architects to show congruence or not?   6. How would show correlation between the two groups UK and US?   7. Will SPSS develop graphs etc for me showing frequency or correlation?


title: Is it allowed to include time as a predictor in mixed models?

text: I always believed that time should not be used as a predictor in regressions (incl. gam's) because, then, one would simply "describe" the trend itself. If the aim of a study is to find environmental parameters like temperature etc. that explain the variance in, let´s say, activity of an animal, then I wonder, how can time be of any use? as a proxy for unmeasured parameters? Some trends in time on activity data of harbor porpoises can be seen here: -> How to handle gaps in a time series when doing GAMM? my problem is: when I include time in my model (measured in julian days), then 90% of all other parameters become insignificant (ts-shrinkage smoother from mgcv kick them out). If I leave time out, then some of them are significant... The question is: is time allowed as a predictor (maybe even needed?) or is it messing up my analysis? many thanks in advance


title: Are there methods to cluster based on pair wise dissimilarity measures?

text: I am facing a problem of learning clusters from a pairwise dissimilarity matrix. The measures i try are not metrics(not symmetric,no triangular equality), like KL-divergence for example. Are there Clustering methods i can use to find clusters from these kinds of matrices? If i have labels available, what kind of cluster validation techniques can i use?


title: Show that $\mathbb{E}(g(T-p)) < \mathbb{E}(g(S-p))$ for any convex function $g$ if $T$ and $S$ are estimators of $p$

text: The more detailed question. ![question](http://i.imgur.com/DVazCqR.png) I'm kinda having some trouble starting out with answering this question. My initial approach would be to $g(x)= x^2$ since that is a convex function and find the expected values of $(T-P)^2$ and $(S-P)^2$. However I fear that that would be too specific since the question would like to prove for any convex function. Is there a way to prove this more generally?


title: How to compare different distance measures in time-series clustering?

text: My aim is to cluster 126 time-series concerning 26 weeks (so each time-series has 26 observation) by partitioning around medoids. Before clustering I wanted to compare which distance measure is the most appropriate: euclidean or dynamic time warping. I used each distance to cluster and compare by silhouette plot. Is there any way I can compare different distance measure? Additional question: is GAP statistic enough to decide how many clusters choose? Or should I evaluate number of clusters with different methods or compare two or three ways how to do it? I would be grateful for any suggestions.


title: continuous vs categorical logistic regression for marks and admission

text: I have a list of marks scored by students in Science (X, between 0 to 100%) and whether they went to college to or not (Y).   High marks in science showed a higher concentration of college admits and low marks had the second best hit rate (students went for Arts degree, etc). Scores in the intermediate range had a lower hit rate. Most students have lower scores in Science. I divided my sample into 5 bins: 0-10, 10-20, 20-80, 80-90, 90-100. Found Chi-Sq to be significant. I transformed these bins into dummy categorical variables and then calculated LR coeff which were significant for 80-90, 90-100(+ives), and 0-10(-ive). I concluded that when scores are higher, the odds of getting admissions are high. When scores are low, it is unlikely that the student would go to college. Q1. Should I instead use marks as `continuous variable`? My main hypothesis is: highest marks result in college admissions, while the lowest marks imply no college admission. Q2. The data was linear before. However, what if the data was shown in the chart below, do you think categorizing it made more sense? Event hit rates are higher for a particular range and then trail off. Categorizing it in bins helps as I just to have to analyze the performance of that bin (main focus of the study). May be using dummy variables removes the affect of non-linear relationships? ![enter image description here](http://i.stack.imgur.com/gQjYY.jpg)


title: Why am I getting information entropy greater than 1?

text: I implemented the following function to calculate entropy:               from math import log          def calc_entropy(probs):         my_sum = 0         for p in probs:             if p > 0:                 my_sum += p * log(p, 2)              return - my_sum      Result:               >>> calc_entropy([1/7.0, 1/7.0, 5/7.0])     1.1488348542809168     >>> from scipy.stats import entropy # using a built-in package                                          # give the same answer     >>> entropy([1/7.0, 1/7.0, 5/7.0], base=2)     1.1488348542809166      My understanding was that entropy is between 0 and 1, 0 meaning very certain, and 1 meaning very uncertain. Why do I get measure of entropy greater than 1? I know that if I increase size of log base, the entropy measure will be smaller, but I thought base 2 was standard, so I don't think that's the problem. I must be missing something obvious, but what?


title: R lm without intercept strange contrast

text: I am actually doing a linear regression without intercept `lm(y~ -1 + a + b + c + d )`, where `b`, `c` and `d` are factors. (I know that it's usually not a good idea to do it without intercept but it's a necessity for my problem) And i noticed something strange in the determination of my coefficient by R: R estimates all the possible values of `b` but it's not a sum constrast ! and if i put `c` as the first variable (which isn't numeric) it does it for `c` but not anymore for `b`? Does anyone know where it comes from? Is it because my model has no intercept? And may i change it without putting the intercept back in my model?


title: Which model for data which has within-effects, between effects and non-parallel lines

text: My data (Panel, 19 countries, 7 years, ordinal outcome variable) has within- effects (countries over time), between effects (between countries) and it seems as if the slope of lines (test of parallel lines) differs between the different outcome categories. How can you estimate the model in SPSS properly?


title: Randomly generated adjacency matrix in R

text: I'm wanting to generate random connected directed acyclic graphs, and am wondering if there is a way of populating an adjacency matrix in R which would represent the aforementioned. Something similar to this would be nice (that is done with Mathematica): http://mathematica.stackexchange.com/q/608 Thanks.


title: Pros/cons of estimating parameters for missing observations?

text: Some people are playing a game online. Every time a person plays, a new game board is generated randomly. On generation of a new board, the player can also choose a special weapon. (The choice of weapon does not affect the board generation.) Over time, observed data are like this:               player weapon win     3      5      0    # player 3 used weapon 5 and lost     4      2      1    # player 4 used weapon 2 and won     8      1      1    # player 8 used no weapon and won      To find out if some players are better with some weapons, and others with other weapons, I've modeled this as a Bayesian logistic regression: $logit(Pr(Win=1)) = \alpha_{p,w}$ $\alpha_{p,w} \sim N(\mu_{w},\sigma^2)$ So, $\mu_w$ is the effect of weapon $w$ across all players, and $\alpha_{p,w}$ is the effect of player $p$ with weapon $w$. Because this is a Bayesian model, the prior for $\alpha_{p,w}$ is $\mu_w$, so in theory, I can estimate $\alpha_{p,w}$ for all combinations of $p$ and $w$ even if some players never use some weapons. (JAGS will do this with nested for loops over players and weapons, for example.) But is that a bad idea? Also, is there a special name for this kind of model? P.S. Please edit the title and tags if you can make it clearer.


title: Scaling of huge data and estimating the distribution

text: Suppose you have some sample of huge numbers and you want to fit some continuous distribution to these numbers. You will get some distribution that is loosely speaking highly "smeared", e.g. it has a huge scale factor. For numerical purposes this might be very inconvenient. Now what happens if one rescales the data and then fits the distribution? In case of scale invariant distributions there should be no difference. You just scale the distribution again. How about distributions that are not scale invariant? Intuitively, I'd say it is hazardous to scale and then fit. One might introduce a serious error if you compute e.g. a quantile from the resulting distribution. So how does one deal with such huge data - is scaling unfavorable?


title: Effective way to visualize net growth/profit/income?

text: A non-profit has donors with monthly commitments, which you can think of as subscriptions. **What is the best way to show for each month new acquisitions, cancels, and net growth**? The relationship is                net growth = new acquisitions - cancels      The same time of chart could be appropriate for net profit (business) or net income (personal finances). The first (and more important) problem is the design: it must be attractive and easily understood by non-technical people. The design is really the answer to this question. The second problem, which is not required in the answer, is the implementation: I prefer to implement this in SAS, Excel, SharePoint, or some JavaScript library--- or maybe R. One story behind the chart is new acquisitions have a high cancel rate, so they both help and hurt net growth. (However, the cancels do not distinguish whether the commitment was new or old.) My boss drew this example chart: ![Example chart](http://i48.tinypic.com/2mq6w07.jpg) Here is a new design with an idea from xan's post. In this design, cancels are implicit, so I was able to simplify by eliminating one chart element. ![design 2](http://i.stack.imgur.com/iBG4n.png) This chart from an EU report is similar: it shows imports, exports, and the trade balance. ![EU chart](https://learnr.files.wordpress.com/2009/09/export_import_graph5.png?w=600) Here is what may be my final version: ![Net growth version 2](http://i.stack.imgur.com/0AOfn.png)


title: How can I know a random small subsample has a good performance?

text: Sorry, if the title is confusing. not native speaker here. I ran a survey study which had 15 7-point likert scale questions. Each question is ask people to judge how creativity an object in the picture. I received 153 responses in total. What I try to understand is whether those people who answered the survey did a similar judgement as experts do. I have 15 experts completed the same survey and compute the mean rating for each picture. For the 153 responses (I call this group as crowd), I also calculated the mean rating for each picture. Here is the mean rating data of the two groups (E-experts, C-crowd)               E   C     3.36    2.33     1.43    1.53     1.07    1.19     2.43    2.69     2.14    1.92     4.64    4.22     4.71    4.36     4.00    3.24     4.21    3.20     5.07    4.78     4.36    3.41     4.64    5.02     6.14    5.50     5.79    5.73     6.00    5.95      my first question about the data is whether the two group did the judgment similar? could I do a simple t-test comparing the mean rating of 15 questions and tell the result? second, since I do the survey for an application design. In the real system, for each picture , I could expect only 15-20 people come to rate it. So I want to know if 153 responses I collected now say they do similar judgements as those experts(n=15) do, whether I could say random 15 people could also do similar judgement? How can I know a small sample (a subsample) has a similar performance as a big sample has? one way I think about is to random subsample 15 people from the 153 responses for 10000times, and use the mean rating to be compared with the mean rating of 153 responses. but I'm not sure whether it is a right way to do it. Please do ask for more information if you get confused by my language.


title: Cluster analysis with ties issue

text: When I perform cluster analysis in SAS, the SAS log sometimes return a warning something like this:                   WARNING: Ties for minimum distance between clusters have been                   detected at 4 level(s) in the cluster history.      Depends on the clustering methods (all hierarchical clustering methods) I choose, the number of ties will change. But regardless of the methods, I want to know if there is a solution to this issue. Or is this OK to ignore since the issue arises from the fact that the distance of values are really close to each other?


title: Using PCA for detecting similar regions in an image

text: I'm trying to understand an algorithm that detects similar regions of an image using PCA. The algorithm essentially divides the image into overlapping square blocks and then does PCA with each block as a dimension. Next it does a lexicographical sort of the principal components to arrange similar blocks close to one another. Finally similar regions are selected based on distance in the list and a few other measurements. The PCA part of the algorithm is described as follows: > An image is tiled with overlapping blocks of $b$ pixles ($\sqrt{b} \times > \sqrt{b}$ pixels in size), each of which is assumed to be considerably > smaller than the size of the duplicated regions to be detected. Let $ \vec > x_i,i=1,...,N_b,$ denote these blocks in vectorized form, where > $N_b=\left(\sqrt{N}-\sqrt{b}+1 \right)^2$. We now consider an alternate > representation of these blocks based on a principal component analysis > (PCA)[15]. Assume that the blocks $\vec x_i$ are zero-mean, and the compute > the covarience matrix as: \begin{equation*} \tag{1.1} C = \sum_{i=1}^{N_b} > \vec x_i \vec x_i^T \end{equation*} The orthonormal eigenvectors,$\vec > e_j$,of the matrix $C$, with corresponding eigenvalues,$\lambda_j$, > satisfying: \begin{equation*} \tag{1.2} C \vec e_j = \lambda_j \vec e_j, > \end{equation*} define the principal components, where $j=1,\dots,b$ and $ > \lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_b$. The eigenvectors, > $\vec e_j$, form a new linear basis for each image block,$\vec x_i$: > \begin{equation*} \tag{1.3} \vec x_i = \sum_{j=1}^b a_j \vec e_j, > \end{equation*} where $a_j = \vec x_i^T \vec e_j$, and $\vec a_i = \left(a_i > \dots a_b \right)$ is the new representation for each image block. > > The dimensionality of this representation can be reduced by simply > truncating the sum in Equation (1.3) to the first $N_t$ terms. Note that the > projection onto the first $N_t$ eigenvectors of the PCA basis gives the best > $N_t$-dimensional approximation in the least squares sense, if the > distribution of the vectors, $\vec x_i$, is a multi-dimensional Gaussian > [15]. This reduced dimension representation, therefore, provides a > convenient space in which to identify similar blocks in the presence of > corrupting noise, as truncation of the basis will remove minor intensity > variations. I've completed translating the algorithm into MATLAB/Octave but it does not appear to be working correctly. Below is my test image with a screen shot of the data in vectorized form and the PCA results. Here $N_b = 225$ and $ b=4$. In the PCA table all the remaining rows are either 0 or -1. ![The test image.](http://i.stack.imgur.com/LGF49.gif) (Please download test image at left. Only two blocks are identical.) ![Input data.](http://i.stack.imgur.com/G05rI.png) ![PCA results.](http://i.stack.imgur.com/n4wGK.png) What I don't understand is how a lexicographical sort on the PCA data will show similarity. My expectation was that row 1 and 4 in the PCA output would be identical and that PCA would allow me to remove higher columns to increase matches. Similar to the way discrete cosine transform moves 'important' information to a smaller number of values of the data set. I've verified that my PCA algorithm mostly matches (signs are different) vs MATLAB's `princomp()` command so I must not understand what PCA is telling me. My question is, how would a sort on PCA results show self-similarity?               %This does principal component analysis     clear all;          bSide = 2; % Size of side of one block          %Read B&W image     chanData = imread('./randS.gif');          %Setup the input values     [imageWidth,imageHeight] = size(chanData);     tic();          %Calculate the derived constants     N = imageHeight * imageWidth;     b = bSide^2;     Nb = (sqrt(N) - sqrt(b) + 1)^2;     printf("ET:%d - ",toc());     printf("Finished deriving constants: N:%d b:%d Nb:%d\n",N,b,Nb);          %A set of observations of M variables     X = zeros(b,Nb); %Block representation of input image     u = zeros(b,Nb);%Temp for zero-mean calculation     coords = zeros(Nb,2); %Coordinate of the block at this location          %Create our matrix holding the dimension data     %X = zeros([Nb,b]); %b columns by Nb rows          %Populate the dimension matrix     iWidth = imageWidth - bSide + 1;     row = 1;     column = 1;     for i = 1:Nb       %Setup local variables       Xtemp = zeros(1,b);            %Each block becomes a column of bSide rows       for j = 1:bSide         lowerX = (j - 1)*bSide + 1;         Xtemp(lowerX:lowerX+(bSide-1)) = chanData(row+(j-1),column:column+(bSide-1));       end            %Add this vector as a column to the array       X(:,i) = Xtemp;            %Now create a mean vector       u(i) = mean(X(:,i));            %Save the coordinates of this block in a two column matrix       coords(i,1) = row;       coords(i,2) = column;            %Update our source matrix indicies       if column == iWidth         row = row + 1;         column = 1;       else         column = column + 1;       end     end     clear Xtemp row column j iWidth lowerX;     printf("ET:%d - ",toc());     printf("Finished matrix construction\n");          %Get the mean for this vector and subtract it from the data     B = zeros(b,Nb);     for i = 1:Nb       B(:,i) = X(:,i) - u(i);     end     printf("ET:%d - ",toc());     printf("Allocated mean-zero matrix\n");     %clear u X;          %Find the covarience matrix     C = cov(B);     printf("ET:%d - ",toc());     printf("Found covarience matrix\n");     %clear B;          %Find the eigenvalues and eigenvectors     %Evec: Nb x Nb matrix of eigenvectors, each column is a vector     %Eval: Nb x Nb matrix where only the main diagonal contains eiganvalues     [Evec,Eval] =  eig(C);     Eval = diag(Eval); %Convert eigenvalue matrix to vector     %clear C;     printf("ET:%d - ",toc());     printf("Found eigenvalues\n");          %Sort by eigenvalue. We do this by appending the eigenvalue vector as the     %first row. MATLAB won't sort by column for some reason, so we must     %transpose.     sortedEvec = [transpose(Eval); Evec];     sortedEvec = transpose(sortedEvec);     sortedEvec = sortrows(sortedEvec,-1);     sortedEvec = transpose(sortedEvec);     printf("ET:%d - ",toc());     printf("Sorted eigenvectors\n");          %Remove the eigenvalues row     sortedEvec(1,:) = [];          %Now do the final multiplication of the two transposed matrices     PCA = transpose(sortedEvec) * transpose(B);     printf("ET:%d - ",toc());     printf("Finished PCA\n");     %PCAt = PCA'; %PCAt matches princomp(B) except signs are reversed for first column          %Now integer quantize the array     PCA = floor(PCA);     printf("ET:%d - ",toc());     printf("Quantized PCA\n");      This is code I used to get the PCA matrix. princomp isn't defined for Octave but I was able to access MATLAB and run it there. My results match except the signs are inverted and the matrix from princomp is the transpose of my results.


title: Is comparing standard deviation a valid way to evaluate the usefulness of a sample?

text: I have data for about 1,800 email messages, and am building a tool in Excel that enables my team to create "benchmarks" on-the-fly -- they can select various characteristics of a message (for example, audience size) and see the mean performance of similar past messages. I use COUNTIF to indicate the number of similar messages being averaged, with the idea that the mean performance of one or two messages is less reliable than five or six (or more). I'm wondering if there is a meaningful way to use standard deviation to indicate whether the sample they've generated is reliable. For example, is it useful to compare the standard deviation of the open rate for the entire population to the standard deviation of the open rate for the sample they're looking at? (It has been several years since my last statistics class, and I'm just starting to piece this stuff together again.)


title: What are the difference between CNN, RBM,and auto-encoder?

text: Recently I am learning about deep learning and I am confused between the terms (or say technologies). What are the different between convolution neural network (CNN), restricted Boltzmann Machines (RBM), and auto-encoders? To me, auto-encoders and CNN are very similar???


title: plotting line graph with error bar in R from cross tabulated data

text: I am wondering how could I line plot mean with error bar of cross tabulated data in R package. My data looks like this               Mean                 Stage     Treatment    breaker        green        pink             red          turning          cont    0.06779080    0.1964575    0.010335260    0.005861540    0.05828315          FRM     0.17211680    0.1853831    0.021794200    0.013183020    0.09467819          KNO3    0.24755945    0.2863050    0.007866763    0.002554247    0.06097948          LFR     0.08053258    0.1287083    0.008566730    0.000000000    0.07710268          salt    0.38564291    0.4419331    0.050551620    0.001115087    0.14508939          Standard deviation              Stage     Treatment    breaker        green        pink            red            turning          cont    0.01504616    0.14157770    0.003498454    0.0037076134    0.032316651          FRM     0.02710538    0.03422274    0.017164023    0.0143484928    0.058139975          KNO3    0.05372195    0.12865178    0.003552846    0.0006738978    0.001983925          LFR     0.04366554    0.05611974    0.009180117    0.0000000000    0.029396442          salt    0.03035888    0.12908778    0.028818513    0.0004681397    0.050566535


title: How to decide which penalty measure to use ? any general guidelines or thumb rules out of textbook

text: A number of regularization measures are available in literatures, which is kind of confusing to beginners. The classical penalty is **ridge** by Hoerl & Kennard (1970,Technometrics 12, 55–67). ![enter image description here](http://i.stack.imgur.com/uMQkC.png) Another modification to this is **lasso** by Tibshirani (1996, Journal of the Royal Statis- tical Society B 58, 267–288), defined as: ![enter image description here](http://i.stack.imgur.com/Y5eY5.png) Another penalty is the **elastic net penalty** (Zou and Hastie 2005, Journal of the Royal Statistical Society B 67, 301–320) , which is a linear combination of the lasso penalty and the ridge penalty. Therefore the penalty covers these both as extreme cases.   ![enter image description here](http://i.stack.imgur.com/rnP5w.png) The another penalty that I could find is **bridge penalty** introduced in Frank & Friedman (1993, Technometrics 35, 109–148). where λ ̃ = (λ, γ). It features an additional tuning parameter γ that controls the degree of preference for the estimated coefficient vector to align with the original, hence standardized, data axis direc- tions in the regressor space. It comprises the lasso penalty (γ = 1) and the ridge penalty (γ = 2) as special cases. ![enter image description here](http://i.stack.imgur.com/KHdyi.png) My question is : **_are there any preferences on type of penalty to use - something from or out of statistical text books_** ? Or this is just trial and error ? Please explain to layman language.


title: Logistic regression with multi-class features in R

text: I'm working with a data set like the following: _X_ =               c1 c2 c3 c4 c5 y     a  c  f  h  j  0     a  d  f  i  k  0     a  c  g  h  j  1     a  c  f  h  k  0     b  d  g  h  k  0     b  e  f  h  j  0      I'm trying to create a logistic regression model that is able to predict `y` (0 or 1) based on the features of _X_. Info on _X_ :   The values of `c1`–`c5` are all factors.   The features have a varying number of levels (e.g. `c1`: 2 levels, `c2`: 4 levels, `c3`: 3 levels, etc.)   Approx 10% of `y` is 1. I've tried using SVM and GLM without any good result.               model <- svm(y ~ ., data = X)       pred <- predict(model, X)        table(pred,X$y)                  y     pred    FALSE TRUE     FALSE   1332   113     TRUE       0     0      and               model <- glm(y ~ ., family=binomial("logit"), data=X)       pred <- predict(model, X, type="response")       table(pred,X$y)            pred                   FALSE TRUE     4.2260288377431e-08        2    0     4.24333100181876e-08       1    0     ...     0.706714407238236          1    1     0.736650629322038          0    1      I'm used to working with features with continuous values when creating a predictive model, but I don't really know how to tackle a problem with factors.   What would you guys recommend for this type of problem? Changing from logistic regression to something else or using other functions besides GLM & SVM?


title: How to design good plots for multiple time series?

text: I have huge (about 100 000) set of time series. I need to show between 5 to 10 time series, chosen semi-randomly on one chart. Chart estate is very limited - plot for each time series is only 100px x 800px. It is not possible o predict which time series will be shown together - they have to be plot independently. Ranges of all variables are very different: for one time series it can be between 0-3000 and for another 0 to 50. General range of all variables is 0-60000. I want to use horizon graphs: http://timelyportfolio.blogspot.com/2012/08/more-on-horizon-charts.html but, if I make plots using 4 bands almost all time series will look flat. If I plot using 32 bands - plots will change from horizon charts, to heatmap charts. Is it good idea, to set bands range not uniform? i.e. 1st band: 0-100, 2nd: 100-1000 3rd: 1000-5000, 4th: 5000-60000.


title: Compute maximum likelihood estimator and confidence interval

text: Let $X$ have a Poisson distribution with parameter $\lambda$. Find the maximum likelihood estimator of $\alpha=\mathbb P(X=0)$. In a sample of size 100 from a Poisson distribution, it is found that the mean is 0.75. Calculate an approximate 95% confidence interval for ${\alpha}$. For the MLE, we want to maximise $e^{-\lambda}$, but isn't that a constant? For the confidence interval, I think it is an interval of radius $\frac{0.075}{0.075/100}$, but I can't quite remember. A few years since A levels...


title: Most common way of evaluating SVM classifier

text: I used a SVM to classify my features. With cross validation, on my testing set, I obtained an accuracy of 81%. I want to measure how good/bad this is. Is the technical term for good/bad statistical significance? With my SVM we predict that an 8 dimensional feature vector is in 1 out of the 2 classes. I wanted to use a permutation test to find the p-value, but I do not know what to use for the test statistic.   1. What can I use for the test statistic?   2. Is this the wrong line of thought? What else can I do to show that 81% is a good/bad baseline? Thanks for all the help!


title: How to interpret dendrogram height for clustering by correlation

text: Given the following data frame:               df <- data.frame(x1 = c(26, 28, 19, 27, 23, 31, 22, 1, 2, 1, 1, 1),                      x2 = c(5, 5, 7, 5, 7, 4, 2, 0, 0, 0, 0, 1),                      x3 = c(8, 6, 5, 7, 5, 9, 5, 1, 0, 1, 0, 1),                      x4 = c(8, 5, 3, 8, 1, 3, 4, 0, 0, 1, 0, 0),                      x5 = c(1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0),                      x6 = c(2, 3, 1, 0, 1, 1, 3, 37, 49, 39, 28, 30))      Such that               > df        x1 x2 x3 x4 x5 x6     1  26  5  8  8  1  2     2  28  5  6  5  1  3     3  19  7  5  3  1  1     4  27  5  7  8  1  0     5  23  7  5  1  1  1     6  31  4  9  3  0  1     7  22  2  5  4  1  3     8   1  0  1  0  0 37     9   2  0  0  0  0 49     10  1  0  1  1  0 39     11  1  0  0  0  0 28     12  1  1  1  0  0 30      I would like to group these 12 individuals using hierarchical clusters, and using the correlation as the distance measure. So this is what I did:               clus <- hcluster(df, method = 'corr')      And this is the plot of `clus`: ![dendogram](http://i.stack.imgur.com/ALfbr.png) This `df` is actually one of 69 cases I'm doing cluster analysis on. To come up with a cutoff point, I have looked at several dendograms and played around with the `h` parameter in `cutree` until I was satisfied with a result that made sense for most cases. That number was `k = .5`. So this is the grouping we've ended up with afterwards:               > data.frame(df, cluster = cutree(clus, h = .5))        x1 x2 x3 x4 x5 x6 cluster     1  26  5  8  8  1  2       1     2  28  5  6  5  1  3       1     3  19  7  5  3  1  1       1     4  27  5  7  8  1  0       1     5  23  7  5  1  1  1       1     6  31  4  9  3  0  1       1     7  22  2  5  4  1  3       1     8   1  0  1  0  0 37       2     9   2  0  0  0  0 49       2     10  1  0  1  1  0 39       2     11  1  0  0  0  0 28       2     12  1  1  1  0  0 30       2      However, I am having trouble interpreting the .5 cutoff in this case. I've taken a look around the Internet, including the help pages `?hcluster`, `?hclust` and `?cutree`, but with no success. The farthest I've become to understanding the process is by doing this: First, I take a look at how the merging was made:               > clus$merge           [,1] [,2]      [1,]   -9  -11      [2,]   -8  -10      [3,]    1    2      [4,]  -12    3      [5,]   -1   -4      [6,]   -3   -5      [7,]   -2   -7      [8,]   -6    7      [9,]    5    8     [10,]    6    9     [11,]    4   10      Which means everything started by joining observations 9 and 11, then observations 8 and 10, then steps 1 and 2 (i.e., joining 9, 11, 8 and 10), etc. Reading about the `merge` value of `hcluster` helps understand the matrix above. Now I take a look at each step's height:               > clus$height     [1] 1.284794e-05 3.423587e-04 7.856873e-04 1.107160e-03 3.186764e-03 6.463286e-03          6.746793e-03 1.539053e-02 3.060367e-02 6.125852e-02 1.381041e+00     > clus$height > .5     [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE      Which means that clustering stopped only in the final step, when the height finally goes above .5 (as the Dendogram had already pointed, BTW). Now, here is my question: **how do I interpret the heights?** Is it the "remainder of the correlation coefficient" (please don't have a heart attack)? I can reproduce the height of the first step (joining of observations 9 and 11) like so:               > 1 - cor(as.numeric(df[9, ]), as.numeric(df[11, ]))     [1] 1.284794e-05      And also for the following step, that joins observations 8 and 10:               > 1 - cor(as.numeric(df[8, ]), as.numeric(df[10, ]))     [1] 0.0003423587      But the next step involves joining those 4 observations, and I don't know:   1. The correct way of calculating this step's height   2. What each of those heights actually means.


title: What type of graphs can be used to show overlap between many sets?

text: I have a dataset with 90 different sets of GO terms (e.g. one set would be ("GO:0014075", "GO:0060742", "GO:2001238", "GO:0071478"). I want to plot the overlap between all of these groups. I don't know what, if any, type of plots are used for showing overlap between many groups. For a smaller set of groups I could use a venn diagram, but this does not work with a large group of comparisons. What type of plot can be used for plotting the overlap between many groups?


title: How to interpret log-likelihood outputs from MASS::fitdistr (R)

text: **AIM:** Fit the best distribution to columns in a dataset (30k records) so that I can to go on to produce test data that is in a similar distribution. **WHAT I'VE DONE SO FAR:** Using R, I have found and implemented the MASS function `fitdistr` and written code that checks a number of distributions and returns the log-likelihood of each. **PROBLEM:** I have the log-likelihoods, and I think that I should be selecting the largest log-likelihood as an indication of the best fit. I am unsure however, if this is the right interpretation of the documentation and/or whether I should be going further, for instance by calculating AIC or BIC and using those. Could someone provide guidance on how to interpret the log-likelihood figure and what (if necessary) further analysis needs to be done to correctly pick the best distribution?                   col   distn      loglik      bestfit      1: colA    normal -163390.570         2: colA         t -163383.684     yes      3: colA   weibull -165727.706      4: colB    normal -111180.070      5: colB        t -111047.812      6: colB  weibull -109978.545     yes      7: colC   normal -339501.693      8: colC        t -334629.199      yes      9: colC  weibull -335920.848     10: colD   normal -291132.858     11: colD        t -289509.408     12: colD  weibull -287746.432      yes     13: colE   normal   -6609.996     14: colE        t   -6610.760     15: colE  weibull     781.283      yes


title: UMP of Folded Normal, Finding the Critical Value

text: Let $X_1,...,X_n$ be a random sample from the folded normal distribution with the density... $f(x)= \sqrt{\dfrac{2}{\pi}} \theta^{-1} e^{(-x^{2} \theta^{-2})/2}$ for $x>0$ and $\theta>0$, 0 otherwise. The question asks... Find the UMP for test size $\alpha$ for $H_0:\theta = \theta_0$ vs. $H_0:\theta > \theta_0$. Determine the critical region for the value. So the likelihood ratio is... $$\dfrac{L(f(x,\theta_1))}{L(f(x,\theta_0))}= \bigg(\dfrac{\theta_1}{\theta_0}\bigg)^{-n} exp\bigg[{\sum{x^{2}_{i}}\frac{\theta_{0}^{-2}-\theta_{1}^{-2}}{2}} \bigg]$$ We know the UMP exists because the function is decreasing in $\sum x_{i}^{2}= T(x)$. So we reject the null if $T(x)<k$. I am unsure about how to find $k$ because I do not know how to find the distribution of $T(x)$. Any ideas on this would be greatly appreciated.


title: Record linkage when sources have different fields

text: I have read a little about record linkage, but it seems to me that a requirement is that all fields in both sources can be compared. For example, with sources A and B, an assumption is that we can create a 'comparator' measure on all their fields. So for example both A and B may include First and Last name, location, etc. and we'd compare them with a string distance measure, geographical distance, and so on. But what if that is not the case? I.e. if A and B do share some, but not all fields? What if these non-shared fields do give some information on whether any two records are a match? For example, source A may include 'household expenditure', and B may include 'number of children' or 'civil status' variables. Is it necessary to create a distance measure to compare these? Why shouldn't they be used separately?


title: Standard Error used in Hypothesis Testing and Confidence Interval construction

text: In the excellent Practical Statistics for Medical Research Douglas Altman writes in page 235: _"Because the standard error used for calculating the confidence interval differs from that used in hypothesis testing it can occasionally happen[...] that the confidence interval excludes the value specified under the null hypothesis when the hypothesis gives a non- significant result"_ Could someone comment why the SE's are different in hypothesis testing than in confidence intervals construction and which formulas are appropriate in each case?


title: Regression model for $f(x_1, x_2) = a + b x_1\log x_2$

text: Which regression algorithm do I need to use to fit the coefficients of $f(x_1, x_2) = a + b x_1\log x_2$? Will linear regression with an independent variable $x_1 \log x_2$ work?


title: How do I perform an IDF calculation?

text: How do I perform an IDF calculation? I am uncertain as to whether IDF should be calculated in per-class level or for the entire document set (that contains multiple classes).


title: Unevenly sampled data and the Lomb-Scargle method

text: I managed to estimate the periodogram of unevenly sampled data using the Lomb- Scargle Method. Analyzing the frequency domain it would be interesting to filter out a frequency band and then apply IFFT and get back a filtered signal (of course this would be evenly sampled and quite different from the original). The Lomb-Scargle method does not allow for this procedure. Is there a common approach? Am I missing something here? Thank you!


title: Why does $r^2$ between two variables represent proportion of shared variance?

text: Firstly, I appreciate that discussions about $r^2$ generally provoke explanations about $R^2$ (i.e., the coefficient of determination in regression). The problem I'm seeking to answer is generalizing that to all instances of correlation between two variables. So, I've been puzzled about shared variance for quite a while. I've had a few explanations offered but they all seem problematic:   1. It's just another term for covariance. This can't be the case, as factor analysis literature differentiates between PCA and EFA by stating that the latter accounts for shared variance and the former does not (PCA obviously _is_ accounting for covariance in that it is operating over a covariance matrix, so shared variance must be a distinct concept).   2. It is the correlation coefficient squared ($r^2$). See:      * http://www.philender.com/courses/linearmodels/notes1/var1.html, or      * http://www.strath.ac.uk/aer/materials/4dataanalysisineducationalresearch/unit6/correlationcoefficient/ This makes slightly more sense. The trouble here is interpreting _how_ that implies it is shared variance. For example, one interpretation of 'sharing variance' is ${\rm cov}(A,B)/[{\rm var}(A)+{\rm var}(B)]$. $r^2$ doesn't reduce to that, or indeed a readily intuitive concept [${\rm cov}(A,B)^2/({\rm var}(A)\times{\rm var}(B))$; which is a 4 dimensional object]. The links above both attempt to explain it via a Ballentine diagram. They don't help. Firstly, the circles are equally sized (which seems to be important to the illustration for some reason), which doesn't account for unequal variances. One could assume it is the Ballentine diagrams for the standardized variables, hence equal variance, in which case the overlapping segment would account for the covariance between two standardized variables (the correlation). So $r$, not $r^2$. **TL;DR:** Explanations of shared variance say this: > By squaring the coefficient, you know how much variance, in percentage > terms, the two variables share. Why would that be the case?


title: Weighting the response variable in an lm

text: I want to do a simple lm of y~x where I weight my response variable. This is because the values of y are actually each in turn the value of a slope of another regression of y~year i.e. rates of change over time, and for each of these original regressions the number of recorded years was variable (some had data for 20 years, some for 40, some for 45 etc). I realise weighting an lm has been discussed here: How to use weights in function lm in R? However I am unsure as to whether using the simple instructions outlined in this post I will be weighting y (which is what I want) or x, or whether the weight "acts" across the whole lm and the distinction I am drawing between weighting y specifically, rather than x, is invalid...


title: Type-1 Error of the Two-Stage Design

text: **Background on Two-Stage Designs**   In clinical trials, we are often interested in the response rate $p$ for an experimental treatment. In a typical trial, we might expose $n$ patients to the treatment and observe the total number of responses $X$ in order to test the null hypothesis $H_0: p \leq p_0$ against a one-sided alternative $H_1: p > p_0$, where $p_0$ could be the response rate for a standard-of-care treatment. In order to limit patient exposure to an ineffective treatment, two-stage designs are popular: in the first stage, $n_1$ patients receive treatment and the number of responses $X_1$ is observed. If there are $r_1$ or fewer responses, the trial ends. If $X_1 > r_1$, a second stage is carried out: additional patients are enrolled until a total of $n$ patients have received treatment, and the total number of responses $X$ is observed. If there are more than $r$ total responses, the null hypothesis is rejected; otherwise, the null is not rejected and the new treatment is considered ineffective (or at least not superior to the standard of care). The parameters $n_1, r_1, n,$ and $r$ are chosen by the investigator before carrying out the study. **The Problem**   While working on designing such a study, I have come across a phenomenon that I would like to better understand. Intuitively it seems that increasing the cut-off $r_1$ should decrease the type-1 error of the test. One might reason that for larger values of $r_1$, it is less likely that the study will proceed to the second stage, and thus less likely that one will ultimately reject the null. However, it appears that this is not always the case. For example, with the parameters $n_1 = 20, n = 30, r = 25,$ and $p_0 = 0.6$, I found the type-1 error to be the same (up to 9 digits) for $r = 10$ as for $r = 15$ (I got $\alpha = 0.001510074$). In fact, I get the same number for all values of $r_1 \leq 15$. See my code below. **My Question**   So my question is, does it make sense that changing the value of $r_1$ does not affect the type-1 error in some cases? Can you help me refine my intuition to understand why this is true? **Code**   Here is a quick function I have written to compute the type-1 error (as well as the power, for a given alternative, and the expected sample size) for two- stage designs.               ##### Function to compute the type-1 error, power, and expected sample size for a     ##### two-stage design. The parameters p0 and p1 are the response rates under the     ##### null and alternative, respectively.          twoStage = function(n, n1, r1, r, p0, p1){            # Type 1 error = P(reject H0 | p = p0)        #              = P(X1 > r1 && X > r | p = p0)]       n2 = n - n1       x1 = (r1 + 1):n1       alpha = sum(dbinom(x1, n1, p0)*(1 - pbinom(r-x1, n2, p0)))            # Power = P(reject H0 | p = p1)        #       = P(X1 > r1 && X > r | p = p1)]       # Same as above, but with p = p1       power = sum(dbinom(x1, n1, p1)*(1 - pbinom(r-x1, n2, p1)))            # Expected number of subjects enrolled under H0       expected.n = n1 + (1 - pbinom(r1, n1, p0))*n2            return(cbind(n = n, n1 = n1, r1 = r1, r = r, p0 = p0, p1 = p1, alpha = alpha,        power =   power, expected.n = expected.n))          }          ##### Example:          > twoStage(n=30, n1=20, r1=5, r=25, p0=0.6, p1=0.85)[,"alpha"]       alpha      0.001510074      > twoStage(n=30, n1=20, r1=10, r=25, p0=0.6, p1=0.85)[,"alpha"]       alpha      0.001510074


title: Difficult to understand the meaning of ergodicity and ensemble averaging

text: Chaotic Signal Processing, pg147--148 describes how a signal (output of laser) operating in chaotic regime is ergodic. Literature says that a stationary signal is ergodic, if its ensemble average = time averages. Should it be the statistics computed by time averaging = statistics computed by ensemble averaging?The way I understood from the book in the link is as follows : Case1: There are N samples in a time series I compute the mean1,skewness1,kurtosis1 . Case2: Next, with the N samples, T experiments are conducted and in each of the experiment is calculate the statistics. After, T experiments are done, we calculate the average of these statistics - mean_avg,skewness_avg, kurtosis_avg. This is called as Ensemble averaging or Probabilistic averaging. If mean1 = mean_avg and so on, the the process/signal is ergodic. Is my understanding correct? The last question which I have about the pdf in the picture. The first figure 6.12 is the pdf of one realization and the second is for a single sample. So, does it mean that the first figure is the pdf of the time averaging and the second Figure 6.13 is that of ensemble averaging?If so then how come from a single sample the pdf is drawn in the second picture? I shall be grateful for the explanation of the terms ergodicity, time averaging and ensemble averaging. Thank you![PDF](http://i.stack.imgur.com/MVezp.jpg)


title: nnetar return negative forecast values

text: I am confused about using `nnetar`. All my training time series data are positive numbers, but the forecast results obtained from `nnetar` returns negative numbers. The following is my code               myts <- ts(data, freq = 5)     fit <- nnetar(myts)     fcast <- forecast(fit, h = 100)      However, if I forecast only short period of time e.g., h = 10 or 20. The returned values are positive. Did I do anything wrong ? Thank you.


title: recommendations to analyze a survey of the entire sample frame with a 20% response rate

text: we surveyed all 10,000 professionals in a particular industry. The industry is highly-regulated, so we have contact information for everyone in our population of interest. We attempted to contact 100% of the population. We now have a data set containing the 2,000 responses, because 20% of our population agreed to complete the survey. When administering this survey, there were no sampling probabilities and no clustering at all. There is a lot of variation in the response rate when broken out by state of residence. Since state variation is important in this industry, we plan to calculate weights for this final data set so that any statistics that we run will generalize to the overall population rather than the 20% who responded. I believe the weights should be treated as post-stratification weights, but I'm not certain. I don't imagine this is a terribly complex data set to analyze, but I'm not sure if it's a special case of some sort -- it didn't involve any sampling whatsoever but at the same time it is not the entire universe. I would appreciate any coding tips (in any statistical language) to recommend the survey analysis setup that makes the most sense for data of this structure. if I had to guess, here's the R code I would use:               # start with data set `x` and add a column of five, since 20% responded     x$wgt <- 5     # give everyone in the data set a weight of five          # provide only a column of 5's to the `svydesign` command     y <- svydesign( ~1 , data = x , weights = ~ wgt )          # create a table with the intended joint distribution, here with just two example states     pop.types <- data.frame( state = c( "state 1" , "state 2" ) , Freq = c( 5000 , 5000 ) )          # create the post-stratified survey design     z <- postStratify( y , ~ state , pop.types )          # have fun running statistics and confidence intervals     svymean( ~ variable.to.analyze , z )     confint( svymean( ~ variable.to.analyze , z ) )      ucla has a post-stratification tutorial in stata that makes me think it might be smarter to create the svyset line like this--               gen total_pop = 10000     gen pststr_wgt = .     replace pststr_wgt = 5000 if state == "state 1":state     replace pststr_wgt = 5000 if state == "state 2":state     svyset _n , fpc( total_pop ) poststrata( state ) postweight( pststr_wgt )      thanks!!


title: Equal-size categories vs unequal-size categories

text: I'm trying to reduce the size of my dataset, which is composed of 200,000 projects. Each project is defined by its size and a binary value that is 1 if the project has active users, 0 otherwise. Most of the project sizes are < 1000 (80% of the dataset). I would like to group the projects, and for each group I would count the number of projects with active users. I'm not sure how to group these projects. In particular, is it better to group the projects into categories with equal intervals (0-500, 501-1000, 1001-1500, ...) or into categories with the same number of projects (as a consequence, I would have unequal intervals like 0-10, 11-15, 16-40, 41-100, 101-120, ...)?


title: Floor effects in Bayesian estimate, can I reparameterize?

text: I'm replicating an old study and I have two sets of existing estimates which measure a similar effect, namely the presence of a studied item in memory over time:               a <-c( .42, .31, .26, .11, .02, .001, .001, .001, .1 )     b <-c( .17, .14, .03, .025, .06, .08, .06, .06, .08)      Both series of estimates are based on very similar data but the estimates seem to deviate very strongly from each other after an initial stage of similarity. The original authors attribute this to floor effects and processing related differences. I would like to estimate those effects in my model since they seem to cause a lot of trouble there. In my model (Bayesian inference), a and b are estimated using a regression like structure, e.g.: $$ a \sim dnorm(\mu_a, \tau_a)\\\ \mu_a \leftarrow \lambda_a t^{-\beta}\\\ \tau_a \sim dgamma(0.1,0.001)\\\ \beta \sim dbeta(0,4)\\\ \lambda_a \sim dunif(0,1)\\\ $$ Where $t$ is the time. As a result of the strange effects in the above estimates, the $\beta$ estimate suffers and the resulting curves do not fit the data as much as I would expect them to. Is it possible to relieve this issue by adding a parameter that encodes the floor effect: $$ \mu_a \leftarrow \lambda_a t^{-\beta} + f\\\ f \sim dbeta(0,1) $$ Where $f$ is the floor-effect parameter. Would it make sense to add this parameter only to those conditions where the floor effect occurs? Or would it be better practice to add it to all conditions?


title: How to test the accuracy of a predictive model?

text: I am trying to find the best way to test the accuracy of a predictive model for the outcome of a game between two random teams. The model presents a chance for each game that team A will win (win chance). I have gathered almost 3500 samples comparing the win chance with the actual outcome. As a crude measure I broke down the win chances into bins of 5% increments (or larger where sample size was small). Within each bin I compared the average win chance with the percentage of games that were won. Here how I organized my data so far:   ![https://lh4.googleusercontent.com/-vOIredJFWq8/Ua- AABHJYPI/AAAAAAAAAMg/DjZHtjaoYbA/w887-h737-no/XVMv-5.jpg](http://i.stack.imgur.com/BTrlV.jpg)   * "bracket" is the same as bin   * "low-high" is the win chance range for each bin, e.g 20% to 24% chance team A will win, etc   * "games" is number of data points or samples in each bin   * "win loss draw" are the possible outcomes of each sample. I'm mainly focused on the wins   * "my win%" are the observed results of the samples in each bin   * "predicted XVM" is the average win chance of all data points within each bin. At first glance I would say it's quite accurate, however I would like to have some concrete method or values to show how accurate it is. I'm guessing breaking it down into bins would be almost like setting confidence (credible?) intervals. How else to approach this?


title: Estimating a distribution from above/below observations

text: Let $P$ be an unknown distribution on $(-\infty,\infty)$. Let $X_1,\ldots,X_n$ be an iid sample from $P$. Let $c_1,\ldots,c_n\in(-\infty,\infty)$ be a known set of constants. We observe $Y_1,\ldots,Y_n$, where $Y_i = 1(X_i < c_i)$. (That is, $Y_i=1$ if $X_i<c_i$ and is $0$ otherwise.) I'm looking for some "reasonable" estimators of the distribution $P$. And what do I mean by "reasonable"? Suppose, for simplicity, that $P$ is supported on [0,1], and the $c_i$ are an iid sample from Uniform[0,1]. $P$ probably has atoms, but we can ignore that if it invites more answers. Call $\hat{P}_n$ a "reasonable" estimator of $P$ if, with probability $1$ over the selection of the $c_i$'s, $\hat{P}_n$ converges to $P$ in distribution. NOTE: We can actually consider the $c_i$ as design points that can be chosen by the experimenter. That seems like a separate issue, and I don't want to complicate the question. But if you have an estimator that requires a particular set of $c_i$'s, that's fine too.


title: ISC exam - cheating or not

text: Background: I read _this_ article on "hackaday" about alleged "large-scale cheating" on the ISC exam. It gives _this_ as source. Here is one of the images from the site: ![enter image description here](http://i.stack.imgur.com/TXLKZ.png) The hack-a-day asks for speculation about the nature of the "cheating" that the data indicates. **My challenge is: determine the actual (non-cheat) distribution from this tampered result and justify your answer with strong and appropriate use of statistics.**


title: Visualise and interpret large set of univariate skewed data

text: I am trying to find out answer for this simple question. I have data (list of numbers). I have 20000 numbers. smallest number is 100 and largest is 15000. I want to interpret this data, how it is distributed or any other suggestions, that how can I play with this data and do different things? The thing is I have to give a presentation on summary stats on this data and also I want to include plots of this data in my presentation so that I can easily show to everyone, what data looks like, to tell them, which is largest number, smallest, which range of numbers are occurring more. So I have made histogram in R but for so many numbers, histogram is not a good choice. Also I know how to calculate all this mean, median and standard deviation. This is not a homework, I want to give presentation about the data, and I am not a statistician.


title: Multi-value categorical attributes in R

text: I have a training data set with both numerical and categorical variables, and one class variable. I want to build a classification model (e.g.,SVM), and for this goal I need to transform all variables into convenient format. I m confused about my categorical variables. Let me give you an example about one of them. The categorical variable in each observation represents a Google search query (usually 3-10 comma-separated words, see example below).               ----------+----------------------------+-------------------+----------------     search_id | **query_words (categorical)**  |..(other variables)| class variable     ----------+----------------------------+-------------------+----------------     1         | how,to,grow,tree           |..                 | 4     2         | smartfone,htc,buy,price    |..                 | 7     3         | buy,house,realty,london    |..                 | 6     4         | where,to,go,weekend,cinema |..                 | 4     ...       | ...                        |..                 | ...     ----------+----------------------------+-------------------+----------------      The words in this categorical variable are disordered and the same words may occur in different observations (that's logical). Number of unique words for all observations = few thousands. Number of observations: ~150.000.000 Since this categorical variable (query_words) is very important for my classification analysis, I need to train my model with it. My question is how to represent it to use for e.g., SVM. In each observation I can sort words alphabetically to order them. If I will use a numeric vector with few thousands elements (one per each unique word) I can represent this variable for each observation as e.g.:               query_words[1] = (0,0,..1,..0,..1,..1,..0,...1,..0) # very big vector      But I don't believe it will work effectively. How should I handle this categorical variable. I m using R for analysis.


title: should one always perform svd before doing KNN?

text: I am trying to perform a Collaborative filtering for recommendation of products to customers in fashion industry.I am using the usual KNN approach to bring similarities among products. I have seen people using SVD(Singular Value Decomposition) before opting for collaborative filtering , but all of those seemed to be dealing with prediction of movie reviews. I want to know if in my case it is suitable to use SVD(svd() in R) prior to collaborative filtering & if so, should I replace zero/missing values by non- zero ones. The second point comes with the idea that normal SVD is not very useful while dealing with sparse data.


title: Detecting Outliers in Time Series (LS/AO/TC) using tsoutliers package in R. How to represent outliers in equation format?

text: **Comments:** Firstly I would like to say a big thank you to the author of the new tsoutliers package which implements Chen and Liu's time series outlier detection which was published in the Journal of the American Statistical Association in 1993 in Open Source software $R$. The package detects 5 different types of outliers iteratively in time series data:   1. Additive Outlier (AO)   2. Innovation Outlier (IO)   3. Level Shift (LS)   4. Temporary change (TC)   5. Seasonal Level Shift (SLS) What is even more great is that this package implements auto.arima from forecast package so detecting outliers is seamless. Also the package produces nice plots for better understanding of the time series data. **Below are my questions:** I tried running few examples using this package and it worked great. Additive outliers and level shift are intuitive. However, I had 2 questions with regards to handing Temporary Change outlier and Innovational outliers which I'm unable to understand. Temporary Change Outlier Example: Consider the following example:               library(tsoutliers)     library(expsmooth)     library(fma)          outlier.chicken <- tsoutliers::tso(chicken,types = c("AO","LS","TC"),maxit.iloop=10)     outlier.chicken     plot(outlier.chicken)      The program rightly detects a level shift and a temporary change at the following location.               Outliers:       type ind time coefhat tstat     1   LS  12 1935   37.14 3.153     2   TC  20 1943   36.38 3.350      Below is the plot and my questions.   * How to write the temporary change in an equation format ? (Level shift can be easily written as a binary variable, anytime before 1935/Obs 12 is 0 and any time after 1935 and after is 1.) The equation for temporary change in the package manual and the article is given as : $$ L(B) = \frac{1} {1-\delta B} $$ where $\delta$ is 0.7. I'm just strugling to translate this to the example above.   * My second question is about innovational outlier, I have never come   across an innovational outlier in practice. any numercial example or a case example would be very helpful. ![outliers](http://i.stack.imgur.com/xxJ3H.png) **Edit:** @Irishstat, the tsoutliers function does an excellent job in identifying outliers and suggesting an appropriate ARIMA model. Looking at the Nile dataset, see below application of auto.arima and then applying tsoutliers (with defaults which includes auto.arima):               auto.arima(Nile)     Series: Nile      ARIMA(1,1,1)                              Coefficients:              ar1      ma1           0.2544  -0.8741     s.e.  0.1194   0.0605          sigma^2 estimated as 19769:  log likelihood=-630.63     AIC=1267.25   AICc=1267.51   BIC=1275.04      After applying tsoutliers function, it identifies an LS outlier and additive outlier and recommends an ARIMA order (0,0,0).               nile.outliers <- tsoutliers(Nile,types = c("AO","LS","TC"))     nile.outliers     Series: Nile      ARIMA(0,0,0) with non-zero mean           Coefficients:           intercept       LS29       AO43           1097.7500  -242.2289  -399.5211     s.e.    22.6783    26.7793   120.8446          sigma^2 estimated as 14401:  log likelihood=-620.65     AIC=1249.29   AICc=1249.71   BIC=1259.71          Outliers:       type ind time coefhat  tstat     1   LS  29 1899  -242.2 -9.045     2   AO  43 1913  -399.5 -3.306      ![enter image description here](http://i.stack.imgur.com/NMq9k.jpg)


title: Using bootstrapping simulation to compare estimators in survey

text: I would like to estimate the mean of a population and select a best estimator with minimum variance of the estimated mean. Suppose that I have two estimators est1 and est2, and they could refer to any type of sampling design, such as simple random sampling, stratified sampling, unequal probability sampling, etc. Here is one way that I see very often to compare the estimators: 1) Compute the sample mean $\hat{\mu}$ from the original sample 2) Conduct a bootstrapping simulation (Nboots=1000) of re-sampling the original samples with replacement. For every simulated sample, compute the sample mean $\hat{\mu}_{i}$ as well as the 95% confidence interval using each of the estimators $\hat{\mu}_{i}+/-1.96\times \sigma_{i}^{k}$, where $\sigma_{i}^{k}$ refers to the estimated variance of mean using the ith sample and estimator k (k=1,2). 3) Compute the lower-bound (LB), higher-bound (HB) and actual confidence interval (CI) for each estimator. The LB for estimator 1 is defined as: $\frac{\text{number of runs that}\ \hat{\mu}>\hat{\mu}_{i}-1.96\times \sigma_{i}^{1}}{Nboots}$ The UB for estimator 1 is defined as: $\frac{\text{number of runs that}\ \hat{\mu}<\hat{\mu}_{i}+1.96\times \sigma_{i}^{1}}{Nboots}$ The actual CI for estimator 1 is defined as: $\frac{\text{number of runs that}\ \hat{\mu}_{i}-1.96\times \sigma_{i}^{1}<\hat{\mu}<\hat{\mu}_{i}+1.96\times \sigma_{i}^{1}}{Nboots}$ 4) The estimator with a larger LB, HB and larger CI would then be selected as the best estimator. My questions are: 1) Can someone provide me some theory or reference about using this method to compare estimators? 2) In case of one of my estimator is a bootstrapping estimator, does it mean that for each bootstrapping simulated sample $i$, I need to conduct again a bootstrapping simulation in order to estimate $\sigma_{i}^{\text{bootstrapping}}$? Thanks


title: Regression on survey analysis

text: I have a survey analysis data which has responses regarding Consumer Satisfaction (on a scale of 1 to 5)and I am trying to fit a linear regression model to it. As per my understanding, the basic assumption for linear regression is that the independent variables must not show significant correlation. In my case however, since the responses are filled by people (homo-sapiens), the responses are showing correlations within a category and across categories (Food, Facility etc). Is this a cause for concern? Can I still go ahead and apply linear regression or should I combine the correlated responses? Also if I were to combine responses, how should I go about it? I have had to make changes to the responses (independent and dependent variables) based on a scorecard: Excellent 100 Very good 90 Good 75 Fair 25 Poor 0 I have two approaches in mind:   1. I can run a linear regression based on the scorecard   2. I keep the responses to my dependent variable as "Excellent, Very good, Good, Fair, Poor", change the responses to the independent variables according to the scorecard and apply an Ordinal Logistic Regression Can you guys help me pick the most proper approach. Thanks!


title: WiFi localization using machine learning

text: I am trying to determine which room a person is in based on WiFi measurements. I currently have WiFi measurements for 3 rooms. The measurement consists of 100 WiFi scans collected from different locations inside each room. Each scan shows the number of WiFi APs visible at at some location inside the room. The number of APs seen in each of the 100 scans is variable. Here is one such scan:               1.SSID: wireless, BSSID: 00:21:6c:63:cc:dd, ss: -54, frequency: 2437      2.SSID: visitor, BSSID: 00:24:6c:61:aa:bb, ss: -58, frequency: 2462      3.SSID: visitor, BSSID: 00:24:6e:6d:ab:ab, ss: -60, frequency: 2437      ss: signal strength This is taken from a single scan out of the 100 scans collected for each room. Each scan is like one feature set for the room. I need to somehow train my system using this data so that it is able to classify a scan (may have multiple APs) taken during testing, to a particular room. What supervised training algorithm can I use? I already tried SVMs but they don't allow variable feature set.