title: Interpretation of categorical variable level interaction in a linear model/ ANOVA

text: This is similar to an earlier question which was never answered... Interpretation of interactions between categorical data... and which is much shorter so no problem if you would like to read/answer that instead: I am doing an experiment where I have varied one parameter (timestep [ **tt** ] - factor, 5 levels: hourly, 3hrly, 6hrly, 12hrly, daily), added in a couple of covariates (release location [ **location** ], release month [ **m** ]) - so far straightforward - and have a response variable (where it starts getting complicated). I am trying to see how timestep affects the track of a particle which is traced over time, but further than that suggest the maximum timestep which can be used with minimal effect on particle fate. "Particle fate" is a complicated concept which varies in direction and distance after release, so I have created a response variable which is already a moving comparison - i.e. distance from the daily timestep position, per day (e.g. below [ **day_3** ]). This allows me to exclude tracking day from the test which gets a bit around the timeseries/autocorrelation issue, but I realise I will need to be careful to remember that I am looking at the same particles at different times and patterns are likely to be lost over time due to the increasing influence of random factors. I figure an ANOVA/ANCOVA derivative would be an appropriate test since I am looking for differences between sample groups, so I have run an lm in R in order to utilise its factor-level specific ANOVA - here is an example output:               > summary(lm(day_3 ~ tt + m * location))     Call:     lm(formula = day_3 ~ tt + m * location)     Residuals:         Min      1Q  Median      3Q     Max      -6.7523 -0.3392 -0.0836  0.2803  2.4548      Coefficients:                     Estimate Std. Error t value Pr(>|t|)         (Intercept)      10.7602     0.7272  14.796  < 2e-16 ***       tt3              -0.1765     0.4719  -0.374 0.710065         tt6              -0.4822     0.4719  -1.022 0.312317         tt12             -1.8280     0.4719  -3.874 0.000345 ***       m4                0.3378     0.9438   0.358 0.722087         m7               -9.1490     0.9438  -9.694 1.36e-12 ***        m10              -5.8576     0.9438  -6.207 1.54e-07 ***       location180      -5.2777     0.9438  -5.592 1.25e-06 ***       location270      -8.6343     0.9438  -9.149 7.82e-12 ***       location360      -7.3264     0.9438  -7.763 7.64e-10 ***       m4:location180   -4.8520     1.3347  -3.635 0.000710 ***       m7:location180    5.5768     1.3347   4.178 0.000133 ***       m10:location180   4.1863     1.3347   3.137 0.003011 **        m4:location270   -0.5704     1.3347  -0.427 0.671166         m7:location270   10.3842     1.3347   7.780 7.21e-10 ***       m10:location270   8.8500     1.3347   6.631 3.60e-08 ***       m4:location360   -1.8426     1.3347  -1.381 0.174234         m7:location360    6.5613     1.3347   4.916 1.22e-05 ***        m10:location360   4.1764     1.3347   3.129 0.003073 **        ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1          Residual standard error: 1.335 on 45 degrees of freedom     Multiple R-squared:  0.8857,    Adjusted R-squared:   0.84      F-statistic: 19.37 on 18 and 45 DF,  p-value: 1.724e-15      In this instance, of all the timesteps, you can see that the intercept (hourly) and the tt12 (12hrly) is significant which I would interpret as hourly is different from daily (which would be an intercept of zero), 3hrly & 6hrly are no different from hourly and 12 hrly is different from hourly. However this does not tell me whether 12hrly is different from daily? I ran the test again with a reordering of levels so that 12hrly is the reference level and got the following output:               > summary(lm(day_3 ~ tt + m * location))     Call:     lm(formula = day_3 ~ tt + m * location)     Residuals:         Min      1Q  Median      3Q     Max      -6.7523 -0.3392 -0.0836  0.2803  2.4548      Coefficients:                     Estimate Std. Error t value Pr(>|t|)         (Intercept)       8.9322     0.7272  12.283 5.69e-16 ***     tt1               1.8280     0.4719   3.874 0.000345 ***     tt3               1.6515     0.4719   3.500 0.001062 **      tt6               1.3458     0.4719   2.852 0.006538 **      m4                0.3378     0.9438   0.358 0.722087         m7               -9.1490     0.9438  -9.694 1.36e-12 ***     m10              -5.8576     0.9438  -6.207 1.54e-07 ***     location180      -5.2777     0.9438  -5.592 1.25e-06 ***     location270      -8.6343     0.9438  -9.149 7.82e-12 ***     location360      -7.3264     0.9438  -7.763 7.64e-10 ***     m4:location180   -4.8520     1.3347  -3.635 0.000710 ***     m7:location180    5.5768     1.3347   4.178 0.000133 ***     m10:location180   4.1863     1.3347   3.137 0.003011 **      m4:location270   -0.5704     1.3347  -0.427 0.671166         m7:location270   10.3842     1.3347   7.780 7.21e-10 ***     m10:location270   8.8500     1.3347   6.631 3.60e-08 ***     m4:location360   -1.8426     1.3347  -1.381 0.174234         m7:location360    6.5613     1.3347   4.916 1.22e-05 ***     m10:location360   4.1764     1.3347   3.129 0.003073 **      ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1     Residual standard error: 1.335 on 45 degrees of freedom     Multiple R-squared:  0.8857,    Adjusted R-squared:   0.84      F-statistic: 19.37 on 18 and 45 DF,  p-value: 1.724e-15      I figure this tells me that 12hrly is different from daily also, but then suggests that all other increments of timestep are different from 12hrly. **So my question is...** is it appropriate to rehash these tests with each categorical level (timestep) as the reference level to see where they overlap and where they differ, or is there some "better"/easier way to cross-compare the timestep levels. I realise there are other clues in each output and there may be a way to interpret the "estimate" (slope)/ "t value" columns to see whether there is a significant difference between timestep values? If so how do you set a threshold value? What would you recommend in my shoes? N.B. as an aside, I realise I am using m * location as an interaction - this is intentional and was tested prior to running the lm. And I realise that day of tracking needs to be carefully cross-considered also as they are autocorrelated results. P.S. Apologies if this is overspecific! P.P.S THANKS FOR READING ALL THIS!


title: Split plot ANOVA or t-test

text: I did a study in which I wanted to see if a teaching intervention would have an effect on student learning. I nonrandomly divided students into two groups, a control and experimental group, and gave them all a pretest at the beginning of class and a postest at the end. Since there are both between and within groups factors I didn't do a t-test but a mixed (split-plot-ANOVA). Is this correct? Thank you very much.


title: How to apply Gaussian kernel to smooth density of points on 2D (theoretically)

text: I have a set of discrete points on a 2D surface and need to build a heat map or a distribution of the density of the points. However, I also need to smooth out the density/distribution by applying some sort of kernel (Gaussian kernel, for example). I Know what Gaussian distribution is, and I have no idea how to mathematically apply the Gaussian kernel to the points on 2D so I can get a smooth distribution or density of points. I know there are functions in Wolfram to do that, but this is more like a research problem, so I need to understand the theory of it. I am a theory person, and I am not in need of programming. I just need to understand the underlying math of it. Please explain it to me, possibly with an example, or take me to a website that I can read about it. * * * I got this answer, but I'm still struggling to understand it: > Since the points are discrete, you simply translate the kernel to each point > and scale it by the height (value) at that point, and then sum the results. > > $$f(x,y)=\sum_i \sum_j p_{i,j} f(x−x_i,y−y_j)$$ where $f(x,y)$ is my kernel > and $p_{i,j}$ is the value of the point at $(x_i,y_j)$. The result will be smooth because it is a finite sum of smooth functions. However, If we multiply the height at each point with $f(x−x_i,y−y_j)$, does it affect the density at all? Say, if $f(x−x_i,y−y_j) > f(x−x_p,y−y_q)$ for two particular points $(x_i,x_j)$ and $(x_p,x_q)$, then this would change the relative original densities at $(x_i,x_j)$ and $(x_p,x_q)$. I think I am wrong, but cannot understand why I am wrong. Maybe it's something in the function $f(x−x_i,y−y_j)$ itself that I misunderstand. Any explanation or links to documentation/tutorial would be much appreciated!


title: Handle large set of features using SVM

text: I have a biological dataset with 30.000 features (genes) and 1000 data points (cells). Basically I have two major classes of cells: 1 and 0 with a distribution of 90/10. Now I am trying to classify these correctly using nested cross validation. The first thing I tried was to manually decrease the number of features by considering biological relevant subsets of the total feature set (reduced to 20 features), which gives me reasonable results (0.7 F2 score). However, I am wondering if I use the whole feature set if I will get big overfitting since I have much less data points than features. Is it true that I would overfit my data if I use the whole feature set? And if so, are there any ways to decrease the feature set without prior biological knowledge? Thanks a lot! Tomi


title: How to do the prediction with text mining and rating score?

text: I have the data as following:               rating  document1    document2     5       some words   some words     3       some words   some words     2       some words   some words      First column is user rating, and other two are the user feedback, which is a cocument. Also, i have `category` is a list of -1 and 1, which stands for negative and positive for each row. I would like to do the prediction for the `category`. I think about document- term matrix, then ,I do not know how to also involve the rating in the prediction? Is there anyone can guide me ? Thanks so much!


title: Absolute variable as dependent variable

text: I have the following model: $$ |X| = B_0 +B_1 \cdot y + B_2 \cdot z , $$ where $z$ and $y$ are normally distributed random variables, and $B_1$ and $B_2$ denote the coefficients. My dependent variable contains either the positive or negative values of $X$. The positive and negative values of $X$ together are normally distributed, with a mean and median of around zero. For statistical reasons I would like to separately test the positive and negative values. This splits the left and right side of the otherwise normally distributed variable $X$. This clearly violates the normality assumption used for linear regression models. If I want to compare groups I would use a nonparametric type of test, but in this case I want to run the equivalent of a linear regression. Is this possible?


title: Model Tuning and Model Evaluation in Machine Learning

text: Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task. I want to perform a manual **features/parameters selection (Model Tuning)** and, after that, perform a **model evaluation** with a cross-validation (k=10) on the entire dataset. What is the best strategy? Intuitively, I would do those steps :   1. Divide dataset in **validation_train** (60%), **validation_test** (20%) and **test** (20%).   2. Process the Model Tuning on the **validation_train** and **validation_test**.   3. Repeat (2) until a good accuracy is reached.   4. Finally, process a CV-10fold on the entire dataset (validation_train, validation_test and test). Or, this strategy conducts to a too optimistic score? ## Ref :   * Cawley, G. C. & Talbot, N. L. On over-fitting in model selection and subsequent selection bias in performance evaluation The Journal of Machine Learning Research, JMLR. org, 2010, 11, 2079-2107.   * N. Japkowicz and M. Shah, Evaluating learning algorithms: a classification perspective, Cambridge University Press, 2011


title: Which statistical test is appropriate for determining whether there is a significant difference in the exam scores of 2 groups(abnormal distrib)?

text: Male exam scores 58 54 61 64 60 60 45 44 53 54 32 56 43 69 69 72 66 45 63 49 48 53 Female exam scores 71 55 42 49 57 51 51 60 60 60 51 43 41 24 43 48 57 79 73 53 53 64 54 59 73 47


title: Considering Age-standardized incidence rates: Will changing the reference population change the incidence rates

text: Im trying to calculate the mortality rates for males and females in two different disease groups (type 1 and type 2 diabetes). In my primary analyses i forgot to use respective gender as the background population and instead used the whole populationen. Now the thing is when i divide the background population in females and males for both type 1 and type 2 diabetes and use these reference populations (instead of the entire population) the incidence rates for type 1 changes dramatically while not even one decimal changes in type 2 mortality rates. Is this even plausible? Truly grateful for all the help i can get!


title: Required: Method of moments fitting routine for the two-parameter generalized Pareto

text: I am currently using the `evd` package which fits a two-parameter GPD by maximum likelihood. Since in small samples the MOM is superior to the ML estimation I'd like to give it a go. However, the POT package - which could do the job - is offline due to memory access errors. There are many extreme value packages around. However I am ONLY interested in the two-parameter GPD given by $G(y)= \begin{cases} 1-\left(1+ \frac{\xi y}{\beta} \right)^{-\frac{1}{\xi}} & \xi \neq 0 \\ 1-\exp\left(-\frac{y}{\beta}\right) & \xi=0 \end{cases}$ or alternatively $g(y)= \begin{cases} \frac{1}{\beta} \left( 1+\frac{\xi y}{\beta} \right)^{-1-\frac{1}{\xi}} & \xi \neq 0 \\ \frac{1}{\beta} \exp\left(-\frac{y}{\beta} \right) & \xi=0 \end{cases}$ Is there any package that can fit such a distribution using a method of moments approach?


title: Term to Describe Strongly Clustered Data

text: I have some data which are strongly gathered into more than one cluster. I am looking for a term to effectively describe this phenomenon: e.g., _multi- clustered data_ , which however seems to me that we have later imposed a cluster analysis to identify more than one cluster. So far I prefer _multi- polarised data_ , but I am open to suggestions.


title: How to change threshold for classification in R randomForests?

text: All the Species Distribution Modelling literature suggests that when predicting the presence/absence of a species using a model that outputs probabilities (e.g., RandomForests), choice of the threshold probabilitiy by which to actually classify a species as presence or absence is important and one should not always rely on the default of 0.5. I need some help with this! Here is my code:               library(randomForest)     library(PresenceAbsence)          #build model     RFfit <- randomForest(Y ~ x1 + x2 + x3 + x4 + x5, data=mydata, mytry = 2, ntrees=500)          #eventually I will apply this to (predict for) new data but for first I predict back    to training data to compare observed vs. predicted     RFpred <- predict(RFfit, mydata, type = "prob")          #put the observed vs. predicted in the same dataframe     ObsPred <- data.frame(cbind(mydata), Predicted=RFpred)          #create auc.roc plot     auc.roc.plot(ObsPred, threshold = 10, xlab="1-Specificity (false positives)", ylab="Sensitivity (true positives)", main = "ROC plot" , color = TRUE, find.auc=TRUE, opt.thresholds = TRUE, opt.methods = 9)       From this I determined that the threshold I would like to use for classifying presence from the predicted probabilities is 0.7, not the default of 0.5. I don't totally understand what to do with this information. Do I simply use this threshold when creating a map of my output? I could easily create a mapped output with continuous probabilities then simply reclassify those with values greater than 0.7 as present, and those < 0.7 as absent. Or, do I want to take this information and re-run my randomForests modeling, using the cut-off parameter? What exactly is the cut-off parameter doing? Does it change the resultant vote? (currently says it is "majority"). How do I use this cut-off parameter? I don't understand the documentation! Thanks!


title: Different results from SAS PROC MIXED procedure and R lme function

text: I have 3(treatment)x3(period)x6(sequence) crossover design data. I have used SAS PROC MIXED procedure and I got the results with following code:               data cross;     input yield id treatment period sequence;     cards;     72  1   1   1   1     73  1   2   2   1     77  1   3   3   1     75  2   2   1   2     78  2   3   2   2     70  2   1   3   2     75  3   3   1   3     77  3   1   2   3     73  3   2   3   3     64  4   1   1   4     68  4   3   2   4     71  4   2   3   4     80  5   2   1   5     72  5   1   2   5     80  5   3   3   5     74  6   3   1   6     76  6   2   2   6     70  6   1   3   6     58  7   1   1   4     62  7   3   2   4     67  7   2   3   4     64  8   2   1   5     56  8   1   2   5     60  8   3   3   5     72  9   3   1   6     69  9   2   2   6     66  9   1   3   6     76  10  2   1   2     79  10  3   2   2     65  10  1   3   2     61  11  1   1   1     50  11  2   2   1     60  11  3   3   1     71  12  3   1   3     72  12  1   2   3     75  12  2   3   3     ;     PROC MIXED data=cross;     CLASSES id treatment period sequence;     MODEL yield= sequence period treatment period*treatment / solution;     RANDOM id(sequence);     *lsmeans period/alpha=0.05 cl diff adjust=tukey;     run;      Results:                              Effect              DF      DF    F Value    Pr > F                         sequence             5       6       0.88    0.5479                    period               2      16       0.24    0.7863                    treatment            2      16       7.23    0.0058                    treatment*period     4      16       3.32    0.0368      But when I used `lme` function from `nlme` package in R, I got the following results: R code:               anova(cross <- lme(yield ~ sequence +treatment + period + period*treatment,data=cross, random= ~1|id/sequence,correlation=NULL,method="REML"))      Results:                              numDF denDF   F-value p-value     (Intercept)        1    16 1086.7422  <.0001     sequence           5     6    0.5755  0.7196     treatment          2    16    7.2270  0.0058     period             2    16    0.2441  0.7863     treatment:period   4    16    3.3186  0.0368      The results are same for treatment, period and treatment-period interaction but I got different results for sequence. And I don't know why? Has anyone got any idea?


title: Can we use ordinal or multilevel predictors directly into logistic regression?

text: Can we use ordinal/multilevel predictors directly into binary logistic regression model? I guess not. we usually here convert them to multiple predictors to have values 1/0 for each category. Also if we have a variable with say 20 levels(e.g. sectors like 'energy' 'IT' 'telecom') or factors, should we created 20 variables with values 1/0 for each type or is there better way?


title: Incomplete questionnaires

text: Is it recommented to delete the cases from a database if the responses are highly incomplete? Is there a percentage that can be taken into consideration? (for example, deleting a case if more than 50% of the items have not been answered).


title: McFadden R Square

text: I did multiple regression analysis, and i have one little problem about interpreting the output. What does McFadden R-Square means, if it is 0,196. What it shows me? Is Mcfadden the best solution, or should i watch others like Nagelkerke or Cox and Snell. What`s the difference between the three of them (beside value)? Thanks


title: Kmeans: Whether to standardise? Can you use categorical variables? Is Cluster 3.0 suitable?

text: I am running kmeans for a market research study, and I have a couple of questions:   1. Should I be standardizing my data, and if so, how? For example, one variable I have is product demand, which is measured on a seven point scale. On the other hand, I also have a variable on age, which is a very different scale. Should I be standardizing these, and how?   2. Can I use categorical variables in kmeans? Specifically, I would like to use gender and ethnicity. If it is possible, how would I prepare this data for the cluster analysis? I suppose I would assign numbers to them, but how would I standardize these with my other data?   3. I downloaded the open source software Cluster 3.0. Is this a good one to use?


title: GEE Repeated Measurements: Specifying Measurement ID

text: Say I have 10 subjects, coming from three groups. In each subject I make two measurements (one on the left, one on the right).               ID <- rep(1:10, 2)     myData <- data.frame(ID)     myData$Group <- rep(c('A','B'), 10)         myData$Measurement <- rnorm(20, mean = 10, sd = 1)     myData$MeasurementLocation <- c(rep('Left', 10), rep('Right', 10))     myData <- myData[with(myData, order(ID)),]      Using the `gee()` function from the `gee` package in R, I can compare `Measurement` by `Group` while controlling for intrasubject correlation using `gee()`:               gee(Measurement ~ Group, id = ID, data = myData)      However, I would like to specify _where the measurement came from_ (i.e., right vs left). I am told that in SPSS you can specify a `WITHINSUBJECT` argument (which I assume would be `myData$MeasurementLocation` in my data). Is there a way to do this using `gee()`? If not, is there another function/package that would allow for this?


title: Looking for definition: Granger Causality with a common explanatory process

text: I am looking for help in defining my problem. Essentially, I have two data processes (both continuous macroeconomic variables: $x$ and $y$). There is evidence of bidirectional causality between $x$ and $y$, proven by a Granger Causality test. What I am alleging now is that $x$ and $y$ are not causing each other, but that they are both caused by a third process, $z$, which is the reason for the causation between $x$ and $y$. So, $x$ and $y$ do not actually cause each other, but they are both simply caused by $z$. I'm thinking about it this way: $x$ and $y$ appear to GC each other, since some events of $x$ precede events of $y$ and vice versa. But if there is a process $z$, whose events always precede those of $x$ and $y$, than $x$ and $y$ do not necessarily cause each other. For now, I am trying to frame this problem properly, i.e. I am looking to define it. I could not find any literature on this particular type of problem, but I am sure there must be some. I assume that I'm simply not searching for the correct terms. Hence, my (hopefully) simple question: How is this phenomenon called? (I would of course appreciate any suggestions on literature as well)


title: Additive vs Multiplicative decomposition

text: My question is a really simple one but those are the ones that really get me :) I don't really know how to evaluate if a specific time series is to be decomposed using an additive or a multiplicative decomposition method. I know there are visual cues as to telling them apart from one another but i don't get them. Take for instance this time series: ![enter image description here](http://i.stack.imgur.com/02pCs.png) How would you describe it? Thanks in advance for your help.


title: one-step-ahead Stochastic Volatility for 5-minute VWAP prices

text: I'm trying to run an SV model against prices of Euro/USD. For those not familiar with SV, its a volatility model in which each point gets its own volatility parameter $h_t$ with 3 main parameters that are derived using a monte carlo simulation (MCMC),   * $\mu$ = average of the volatility across the entire sample set   * $\phi$ = the weighting that the volatility of the last point has on thenext (predicted) volatility   * $h_t$ = the most recent - {time interval}'s volatility.   * $\sigma$ = volatility of the volatility Once we have $\mu$, $\phi$, and $h_t$ we can predict $h_{t+1}$, that is, the predicted volatility of the next point, by $h_{t+1} \sim \text{Normal}(\mu+\phi(h_t-\mu), \sigma)$. I'm wondering if I'm doing the next part correctly: To put this into practice, we run a MCMC every 5 minutes, gathering those variables and predicting $h_{t+1}$. This results in a graph like this, for CME futures contract 6EU4 (September euro) with 5-period BBands also displayed. Ignore the shapes that appear on the graph. ![enter image description here](http://i.stack.imgur.com/QrNaj.jpg) It feels inaccurate, so I'm not sure if I'm doing something wrong. Did I understand the process correctly? `phi` is around .55 for the first half of the day, then jumps to .95 and stays there, which seems wrong, but I guess it isn't too surprising given the data... R's package stochvol is taking care of the parameter estimation, so assume that the numbers themselves are accurate. Should I paste the 5-minute Volume Weighted Average Prices here? its a pretty long data set. I'll edit it to do that if so.


title: Calculating Bias from bivariate data

text: ![enter image description here](http://i.stack.imgur.com/z71AL.jpg) My data consists of one column of real temperature, and one column of calculated temperatures. I want a single number which quantifies the 'bias' in the real temperatures. Basically, if most of my data points lie below the ref line, then I should expect a negative bias, and if they lie above the line then I will expect a positive bias. And this bias quantified by the spread in the scatter plot. What is the best statistic to quantify this? Any help would be greatly appreciated. Edits: The objective of this project is to obtain an estimate of bias in summer temperatures due to unequal land distribution in the Northern and Southern Hemispheres. The Northern Hemisphere has about 68% of the total landmass on earth, while the Southern hemisphere has less than half of the northern hemisphere land (~32%). The data for temperatures here is obtained from GCM simulations. Temp_Real is obtained from a simulation with real earth, and Temp_symmetric is obtained from a GCM simulation using a symmetric landmass earth model. By comparing the climates from Real Earth and a Symmetric Earth, I want to calculate a bias caused by the asymmetry in the real earth. This scatter plot is for Northern Hemisphere.


title: sample generative model from a chain/tree

text: I have a tree of states and I would like to sample from this tree based on pure birth process; however, I don't know how exactly I can do this; so far I have done this; I simplified my problem; the probability of the first state is $A$ and each state obtain the parent probability minus $e^{-\lambda}$; here is a picture I drew![enter image description here](http://i.stack.imgur.com/BGM3P.jpg) as I said, each state inherit the same probability of the parent minus $e^{-\lambda}$. therefore, for example $P(n_5) = A - 2e^{\lambda}$. since we know $\sum_{i=1}^{5}P(n_i) = 1$; we can solve the equation with respect to A and and calculate the probability of every single point; However, this is simple; I would like to get advantage of the pure-birth process; I appreciate any comment and suggestions.


title: Lévy process with a negative starting time index

text: Normally, a Lévy process, $X_t$, is defined for $t\geq 0$, so $t_0=0$. I am applying a transformation on the time index so that it starts from a negative value instead of zero. Now, is it possible to define a Lévy process with $t_0<0$? If yes, then can we treat this new stochastic process exactly like a Lévy process as defined here? Is there any reference that explains such a Lévy process? Thank you very much.


title: Trace test or maximum eigen value in Johansen's cointegration test?

text: Which test to rely on for a time series data of 30 years for instance(when they offer different cointegrating equations in Johansen's cointegration test)? Trace test or maximum eigen value? Say trace test says there are 8 cointegrating equations. Is this the number we input in the "number of cointegrating" box when estimating the VECM? Does it make a huge difference if we put 1? Thanks!


title: A priori justification for using a quartic regression

text: I am reviewing some papers relating school performance to socio-economic background:   * http://archive.treasury.gov.au/documents/1421/HTML/docshell.asp?URL=05%20How%20much%20of%20the%20variation%20in%20Literacy%20and%20Numeracy%20can%20be%20explained%20by%20School%20Performance.htm   * http://www.bristol.ac.uk/cmpo/publications/papers/2012/wp287.pdf   * http://datatopics.worldbank.org/hnp/files/edstats/KHMimp09b.pdf These papers use quartic regression to increase the explanatory power of their regression analysis, but do not appear to say why a quartic is a "natural" choice. Why would these authors have used a quartic relationship in particular? Of course this could be an orthodox choice for this particular area of study, but I can't immediately see that this is the case; I'm interested when quartic relationships should be investigated in general.


title: Can I still interpret a Q-Q plot that uses discrete/rounded data?

text: I have a data set with only discrete/rounded values in it. As a result, when I produce a Q-Q plot a "stair-case" pattern appears. Can I still interpret this just like a normal Q-Q plot even though it is a lot harder to do so? Are there any limitations to interpreting a plot like this?


title: Regression using dummy variable

text: I would like to regress total energy expenditure on weight and gender. Is it better to consider gender as a dummy variable or find separate regression equations for men and women?


title: Can I use weights generated by robust regression in a quasipoisson glm in R?

text: I have response variable count data that should be treated as quasipoisson or something similar. This data also contains outliers which are important to the dataset. I cannot find an r package that will let me do robust regression with quasipoisson-type distribution. I could, however, collect weights for each model using robust regression and then try to use those weights in glm. Note those weights are always > 0. My problem is that I cannot find a clear definition of how weights are used in glm with family=quasipoisson. It sounds similar but I can't figure out if actually does what I want it to do (e.g. downweight outlier response variables to decrease their impact). The response data are numbers of moths caught daily at pheromone traps, and the predictor variables are weather data. All variables are diffed to handle temporal autocorrelation. The outliers are high numbers of migratory moths in response to cold front passages. I just found this reference which explains that low weights are interpreted as representing observations with a high variance, which does make sense in my case. But does that mean the effect is to reduce the impact of that observation on the regression? http://r.789695.n4.nabble.com/weights-in-glm- PR-8720-td910336.html


title: Prediction variance in R

text: Is there a way to get the variance of prediction for a linear regression model in R? The variance that I need is $s_f^2=s^2\left(1+\frac{1}{n}+\frac{(x_{n+1}-\bar{x})^2}{\sum_{i=1}^n (x_i-\bar{x})^2}\right)$.


title: Confidence interval estimation from sample

text: I have a sample of data that follows a certain distribution with known parameters. How can I estimate the confidence interval based on this sample, without using the quantile function?


title: Insignificant interaction makes variable of interest significant

text: I am having some strange results in a regression I am trying to run and I hope someone can help me to interpret them. Basically, first I regressed my dependent variable on a set of regressors which were all significant, then I added a control and some of them became not significant anymore. So far so good, I explained it with an omitted variable bias. The problem is that when I add an interaction between two of the regressors the coefficients become significant again, while the interaction coefficient is not significant at all. How could this be explained? thank a lot for the help!!


title: Sampling distribution of regression coefficients for normally distributed random variables

text: Based on $N$ realizations of two random variables $X \sim N(0,\sigma_X^2)$ and $Y \sim N(0, \sigma_Y^2)$ with correlation $\rho$, I conduct a simple linear regression $Y = \beta_0 + X\beta_1 + \epsilon$. Using the known distributions of $X$ and $Y$, their correlation $\rho$ and the number of realizations $N$, I am trying to find the sampling distribution of the regression coefficients. Estimates of the coefficients can be caluclated as:   $\hat \beta_1=r\frac{s_Y}{s_X}$ where $r$ is an estimate of $\rho$ and $s_X, s_Y$ are estimates of $\sigma_X, \sigma_Y$   $\hat \beta_0=\overline{Y} - \hat \beta_1 \overline{X}$ where $\overline{X}, \overline{Y}$ are estimates of the means of $X$ and $Y$ Intuitively, I thought the regression coefficients should follow a normal distribution. However, after some additional thoughts, I am not sure of anymore. Plotting the distribution of $\beta_1$ from 10,000 experimental runs with $N=12, \sigma_X^2=1, \sigma_Y^2=2, \rho =0.6$ (black) vs. fitted normal distribution, a deviation from normal distribution can be seen. This is also the case more iterations (I tested up to 100,000) for which the normal distribution should be quite stable. For more detail on the experiment, see below. ![Distribution of $\\beta_1$ from 10,000 experimental runs \(black\) vs. fitted normal distribution](http://i.stack.imgur.com/CDFjN.png) Finding the specific distribution of $\beta_0, \beta_1$ - if there is one - is however only the second step. In the first step, I am trying to find the expected value and variance of the regression coefficients. It is clear that $E[\hat \beta_0]= 0$ and $E[\hat \beta_1] = \rho \frac{\sigma_Y}{\sigma_X}$. However, I have trouble deriving the variance of both parameters. I have found some definitions for the coefficient variance in the literature. $Var[\hat \beta_0] = \frac{s_\epsilon^2\sum x_i^2}{N \sum (x_i - \overline{x})^2}$ and $Var[\hat \beta_1] = \frac{s_\epsilon^2}{\sum (x_i - \overline{x})^2} = \frac{s_\epsilon^2}{N \sigma_X^2}$. However, as stated correctly by whuber, these definitions are conditioned on $X$. A calculation is therefore only possible for specific relizations and the definitions are therefore not applicable to my use case. In the literature, I found that the restriction on fixed $X$ is going back to Fisher (e.g. his work 'Asymptotic distribution of the reduced-rank regression estimator under general conditions' from 1922). However, I did not find a consideration of the case with random $X$ in more recent literature. I have only found refrences to the (non central) Wishart distribution. I'm not sure whether and how the Wishart distribution can be used in my use case, though. Overall, I am completly stuck on how to derive the variances in the described case, $Var[\hat \beta_0], Var[\hat \beta_0]$ with $X, Y$ being random variables. For cases with conditioning on $X$, I have found various answers on how to derive the variance of the regression coefficients, for instance http://stats.stackexchange.com/a/89155/48067. However, since not only $\epsilon_i$ is a random variable, the approach described in the answer is not easily transferable to my problem. Consdering that $\hat \beta = r \frac{cov(X, Y)}{s_X^2}= r \frac{s_Y}{s_X}$, I assume that the distributions of $r, s_Y, s_X$ may for some reason be problematic for the calculation of the variance. I have found that $s_X^2, s_Y^2$ follow a gamma distribution. $s_X, s_Y$ should therefore follow a generalized gamma distribution, which have a well defined variance. However, I am not sure about the quotient of two generalized gamma distributed random variables. $r$ has a skewed distribution on which the Fisher transformation can be used - I am however not sure whether this helps to calculate the variance. Lastly, the sample covariance has a strange distribution, approximately like a shifted gamma distribution. Overall, I have already derived some information about the distributions of characteristics of the samples. Yet, I have not found a way to use these information to derive the variance of the sample coefficients in my use case. Can someone point me in the right direction? --   Some additional detail about my "simulations", as requested:   * Samples of size N (e.g. N=12) are drawn from $X \sim N(0, \sigma_X^2)$ and $Y \sim N(0,\sigma_Y^2)$ (e.g. $\sigma_X^2 = 1$, $\sigma_Y^2 = 2$) so that the samples have a specific correlation $\rho$ (e.g. $\rho = 0.6$). This is achieved by using two uncorrelated random variables $A, B \sim N(0,1)$, and constructing a new (correlated) one as $C = \rho^2 A + \sqrt{1- \rho^2} B$ and correcting the variance of $A$ and $C$ in order to have $X$ and $Y$    * A regression of $Y$ on $X$ is executed leading to coefficients $b_0$ and $b_1$    * Steps one and two are repeated (for instance 10,000 times)    * The distribution resulting of $b_0$ and $b_1$ resulting from the runs are analyzed  The distributions resulting from the last step are the distributions for which I want to derive exact distribution parameters in order to analyze the uncertainty in the regression parameters resulting from the sampling of the random variables.


title: Solving a probability distribution problem in profit contract

text: A contractor has submitted bid on four state jobs: a car parking garage, an office building, a theater, and a shopping mall. State rules do not allow a contractor to be offered more than one of these jobs. If this contractor is awarded any of these jobs, the profits earned from these contracts are: RM2 million from the parking garage, RM3 million from the office building, RM5 million from the theater, and RM7 million from the shopping mall. He gets no profit if he does not get any contract. The contractor estimates that the probabilities of getting the car parking garage contract, the office building contract, the theater contract, the shopping mall contract, or nothing are 0.25, 0.15, 0.35, 0.13, and 0.12 respectively. Let Y be the random variable that represents the contractor’s profits in millions of ringgits.   1. Write the probability distribution of Y. (4 marks)    2. Construct the cumulative probability distribution of Y (4 marks)    3. Find the expected profit of the contractor. Give a brief interpretation of the value obtained (5 marks)    4. Estimate the probability of the profit more than what is expected by the contractor. (2 marks)


title: Several questions about statistical financial timeseries models from "machine-learning person"

text: In order to explain why I have those stupid question you'll find below I have to say that I am more a machine-learning person. While I worked on problems in bioinformatics everything was fine. When I heard words like "regression" or "kurtosis and skewness", in first case I just smiled, in second case I just made some clumsy move with my shoulders trying to say something like: "yes, I heard of it, and even know how to calculate it, but why on Earth anyone will need it?". Situation dramatically changed when year ago, just for fun, I tried to apply my machine-learning knowledge to some financial time series. I started with idea of making a Bayesian Network from signals provided by "technical" "analysis" "indicators". Idea failed. Also it was somewhat pleasing to find at least two topics with similar idea on this site (which used neural networks instead of bayesian). Next, after a lot of effort, I was able to build a mixture of kNN and symbolic regression which I trained on 1-hour data from 2000 to 2006 and tested on data from 2007. This model actualy gave a grate profit. But then when I applied it to latest data I realized that it's accuracy dramatically droped because of economic crysis and it doesn't work anymore because something changed in the market and I need more new data, which I can obtain only in a 2-5 years. Well, lot of stuff was tried later and if this whole thing started as "just for fun", it was not fun anymore. Untill I found online lectures of Ruey S. Tsay on ARIMA, GARCH, TAR and all other completely new stuff to me. Basically I found a whole new world and I really enjoy it. Right now I was able to fit my first ARIMA model, and then tuned it to reduce rms-error twice by looking at ACF, PACF, playing with seasonality, and so on. Well, the fun is back, I had a lot of it and I expect having even more. But I had some questions and found this great site. Read almost all topics about ARIMA and other related techniques here, along with many other general topics related to similar approaches. For sure will be reading more. I am still thinking in a mind frame of machine learning approach, which leads to a lot of stupid questions, for most of which I find answers on this site. So, after this long introduction, here are my remaining stupid questions:   1. While machine-learning approach is more concerned about finding "pattern" in data I find it in contradiction with statistical models for financial timeseries which extensively use random walk theory (which makes existance of patterns at least questionable). I realize that its very naive and incorrect description, but what I am trying to say, is that most machine learning techniques are in conceptual contradiction with statistical approach to the problem. I am not saying that some approach is better, I am just saying that they have contradictions. Is that correct and how big is this contradiction?   2. I really liked description and idea of TAR model which for me looks like marriage of machine-learning with statistics. This is a model I want to try next after I add GARCH to my ARIMA. But I have some questions about it:     * TAR definitely uses both statistical and machine learning approaches. So, keeping in mind my first question, aren't there an error when triyng to find pattern for set of models, which are basically built on theory which excludes patterns? Or it is just an idea of how to combine two models which study different aspects of same problem, into one even more powerfull model?     * When you make search by keyword "ARIMA" on this site, you'll hit 15 pages of topics, while for TAR there is only one. Also why do people stopped just on applying AR? Why not expand this idea for more complex models (like ARIMA) ? Is it because TAR didn't give expected improvement over AR ?   3. I know that MCMC methods and other machine-learning stuff is currently being mixed with statistical models. I am personally also a big fan of Hidden Markov Models and Conditional Random Fields. Do you know about any mixtures of any of these methods with statistical models ?


title: What is the non-asymptotic relative power of non-parametric tests?

text: The relative asymptotic power of non-parametric tests is reported to be high. What is the power of tests such as Wilcoxon Signed Rank Test and Mann-Whitney U with N's of 10, 20, 50, for example, as compared with t-tests?


title: Combining classifiers by flipping a coin

text: I am studying a machine learning course and the lecture slides contain information what I find contradicting with the recommended book. The problem is the following: there are three classifiers:   * **classifier A** providing better performance in the lower range of the thresholds,   * **classifier B** providing better performance in the higher range of the thresholds,    * **classifier C** what we get by flipping a p-coin and selecting from the two classifiers. **What will be the performance of classifier C, as viewed on a ROC curve?** The lecture slides state that just by flipping this coin, we are going to get the magical " **convex hull** " of classifier A's and B's ROC curve. I don't understand this point. **Just by simply flipping a coin, how can we gain information?** **The lecture slide** ![lecture slides](http://i.stack.imgur.com/Scvo3.png) **What the book says** The recommended book ( _Data Mining... by Ian H. Witten, Eibe Frank and Mark A. Hall_ ) on the other hand states that: > To see this, choose a particular probability cutoff for method A that gives > true and false positive rates of tA and fA, respectively, and another cutoff > for method B that gives tB and fB. If you use these two schemes at random > with probabilities p and q, where p + q = 1, then you will get true and > false positive rates of p . tA + q . tB and p . fA + q . fB. This represents > a point lying on the straight line joining the points (tA, fA) and (tB, fB), > and by varying p and q you can trace out the whole line between these two > points. In my understanding, what the book says is that to actually gain information and reach the convex hull we need to do something more advanced than simply flipping a p-coin. AFAIK, the correct way (as suggested by the book) is the following:   1. we should find an optimal threshold Oa for classifier A   2. we should find an optimal threshold Ob for classifier B   3. define C as following:     * If t < Oa, use classifier A with t      * If t > Ob, use classifier B with t      * If Oa < t < Ob, pick between classifier A with Oa and B with Ob by the probability as a linear combination of where we are between Oa and Ob. **Is this correct?** If yes, there are a few key differences compared to what the slides suggest.   1. It's not a simple coin flipping, but a more advanced algorithm which needs manually defined points and picks based on what region we fall into.   2. It never uses classifier A and B with threshold values between Oa and Ob. **Can you explain to me this problem** and **what is the correct way to understand it** , if my understanding was not correct? **What would happen if we would just simply flip a p-coin like the slides would suggest?** I would think that we'd get a ROC curve that is between A and B, but never "better" than the better one at a given point. As far as I can see, I really don't understand how the slides could be correct. The probabilistic calculation on the left hand side doesn't make sense to me. **Update:** Found the article written by the original author who invented the convex hull method: http://www.bmva.org/bmvc/1998/pdf/p082.pdf


title: How do you control FWER when testing multiple nested models with likelihood ratio tests?

text: Suppose I have three different models $M_1, M_2$ and $M_3$, these models are nested such that $M_1 \subseteq M_2 \subseteq M_3$. I perform two likelihood ratio tests, $M_1$ against $M_3$ and $M_2$ against $M_3$. In this case how do you control the familywise error rate (feel free to imagine that there is a long chain of nested models)? Intuitively it seems that these test statistics will be dependent, since $M_1$ and $M_2$ are nested and they both test against the same model $M_3$. Is there a concept or paper that describes this situation?


title: Subscales (not items) as indicators of latent variables in SEM

text: I have a sample (n=200) that I have collected questionnaire data from. Each participant will complete 5 questionnaires that capture different behaviours and all of my 5 questionnaires include between 10 and 27 items. I am planning to use SEM to construct a model that investigates whether scores on 4 of my questionnaires (IVs) predict scores on the 5th questionnaire (DV). The IVs and DV will be treated as latent variables in the model, each with a number of indicators attached. However, because my questionnaires have so many items I want to engage some sort of process of reduction so that I have less indicators in my SEM. I'm wondering how my rationale holds, as I'm new to this. What I am considering at the moment is conducting an exploratory factor analysis and then confirmatory factor analysis on each of the 5 questionnaires separately to extract how many factors are captured by the items (lets say for example, that magically each questionnaire has 3 underlying factors). I would then sum the scores of the items that are associated with each of the 3 factors separately, so each participant now has 3 scores from each questionnaire rather than the 1 total sum score. The 3 scores would then form 3 indicators used in the final SEM predictive model. In my predictive SEM model, I would have 5 'latent variables', each loading on 3 indicators. I guess the central question is: If you have in excess of 10-items associated with a latent variable, is it acceptable to use derived factor scores as indicators in SEM rather than the original items?


title: How to test against a mixed distribution?

text: There is a population $D$ in which each data point has two attributes $X$ and $Y$ that are randomly distributed. While they are probably not exactly normally distributed I imagine they are not too pathological. I don't know their means and $s.d.$'s, but they are likely to be somewhat similar. I also don't know a priori whether $\mu_X$ or $\mu_Y$ is bigger; I also don't know whether $X$ and $Y$ are correlated. There is another set of data $D'$, with the same two attributes but slightly differently distributed: $X'$ and $Y'$ are distributed $X' \sim X + c1$ and $Y' \sim Y + c2$ where $c1$ and $c2$ are assumed to be constant but unknown. I can sample directly from $X$ (without knowing $Y$) or from $Y$ (without knowing $X$); and, I can also sample from $max(X', Y')$ in which I can get identifiers that state whether each value comes from $X'$ or $Y'$, i.e., whether $x' > y'$ or $y' > x'$ for this data point. Although the variables are real numbers, because of the sampling procedure they are rounded. As a result, $x'$ and $y'$ can tie and in which case the identifier would report either $X'$ or $Y'$ randomly. I have two questions. What is the best way to test:   1. whether $X' > X$ and $Y' > Y$?   2. whether $max(X', Y')$ > $max(X, Y)$?


title: Exponential Distribution Data

text: The life time of a bulb follows exponential distribution. Such types of examples frequently occurs in many areas of science and engineering. Is there any source of practical data set of two exponential distributed data with a location and scale parameter?


title: Survival analysis with unequal repeated measures, time variable

text: I'm modelling propensity to purchase a particular product or service following exposure to online advertising. I'm attempting to get something like the analysis from John Chandler- Peplnjak's thesis "Modeling conversions in online advertising" A similar method has also been used by Andrie de Vries in this presentation. I have included a sample of the data at the bottom of this question, it is quite cumbersome, sorry about that! The essential point is that both De Vries and Chandler-Peplnjak apply a cox proportional hazards model to data with unequal numbers of repeated measurements. There is no attempt to identify consumer exposure to advertising by user, so it appears that each row of the data becomes a separate survival record. I have created a couple of custom time variables (times are UNIX times in seconds) to measure time since the first advertising exposure (time_from_start), and time since the last advertising exposure (interval_time) and can use these to produce the model, but surely I will be:-   1. modelling far too many paths   2. artificially shortening paths if I use the time between events as the survival time   3. modelling overlapping events if I use the time since the first exposure De Vries provides sample code in his presentation and it doesn't appear that he takes any account of this in his analysis (such as downweighting events according to the length of the conversion path, I don't think Pepelnjak does either). It has been suggested that the interval time variable should produce reasonable results because the time slices are assumed to be independent of each other in the Cox-PH framework? Does this also get around the problem of there being too many paths? EDIT 1: So it looks like this is ok. I've bought myself a copy of "Applied Longitudinal Data Analysis" by Singer & Willet, and these kinds of data sets are called "person- period" data sets. So far I've only found info on the discrete time case, but I would assume it must be possible to extend to the continuous time case, even if only as a limiting case. Do shout if you've any experience with person-period data sets on continuous time variables, let me know what kind of time variable you used for your analysis.                    Event_Time Start_time  Time_from_start Interval_time   Buy_ID  User_ID Ad_Id   Ad_Type Cre_Id  Creative_size_lookup    Page_ID Site_ID Imp_ord Click_ord   Type    Interaction_ord Activity_ord    State_Ord   Path_Ord    Fixed_path_ord  Path_max_ord    prev_clicks prev_impressions    Is_Conv Weight          1384011915 1384011915  0   0   1234    86549917    2739    Standard    54876   300x250 1003    9922    1       Imp 1           1   1   15  0   0   0   0.0001          1384023420 1384011915  11505   11505   1234    86549917    2739    Standard    54876   300x250 1003    9922    2       Imp 2           2   2   15  0   1   0   0.0001          1384123446 1384011915  111531  100026  1234    86549917    2739    Standard    54876   300x250 1003    9922    3       Imp 3           3   3   15  0   2   0   0.0001          1384123493 1384011915  111578  47  1234    86549917    2739    Standard    54876   300x250 1003    9922    4       Imp 4           4   4   15  0   3   0   0.0001          1384251680 1384011915  239765  128187  9876    86549917    2491    Search  0   0x0 7459    9027        1   Click   5           5   5   15  1   3   0   0.0001          1384255778 1384011915  243863  4098    1234    86549917    2739    Standard    54876   300x250 1003    9922    5       Imp 6           6   6   15  1   4   0   0.0001          1384256704 1384011915  244789  926 1234    86549917    2739    Standard    54876   300x250 1003    9922    6       Imp 7           7   7   15  1   5   0   0.0001          1384287309 1384011915  275394  30605   9876    86549917    2747    Standard    55758   160x600 1010    9922    7       Imp 8           8   8   15  1   6   0   0.0001          1384288823 1384011915  276908  1514    1234    86549917    2764    Standard    55855   300x250 1036    9922    8       Imp 9           9   9   15  1   7   0   0.0001          1384342474 1384011915  330559  53651   1234    86549917    2739    Standard    54876   300x250 1003    9922    9       Imp 10          11  10  15  1   8   0   0.0001          1384381908 1384011915  369993  39434   9876    86549917    2764    Standard    56137   300x250 1036    9922    10      Imp 11          12  11  15  1   9   0   0.0001          1384462442 1384011915  450527  80534   1234    86549917    2739    Standard    54876   300x250 1003    9922    11      Imp 12          13  12  15  1   10  0   0.0001          1384543045 1384011915  531130  80603   1234    86549917    2739    Standard    54876   728x90  1003    9922    12      Imp 13          14  13  15  1   11  0   0.0001          1384543067 1384011915  531152  22  1234    86549917    2739    Standard    54876   728x90  1003    9922    13      Imp 14          15  14  15  1   12  0   0.0001           1384545213    1384011915  533298  2146    9876    86549917    2773    Standard    56271   728x90  1028    9922    14      Imp 15          16  15  15  1   13  1   0.0001           1383856924    1383856924  0   0   9876    86949417    2491    Search  0   0x0 7459    9027        1   Click   1           1   1   1   0   0   1   0.0001           1383408344    1383408344  0   0   9876    87159517    2760    Standard    55705   728x90  1042    9585    1       Imp 1           1   1   7   0   0   0   0.0001           1383408509    1383408344  165 165 9876    87159517    2760    Standard    55705   728x90  1042    9585    2       Imp 2           2   2   7   0   1   0   0.0001           1383486599    1383408344  78255   78090   9876    87159517    2756    Default 55567   1x1 1041    8897    3       Imp 3           7   3   7   0   2   0   0.0001


title: ANOVA with repeated mesures - No random groups

text: I have 4 groups, 3 of them receiving a treatment and the last nothing. I will do two mesures, before and after the treatment. But my groups are pre-defined, not resulting of a random selection. Is it nonetheless valid to use an ANOVA with repeated mesures ? Thank's beforehand


title: Heavy tailed distribution results in sparse random variable?

text: I often come across that, in Bayesian inference, to encourage sparseness, we should choose a heavy-tailed probability distribution as prior. AFAIK, in heavy-tailed distribution, it's more likely for a random variable to take on large or extreme values. Now for a multivariate random variable $\bf{x}$, to make it sparse means most entries of $\bf{x}$ should be $0$, then how could a heavy-tailed distribution work? Does it mean, for each entry in $\bf{x}$ (independent of each other) there is a heavy-tailed distribution, and the joint probability of most entries being $0$ is high? If so, why wouldn't a typical Gaussian distribution fit in?


title: How should I handle variables whose data points have varying degrees of predictive power?

text: I'm trying to determine which type of learning algorithm is best for making predictions on my data. My data set consists of several independent variables, each of which is accompanied by an "indicator" variable that represents the sample size used to acquire its specific value. Let's say I'm attempting to assign expected future conversion rates to an assortment of web advertisements (conversion rate = orders/clicks). For some ads, I have a lot of specific traffic history to draw from. But for newer ads that have minimal to no traffic history, I need to rely on higher level characteristics. My data set consists of historical conversion rates at different levels/traits of the ad: ![enter image description here](http://i.stack.imgur.com/uM4Gl.jpg) For ad #5, I would want my model to rely almost completely on the historical conversion rate for the specific ad (4.23%) since it has enough past clicks for me to trust that value. For ad #1, however, I would like the model to ignore the "specific ad" value and rely mostly on the historical rates for the "brand" and "product category" of the ad (since 15 clicks is not enough to establish a reliable conversion rate). I've had some success with separating ads into several different groups by similarity of click volume, then using a different regression formula for each group. This allows each variable to be properly weighted. However, this approach ends up spreading the data too thin and likely does not learn as well as one consolidated model could. What is a good approach for using one model to make predictions on this data? I need the model to recognize that the "clicks" variables should be used to weight their respective conversion rates accordingly. (I've also had some small success with the nnet package in R, but the results are very hit-and- miss). Please don't hesitate to ask for clarification. Thanks!


title: Can a two-sample t-test be used with data that doesn't follow a normal distribution?

text: One of the assumptions for t-tests is that the data must follow a normal distribution. However, due to the Central Limit Theorem (and this thread): _"if the sample is large enough you can use t-test (with unequal variances)"_. I'm trying to sort out what this means for my case. I think my sample **should be** large- enough, but how to confirm it? A Levene's test showed that the two samples don't have an equal variance, hence I plan to use Welch's test (the unequal variance version of the t-test). I've also ran the Shapiro-Wilk test to confirm that one of my two samples doesn't, in fact, follow a normal distribution. * * * ## Additional information I need to run the tests for a few different cases, but to keep things short I'm detailing only two of them. Sample sizes are 19 and 15, respectively for group1 and group2 (this happens on both the examples: Case1 and Case2). **Results of Shapiro-Wilk's test for normality**               Case1     sample | p_value   | w     | Result     group1 | 0.104     | 0.918 | Normal     group2 | 0.027     | 0.863 | Not Normal (p<0.05)          Case2     sample | p_value   | w     | Result     group1 | 2.054e-05 | 0.663 | Not Normal (p<0.05)     group2 | 0.006     | 0.814 | Not Normal (p<0.05)      **Results of Levene's test for equality of variances**               Case1     p_value | w    | Result     0.154   |2.128 | Equal Variance          Case2     p_value | w    | Result     0.0251  |5.521 | Unequal Variance (p<0.05)      **Result of the one-tailed (Welch) t-test (H1: group1>group2)**               Case1     t_statistic | p_value | Result     3.073       | 0.002   | Significant (p<0.05)          Case2     t_statistic | p_value | Result     2.475       | 0.012   | Significant (p<0.05)


title: Estimating error in ARIMA(p,d,q)

text: I am trying to model my data with ARIMA(1,2,12) and since the variance is not stationary I have also included GARCH (2,3). I have saved all the parameter of my model in a variable called mdl. I have used 1000 time steps as initial values and I want to do a 10 time step ahead for forecasting and then I want to compute the residuals. this is my code:               [res,V,LOGL] = infer(mdl,data(1:1000));          [YF, YMSE,VV] = forecast(mdl,10,'Y0',data(1:1000),'E0',res,'V0',V);          ERROR=data(1001:1010)-YF;      My question is: do I have to double difference my data to compute the ERROR or should I don't need to do that? Because D=2 is my ARIMA I don't know what forecasts are giving me, the data after being differentiated two times or without differencing.


title: Computing $\eta_P^2$ from $\eta_G^2$ generalized eta for G*Power

text: I am trying to perform a power calculation using G*Power. I need $\eta_P^2$, but the ez package for R gives me $\eta_G^2$. I understand the reasons for it doing this, but I still need $\eta_P^2$ for a power calculation for a replication. Is there a way that I can calculate $\eta_P^2$ from the ezANOVA output, or convert my $\eta_G^2$ into a $\eta_P^2$?


title: Evaluating clusters of variables produced by varclus & hclustvar

text: Background - I want to cluster analyze a mixed dataset, clustering the variables on the basis of correlational similarity. SPSS gives me this option, but doesn't allow me to evaluate the clustering solutions by providing statistical measures of heterogeneity change (e.g. pseudo F statistic) or direct measures of heterogeneity (e.g. CCC). As such, I'm learning R. I've managed to cluster my variables using both the varclus and hclustvar procedures, both of which generate nice dendrograms that I can interpret visually. However I'm struggling to get R to provide me with some actual numbers, such as the statistics mentioned above, which might indicate what constitutes the "best" clustering solution. How can I do this? I've been through the documentation for both the Hmisc and ClustOfVar packages and can't find any way to do this. I've read elsewhere that clustering criteria can be applied using the NBClust package, but as far as I can see the NBClust function only works on dissimilarity matrices / distance measures, which aren't an option for me as I'm interested in correlational similarity. Any suggestions?


title: Chi-squared test where observations can have multiple categories

text: I am investigating patterns in click-order on images. A user is asked to click on an image in 5 places (can be the same place). This sequence of 5 clicks makes up a graphical password. I am looking for patterns in the order in which users click e.g. they scan from left to right i.e. x_1 < x_2 < x_3 < x_4 < x_5 where x_i is the x co-ordinate of the ith click. The categories that I am considering are left to right, right to left, top to bottom, bottom to top, clockwise and anticlockwise. I want to see if there is a difference in click-order for populations with different attributes e.g. are women more likely to have a click-order of left to right than men? For this I planned a chi-square test of independence probably Fisher's exact test because of a low sample size. My problem is that a user can be in multiple categories e.g. they may click left to right and bottom to top making a diagonal pattern. I am assuming that this isn't allowed in a contingency table so will I have to split the patterns into mutually exclusive sets i.e. test left to right against right to left, top to bottom against bottom to top etc.? Also would I have to include users that do not fit into any category e.g. have left-right, right-left and neither as my categories for testing the horizontal plane? Is there a way to test the independence of the pattern as a whole (may include more than one category) from an attribute (such as gender)?


title: Wilcoxon Signed Rank when subjects appear in each sample n times.

text: I have sales data and would like to see if the median sales for an employee has increased as a result of this training. Thus, I have pre and post data, since it is only six months and am interested in the median difference I plan to use the Wilcoxon signed rank. When performing this analysis I took the difference of the median of the pre training 6 months and the median of the post training 6 months (Let's say there is 30 subjects, since I am taking the median for each, n still equals 30). I thought that this would give a reliable estimate for the medina difference, however, another employee (with more experience) paired each subject (month 1 vs month 1) and did not take the median for each subject before running the Wilcoxon signed rank and received a higher point estimate than mine. Both are still statistically significiant, but I'm curious as to which approach would give a more reliable estimate? Can both work, should you take the more conservative estimate?


title: Calculating PDF given CDF

text: I know that the PDF is the first derivative of the CDF for a continuous random variable, and the difference for a discrete random variable. However, I would like to know why this is, why are there two different cases for discrete and continuous?


title: Significance testing on weight of people from 2 populations given their height

text: Suppose we have $N_A$ and $N_B$ samples from population A and B respectively. For each sample, we have his/her weight and height. What interests us is whether there are a significant difference in weight between A and B. A _t_ -test can do this easily. But we also know there is strong correlation between height and weight. If weight is not influenced by population but the height of samples are different between A and B, then we may reject $H_0$ by mistake. Is there a systematic way to correct samples' weight using their height so we can prevent its interference?


title: Recurrent Neural Network (Elman Network). Context to hidden / hidden to context layers weights need update?

text: I've just started look into recurrent neural network. I found three sources of information on Elman's network(Elman 1991). (Example and code) http://mnemstudio.org/neural-networks-elman.htm (Paper) http://www.sysc.pdx.edu/classes/Werbos- Backpropagation%20through%20time.pdf (Q&A) http://stackoverflow.com/questions/16739299/elman-and-jordan-context- values-during-training-for-neural-network According to the first resource, the weights from hidden to context / context to hidden layers are not updated. From the second resource, it also set these updates to 0, which means it doesn't updates the weights. But from the third resource on Stackoverflow, the user claimed that "The context neurons neuron values themselves are not updated as training progresses. The **weights** between them and the next layer **ARE updated** during training. " I understand the context neuron saves the value of hidden neural at time t, and feed it (together with input neuron) to hidden neuron at t + 1. But do we have to update the weights in between?


title: Using Beta to interpret interaction in general linear model

text: Following the question here, someone suggested that I could just look at the B column in SPSS "Parameter Estimates" table to interpret the interaction. For instance, the B column for one of my tests is as following:   * Intercept: .724   * [Gender = 1.00]: .140   * [Gender = 2.00]: 0   * [Gender = 1.00] * Income: .620    * [Gender = 2.00] * Income: .690 The dependent variable is purchase of a particular product. 1 = Males and 2 = Females On the basis of the above, can I say that the gender of female has a stronger effect between income and purchase.


title: N-gram smoothing implementation in R

text: Is there an implementation of n-gram smoothing like Kneser-Ney in R? I'm using the `tm` package.


title: Variable selection with LASSO

text: I am trying to fit a predictive gene-based model in survival analysis. My question is: **Can I use LASSO as a variable selection method, and then run a multivariate Cox regression to get the coefficients of those variables (genes) instead of using the coefficients from LASSO?** In that case (if I run a multivariate Cox regression), I can get p-values, and make an inference for the coefficients. I am pretty new to this area. Please bear with me if my questions are a bit dumb.


title: Skewness, kurtosis and normality of a time series

text: I have a sample size of $21$ with $496$ observations.Can I presume an approximately normal distribution,and use a $t$-test to compare the difference in means, and difference in various financial features (Financial data daily prices)? In addition, sometimes I get insignificant differences on the daily basis, however, if I annualize the mean and the standard error, it becomes highly significant, like a difference between two means per day is $0.10%$ at the end of the year using this formula it will be extremely huge $((1+R)^{240}))-1$, any converting the standard deviation by the formula $\sqrt{240}* SD_{daily}$. First it was a typo yes, the difference in the daily return is 0.10%, then the data will be auto-correlated for sure, since the market regulations allows the price to fluctuate by 2% only per trading session so the pattern will be there, what I am doing is Mainly constructing indexes in the market, grouping the 21 stocks into two groups, 16 stocks and 5, and then I will be comparing the indexes returns, Sharpe ratio etc, so when I'm using t-test to check statistical difference as this formula (x1-x2) difference in means / SQRt(SE^2+SE^2) the results were not statistically significant but financially 0.10% per day will be at the end of the year 15-17% difference which is MASSIVE!!!!!   and I provided the statistics for the daily data Kurtosis: the whole market (7.392221073) first group (6.344474009) second group (14.0065786) Skewness the whole market ,(0.930473559), first group (1.173843213), second group(1.325395262). So I'm actually taking the whole population which is only 21 firms in the emergent market, so in my research introduction chapter I have to talk about this issue so do you think I have to say by the central limit theorem, I have more than 20 sample size, and I would use t-test for comparing, if not it is not acceptable, tell me any other test for comparing the differences between the means and in details please.               Annual stats                                  market        set1          set2      Average Annual Return   0.182450075   0.344723515   0.149800603     Standard Deviation      0.134419078   0.202235211   0.138592387          per session stats      market   set1     set2      Mean                    0.10%   0.18%   0.08%     Standard Deviation      1.05%   1.57%   1.08%      the conversion done by assuming 165.33 session per annum


title: Real-life examples of common distributions

text: I am a grad student developing an interest for statistics. I like the material over-all, but I sometimes have a hard time thinking about applications to real life. Specifically, my question is about commonly used statistical distributions (normal - beta- gamma etc.). I guess for some cases I get the particular properties that make the distribution quite nice - memoryless property of exponential for example. But for many other cases, I don't have an intuition about both the importance and application areas of the common distributions that we see in textbooks. There are probably a lot of good sources addressing my concerns, I would be glad if you could share those. I would be a lot more motivated into the material if I could associate it with real-life examples.


title: How to simulate two correlated AR(1) time series?

text: Is it possible to simulate two time series AR(1) for example 0.5 and 0.8 and at the same time these time series to be correlated with $\rho = 0.8$ using R?               ts.sim <- arima.sim(list(order = c(1,0,0), ar = 0.9), n = 100)     ts.sim2 <- arima.sim(n = 100, list(ar=c(0.9)),n.start = 25)     et<-rnorm(100)     yt<-0.8*ts.sim+et     yt2<-0.8*ts.sim2+et      I am not sure if `yt` and `ts.sim` or `yt2` and `ts.sim2` are indeed correlated. Note that `ts.sim` and `ts.sim2` are slighty different approaches of the same idea. Just wanted to see if the results would be different.


title: What are some better ways to encode these symbols?

text: I was going through some tutorials on information theory. It had the following example concerning transmission of three symbols $A, B$ and $C$ such that $P(A) = 1/3 = P(B) = P(C)$ If we encode the events as $A = 0, B = 10, C = 11$ ; then we get on an average $1.66$ bits per symbol. But it mentioned that theoretically it's possible to send $1.58$ bits/symbol. How would one go below $1.66$ bits / symbol ? I thought of group encoding i.e. encoding $2$ symbols together. But it didn't give any decrement in bits/symbol. Thanks.


title: Is there a way to compare linear regression slopes by permutation tests?

text: I have some species $\times$ year matrices and analyse the temporal dynamics by regressing distance between years on $\sqrt{ {\rm time \ lag} }$. Since the data points are not independent and since I get $(n^2-n)/2$ distances for a time series of length $n$, it's problematic to use the standard way of determining the significance of the regression slope. I thus applied a Monte Carlo permutation procedure and compared the distribution of the random slopes with the observed slope. No problem so far; most of my regression slopes are significant at $p<0.0001$. My question: How do I compare the slopes for different species $\times$ year matrices? The parametric approach is still not a solution due to the problems outlined above, but I am not aware of a solution based on permutation procedures.


title: Automating tree-based classification by using different subset of attributes

text: First of all, I am very new to this subject. I have been using weka some days to familiarize myself and reading about data mining to understand it. I have a dataset (from UCI, german credit data) and I need to classify it. I have been trying different tree algorithms using different attributes instead of all and I wish to find which one gives me the best tree. (splitting my data 66/34 for training and testing) Is there a way to automate this, that is make weka try all combinations of attributes to find which ones give the best tree? I would prefer to use weka since is the tool I have been using, but this is not compulsory.


title: Where does the square root for a polynomial kernel mapping function come from?

text: I'm trying to understand how polynomial kernel functions work, in my textbook it shows an example with a degree of 2, with an input dimension of 2: $K(\vec{x}, \vec{y})$ = $(1 + x_1y_1 + x_2y_2)^2$ = $1 + 2x_1y_1 + 2x_2y_2 + x^2_1y^2_1 + 2x_1x_2y_1y_2 + x^2_2y^2_2$ I understand all of the above. It then goes on to mention, 'the image of the input vector $x$ induced in the feature space is therefore...' = $(1,\sqrt{2}x_1,\sqrt{2}x_2,x^2_1,\sqrt{2}x_1x_2, x^2_2)$ I don't understand this line, in particular, where did the square root come from?


title: Comparing two exchange rate models with different dependent variables

text: I want to compare two exchange rate models namely Balassa-Samuelson model and monetary model of exchange rate. Both models are totally different from each other and their dependent variables are also different. I don't know how to compare these two models and which econometric technique I should use for this comparison.


title: Fisher's criterion distribution

text: My task is to show that Fisher's criterion distribution when the null hypothesis is true follows $F(k-m,N-k)$ when df increases. Here $k-m$ is number of restriction. So I can write a null hypothesis H0: $b_1=b_2=\ldots b_k=0$.   And I think that I can show that F-statistic follows $F(k-m,N-k)$ using restricted and unrestricted models. where $F=R^2u-R^2r/(1-R^2u) (k-m/N-k)$ or I can find this value from the table of Wald test. When I compare it critical value =@qfdist(.95,eq.@ncoef-eq2.@ncoef,eq.@regobs- eq.@ncoef). Is my thinking right? How use should use the condition of task that degrees of freedom increases?


title: Which regression to use for ordinal variables?

text: I have 10 independent ordinal variables, each having 5 levels, all intended to measure the same latent construct, and one ordinal dependent variable named `rank` with 5 levels. I have read somewhere that for using regression, it is necessary to convert categorical variables with multiple levels to dummy variables, which will be very cumbersome. Is there any way I can use them as is in regression?


title: Calculate expectation from empirical cdf

text: > Mirror thread on Math.SE I have a empirical cumulative probability distribution function for a random variable. The random variable is "time to failure" and I have the full curve i.e till the probability reaches 1. I want to know Mean Time To Failure i.e expectation of that random variable. Is there any standard method to find mean from an empirical distribution. I am getting the empirical CDF (as discrete values) as output from a "model checking tool" which uses iterative numerical computation techniques to get those probabilities. For example, let F(x)=P(X<=t) is the CDF of the random variable X where X stands for time between failure. To plot the curve of F(X) vs t I am varying t with some step size, calculating F(X) for that t using the "model checking tool" and adding the points to get the curve. I can use small step size to get the more accurate curve. So, I have access to only this CDF values at different t. From this values I want to do a good estimate of mean value of X.


title: How to create a dataset with conditional probability?

text: Suppose that a certain disease (D) has a prevalence of 3/1000. Also suppose that a certain symptom (S) has a prevalence (in the general population = people with that disease D and people without that disease [probably with other disease, but it's not important]) of 5/1000. In a previous research, it was discovered that the conditional probability P(S|D) = 30% (the probability to have the symptom S, given the disease D is 30%). **First question** : Could be P(S|D) interpreted as equivalent to the prevalence of the symptom S in the group of people having the disease D? **Second question** : I want to create in R a dataset, which shows that: $$P(D|S) = \frac{P(S|D)P(D)} {P(S)}$$ With my fictional data, we can compute P(D|S)=0.18, that is interpreted in this way: given a patient with the symptom S, the probability that he has the disease D is 18%. How to do this? If I use simply the `sample` function, my dataset is lacking of the information that P(S|D)=30%:               symptom <- sample(c("yes","no"), 1000, prob=c(0.005, 0.995), rep=T)     disease <- sample(c("yes","no"), 1000, prob=c(0.002, 0.998), rep=T)      So my question is: how to create a good dataset, including the conditional probability I desire? _EDIT_ : I posted the same question also on stackoverflow.com (http://stackoverflow.com/questions/7291935/how-to-create-a-dataset-with- conditional-probability), because, in my opinion, my question is inherited to the R language program, but also to statistical theory.


title: Variable transformation for factor analysis

text: When conducting a factor analysis, we need to check the normality and constant variance assumption of the original variables.   If there is HSK existing in the data, can I do different transformations(for example log transformation to x1, squared x2...) to different variables and then conduct the analysis?   Or I should do the same transformation to all variables? Thanks a lot.


title: Problem with k-means used to initialize HMM

text: I have a sequence of two possible observations ($A$, $B$) and want to train an HMM with $h$ states, namely $\lambda_h$, to predict the probability of the next observation using the Baum-Welch algorithm. Output probability distributions are Bernoulli with parameter $p_i$ ($i$ is the state number). My problem is mainly with the initialization. If $\lambda_h$ is initialized to a totally random HMM, results of learning are poor. On the other hand, the K-Means initialization algorithm assigns zero initial probability ($\pi_i$) to some states if $h > 2$, preventing BW from improvements. I use a 4-state HMM ($\lambda^0_4$) to generate my test sequences. Hence trying $h=4$ should be totally possible! Any ideas? * * * Parameters of $\lambda^0_4$: Transition probabilities: $A_{i,i}$ = 0.4, $A_{i,j\neq i}$=0.2 Observation probability distributions: $p_0$=1.0, $p_1$=0.7, $p_2$=0.3, $p_3$=0.0 Initial state probabilities: $\pi_i$ = 0.25


title: How to interpret coefficients produced by the sem function in R?

text: I have performed the path analysis using the `sem` function in R. The model which I fitted consists of both direct and indirect paths. I have some trouble in interpreting the estimates of the SEM coefficients.   * Does R gives the value of total effect = (direct effect + indirect effect) directly or do I have to multiply the coefficients which are on the indirect path and then add them to the coefficients which is on the direct path? This is the usual way of doing path analysis with the raw/absolute correlation coefficients. For example consider X (independent variable), Y (dependent variable) and M (Mediating variable). The raw/absolute correlation/ standardized regression coefficients between them are X and Y -0.06; X and M 0.22 and M and Y 0.28 whereas on the path analysis/sem in R, the above coefficients are X and Y -0.13; X and M 0.22 and M and Y 0.31.   * Thus is the total effect of X and Y equal to -0.13?   * Alternatively how should I interpret this coefficient considering the effect of variable M into the account?


title: Varying Non-Linear Parameters Based on Groups in R

text: I'm trying to develop a non-linear model, but I'd like to have the values for the parameters vary by group. To give you an example, a section of my data (just random numbers here) looks like:                 MONTH FUTURE.PC FUTURE.SALES     1   APR        35         1498     2   APR        22         1124     3   MAY        24          744     4   MAY        45          453     5   MAY        13         1024       6   JUN        26          689      Say I wanted to make a non-linear model that looked like               nls(FUTURE.SALES ~ a + b * FUTURE.PC ^ c)      Is there a way to change the values of $a$, $b$, and $c$ based on which month the record belongs to? Alternatively, is there a way to incorporate factor variables in non-linear regression in $R$?


title: Detect statistical anomalies

text: I've read these SO questions and this is not duplicate of them   * best way to statistically detect anomalies in data   * Statistical calculations   * Algorithm(s) for spotting anomalies ("spikes") in traffic data * * * I've collected statistical data about people Data hierarchy is like this:   * Region      * Street        * Building number       * Entrance number          * [Statistical package] [Statistical package] contains (in this example)   * floor (stock) number   * UUID (defining flat)   * Religion   * Appearance of toilete **What algorithm or procedure should I use** to discover anomalies like: or **What statistical programming framework should I use?**   (including what is best underlaying technology - like SQL or Document oriented DB, interpreted or compiled language, and so on) 1-a :: Only one floor (of every floors in building) has no toilets   1-b :: One flat (UUID) has no toilet although all other flats in entrance/building has at least on   2-a :: There is one flat claiming Religion X although whole Region has Religions Y and Z   2-b :: There is one building claiming Religion X although whole Region has Religions Y and Z But this is only example on limited number of Statistical package attributes, I should find many types of anomalies on around 15 attributes in every Statistical package Note: this question is not about how should I find anomalies for provided examples, those examples are just illustrative, I'm looking for common solution/algorithm Thanks beforehand for any response


title: Universal, simple to use method of estimating 1 variable from 3 input variables

text: Im looking for a method that would allow me to estimate the value of 1 variable, described by 3 other variables. I have a set of measurement data collected with my camera. Each record consists of:   1. Pixel brightness (describing the ammount of infrared radiation emitated by a body during picture capture)   2. Gain factor (% of Gain set in camera hardware during picture capture)   3. Exposure time (used during given picture capture   4. Temperature of the body (at the moment of image capture) The number of measurements in set is of course finite. I want to be able to estimate the temperature of the body given any combination of gain, exposure and pixel brightness values. This is only a little part of a big project and I dont want to dive deep into statistics to solve my problem. What I need is a method that is universal i.e. will work farily ok with any set of data without requiring me to attune the method parameters or to know the details about statistics or what is the shape of relationships between variable. Can you suggest me anything? Also if on top of that anyone knew a library in c++ that has this method implemented and shared the information I would apprecieate it greatly! Best regards Marcin


title: Why do we prefer white noise and iid property?

text: **EDIT**   1. When we add noise to system we say it is white noise. White noise will have a constant power spectral density, flat power spectrum. But what is the advantage of this?   2. Why do we need iid property?   3. When we find out the probability density function or is it the distribution function that we need the data to be iid? Why? Thank you


title: Sampling distribution of the mean of population that assume values only between 0 and 1

text: I'm trying to use central limit theorem to compute the mean of the sample, however the population where I'm sampling has value only between 0 and 1, can I use mean of the sample as mean of the population and standard error = (population's standard error / sqrt(n))? I have done it, but the distribution does not seem a Normal, for example, I have a population with mean = 0.9895348 and SD = 0.05908021, then I'm assuming that mean of the sample should be 0.9895348 and for a sample of n = 78 the SD of the sample mean = 0.006689516. I sampled 2000 times a random sample of n = 78, and the histogram I attached below. It does not seems a Normal, and the mean seems to be on the left where it should be, and the SD seems to be greater than 0.006689516. ![enter image description here](http://i.stack.imgur.com/pPiPQ.png) * * * Edit: What I want is the best way to calculate the expected value of mean of a random sampling from this population. When the mean is close to 0 or 1, in my opinion, the expected value does not seem fine. When it is close to 0 the expected value seems to be greater than it should be, and when it is close to 1 it seems to be lower than it should be. It is a synthetic dataset, where every point has a probability, below has a histogram of the data: ![enter image description here](http://i.stack.imgur.com/3rtQe.png) Another dataset, when the mean is close to 0, here the mean is ploted as a point, but I think the expected value should be lower ![enter image description here](http://i.stack.imgur.com/jPveV.png)


title: Parameter Estimation for Naive Bayes - Maximum a posteriori and Maximum Likelihood

text: I am wondering if I understand those terms correctly. To summarize my thoughts: In naive Bayes, our decision rule is basically the Maximum a posteriori (MAP) estimate of our hypothesis. We assign an observation $\pmb x$ to the class $\omega_j$ that has the largest posterior probability: \begin{equation} \underset{j = 1 ..., m} {\mathrm{argmax}} \; P(\omega_{j} \mid \pmb x) \end{equation} This is called MAP since we are incorporating prior knowledge (the prior probabilities) to calculate the posterior probability: \begin{equation} P(\omega_j \mid \pmb x_i) = \frac{P(\pmb x_i \mid \omega_j) \cdot P(\omega_j)}{P(\pmb x_i)} \end{equation} where   * _$i$ = 1, 2, ..., n_ (samples)   * _$j$ = 1, 2, ..., m_ (class labels)   * $\omega_j$ = class $j$   * $\pmb x_i$ = features of sample $i$ Now, if we use the training data to estimate the parameters for the priors $P(\omega_j)$ based on the the frequency of the classes in the training data, this would be a Maximum Likelihood Estimate (MLE). Similarly, we can use MLE to calculate the class conditional probabilities $P(\pmb x_i \mid \omega_j)$ under the conditional independence assumption \begin{equation} P(\pmb x \mid \omega_j) = \prod_{k=1}^{d} P(\pmb x_k \mid \omega_j) \end{equation} Does this make sense at all?


title: multiple imputation for categorical data question

text: How do i design the use of Multiple Imputation based on Bayesian Inference when I am dealing with categorical data and my dataset does not contain complete prior observations at every combination ? For example i have three categorical variables                 x1, x2, x3,        x1 {1,2,3}       x2 {1,2}       x3 {1,2}      This is 3*2*2 = 12 different combinations I dont have have complete data for combinations (x1=2, x2=1, x3=2) and similarly two other combinations, so can I use MI based on Bayes Inference ? Please advise.                x1 x2 x3 freq      1  1  1   10      1  1  2   4      1  2  1   5      1  2  2   7      2  1  1   9      2  1  2   -      2  2  1   -      2  2  2   8      3  1  1   4      3  1  2   -      3  2  1   3      3  2  2   2


title: How to compare two clusterings generated by two clustering approaches

text: I am currently working on a modification of a clustering algorithm to suit my problem domain. I want to know which methods are available for me to compare the centroids generated from the two methods? That is, **I want to know how well my (modified) clustering method agrees with the current method, the current method being the gold standard.** Would highly appreciate any help.


title: How to represent categorical data in a pie graph form?

text: I have been asked to produce a pie graph of some categorical data. On a simple scale it look like this:               House 1: Speak a foreign language , own a computer     House 2: Only lived in by women     House 3: Only lived in by women, speak a foreign language     House 4: Has pets     House 5: Has pets, own a computer     House 6: Speak a foreign language.      They want a pie graph that shows what percent of the different categories are represented.               Well if I go:     Speak a foreign language  = 3/6     Own a computer = 2/6     Only lived in by women = 2/6     has pets = 2/6      This totals 9/6 (i.e more than 100%) Should I be totalling how many individual items there are and then creating my %'ages from that i.e               Speak a foreign language  = 3/9     Own a computer = 2/9     Only lived in by women = 2/9     has pets = 2/9      this makes 100% but i am unsure if this is how you would do it so all the data is shown in one pie graph.


title: how to compare groups and percentages within

text: I am new to statistics and would really appreciate help. The people within the groups are different - how do I compare the percentages to see if there is a statistical significance between the groups that had training and between the group that did not It looks something like this:               group 1           had training and worked     age group    number of tasks         percentage          20-25             40                  40     26-30             30                  30     31-35             30                  30           group 2       did not have training and worked     age group    number of tasks         percentage          20-25             50                     50     26-30             10                     10     31-35             40                     40


title: How to apply Laplace rule of succession for any possible value in a range?

text: I am looking to have People rate certain items on a scale of 0-10 based on those items perceived utility to the particular Person performing the rating. Fractional values are permitted. Giving no rating on an item is also permitted; however, I prefer to rank "highly rated" items with more ratings higher than "highly rated" items with fewer ratings; so, I know I do not want to use the arithmetic mean. [Note: While this question may seem similar to one asked a few years back, the scenario is distinctly different in the fact that prior question limited ratings to 4 discrete values where this scenario does not involve discrete values.] I also want to take the ratings and come up with an average based on the Laplace rule of succession. However, the rule, as far as I can tell presumes discrete possible values, as opposed to the infinite choices in this particular case. Does a method exist for extending the rule to cover such a scenario?


title: How to compare two models' MSE if the answer range differs?

text: I have two prediction models. The first one returns answers in the range 0 to 1, where the correct answer is 0 or 1. The second returns answers in the range -1 to 1, and the correct answer is -1 or 1. I have the MSE for both modes, and I want to compare performance. What is the correct way to do this? Simply divide the MSE of the second model by 4?


title: How to solve the expectation of continuous random variable

text: I encountered some problem while getting the $\mathbb{E}[x]$ of a continuous random variable.   The following equation is the PDF that im trying the expectation. $\mathbb{E}[x] = \int_0^\infty x^2\alpha^2e^{-\alpha x}dx$ I got stuck of how to get next step of   $[-\alpha x^2e^{-\alpha x}]_0^\infty + \int_0^\infty 2x \alpha e^{-\alpha x} dx $ Can someone explain to me how to get to this step as I have tried many combination of integration by parts (such as $u=x^2, u=x, v=\int x e^{-\alpha x} dx$ but I just couldnt get to the step above :(


title: Rule based pruning clarification

text: I'm trying to answer a homework question and have found the following example online similar to what I need to accomplish for classifying unknown data in a decision tree. I don't quite understand why on slide 5 "Wind =String" is removed from the rule? I've read elsewhere that it's to do with removing information without effecting the outcome but I struggle to understand how you decide which rules to remove sections, what to remove and then what order to sort them in as shown on slide 7. Thanks for any clarification. Full slides here: http://isites.harvard.edu/fs/docs/icb.topic540049.files/cs181_lec07_handout.pdf ![Rule-Pruning example](http://i.stack.imgur.com/WEt1P.png) ![pic2](http://i.stack.imgur.com/nZ2Bl.png)


title: How is approximately unbiased bootstrap better than a regular bootstrap with regards to hierarchical clustering?

text: _I askedthis question at BioStar but did not get a reply, so Im posting the question here._ **What is a simple explanation of what an approximately unbiased bootstrap is with regards to hierarchical clustering?** From what I read, it alters the sample size during randomization to calculate p-values. **How is this approach better than the regular bootstrap which keeps the sample size intact while randomizing and also is it randomization with replacement?** Edit: There is an R package pvclust that calculates p-value and approximately unbiased p-value. My apologies for being unclear as I thought this was due to a difference in the bootstrap method. Thanks for all the answers and comments!


title: Relationship between $H_{0}$, $H_{1}$ and the Power of the (one sided) hypothesis test

text: Suppose we wish to test      $H_{0} : p = 0.5$ vs $H_{1} : p = p_{1}$ (where $p_{1} \gt 0.5$      What are the relative difficulties of testing for $p_{1}$ close to $0.5$ or $p_{1}$ far from $0.5$?      I know that the further $p_{1}$ is from $p_{0}$, the closer to one the power will be, and the further away they are from each other, the closer the power will be to the significance level of the test. I'm not sure I understand why these facts would imply difficulties when testing for $p_{1}$ in the two instances however?