title: Survival Model for Predicting Churn - Time-varying predictors?

text: All, I am looking to build a predictive model for predicting churn and looking to use a discrete time survival model fitted to a person-period training dataset (one row for each customer and discrete period they were at risk, with an indicator for event – equaling 1 if the churn happened in that period, else 0).   * I am fitting the model using ordinary logistic regression using the technique from Singer and Willet.   * The churn of a customer can happen anywhere during a month, but it is only at the end of the month that we know about it (i.e. sometime during that month they left). 24 months is being used for training.   * The time variable being used is the origin time of the sample - all customers active as of 12/31/2008 - they all receive t=0 as of Jan 2009 (not the classical way to do it, but I believe the way when building a predictive model versus a traditional statistical one). A covariate used is the tenure of the customer at that point in time.   * There are a series of covariates that were constructed – some that do not change across the rows of the dataset (for a given customer) and some that do.   * These time variant covariates are the issue and what is causing to me question a survival model for churn prediction (compared to a regular classifier that predicts churn in the next x months based on current snapshot data). The time-invariant ones describe activity the month prior and are expected to be important triggers. The implementation of this predictive model, at least based on my current thinking, is to score the customer base at the end of each month, calculating the probability / risk of churn sometime during the next month. Then again for the next 1,2 or 3 months. Then for the next 1,2,3,4,5,6 months. For the 3 and 6 month churn probability, I would be using the estimated survival curve. **The problem:** When it comes to thinking about scoring, how can I incorporate time-varying predictors? It seems like I can only score with time-invariant predictors or to include those that are time invariant, you have to make them time invariant – set to the value “right now”. Does anyone have experience or thoughts on this use of a survival model? **Update based on @JVM comment:**      The issue is not with estimating the model, interpreting coefficients, plotting the hazard/survival plots of interesting covariate values using the training data etc. The issue is in using the model to forecast risk for a given customer. Say at the end of this month, I want to score everyone who is still an active customer with this model. I want to forecast that risk estimate out x periods (risk of closing the account at the end of next month. risk of closing the account at the end of two months from now, etc.). If there are time varying covariates, their values are unknown out any future periods, so how to utilize the model? **Final Update:**      A person period data set will have an entry for each person and each time period they are at risk. Say there are J time periods (maybe J =1...24 for 24 months) Lets say I construct a discrete time survival model, where for simplicity we just treat time T as linear and have two covariates X and Z where X is time-invariant, meaning it is constant in every period for the ith person and Z is time varying, meaning that each record for the ith person can take on a different value. For example, X may be the customers gender and Z might be how much they were worth to the company in the prior month. The model for the logit of the hazard for the ith person in the jth time period is :   $logit(h(t_{ij}))=\alpha_{0}+\alpha_{1}T_{j}+\beta_{1}X_{i}+\beta_{2}Z_{ij}$ So the issue is, when using time varying covariates, and forecasting (into the yet unseen future) with new data, the $Z_{j}$ are unknown. The only solutions I can think are:   * Don't use time varying covariates like Z. This would greatly weaken the model to predict the event of churning though since, for example, seeing a decrease in Z would tell us the customer is disengaging and perhaps preparing to leave.   * Use time varying covariates but lag them (like Z was above) which allows us to forecast out however many periods we have lagged the variable (again, thinking of the model scoring new current data).   * Use time varying covariates but keep them as constants in the forecast (so the model was fitted for varying data but for prediction we leave them constant and simulate how changes in these values, if later actually observed, will impact risk of churning.   * Use time varying covariates but impute their future values based on a forecast from known data. E.g. Forecast the $Z_{j}$ for each customer.


title: Time series regression with overlapping data

text: I am seeing a regression model which is regressing Year-on-Year stock index returns on lagged (12 months) Year-on-Year returns of the same stock index, credit spread (difference between monthly mean of risk-free bonds and corporate bond yields), YoY Inflation rate and YoY index of industrial production. It looks thus (though you'd subsitute the data specific to India in this case):               SP500YOY(T) = a + b1*SP500YOY(T-12) + b2*CREDITSPREAD(T) +         b4*INDUSTRIALPRODUCTION(T+2) + b3*INFLATION(T+2) + b4*INFLATIONASYMM(T+2)      SP500YOY is the year-on-year returns for SP500 index To compute this, monthly average of SP500 values is computed and then converted to year-on-year returns for each month (i.e. Jan'10-Jan'11, Feb'10-Feb'11, Mar'10-Mar'11, . . ). On the explanatory variables side, a 12-month lagged value of the SP500YOY is used along with the CREDITSPREAD at time T and INFLATION and INDUSTRIALPRODUCTION two period AHEAD. The INFLATIONASYMM is a dummy for whether the Inflation is above a threshold value of 5.0%. The index in the parenthesis shows the time index for each variable. This is estimated by standard OLS linear regression. To use this model for forecasting the 1,2 and 3-months ahead YOY returns of SP500, one has to generate 3,4 and 5-month ahead forecasts for Inflation and the Index of Industrial Production. These forecasts are done after fitting an ARIMA model to each of the two individually. The CreditSpread forecasts for 1,2 and 3 month ahead are just thrown in as mental estimates. I'd like to know whether this OLS linear regression is correct/incorrect, efficient/inefficient or generally valid statistical practice. The first problem I see is that of using overlapping data. i.e. the daily values of the stock index are averaged each month, and then used to compute yearly returns which are rolled over monthly. This should make the error term autocorrelated. I would think that one would have to use some 'correction' on the lines of one of the following:   * White's heteroscedasticity consistent covariance estimator   * Newey & West heteroscedasticity and autocorrelation consistent (HAC) estimator   * heteroscedasticity-consistent version of Hansen & Hodrick Does it really make sense to apply standard OLS linear regression (without any corrections) to such overlapping data, and more so, use 3-period ahead ARIMA forecasts for explanatory variables to use in the original OLS linear regression for forecasting SP500YOY? I have not seen such a form before, and hence cannot really judge it, without the exception of correcting for the use of overlapping observations.


title: How to visualize/summarize a matrix with number of rows $\gg$ number of columns?

text: I have a matrix of positive real numbers between 0 and 1; the rows represent genes and columns represent samples. Number of rows is greater than the number of columns by a magnitude of $10^4$. I am wondering how to visualize this in `R`. I know heatmap is one of the ways to do this, but are there other ideas. Here are a few points which I want to emphasize in the visualization: **Data:**   1. The rows and columns have no order to them (as you may have already realized); specifically rows and columns are exchangeable.   2. The entries of the matrix are positive real numbers between 0 and 1.   3. A small fraction of the data (10% of the rows or genes, around 1000) are actually "interesting".   4. The matrix represents the estimated probability of genes being more active in a sample.  **Aim:**   1. I want show: which genes are more active and in which sample. The matrix has a lot of rows in which the probabilities are very similar across columns.   2. I am ok with ordering the rows (genes) to make the pattern clearer. **My thoughts:** At the moment I can determine active genes in a sample by choosing a cutoff (say $\ge 95\%$) and arrange the genes in such a way that first set of rows are active genes in sample 1, second set of rows are active genes in sample 2, ... I was also thinking about visualizing a subset of the data, may be by sampling rows. But I did not have any success. I know these ideas may not be very elegant but rearranges my data in a way which makes the pattern more recognizable. I know similar questions have been asked before, but I thought my query was a bit more specific, so hopefully I can get better inputs from the members of this forum.


title: Do confidence intervals apply to quota sampling?

text: French polling institutes are currently facing a major crisis after they recently published what can only be called the most ridiculous poll so far on the 2012 presidential election horse race. The French Senate is now considering to legislate on the issue by forcing polling institutes to publish, among other things, the confidence intervals for their results. However, some pollsters are opposing the measure, claiming that confidence intervals do not apply to quota sampling, which is the method used by polling institutes in France. Since quota sampling is formally non-probabilistic sampling, there is some truth to the claim. But since quota sampling is fundamentally stratified sampling, confidence intervals should apply, right? May I ask for experiences about this issue outside of France, in countries where pollsters also use quota sampling?


title: Given a 95% confidence level, how do I demonstrate 95% of the intervals actually contain the population mean?

text: Let me say first off that I'm consistently impressed with the quality of responses on this site. You folks have done a better job of explaining difficult concepts than most instructors or textbooks I've encountered. On to my question/scenario. My basic goal is to prove that confidence intervals actually work, and by work I mean that that a certain percentage of them, say 95%, actually include the population mean (assuming that we know the population mean in the first place). Here's my set up. A) I’ve generated a data set in Excel consisting of 20 samples with 64 observations each (i.e. n = 64) that represent prices. B) Prices are bound between a minimum of \$25 and a maximum of \$100. C) Below is the best Excel formula I found that can generate something approaching a normal distribution. If anyone has any thoughts on this, let me know as I’m not wholly convinced as to how/why it works. I only know that RandBetween produces a uniform distribution so it’s useless for these purposes.               =NORMINV(NORMDIST(MinX,Mean,StDev,TRUE)+RAND()*NORMDIST(MaxX,Mean,StDev,TRUE)-NORMDIST(MinX,Mean,StDev,TRUE)),Mean,StDev)      D) I chose \$62.50 as the population mean simply because it’s the average of 25 and 100. E) I chose \$12 as the standard deviation of population. F) I chose 95% (i.e. z-score 1.96) as the confidence level. Here are my 20 confidence intervals. The ones with the asterisks fail to contain the population mean of \$62.50.               S1  66.19   60.31     S2* 62.30   56.42     S3  66.46   60.58     S4* 70.38   64.50     S5  63.24   57.36     S6* 61.23   55.35     S7  63.95   58.07     S8  65.89   60.01     S9  67.86   61.98     S10 64.17   58.29     S11 65.59   59.71     S12 67.31   61.43     S13 67.61   61.73     S14 66.55   60.67     S15 66.03   60.15     S16 63.77   57.89     S17 64.63   58.75     S18 65.35   59.47     S19 66.19   60.31     S20 64.32   58.44      As you might have already guessed, here’s the issue: theoretically if 95% of these intervals contained the population mean, then only 1 interval out of 20 would have failed to include it. Instead there were 3. And this wasn’t an isolated incident. If I regenerate the prices repeatedly using F9, it only meets the 19 or greater threshold around 70% of the time with some iterations producing 20 out 20 intervals that contain the mean to as low as only 16 out of 20. My question is why? Here’s what I’ve eliminated as possible issues thus far. a) I boosted the number of samples to 100 and sample size to 1281 and got the same result so I don’t believe it’s an issue of sample size. 128,100 data points would seem to be sufficient. b) I performed 10,000 iterations and recorded them using a macro and still got a “success rate” of about 70% same as before. c) I also replicated everything above using 90% and 99% confidence levels. 90% only resulted in 18 or more intervals containing the population mean 63% of the time. A 99% confidence level only managed this about 87% of the time. d) I’ve graphed the data in the form of a histogram to visually inspect it for normality and I’ve found no visual correlation between skewness, spikes, etc. and whether more or fewer intervals fail to contain the population mean. Also, I tried a Normal Quantile Plot Test for Normality that I found online on the 128.1k data point example and I did find a very slight amount of leptokurtosis for anyone who knows what to make of that. Here are possible issues that may be impacting the result in ways I don’t understand. a) The distribution is “truncated.” I found very little that was intelligible for a lay person like myself that explained how to calculate confidence intervals for this type of distribution, or even if this method is appropriate for truncated data. I couldn’t even find agreement on whether the truncation disqualifies it as normal in the strict sense. b) Excel’s Norminv, Normdist, and Rand functions. Is Excel simply incapable of generating random normally distributed data with the level of accuracy needed to make this example work? Note that I also tried Excel’s data analysis->random number generation tool and got virtually identical results as the Excel formula I included above. Lastly, the only way I was able to consistently get 90% of the confidence intervals to contain the population mean was to use a 95.7% confidence level (i.e. z-score 2.025), 95% needed a CL of 99%, and 99% needed a CL of 99.5%. Any thoughts on what’s going on here would be greatly appreciated. If anyone can figure this out, it’s you guys. Thanks, * * * In case someone happens to see this thread, here are screen shots of the 2 main excel spreadsheets; one version with just the values, the other version with just the formulas. ![Random number generator values](http://i.stack.imgur.com/BKrHq.jpg) ![Random number generator formulas](http://i.stack.imgur.com/YQUkl.jpg) ![enter image description here](http://i.stack.imgur.com/RLzM9.jpg) ![enter image description here](http://i.stack.imgur.com/PbwrB.jpg) ![enter image description here](http://i.stack.imgur.com/8RMjr.jpg) ![enter image description here](http://i.stack.imgur.com/pI0MD.jpg) ![enter image description here](http://i.stack.imgur.com/d9sYG.jpg)


title: How can I get a significant overall ANOVA but no significant pairwise differences with Tukey's procedure?

text: I performed with R an ANOVA and I got significant differences. However when checking which pairs were significantly different using the Tukey's procedure I did not get any of them. How can this be possible? Here is the code:               fit5_snow<- lm(Response ~ Stimulus, data=audio_snow)     anova(fit5_snow)          > anova(fit5_snow)     Analysis of Variance Table          Response: Response               Df Sum Sq Mean Sq F value  Pr(>F)       Stimulus   5  73.79 14.7578  2.6308 0.02929 *     Residuals 84 471.20  5.6095                       ---     Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1           df<-df.residual(fit5_snow)     MSerror<-deviance(fit5_snow)/df          comparison <-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)          > comparison <-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)          Study:          HSD Test for audio_snow$Response           Mean Square Error:  5.609524           audio_snow$Stimulus,  means                          audio_snow.Response   std.err replication     snow_dry_leaves            4.933333 0.6208034          15     snow_gravel                6.866667 0.5679258          15     snow_metal                 6.333333 0.5662463          15     snow_sand                  6.733333 0.5114561          15     snow_snow                  7.333333 0.5989409          15     snow_wood                  5.066667 0.7713110          15          alpha: 0.05 ; Df Error: 84      Critical Value of Studentized Range: 4.124617           Comparison between treatments means                                        Difference   pvalue sig        LCL      UCL     snow_gravel - snow_dry_leaves  1.9333333 0.232848     -0.5889913 4.455658     snow_metal - snow_dry_leaves   1.4000000 0.588616     -1.1223246 3.922325     snow_sand - snow_dry_leaves    1.8000000 0.307012     -0.7223246 4.322325     snow_snow - snow_dry_leaves    2.4000000 0.071587   . -0.1223246 4.922325     snow_wood - snow_dry_leaves    0.1333333 0.999987     -2.3889913 2.655658     snow_gravel - snow_metal       0.5333333 0.989528     -1.9889913 3.055658     snow_gravel - snow_sand        0.1333333 0.999987     -2.3889913 2.655658     snow_snow - snow_gravel        0.4666667 0.994348     -2.0556579 2.988991     snow_gravel - snow_wood        1.8000000 0.307012     -0.7223246 4.322325     snow_sand - snow_metal         0.4000000 0.997266     -2.1223246 2.922325     snow_snow - snow_metal         1.0000000 0.855987     -1.5223246 3.522325     snow_metal - snow_wood         1.2666667 0.687424     -1.2556579 3.788991     snow_snow - snow_sand          0.6000000 0.982179     -1.9223246 3.122325     snow_sand - snow_wood          1.6666667 0.393171     -0.8556579 4.188991     snow_snow - snow_wood          2.2666667 0.103505     -0.2556579 4.788991


title: ANOVA result indicated that there were sig. differences while Tukey post-hoc test revealed that there were no statistically sig. differences

text: I was conducted one-way ANOVA analysis to examone the level of knowledge shaorng among the employees based on there age differences. One-way ANOVA indicated that there were significant differences in the level on KS while Tukey post-hoc test revealed that there were no statistically significant differences on the level of knowledge sharing between the groups based on the respondent's age differences. So, _what is the conclusion of this result and why the result shows a different result?_ AND, how do i interpret this result? Table: Mean Scores on Level of Knowledge Sharing among Employees - Based on Age.               Age No. Mean    SD     20-29   34  3.82    1.13     30-39   54  3.87    0.92     40-49   56  3.41    1.06     over 50 45  3.32    1.24     Total   189 3.59    1.10               Table: ANOVA              Sum of Squares  df  Mean Square F   Sig.     Between Groups  11.029  3   3.676   3.120   .027     Within Groups   218.021 185 1.178            Total   229.050 188                        Table: Multiple Comparisons          (I) Age (J) Age Mean Difference (I-J)   Std. Error  Sig.    95% CI Lower Bound  Upper Bound     20-29   30-39   -.047-  .238    .997    -.66-   .57             40-49   .408    .236    .311    -.20-   1.02             over 50 .501    .247    .180    -.14-   1.14     30-39   20-29   .047    .238    .997    -.57-   .66             40-49   .455    .207    .127    -.08-   .99             over 50 .548    .219    .063    -.02-   1.12     40-49   20-29   -.408-  .236    .311    -1.02-  .20             30-39   -.455-  .207    .127    -.99-   .08             over 50 .093    .217    .974    -.47-   .66     over 50 20-29   -.501-  .247    .180    -1.14-  .14             30-39   -.548-  .219    .063    -1.12-  .02             40-49   -.093-  .217    .974    -.66-   .47      Many thanks for helping me to find the answer of my question.


title: Bonferroni post hoc test does not reveal significant differences between groups

text: > **Possible Duplicate:**   >  How can I get a significant overall ANOVA but no significant pairwise > differences with Tukey's procedure? I used a repeated measures ANOVA for my dissertation, with 4 warm-up protocols acting as levels of the independent variables and time to 50m freestyle as the dependent variable. The ANOVA reveals that there is a signifficant difference between means but the Bonferroni doesn't tell me which specific means differed, pairwise comparisons don't reveal any signifficant different. Is it possible? Why? Thanks.


title: Down-sampling the majority class-How can I assure that all rows are been picked?

text: My doubt is about Down-sampling. I have an imbalanced data and I have use down-sampling, but I am not sure if all rows has been picked at any fold I have been use 10 folds.


title: What to use: Nested Case-Control or Case-Cohort?

text: I am following a cohort of patients over time from which I am planning to sub- sample a sub-group of patients to pilot a new intervention. The plan is to compare the effectiveness of the new intervention on improving specific outcomes for a sub-sample compared to those did not receive the intervention. the original study is an inception prospective cohort with several followup assessment time points (3 evaluations time points). My question: Which design is better to use   1. **Nested Case-Control** (the intervention group [participants who have completed the intervention] will be compared to control group [those who didn't receive an intervention] and maybe a third group of normal people from the community) in this case the 2 (or 3) study groups will be matched on age and co-morbidities.   2. **Case-Cohort design** (my understanding for this is that a sub-sample will be randomly selected at the baseline for an intervention and then compared to the rest of the cohort). Both are new to me and not sure if any of these is a superior design over the other one and better to use, and what would be the factors to consider? Many thanks in advance for your help!


title: Differenence between hierarchical linear regression and moderated multiple regression

text: What is the difference between hierarchical linear regression and moderated multiple regression? If I have one and two moderator variables and my models are:   * model 1- IV   * model 2- adding two ModVs   * model 3- adding all the interactive terms, is my model HLM or MMR?


title: Interpreting coefficient errors in regression from uncertain population

text: I'm working on some regressions for UK cities and have a question about how to interpret regression coefficients. In a typical regression, one would be working with data from a sample and so the standard errors on the coefficients can be interpreted as reflecting the uncertainty in the choice of sample. In my case, I'm working with every city in the UK so the error interpretation isn't as clear. There are two sources of confusion:   1. Since I'm working with a full population, can I just ignore the coefficient errors or do they have an additional interpretation that might be relevant? I've seen some mention of finite populations but am not sure how this might apply in a regression.   2. The definition of a city is itself somewhat uncertain. In my study I'm looking at about six definitions, each one consisting of a full population of UK cities (though each definition has a different number of cities; it's not just the attributes of each city that change but the population size itself). Would it be sensible to interpret regression coefficient errors as capturing this uncertainty, or would an alternative model formulation be more appropriate? What I'm looking for is some guidance on best practice in this situation.


title: What is task-loss function?

text: I looked into "Multi-Output Learning for Camera Relocalization" research and faced with the following part (2.2 The Direct Regression Approach): > Given a set of RGB-D frames with known camera poses S = {(Ij , Hj ): j ∈ > [n]}, we want to learn a mapping g, taking an RGB-D frame as argument and > predicting a 6 d.o.f. camera pose, such that the the empirical risk is > minimized > > $$ L(\Theta;S)=\frac 1 n \sum_{j=1}^n l(H_j,g(I_j,\Theta)) $$ > > Here, the task-loss $l$ measures the error between the ground-truth pose > $H_j$ and the predictor’s output $g(I_j; \Theta)$, and $\Theta$ is the set > of parameters to be learned. So, my question is -- what is task-loss $l$? Is it any appropriate function for error measurement? Actually, the question is about terminology but the form of the function $l$.


title: Generating hyperbolic variates

text: How do you simulate from the hyperbolic distribution? The Wikipedia article on it does not describe how to do so. I know that the hyperbolic distribution is a special case of the generalized hyperbolic distribution (GHD), and that GHD variates can be generated given normal and generalized inverse Gaussian (GIG) variates. But I am not too familiar with the GIG distribution and am hoping there is a simpler procedure for the "plain" hyperbolic distribution. In other words, what is the mixing distribution for the hyperbolic distribution?


title: Is it necessary to scale the target value in addition to scaling features for regression analysis?

text: I'm building regression models. As a preprocessing step, I scale my feature values to have mean 0 and standard deviation 1. Is it necessary to normalize the target values also?


title: Specifying a linear mixed model in lmer with replications nested within a fully crossed design

text: I’m trying to specify a linear mixed model for a somewhat complicated, nested & crossed method comparison study with replicated measurements. The goal is to partition and compare variances. It’s more complicated than the typical introductory examples I have seen and I’m not quite sure how to do it correctly. The data I have collected comes from a comparison of different laboratory measurement _methods_ (4 levels), each evaluated for all combinations for _parameter1_ (10 levels) and _parameter2_ (40 levels). Measurements (repeated for each unique combination of method, parameter1 and parameter2) were done on 60 different _subjects_ characterized by _sex_. Each measurement was replicated 800 times (almost 80 million rows of observation) and the difference (true value determined beforehand by a reference method - measured value) was recorded. **Therefore: method x parameter1 x parameter2 are fully crossed (all combinations of factor levels occur once), subjects are nested within sex, and replicated measurements are nested within subject.** Following some advice (https://mailman.ucsd.edu/pipermail/ling-r- lang-l/2011-February/000225.html, and Barr et al., 2013), I would like to initially specify a maximal random effects structure for this, i.e., to account for between-subject differences in the sensitivity to all of the conditions, and then strip down the model from there. But I’m having a really hard time wrapping my head around the formula syntax and the model logic at the same time. I’m shooting in the dark here, but what about:               (1 + method*param1*param2*sex | sex/subj_id)  # which expands to     (1 + method*param1*param2*sex | sex) + (1 + method*param1*param2*sex | sex:subj_id)      I don’t think that’s correct, though. It seems strange to have sex on both sides of the equation. I would be very grateful for any help. On a related issue: I intend to regress on the difference between measured value and (known) true value: diff ~ …. Is there any reason to instead regress the known true value onto the measured value: measured ~ true + …?


title: S and N parameters of Ridor

text: I am using WEKA and in particular their implementation of Ridor. The documentation says this about the parameters S and N: -S Set number of shuffles to randomize the data in order to get better rule. (default 10) -N Set the minimal weights of instances within a split. (default 2.0) For S: I assume this pertains to the input data. When and why is it randomized? Why would it give a better rule set? I tried reading through the reference at http://pages.cpsc.ucalgary.ca/~gaines/reports/ML/JIIS95/JIIS95.pdf but it seems the only random part there has to do with error analysis rather than the method itself. For N, I do not understand the description at all. What I believe it could mean is that if a rule would induce a split such that one part would not have enough instances that rule will be ignored. But how are the instances given weights then? I have tried searching but only come up with the same manual page I got the above from. Any help would be appreciated, thanks in advance!


title: Two-predictor regression vs partial correlation

text: Say I want to see whether two variables y and x1 are linearly related, while controlling for the effect of an x2 "nuisace" variable. What is the difference between (a) regressing y over x1 and x2 and just looking at x1's coefficient; and (b) doing a partial correlation for y and x1 while controlling for (co- varying) x2?


title: How can I use the sufficient statistics (variances, covariances, means) to estimate a linear regression model in R?

text: My question is simple: is there a function in `R` which estimates the linear regresion model in a similar fashion as `lm`, but only using the means, variances, and covariance (correlations), i.e. the sufficient statistics? I am looking for a function to which I can input these statistics (plus sample size) and it returns regression coefficients and tests.


title: Association Rule Mining

text: I am using association rule mining on a large transaction data-set (about 100,000 items). Till now I have done frequent item set mining and constructed some rules using a confidence and lift measure. What I wish to do now is check how valid they are on a new transaction data- set which contains the same items as well as some new items. What is the proper way to do it ? Is there any R package or Java library which allows to measure accuracy of association rules on new data sets


title: Probit and probability of being a member of a group

text: I was wondering if there was a specific way to convert probit regression results into a probability that a test subject is a member of group 1 (vs group 0) if you break up the dependent variable into 2 groups (e.g. success vs. failure.) I know that for logistic regression you use the inverse logit of the sum of the coefficients * the subject's values + the constant but I am unaware of how to do it with probit. I'm sure there is a way because Stata can check the model's ability to classify with estat classification. I primarily use R now for my data analysis, so if you know how to do this in R, I would appreciate your help even more (currently I use inv.logit from the boot package.)


title: Support Vector Machine Question

text: I need help with the following problem. I provided my current (partial) solution, and I hope someone can correct me and/or give me suggestions as to how I should solve the parts that I've left out. > Given an SVM with kernel: > > $$K(x,z) = \theta(x)^T \theta(z) = \left\\{ \begin{array}{ll} 1 & \text{if } > x = z \\\ 0 & \text{otherwise} \end{array} \right.$$ > > We are given $N$ training examples $(x_1, y_1) \ldots (x_N, y_N)$ with $y_i > = \pm 1$. For simplicity, assume that the $x_i$'s are distinct, and that we > only consider the SVM where the hyperplane in the feature space goes through > the origin, i.e., the intercept $b=0$. > >   1. Recall the weight vector $w$ used in SVM has the form   >  $$w = \sum_{i=1}^N \alpha_i y_i \theta(x_i)$$ Compute the $\alpha_i$'s > explicitly that would be found using SVMs with this kernel. > >   2. Recall that the SVM algorithm outputs a classifier that, on input $x$, > computes the sign of $w^T \theta(x)$. What is the value of this inner > product on the training example $x_i$? What is the value of this inner > product on any example $x$ not seen during training? Based on these answers, > what kind of generalization error do you expect will be achieved by SVMs > using this kernel? > >   3. Recall that the generalization error of SVMs can be bounded using the > margin $\delta$ (which is equal to $1/\|w\|$), or using the number of > support vectors. What is $\delta$ in this case? How many support vectors are > there in this case? How are these answers consistent with your answer in > part (2)? > > For part (1), I maximized the dual form of the objective function $$ \max_{\alpha} \sum_{n=1}^N \alpha_n - \frac{1}{2} \sum_{n=1}^N \sum_{m=1}^N y_n y_m \alpha_n \alpha_m K(x_n, x_m) $$ subject to the constraints $\alpha_n \geq 0, n=1,\ldots,N$ and $\sum_{n=1}^N \alpha_n y_n = 0$. I did that by forming the Lagrangian $$ \mathcal{L}(\alpha, \beta, \lambda) = \sum_{n=1}^N \alpha_n - \frac{1}{2} \sum_{n=1}^N \sum_{m=1}^N y_n y_m \alpha_n \alpha_m K(x_n, x_m) - \sum_{n=1}^N \beta_n \alpha_n - \lambda \sum_{n=1}^N \alpha_n y_n $$ then taking the derivative with respect to $\alpha_i$, and setting it to zero. The solution I got for $\alpha_i$ is $$ \alpha_i = \frac{1}{y_i} \left[ \frac{1}{y_i} - \frac{1}{N} \sum_{n=1}^N \frac{1}{y_n}\right] $$ For part (2), I started by replacing my $\alpha_i$ found in part (1) into the inner product $$ w^T \theta(x) = \sum_{j=1}^N \alpha_j y_j \theta(x_j)^T \theta(x) $$ When $x_i$ is in the training set, i get $$ w^T \theta(x_i) = \frac{1}{y_i} - \frac{1}{N} \sum_{n=1}^N \frac{1}{y_n} $$ When $x$ is not in the training set, i get $$ w^T \theta(x) = 0 $$ Is it true that this shows that the kernel will yield a poor generalization performance? Is there any way I can show it formally? Finally, for part (3), I got $\delta$ by $$ \delta = \frac{1}{\|w\|} = \frac{1}{\sqrt{\sum_{n=1}^N \alpha_n^2 y_n^2}} $$ How do I calculate the number of support vectors? And how does this relate to part (b)?


title: Is this a valid application for a Kalman filter?

text: I have 2 time series; ![](http://i.stack.imgur.com/fFfPq.png) Both the series are tracking inflation.(they have different sampling frequencies) Blue is the official CPI released by the US government. Red is an independent group's measure of inflation If I let the Blue line be the underlying system state, and the red line be the noisy input data, can I estimate the system state using the red line(and perhaps some other explanatory variables that I think are valid) I am already fitting a Mixed Data Sampling (MIDAS) regression model to this data, but I want some other models so that i can compare forecasts.


title: Confidence interval for chi-square

text: I am trying to find a solution to compare two "goodness-of-fit chi-square" tests. More precisely, I want to compare results from two independent experiments. In these experiments the authors used the goodness-of-fit chi- square to compare random guessing (expected frequencies) with observed frequencies. The two experiments got the same number of participants and experimental procedures are identical, only the stimuli changed. The two experiments results indicated a significant chi-square (exp. 1 : X²(18)=45; p<.0005 and exp. 2 : X²(18)=79; p<.0001). Now, what I want to do is to test if there is a difference between these two results. I think a solution could be the use of confidence intervals but I don't know how to calculate these confidence intervals only with these results. Or maybe a test to compare effect size (Cohen's w)? Anyone have a solution? Thanks a lot! F.D.


title: Options for comparing logistic regression models

text: I want to choose variables for building a logistic regression model by comparing the GOF of different types. The problem is that some of the candidates have some missing values so I can´t use the likelihood ratio test for comparing models, given that when I want to compare models with those variables they are not nested anymore. So after exploring some options, I found that AIC or even BIC might be an option for that purpose. Is either of those a valid option? I am not familiar with the use of BIC, is it the same as AIC, where the smaller the value is, the better?


title: Compound risk lottery

text: We wish to know if a gambler will enter a lottery and what he expects to win if he does enter, based upon our uncertain knowledge about his beliefs regarding the lottery. The gambler believes that his probability of winning the lottery is p and also believes that the lottery prize will be W if he wins. But we do not know the gamblers belief’s p or W, only that they are distributed with density functions f(p) and f(W) and cumulative distribution functions F(p) and F(W) respectively. The variables p and W are independent of each other. We assume that f(p) is positive in the interval (a,b) and is zero outside this interval. We assume that 0 < a < b <1. f(p) is continuous and differentiable throughout. There is some neighbourhood of b where f(p) is nonincreasing. We also assume that f(W) is positive in the interval (c,d) and is zero outside this interval. c and d are both large dollar amounts where d>c.   f(W) is continuous and differentiable throughout and there is some neighbourhood of d where f(W) is nonincreasing. We assume that distributions f(p) and f(W) are both triangular in shape for the sake of simplicity. **The gambler will only invest in the lottery if pW is > S** I have two questions I need help with:   1. What is the probability that the gambler will invest in the lottery?   2. What is the expected value of the lottery if he chooses to invest? Thank you.


title: How to find group differences with irregularly measured, nested, and missing data?

text: **Background:** I have a dataset, which consists of waterlevel measurements over time at different locations. However, the locations are irregularly measured in different years. Therefore my data consists of more gaps than actual data. Additionally, some of my data is nested. That means that some of my measurements can be grouped under another measurement, while others are not. **How can I best test whether there is a difference between the waterlevels of various locations given these features of the data?**


title: What do you do about negative values within a component in a PCA analysis?

text: > **Possible Duplicate:**   >  PCA analysis with centered variables In my PCA analysis I have in a component both negative and positive variables. This is probably very basic, but I've been told different things about the interpretation of the variables within a component, so I just need some clarification. First; Is it so that the negative values within a component moves in the same direction and the positive in the opposite? Or is it so that I should only look at the absolute values of the variables within a component?


title: Bootstrap data in order to get an estimate of the variance

text: I have a return series of 60 obs. From this returns I compute a non parametric measure. What is the best way in your opinion to get a consistent estimate of the variance of this measure? I did it in this way: I compute a number of X block bootstrap (to take into account of returns dependence) and then for each bootstrap a compute the measure than I get X measure and I compute the variance. It is correct?


title: What is the meaning of regularization path in Lasso or related sparsity problems?

text: If we select different sparsity levels (i.e. that the $\lambda$ controls), we could obtain different solutions with different sparsity levels. Does it mean the regularization path is how to select the coordinate that could get faster convergence? I'm a little confused although I heart about the sparsity often. In addition, could you please give a simple description about the existed solutions of `LASSO` problem? Thanks in advance!


title: How does "rlm" in R decide its "w" weights?

text: Following user603's advice, I opened a new question thread for this question. For your reference, the original question is here: Pointers to understand "rlm" in R better? * * * My question is: How does "rlm" decide its "w" for each IRLS iteration then? Also, is my understanding below correct? The input argument "w" is used for the initial values of the rlm IRLS weighting and the output value "w" is the converged "w". The "weights" input argument is actually what I want to apply. And the real/actual weights are the product of "weights"(I supplied) and the converged output "w" (an output). Thanks to all.


title: Cross-correlation significance in R

text: How do you tell if the correlations at different lags obtained from the cross- correlation (ccf function) of two time series are significant.


> **Possible Duplicate:**   >  Cross-correlation significance in R How can I test the significance of the correlation coefficient? I have two time series and I want to test if they are cross correlation. Should I do prewhitening the two series before comuputing the ccf or there are an easy way?


title: How to calculate confidence interval for autocorrelation- and cross correlation functions?

text: I would like to calculate the confidence interval for my autocorrelation function. Does this calculation differ in any way from the normal way of calculating confidence intervals? How would I calculate the CF for cross correlation between two time series? Thank you for any help =)


title: Method of Additional Events for the two-state Markov Process

text: I'm trying to understand a research paper but am facing a few difficulties, any help would be really appreciated. A three state markov process with transition probabilities for states being: 0 --> -1 = λ(1-u)+s 0 --> 1 = λu 1 --> 0 = μ I have a system of Kolmogorov Forward Equations P'-1(t) = (λ(1-u)+s)P0(t) P'0(t) = -(λ+s)P0(t)+ μP1(t) P'1(t) = -μP1(t) + λuP0(t) and I need to find a solution for P0(t) and P1(t) which is probability of the process being in either state 0/1 at time epoch 't'. I was wondering if anyone could go through how to obtain a solution for this system of equations. (If it is relevant the research paper utilises van Dantzigs' method of collective marks, however this is the one step which I do not follow) Any relevant insight would be greatly appreciated! Thanks.


title: Comparing groups at different time points within repeated measures study using SPSS

text: I have done a RCT (randomized control trial) of an intervention (which is of 6 week duration) versus control. The outcome measure (an exercise test) has been tested pre-intervention ( **Pre** ), immediately after ( **Post-1** ), 6 months after ( **Post-2** ), and one year after intervention ( **Post-3** ). I have done a repeated measures ANOVA in SPSS. There is no group difference at **Post-3**. However, looking at the means/plots at each time point there appears to be a difference between the groups at **Post-1**. How would I test to see if there is a significant group difference at **Post-1**?


title: Simple Multivariate Regression with Octave

text: For the last 2 months, I've been trying to do _any_ kind of multivariate regression, but I haven't had much luck. I hear Octave is simpler and more intuitive for these kinds of tasks, so I thought I'd give it a try. Here's a simplified version of the kind of data I'm using:               day = [4, 5, 6, 8]     temp = [97, 100, 98, 80]     humidity [62, 46, 50, 55]      I'd like to use this data and extrapolate the temperature for day 7 and day 9. Are there any tutorials that clearly and concisely explain how to do this with Octave? Or perhaps someone here could tell me which Octave commands to use to do this? Your help would be greatly appreciated.


title: Can someone point me to a good example of using bayesian models for making marketing decisions?

text: I am tasked to build a bayesian model to support decision making for paid search marketing. I've researched online and found several scholarly articles on using Hierarchical Bayesian model or MCMC in estimating model parameters, but I still don't understand details. Is there a good example somewhere, which explains both the model, and how to use a software package to estimate the parameters step by step? Thank you!


title: Jeffreys prior for inverse gamma distribution

text: Does anybody have the experience of dealing with Jeffreys prior? I am working with hierarchical model at the moment where the parameter σ^2 from normal distribution is said to be chosen according to inverse gamma distribution from Jeffreys prior. How should it work? And why are inverse gamma distributions used so often as the prior for variances? I found in some presentations that I can choose Jeffreys prior as, for example, IG(0.001,0.001) But it looks strange cause when both alpha and beta are <1 the gamma function behaves in a strange manner. If I try to generate values from IG(0.001,0.001) in R with code like               a=rgamma(1000,0.001,0.001)     aa=1/a     mean(aa)      I recieve Inf value since all values of a are close to 0 and some of them are exactly 0. So how should one choose Jeffreys prior?


title: Rank features by predictive power in trained classifier (LibSVM)

text: I'm training a SVM classifier on a dataset using LibSVM through Weka, but after I have my trained classifier I'd like to know which features are important to my classifier and which features don't have much use. Currently a pretty mediocre way I'm doing this is running supervised attribute selection in Weka using the Ranker search and some search method (not sure which one would be best, currently using CorrelationAttributeEval), but this isn't specific to my model, it's just stating how well correlated my features are to my classification attribute. I'm not that familiar with SVM, but I'm aware that the trained classifier has some weight values for each attribute. Could I rank attributes by the absolute value of their weights and it'd tell me how well those attributes predict the classification attribute? If that's a valid method, is anyone also familiar with how I could do that (code wise) using LibSVM in Weka? Thanks.


title: How to further reduce predictive error in a regression tree model

text: I had fitted 480 records to a regression tree model, and validated it with a validation data set of 120 records ( I used a 80/20 split ), I calculated the predictive error of MSE to be about 5.6037. Regression tree:               rpart(formula = HeatLoad ~ RelComp + SurfA + WallA + RoofA +          Height + Orient + GlazeA + GlazeD, data = energyE.train)      Variables actually used in tree construction:               [1] GlazeA SurfA       Root node error: `49168/480 = 102.43` n= 480                       CP nsplit rel error   xerror      xstd     1 0.793780      0  1.000000 1.003270 0.0393036     2 0.090992      1  0.206220 0.207071 0.0163478     3 0.025641      2  0.115228 0.118873 0.0101955     4 0.012391      3  0.089587 0.098468 0.0072328     5 0.012206      4  0.077196 0.091905 0.0070751     6 0.011550      5  0.064990 0.087126 0.0069405     7 0.010000      6  0.053440 0.068864 0.0054837      My question is, what can I do to further reduce the predictive error of the mean square error when compared with a validation data set (MSE)


title: Link between linear regression and correlation

text: I happen to do a lot of linear regression, the typical use case is predicting $Y$ with a few predictors say $X = [X_1;X_2;X_3]$. So we we find $\hat{\beta}$ such that $$\hat{\beta} = \underset{\beta}{argmin} ||Y-X.\beta||^2$$ But actually what I am more interested in in finding the linear estimation of $Y$ such that $$corr(Y,\hat{Y}) = corr(Y,X.\hat{\beta})$$ is maximal. I am bothered by the fact that for a given $\beta_0$, $corr(Y,X.a.\beta_0)$ is the same for all $a$, questions are then Is the optimal linear estimator of $\beta$ in the least square sense also optimal in my correlation term, and if yes should I be bothered that the optimal $\beta$ is not unique ?


title: Chi Square Testing - binary outcome analysis

text: I have formed a contingency table for my binary outcome experiment that has been discussed in my previous question : binary outcome testing One of my hypothesis says that females are better at identifying cars and I am trying to use **chi-square testing**. So I get a table like this where I have summed up all predictions:                               | Bike Predicted (actual 100) |     Car Predicted(actual 100)|           Male      |        50                   |             50               |           Female    |        40                   |             60               |      I have 2 questions regarding the above formulation.   1. Is there any way I can account for what the actual image was? I basically want the hypothesis to say that females are better at identify **correct** car images.   2. Using Chi-Square testing I can prove that the tyre prediction is independent/dependent of gender. However, if it is dependent, then can the actual number prove that females are better than males ? If not, can someone help me out on this !


title: Good distribution for modelling age in a population

text: What is a good distribution for describing the age structure of a population? For example, if I want to know the probability that a member of a population is less than or equal to 20. How can I parameterize this given that I know the mean and perhaps the standard deviation of a sample?


title: Relation between logistic regression coefficient and odds ratio in JMP

text: From the output of a logistic regression in JMP, I read about two binary variables:               Var1 estimate -0.1007384     Var2 estimate  0.21528927      and then               Odds ratio for Var1 lev1/lev2 1.2232078 reciprocal 0.8175225     Odds ratio for Var2 lev1/lev2 0.6501329 reciprocal 1.5381471      Now I obtain `1.2232078` as `exp(2*0.1007384)`, and similarly for the other odds ratio. So, my question is: why I have to multiply by two? Is the relation between a coefficient c and its odds ratio r equal to r=exp(c)? Isn't it?


title: Plugging in mean values/proportions to a logistic regression with continuous-discrete interaction

text: I have a logistic regression (in SAS, for reference) with continuous and categorical predictors (with reference coding), and an interaction term between one of each type (assume for now that the categorical variable in question has three response levels, reference coded to $c_1$ and $c_2$): $logit(p) = a + (continuous terms) + (categorical terms) + b_1 (c_1 x) + b_2 (c_2 x)$ where $b_1$ and $b_2$ are the estimated coefficients from my code. From this I can obviously get an expression for the probability $p$ of my outcome. I want to estimate the average p by plugging in the means of the continuous terms and the proportions of the categorical terms. But what do I do with the interaction term(s)? Do I set $(c_i x) = mean(x)$? Or do I set it to $proportion(c_i)mean(x)$?


title: Akaike test and random effect model

text: I am analyzing a panel data (25 years x 30 countries) through OLS, FE, RE and mixed model (xtmixed). The results I get from the LM test and the Hausman show that I have to focus on the RE model. For the hoice of the best-fit model between the RE and the mixed model I have learned that the Akaike test should be implemented. Now, when I run the AIC / BIC test (through the use of commands "est stats" or "fitstat") for OLS, FE and mixed model I have estimated I get the result. I do not get any information when I run the AIC / BIC test for the RE model (apart from the no. of observations and the degree of freedom). I also tried various options to run the AIC test on the RE model but again I receive zero information. Do I miss to do something? I would be grateful if you could help me to catch the right route. Thanks in advance.


title: McFadden's Pseudo-R2 is very high

text: I was running a logistic regression and to measure the relationship between two variables (Categorical). I used the MacFadden's pseudo R-squared as a measure. I found that certain variables show an R-squared value of close to 1. How should I interpret this value? Basically, MacFadden's R = 1-ln(Lm)/ln(L0) Here Lm - Likelihood of the model L0 - Likelihood when using only Intercept So if R is 0.99, it means that Lm is very small compared to L0. Does this mean that the two variables are perfectly correlated?


title: PERMANOVA on categorical data set

text: I am trying to perform a multivariate analysis using R and as I have a ecololgical dataset of categorical data only, is it possible to use PERMANOVA on this type of dataset? Thanks


title: What is meant by categorical distribution?

text: Is this separate type of distribution (EX: Binomial,bernoulli, Multinomial) or any distribution can be represented by this way. Can someone elaborate with simple example


title: Preliminary crosstabs in survival analysis?

text: I am new in survival analysis. I have been taught that you analyze using survival because you have a special dataset: one with censored data. And that if I work only with not censored data, I can have severe bias. But now, when working with a dataset collected for a survival analysis, I am required to crosstab some dichotomous risk factors and death/censored data. I also have seen advice in internet to use logistic regression to predict death with the risk factors. This is supposed to be preliminary work to the real survival analysis. My questions are: ¿Is it a good idea to run crosstabs? Why not univariate Cox if you want crude ratios? Or exploring the data, why not Kaplan Meier instead of crosstabs? (I am not a native english speaker. Please excuse my bad english). Thanks in advance. Florentino Menéndez


title: Caret varImp for randomForest model

text: I'm having trouble understanding how the `varImp` function works for a randomForest model with the `caret` package. In the example below feature var3 gets zero importance using caret's `varImp` function, but the underlying randomForest final model has non-zero importance for feature var3. Why is this the case?               require(randomForest)     require(caret)               rf <- train(x, y,            method = "rf",           trControl = trainControl(method = "oob"),           importance = TRUE,           verbose = TRUE,           tuneGrid = data.frame(mtry = num.predictors) )               fm <- rf$finalModel               > varImp(f)     rf variable importance                 Overall     var1    100.00     var2    80.14     var3    0.00               > importance(fm)             %IncMSE IncNodePurity     var2    872.7935      40505276     var1    1021.4707      55682866     var3     273.0168       3078731


title: Comparing proportions in the same sample

text: I gathered 11 responses to a question where the options were A, B, or C. 9 people answered A. 2 people answered B. 0 people answered C. Using Clopper-Pearson 95% confidence intervals, I obtained that P(A) ranges from (0.48, 0.97), P(B) ranges from (0.2 to 0.52), and P(C) ranges from (0, 0.28). What statements can I make these results? Can I say that P(A) > P(C) at 95% confidence level? Seems like it would be more than that.


title: Proper method for time-staggered/repeated measures data?

text: I have a data set with 300 workers/subjects. There is one dependent variable: a worker's performance, by some metric, during the current week. There are three independent variables (let's call them IV1, IV2, IV3), which are all descriptors of something from the previous week pertaining to that worker. The data is over the course of 52 weeks. What method would you use to determine if any of IV1, IV2, and IV3 influence the DV over the course of the entire time span? Just a standard multiple regression?


title: Modeling stimulus-response output

text: I am trying to analyse some data and I am a bit lost as to what is the correct approach to take. The data are blood hormone levels, taken repeatedly from various subjects, after a certain treatment, which is given at 10', after 2 control samples are taken at time 0' and 10'. So essentially I have some data that looks like this:               time <- c(0, 10, 20, 30, 45, 60)     response.1 <- c(8, 10, 66, 70, 30, 12)     response.2 <- c(5, 2, 20, 30, 10, 12)     response.3 <- c(10, 8, 80, 70, 40, 22)      As you can see the levels are low for the first two control points, then the stimulus is applied, the levels quickly rise and then slowly descend back to baseline over time. I am a bit stuck as to how to model such a situation, especially what I am not sure how to include in the model is the fact that not all the points are equal in terms of exposure to the stimulus (for the first two points the subject did not have the stimulus at all, and the others are at different points in time after the stimulus). The ultimate goal of this analysis would be to tell whether the response is the same for all of the subjects or not (depending on other factors that I will, of course, include in my model). In the example above, subject 2 had a very reduced response.


title: How to report the results of cross-validation for comparing two models?

text: I want to compare the predictive power of two models. For this, I calculated the difference in some measure of predictive performance over many cross- validation replications. Now I have a distribution of this statistic. How should I summarize it for a publication? The distribution "looks normal", the mean is very close to the median and both are positive, indicating that on the average one model outperformed the other. However, the 0.025 quantile crosses zero. Is this relevant? Should I worry about some sort of confidence or prediction interval, and if so, what kind? It is some sort of habit to want a nice interval around each statistic, but isn't the fact that over many thousand replications one model's predictions were superior evidence in itself? Side question: One model is nested within the other. This doesn't mean that the bigger model necessarily performs better on the test partition, does it?


title: Period detection of a generic time series

text: This post is the continuation of another post related to a generic method for outlier detection in time series. Basically, at this point I'm interested in a robust way to discover the periodicity/seasonality of a generic time series affected by a lot of noise. From a developer point of view, I would like a simple interface such as: unsigned int discover_period(vector v); Where v is the array containing the samples, and the return value is the period of the signal. The main point is that, again, I can't make any assumption regarding the analyzed signal. I already tried an approach based on the signal autocorrelation (detecting the peaks of a correlogram), but it's not robust as I would like.


title: Kmeans cluster size change quite a bit on each run

text: I am running a kmeans on a sample size of 1000 data. The data is scaled (z). When I run kmeans(df, nstart=25, centers=5)- it runs and I can get the size of each cluster. The largest group has 620 in it. I ran it again (I did not mean to), but noticed the largest cluster then had 450 in it. Out of interest, I kept trying and got the largest cluster to be anywhere between 450-800. Now I know that the initialization can effect the clusters and cluster sizes. HOWEVER, I am surprised it varies so much. If my cluster sizes changed by 10-20, then I would get it. But these are large swings. Of course I can set.seed to stop it- but it seems too odd to get that wide variation of results. It seems too random that I might get the largest cluster of 450 or 600 or 775 based on when I ran it (or where it initialized). I feel uncomfortable with the results now. Thoughts or explanations.


title: R implementation of Internal Rate of Return and NPV

text: I'm quite new to R and I'd like to know how to implement the Internal Rate of Return and Net Present Value of a series of cash flows efficiently. Right now I'm porting apache's implementation to R as-is, but I feel I could use the apply functions to get faster results.


title: Error in `ezANOVA` with balanced dataset with no missing data

text: This is cross-posted on StackExchange, but as it has to do with statistics, I thought I'd try here. I'm trying to analyze experimental data with one between-subjects factor (`rank`) and two within-subjects factors (`cost`,`msg`) using the `ez` package. Participants were randomly assigned to both `cost` and `msg` at two points in time, so theoretically they could have the same combination of `cost` and `msg` at both points in time. `ezDesign` indicates that **my data is balanced with no missing values.** Suppose my dataset `ez_data` looks like this:                    id choice_num choice        msg           cost   product            rank id_r     1.1   1    1      1 Provincial      Same cost   Cleaner Least important  1.1     2.1   2    1      1    No norm More expensive   Cleaner  Most important  2.1     3.1   3    1      1    No norm      Same cost      Soap Least important  3.1     4.1   4    1      1       Norm      Same cost    Coffee  Most important  4.1     5.1   5    1      1 Provincial      Same cost      Soap  Most important  5.1     6.1   6    1      1    No norm      Same cost Chocolate  Most important  6.1     7.1   7    1      1 Provincial      Same cost   Cleaner  Most important  7.1     8.1   8    1      0 Provincial      Same cost    Coffee Least important  8.1     9.1   9    1      1 Provincial      Same cost   Cleaner Least important  9.1     10.1 10    1      1       Norm More expensive Chocolate  Most important 10.1     11.1 11    1      1       Norm      Same cost      Soap  Most important 11.1     12.1 12    1      0 Provincial      Same cost    Coffee  Most important 12.1     13.1 13    1      1       Norm      Same cost      Soap  Most important 13.1     14.1 14    1      1 Provincial      Same cost      Soap Least important 14.1     15.1 15    1      0       Norm More expensive    Coffee  Most important 15.1     16.1 16    1      1 Provincial      Same cost   Cleaner Least important 16.1     17.1 17    1      1    No norm More expensive Chocolate Least important 17.1     18.1 18    1      0 Provincial More expensive      Soap Least important 18.1     19.1 19    1      1       Norm More expensive Chocolate  Most important 19.1     20.1 20    1      1 Provincial      Same cost   Cleaner Least important 20.1      Based on the comments here, I've created a new ID variable that concatenates the `choice_num` with the `id` variable so that there are no repeated values for the labels for "id" across the different levels of the factors. Then, I simply run the command:               ezANOVA(data=ez_data, dv=.(choice), wid=.(id_r), within=.(cost,msg), between=.(rank), type=3)      But this still yields the error:               Error in ezANOVA_main(data = data, dv = dv, wid = wid, within = within,  :        One or more cells is missing data. Try using ezDesign() to check your data.      Thoughts?


title: Confusion in model representation

text: Will the functional form of a Moving Average model having coefficients `h = [1 0.2 -0.5]` be               y(t) = 0.2e(t-1) - 0.5e(t-2)      Or               y(t) = 1+ 0.2e(t-1) - 0.5e(t-2)


title: e1071 svm predicted probabilties are all 0.5

text: My svm classifier model always predict 0.5 as probabilities.               svm.model <- svm(repeater ~ idRepeatBuyRatio + idTotalPurchase + c + d, data = trainData, cost = 100, gamma = 1)          svm.pred <- predict(svm.model, testData, probability = TRUE)     head(attr(svm.pred, "probabilities"))         t   f     1 0.5 0.5     2 0.5 0.5     3 0.5 0.5     4 0.5 0.5     5 0.5 0.5     6 0.5 0.5      which is strange because the same call without probabilities actually makes different classifications:               > svm.pred <- predict(svm.model, testData, probability = FALSE)     > head(svm.pred)     1 2 3 4 5 6      f t f t f f      Levels: f t      Can someone explain what I am doing wrong?


title: Understanding Support Vector Machine Algorithm

text: I understand the SVM classifying algorithm in my book given the assumptions. My book says, suppose we have these two lines passing through the support vectors of the two classes. They then give the minimizing generalization error argument and proceed to make maximize the margin between the lines passing through the support vectors. **What I do not understand, is how to get the original two lines passing through the support vectors of the two classes?** These lines are really defined by the support vectors.   1. When implementing the algorithm, how do you pick the support vectors?   2. What is commonly done in practice or in many libraries? **EDIT:** What I meant by 'original two lines' is the decision boundary lines. Thanks for all the help!


title: Standardizing data with categorical and quantitative data

text: I'm analyzing a data set as a final project. This is a categorical data analysis course so I will be focusing on a logistic regression analysis. I scrapped together the data set from flight data and weather data. Data is for flights coming into ORD. I want to model flight delay (whether a flight is delayed or not). Here is the a summary of the variables involved:               > summary(flight_data)       DAY_OF_MONTH    DAY_OF_WEEK      AIRLINE_ID    CRS_DEP_TIME      DEP_DELAY          TAXI_OUT         TAXI_IN             Min.   : 1.00   Min.   :1.000   19977  :7838   Min.   : 333.0   Min.   :-21.000   Min.   :  1.00   Min.   :  1.000        1st Qu.: 6.00   1st Qu.:2.000   19805  :6082   1st Qu.: 510.0   1st Qu.: -4.000   1st Qu.: 10.00   1st Qu.:  5.000        Median :17.00   Median :4.000   20398  :3974   Median : 761.0   Median : -1.000   Median : 12.00   Median :  7.000        Mean   :15.17   Mean   :4.058   19386  : 576   Mean   : 760.2   Mean   :  9.072   Mean   : 15.23   Mean   :  8.009        3rd Qu.:24.00   3rd Qu.:6.000   19790  : 553   3rd Qu.: 985.0   3rd Qu.:  7.000   3rd Qu.: 17.00   3rd Qu.:  9.000        Max.   :30.00   Max.   :7.000   20355  : 453   Max.   :1439.0   Max.   :931.000   Max.   :301.00   Max.   :179.000                                        (Other): 866                                                                               CRS_ARR_TIME       AIR_TIME        DISTANCE            Weather.Type     Wind.Speed        Wind.Dir           region          Min.   :   1.0   Min.   : 11.0   Min.   :  67.0   DRIZZLE     :  219   Min.   : 0.000   Min.   : 1.00   midwest  :5856        1st Qu.: 625.0   1st Qu.: 55.0   1st Qu.: 334.0   FOG         :   35   1st Qu.: 6.000   1st Qu.: 8.00   northeast:4487        Median : 865.0   Median : 99.0   Median : 647.0   MIST        :  569   Median : 8.000   Median :20.00   south    :5833        Mean   : 849.9   Mean   :105.8   Mean   : 762.1   None        :17823   Mean   : 8.462   Mean   :19.16   west     :4166        3rd Qu.:1075.0   3rd Qu.:130.0   3rd Qu.: 925.0   RAIN        : 1420   3rd Qu.:11.000   3rd Qu.:29.75                         Max.   :1438.0   Max.   :486.0   Max.   :4244.0   THUNDERSTORM:  276   Max.   :25.000   Max.   :37.00       I have a few questions:   1. I have categorical variables, as well as quantitative variables. Clearly many of the variables are on different scales (time, degrees, etc). Should I standardize my variables? Does it matter? Do I do this to just the quantitative ones?    2. We learned a bit about GAMs in our class. I would like to check whether some of my variables appear to be linear in the log-odds of arrival-delay (if not that would justify use of a GAM). Is this the following code an appropriate approach to this question? Windspeed: `logodds <- NULL x<-NULL for( i in unique(flight_data[,13])){ idx <- which(flight_data[,13] == i) x <- c(x,i) logodds <- c(logodds,log( sum(flight_data[idx,9])/(nrow(flight_data) - sum(flight_data[idx,9])))) } plot(logodds~x)` ![enter image description here](http://i.stack.imgur.com/65Ky0.jpg)


title: multiple chi-squared tests for pairwise comparisons

text: I have one between-subject factor with four levels, and one dichotomous dependent variable. I want to test if each pair within the between-subject factor is related differently with the dependent variable (So if the first level scores more ‘ones’ on the dependent variable than the second level, if the first level scores more ‘ones’ on the dependent variable than the third level etc.). The only way I can think of to do this, is to perform six pairwise (2x2) chi- squared tests. Is this the way to answer my question, and if it is, how can I control for type-1-error inflation?


title: Best way to test for co-occurrence of measures

text: I have some data with temporal measures over time. I'd like to test whether two binary variables co-occur more often than chance would predict, and I'm wondering the best (simple) way to do that. The data looks something like               Time   UID    Variable1    Variable2     1      1      1            0     2      1      1            1     ...     1      2      0            0     2      2      0            1     ...      I'm wondering what the best way is to control for the fact that these are repeated measures within the same individuals. I started with just a chi-squared test of the counts of when the variables co- occurred.


title: fitting an exponential decay onto a regression line

text: I have data for adherence to medicines which follows a downward linear trend for about 6 months (from 100%) and then plateaus at about 50%. Another way of describing it is by saying that adherence reaches steady state (=50%) at about 6 months after starting a medicine (http://www.ncbi.nlm.nih.gov/pubmed/12132975). I have been exploring the effect of an intervention on adherence to medicines and have been using a longitudinal study design and a linear model to do this. However, I would like to try fit an exponential decay into the model to see if I can improve the fit. I'm using geeglm in R. My original model is straightforward y= Bo + B1(time) +B2(trt group) + B3(post) + B4 (time_post) +b5(post*trt_group) + B6 (time_post*trt_group) where post= the intervention occurring at a defined point in time (change in intercept) time_post = change in slope after the intervention. A function that may work to fit the decay is A+Be-kt where A = value at plateau (estimate at 50%) B = difference between plateau value and original intercept value (intercept should typicall be 100% - so B=50%) -k = elimination rate constant t = time I'm not sure how I would fit this into my model. If anyone has any suggestions for how I could work this out, or could point me to some web-help, that would be really helpful. Thank you. ![Plot of raw data and predicted values from linear model](http://i.stack.imgur.com/diKHK.jpg)


title: Rademacher bounds for unbounded loss functions

text: All common treatment of PAC bounds based on Rademacher complexity assume a bounded loss function (for a self-contained treatemnt, see this handout by Schapire. However, I could not find any result for unbounded losses (with finite moments or possibly subgaussian). The only paper I know of is Cortes et al, but it is about _relative_ bounds, not absolute ones. Does anyone know of a result that explicitly covers Rademacher bounds for general loss functions?


title: MAE/MSE with or without square root

text: I read some papers about recommender systems and information retrieval, where Mean Absolut Error and Mean Squared Error are mentioned. But I've found some differences between the formal definition of this two formulars. At one paper the MAE and MSE are WITH a square root and in other papers (or only resources) without. Basically, I saw this versions: $$ MAE_1 = \frac{1}{|TS|} \sum_{(u,i) \in TS} |\hat r_{ui} - r_{ui}| $$ $$ MAE_2 = \sqrt{ \frac{1}{|TS|} \sum_{(u,i) \in TS} |\hat r_{ui} - r_{ui}| } $$ $$ MSE_1 = \frac{1}{|TS|} \sum_{(u,i) \in TS} (\hat r_{ui} - r_{ui})^2 $$ $$ MSE_2 = \sqrt{ \frac{1}{|TS|} \sum_{(u,i) \in TS} (\hat r_{ui} - r_{ui})^2 } $$ I think that the MSE is without the square root, because otherwise the RMSE would be the same. But I'm not sure. With the MAE I'm also not sure. Is the MAE **with** a square root, or is there something like Root Mean Absolute Error (RMAE) as RMSE is for MSE? Hope someone can help me. thx!


title: Using F-tests for variance in non-normal populations

text: I'm fairly new to stats, so please excuse me if this problem is hopelessly elementary or misinformed. Basically, I'm wondering if you can help me understand whether I'm using the F-Test for variance correctly. I have two fairly small populations (n=15), and I'd like to employ a statistical test to determine their respective variances differ significantly. Group A is relatively normally distributed, and group B is skewed left. Originally, I used an F-Test for variance to test for a difference in variance, but then I learned that the F-test can yield false positives if a population isn't normally distributed (as is the case with group B). To remedy this, I created a distribution of sample means for each of the two groups and then performed the F-Test to compare the variance of those distributions. Those distributions of sample means are obviously normal. However, is it appropriate to do an F-Test for variance on those distributions? And how exactly should I interpret the test results? Just in case it's important, the software I'm using is R and the F-test in R is accessed through var.test()


title: How to store (and analyse) multi-answer multi-choice questionnaire data

text: I have just carried out a questionnaire which has a number of questions which are multi-answer and multi-choice. That is, the respondent can tick as many of the options as apply to them - so they may choose no items, one, two, or seven! There is also an 'Other' field for those who find the options don't apply to them. My question is: how should I store this data (within the R language) so that I can analyse it most easily? The sort of analyses I'd like to do are generally fairly simple - looking at the percentages of different responses within the questions, and won't involve much that is hugely deep statistics (I don't really have a large enough sample size to do this). I can think of a number of ways to store the data, as listed below, but I'm not sure which will work best and be easiest to deal with. Other people must have dealt with these issues before and found out how best to do it - so what do you think? Possible ideas I've had include:   * Use a single column of a dataframe for each question, which contains some sort of comma-separated list - easy to input the data, but I suspect it'll be hard to analyse   * Multiple columns for each question - up to the maximum number of answers that I think anyone will tick (5-6 I suspect - but what if I get more?) and put the name of the answer in each of these columns   * A column for each possible answer and use boolean values in the columns to signify what has been ticked. This probably sounds better, but I'm not sure how I'd then go about analysing it. What do people think?


title: Does adding more variables into a multivariable regression change coefficients of existing variables?

text: Say I have a multivariable (several independent variables) regression that consists of 3 variables. Each of those variables has a given coefficient. If I decide to introduce a 4th variable and rerun the regression, will the coefficients of the 3 original variables change? More broadly: in a multivariable (multiple independent variables) regression, is the coefficient of a given variable influenced by the coefficient of another variable?


title: Statistical significance in an underpowered study, false positive?

text: So, I'm actually a biologist trying to wrap my head around the idea of power of analysis to help design an experiment with the proper sample size. I understand that power of analysis is used to help avoid type II errors, but I came across this paper: http://www.benthamscience.com/open/toepij/articles/V003/16TOEPIJ.pdf which seems to say (if I'm not mistaken) that in underpowered studies, you also increase your risk for false positives. The other thing that confuses me, is that if you keep adding more samples to increase your power, of course you make any finding statistically significant, even if your effect is small. Is there some sort of balance between power and number of samples so that one avoids the pitfalls of an underpowered experiment, and also an overpowered experiment where the results will produce a statically significant result that might not be interesting because the effect size is so small?


title: Why does the type I error not depend on sample size?

text: I have read now a couple of times that the rate of false positives is upper bounded by the significance level alpha regarding hypothesis testing, and does not depend on sample size. But how so? As far as I can see, the probability of drawing only extreme values is much smaller when the sample size is 1000 than say 3.. Thanks


title: Emulate correlation in loss data during aggregation

text: I have n (~100) assets with loss data in R, each drawn from its own general Pareto distribution. Let's say that I have 100 observations each, giving us a 100*100 matrix. In order to aggregate these series to a "portfolio", I need to emulate the diversification benefits, i.e., adjust for correlation. I started by creating a random correlation matrix from this question: Can I multiply my 100*100 matrix of uncorrelated data with the transposed Cholesky decomposition of the p.d correlation matrix (also 100*100) in order to achieve my desired effect? (Based on http://www.sitmo.com/article/generating-correlated-random-numbers/) Is this a sensible solution? (Note. I used the nearPD{Matrix} to assure p.d) Alternative solutions are OK, too.


title: Standard error of mean

text: I am learning statistical analysis, and am now confused about calculating the standard error of the mean. My dataset looks like.. Condition A:               animal 1 - 3,4,4,3,6     animal 2 - 5,5,5,8,7      Condition B:               animal 3 - 3,4,5,1,6     animal 4 - 3,1,1,4,8      How can I calculate the standard error of the mean for condition A? I am confused by two methods 1) Simply taking the mean for two animals and use the means of them, like mean for animal 1: 4 mean for animal 2: 6 overall mean : 5 SD: $\sqrt{\frac{\left[\left(6-5\right)^{2} + \left(4-5\right)^{2}\right]}{2-1}} = \sqrt{2}$ SE: $\frac{\sqrt{2}}{\sqrt{2}} = 1$ or maybe using all the measurements like SD: $\sqrt{\frac{\left[\left(3-5\right)^{2} + \left(4-5\right)^{2} + \left(4-5\right)^2 + ...... + \left(8-5\right)^2 + \left(7-5\right)^{2}\right]}{\left(10-1\right)}} = \sqrt{\frac{24}{9}}$ SE: $\frac{\sqrt{\frac{24}{9}}}{\sqrt{10}} = 0.51$ Whichi is right? Thanks in advance.


title: Data about the concentration of sulfur dioxide (linear regression)

text: > > At a measurement station in Munich for 14 days the sulfur dioxide > concentration was measured. Analyze the influence of the mean temperature > per day in grad Celsius ($X_1$) on the $SO_2$-concentration $(Y)$. Is there > a special effect on weekends? The variable $X_2$ is $1$ if it was measured > on a saturday or sunday otherwise it is $0$. The following data were > measured: $$ \begin{array}{c|c|c|c|c|c|c|c} \hline y & -3.15 & -2.83 & -3.02 > & -3.08 & -3.54 & -2.98 & -2.78\\\ x_1 & 16.47 & 16.02 & 16.81 & 22.87 & > 21.68 & 21.23 & 20.55\\\ x_2 & 0 & 0 & 0 & 1 & 1 & 0 & 0\\\ \hline > \end{array} $$ \begin{array}{c|c|c|c|c|c|c|c} \hline y & -3.35 & -2.76 & > -1.90 & -2.12 & -2.45 & -1.97 & -2.23\\\ x_1 & 18.32 & 15.96 & 15.36 & 12.47 > & 12.46 & 11.77 & 11.72\\\ x_2 & 0 & 0 & 0 & 1 & 1 & 0 & 0 \end{array} > **(a)** Estimate the regression coefficients of the multiple linear model > and comment your results. **(b)** As $R^2$ one gets $R^2=0.5805$. Does the > regressors have any role by explaining the $SO_2$-concentration? Make a > global F-test of level $\alpha=0.01$. **(c)** The estimated variances are > $\hat{\sigma}_1=0.0267$ and $\hat{\sigma}_2=0.2166$. Test the hypothesis > $\theta_i=0$ for $i=2,3$ to the level $\alpha=0.05$ by confidence > intervalls. Remove the covariable that does not have an influence and then > make a simple linear regression. Compare the multiple and the simple > regression graphically. Hello and good day! I managed most things of this task already. First of all I examined by ANOVA if there is a weekend effect. I chose $\alpha=0.05$. For the $F$-statstic I got $F=0.1$ and a p-value of $0.76$. So the null hypothesis that $\theta_1=\theta_2$ can not be fefused and so there is no weekend effect. **(a)** The estimated coefficients are $$ \hat{\theta}_1=-1.0019,~~~\hat{\theta}_2=-0.1032,~~~\hat{\theta}_3=-0.0025, $$ where the first coefficient is the intercept, the second coefficient belongs to $X_1$ and the thirds one to $X_2$. Now I do not know what I should comment here? What could be meant? What is to be commented? **(b)** The global $F$-test says that the hypothesis $H_0: \theta_2=\theta_3=0$ can be refused. So the regressors play a role for explaining the $SO_2$-concentration. **(c)** For $\theta_2$ the confidence interval is $$ C_2^{0.05}=(-0.156863,-0.049537), $$ so because $0\notin C_{2}^{0.05}$, the hypothesis $H_0^2: \theta_2=0$ can be refused. For $\theta_3$ the confidence intervall is given by $$ C_3^{0.05}=(-0.437834,0.432834), $$ so because of $0\in C_3^{0.05}$, the hypothesis $H_0^3: \theta_3=0$ cannot be refused, so the covariable $X_3$ does not have an influence and can be removed from the model. I now made the simple linear regression, again using R, getting now $$ \hat{\theta}_1=-1.0020,~~~\hat{\theta}_2=-0.1033. $$ What I see is, that the coefficients are nearly the same compared to those of the multiple regression. But I do not know how to compare the simple linear regression and the multiple linear regression graphically. Of course, I can plot the simple linear regression: ![simple linear regression](http://i.stack.imgur.com/K6suM.png) but I do not know how to plot the multiple regression in order to compare it then with the simple linear regression. Can you help me with that? * * * So to sum it up: Concerning **(a)** I do not know what is to be commented and concerning **(c)** I do not know how I can compare the multiple and the simple regression graphically. Would be great if you could help me with that! See you Miroslav


title: Random Forest mtry Question

text: I am just looking to understand how mtry works in random forests. Please correct me if I am wrong. When you specify mtry (say 10), it takes 10 random variables from your data set and examines them for one tree. So the next tree would take 10 more random variables, examine them, so on and so forth until it runs through the ntrees that you specify and then returns the average estimates for the best/most important variables? Thanks!


title: does a log transformation of the dependent variable affect autocorrelation?

text: I have panel data and have used the xtserial command in Stata to test whether there is autocorrelation. When I take the log of the dependent variable, the test shows that there is autocorrelation. When omitting the log transformation there is no autocorrelation. I cannot see why this is the case? Transformation of the dependent variable should have nothing to do with autocorrelation, right?


title: How to do statistical inference based on randomly sampled discovery/replication sample?

text: I want to compute a score based on the association of several variables with my dependent variable and subsequently test the score's association with the dependent variable in an independent sample. Unfortunately, I don't have a natural discovery and replication sample. However, any split of the sample would seem arbitrary and reviewers might suggest, that I picked the split that delivers the best results. Hence, I thought the straightforward way to approach this would be to randomly sample like 1000 different splits. This means I compute the score in a discovery sample and test it in an independent sample 1000 times. Thereby I get 1000 beta values for the association strength. I would then perform a one sample t-test to see if the 1000 beta-values I get are significantly greater than 0. I've already implemented this in R and it seems to work fine but I'm not totally confident, that this is a valid strategy or if there are any alternative/better strategies. Is it possible to inflate my t-statistic by simply increasing the number of random samples to like 10000 (I guess at some point it becomes asymptotic)? Is this legit? Thanks for your help! I appreciate any pointers since this kind of analysis is rarely done in my field of research.


title: Simple method for cluster analysis

text: One of the academic's at the university where I am studying has conducted research on organisational sustainability. He approached me to turn his research into software that can be used for consulting purposes. The general idea is that the members of an organisation sign up and take the survey online. The survery has 8 dimensions (diversity, innovative potential, conplexity and so on), and each dimension has 10 questions. Each question must be answered on a scale of 1 -5 agree/disagree. There are no averages/totals or calculations taken on the scores. I then need to do a cluster analsysis of some sorts. The requirement is that the number of clusters can be selected, and then a nice diagram is presented showing the clusters, and this will then be used to show if there is divergence/convergence in the organisation, and as a basis for further consulting (i.e remedies if there are clusters of differing values on say diversity). What I need is a suitable simple and mainstream method for the cluster analysis. The academic suggested neural networks but this seemed to be really difficult to use for cluster analysis (statistics is not my major so I am skating on thin ice here). Wikipedia throws up things like K Means and so on, but I am too green in stats to decide what is what. The only requirement that I have is that it must be mainstream and give results that are easy to visualize using a javascript library such as D3.


title: How to determine the Forecast Interval beyond the next period.

text: I like to establish a confidence interval my custom-made forecasting model (not one of the standard ones in the `forecast` package, which include the interval). The empirical record indicates that the forecast errors are normal distributed and thus I can take the point forecast and calculate intervals like you would do with regular normal distribution intervals:               mean +/- z * sqrt(mean squared error)      Can anybody shine some light on what the appropriate way is for treading the error when making forecasts for periods beyond the immediate future?


title: Multiple imputation for variables used to calculate regression weights

text: My basic question: is there anything that you **_can't_** impute using MI? My more complicated question: Consider the regression $Y=\rho T+X'\beta+\epsilon$. For whatever reason, you want to weight the regression. Say that you are weighting by an inverse propensity score, assuming that such would get you to conditional independence and causality. Specifically, $pr(T==1) = logit(Z'\gamma + \eta)$, which gives you weights as $$ w = (T_i/pr(T)+(1-T_i)/(1-pr(T))^.5 $$ Say you're missing elements of Z. Could you set up imputation models for them (using, say, a MICE approach), and use them to construct $w$ which you use to fit the main regression many times? After which you'd combine using Rubin's rules? Is this invalid for any reason? Is there something special about missing model covariates such that those are the only things that can be imputed? I've got a dataset with good coverage in the $X$'s, but a lot of missing values in the $Z$'s.


title: How can a distribution of cross-validated $R^2$ scores be used to determine whether one model is significantly better than another?

text: I have two models, A and B. I have performed 10-fold cross-validation on both of them, so now I have 10 $R^2$ scores for each. How can I determine whether one is significantly better than the other? I fear that calling $A$ the winner iff $$\text{mean}(A)-\text{SEM}(A) > \text{mean}(B)+\text{SEM}(B)$$ is perhaps not the correct way to do it.


title: Tests behind comparison of a sample's survival to that of the general population

text: In the literature there are often reports of comparisons of a sample's survival to that of the general population. Life and mortality tables are cited however the tests and statistical procedures behind this comparison are not. How is this comparison performed? Which tests, procedures and calculations are involved? I am familiar with this question Compare survival of one population to the general population but it does not answer my question. I am seeking more details.


title: In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

text: Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?


title: How do I know when I should use a log transformation on a variable by multiple regression?

text: I am very new to regression analysis. I have a dependent variable and 9 independent variables. I want to build the best possible model from them, and I don't know when to log transfrom a variable.


title: What are the pros and cons of standardizing variable in presence of an interaction?

text: I put this question because while reading the benefits of standardizing explanatory variables or not, I read _good but contrasting_ opinions about standardizing when there are interaction in the model. Some talk about how problems of collinearity are removed when standardizing (e.g. Collinearity diagnostics problematic only when the interaction term is included), which is basically the case of my GLMM. However, others claim that standard errors and p-values of interactions of standardized models are not reliable... (e.g.Variables are often adjusted (e.g. standardised) before making a model - when is this a good idea, and when is it a bad one? or http://quantpsy.org/interact/interactions.htm) So, any ideas on what is the right thing to do?


title: Why are time-related covariates log transformed in modelling?

text: I have seen a lot of examples of transforming time-related variables (e.g., age, year, days, etc.), but I don't understand the reasoning behind doing this. I don't think it's for stabilizing the variance (because it is a covariate). I suppose it's because of something related to linearity. This is even more confusing when the transformed covariate is in a mixed model. **Why are temporal variables often log transformed?**


title: When should you center your data & when should you standardize?

text: In some literature, I have read that a regression with multiple explanatory variables, if in different units, needed to be standardized. (Standardizing consists in subtracting the mean and dividing by the standard deviation.) In which other cases do I need to standardize my data? Are there cases in which I should only center my data (i.e., without dividing by standard deviation)?


title: linear regression scaling variables

text: Should the target variable in a linear regression be scaled along with the predictor variables? It seems I am getting garbage out if not? Also what are the differences in log transformation vs the z-score transformation for the variables? Are there situations where one is more appropriate than the other


title: Do I center an already transformed variable?

text: My chair would like me to center my data for my hierarchical multiple regression, including several moderators and interaction effects. It seems recommended to do this, if not to help minimize multicollinearity, but additionally (and more crucially) to help with interpretability of my interactions. However, where I am uncertain is regarding what to do with some variables I had already thought to transform due to problems with skewness. Do I center these variables? I feel like that might mean transforming them, in essence, twice, but remain unsure. If I don't transform them, how can I use them in an interaction with a centered variable?


title: Which skewness/kurtosis figure do I use in a spatial regression analysis

text: > **Possible Duplicate:**   >  In linear regression, when is it appropriate to use the log of an > independent variable instead of the actual values? I'm currently conducting a spatial regression analysis. I'm trying to decide whether my response variable needs to be log transformed. I understand that regression analyses assume normality of the residuals. I also understand that the best way to calculate how close to normality the data is is by inspecting the skewness/kurtosis figures (closer to 0 the better). The stats package I'm using (S.A.M.) gives two different figures for both skewness and kurtosis. One is for the response variable data and the other is the skewness/kurtosis of the residuals. I'm a bit confused as to which of these figures is the one I should be concentrating on, the response variable skewness or the residuals skewness? I hope this all makes sense. Many thanks for any help Ben


title: Standardizing the variables

text: I am working with an anaerobic treatment plant where I have got main 3 variables, gas production (l/d), flow rate (l/h) and COD influent (g/l). if I want to do multiple l.regression, is it necessary to standardize (gas ~ flow+ COD) the variables?


title: Is it necessary to log transform positively skewed DV’s even though I am using bootstrapping?

text: > **Possible Duplicate:**   >  In linear regression, when is it appropriate to use the log of an > independent variable instead of the actual values? I am running a series of multiple mediation models. Each model includes one IV, two mediators, and a DV. I’m using a macro created for SPSS (provided by Preacher and Hayes) that uses bootstrapping to examine the indirect effects of the mediators. My DV’s are depression and social anxiety (each run in a separate model) and are both positively skewed. Normally I would perform a log transformation on these variables, however, bootstrapping is a nonparametric resampling procedure that does not hold the assumption of normality of the sampling distribution. Therefore, my question is: Is it necessary to transform positively skewed DV’s even though I am using bootstrapping? Note: I have run my models with the log transformed means vs. untransformed means for the DV’s and the pattern of results are essentially the same. However, I would prefer to use the untransformed means as the unstandardized regression coefficients more strongly support my predictions.


title: When do I need to normalize the data?

text: When do I need to normalize the data? What's the problem if I always normalize the data? For example, I am given a set of multivariate gaussian data. I use max. likelihood to estimate the estimate the mean and co-variance matrix. And then, I want to use Bayes classifier to classify new data point. In this situation, is it always good to normalize the data before doing the max. likelihood.


title: Treat categorical variables as continuous under interactions/

text: there are 13 variables which have been regressed using the logit. All the variables are categorical variables. Now my aim is to see the interaction between two variables. When I run my logit, it runs out of degrees of freedom and hence the STATA does not deliver any reslut. Example Q= i.A i.S i.Ec i.Pr i.xx i.dd#i.ii Can I treat my 11 variables as continuous and add the interaction term. If yes, how do I support my doing Example Q = A S Ec Pr xx i.dd#i.ii


title: interpretation of Goodman Kruskal Gamma

text: Value of Goodman-Kruskal Gamma = 1 or -1 means that X and Y are monotonic but not strictly so i.e., $X_1 < X_2$ implies $Y_1 \leq Y_2 $. Can someone please explain this to me with an example?


title: Bayes theorem example - Is a conversion rate the same as a user's probability to convert?

text: In web analytics we often calculate conversion rates for groups of users (number of users who bought / number of users). If we turn this around and a new user lands on a site I believe that this user's probability of converting (given that the user is in a certain group) is not the same as the conversion rate for that group. Looking at conversion rate for the segment of users in the 30-35 age group we have:   * P( conversion | in the 30-35 age group) - What we want to know: Given a new user in the 30 - 35 age group what is his probability to convert?   * P(in the 30-35 age group | conversion) - this is from our records (number of conversion by users in the 30-35 age group / total number of conversion)   * P(converting) - overall conversion rate (users who converted / all users)   * P(in the 30-35 age group) - proportion of all users who are n the 30-35 age group Intuitively we might think that a new user in the 30-35 age group's probability to convert [P(conversion | in the 30-35 age group)] is the same as the conversion rate [P(in the 30-35 age group | conversion)]. But from Bayes theorem: P(conversion | in the 30-35 age group) = P(in the 30-35 age group | conversion) * P(conversion) / P(in the 30-35 age group) Is this the correct way to calculate the probability to convert for a new user in the 30 - 35 age group?


title: P-value multiplied by 2 for two tail hypotheses F-test

text: There is a following answer to an exercise in a text book: The hypotheses are \begin{align*} H_0 &: Var_1 = Var_2\\\ H_1 &: Var_1 \neq Var_2 \end{align*} Computation: $f = 78.800/913.333 = 0.086$. Since $\text{P-value} = 2P(f < 0.086) = (2)(0.0164) =0.0328$ for $4$ and $6$ degrees of freedom, the variability of running time for company $1$ is significantly less than, at level $0.0328$, the variability of running time for company $2$. So the question is why do they multiple $\text{P-value}$ by $2$? I saw they did that when they were using $T$ or Normal distributions but the $F$ distribution is not symmetrical.