title: Ancova: What I can do when assumptions aren't met?

text: > **Possible Duplicate:**   >  ANCOVA and its disturbing assumptions I am testing whether prenatal smoking affects intelligence, while controlling for social economic status. I want to use an ANCOVA (Intelligence - DV, Prenatal smoking - IV and SES -covariate) but I carried out a preliminary homogenity of regression slopes but the results are significant for interaction effect which suggests I should not use an Ancova. I don't think I can use a regression either as I only have one IV. Any ideas?


title: Probability of independent events given the history

text: I am having some trouble understanding something that feels very basic in probability theory concerning the past history of repeated independent events. Consider the tossing of a fair coin. $P(\text{head}) = P(\text{tail}) = 0.5$. Repeated tosses are independent events. Let's say that we already have four Heads in a row. Here's my dilemma: > The probability of the next toss result being head = $0.5$, if I think of > this as an independent event. But the probability of getting five heads in a > row = $0.5^5$ given the history of the tossing. Aren't these statements conflicting? Does the probability of the next toss being a head really differ? So, if were to bet on the outcome, would it be any better to bet on tails? I can think of rationales for both cases but I can't wrap my mind around how both probabilities for a head can co-exist. An explanation would be deeply appreciated.


title: "Correlation" terminology in time series analysis

text: From basic statistics and hearing "correlation is not causation" all the time, I tend to think it's fine to say that "X and Y are correlated" even if X and Y aren't in a causal relationship. For example, I'd normally think it's perfectly okay to say that ice cream sales and swimsuit sales are correlated, since high swimsuit sales probably means high ice cream sales (even though increases in swimsuit sales don't cause an increase an ice cream sales). However, when studying time analysis, I get a little confused about this terminology. It seems like a time series analyst would _not_ say that ice cream sales are correlated with swimsuit sales, but rather that ice cream sales are _spuriously_ correlated with swimsuit sales. An unmodified "X is correlated with Y" seems to be reserved for the case where X actually _causes_ Y, so it's fine to say _temperature_ (but not ice cream) is correlated with swimsuit sales. Is this correct? My problem is that there seem to be two meanings to spurious correlation:   1. Regress two independent random walks against each other, and ordinary statistical tests will say that they're correlated, even though the two random walks are obviously unrelated in any fashion. (I'm fine with this meaning of spurious correlation, since there really is no relationship.)   2. Regress ice cream sales against swimsuit sales. It confuses me that _this_ correlation is called spurious, since there really _is_ a relationship between ice cream sales and swimsuit sales, even though this relationship isn't causal. So I guess my question is: do time series analysts reserve the term "(non- spurious) correlation" for causal relationships -- so that for time series analysts, correlation _is_ meant to suggest causation! -- while statisticians in general are fine with using "correlation" to indicate any kind of (possibly non-causal) relationship?


title: Most powerful GoF test for normality

text: It's well established that both Anderson-Darling and Shapiro-Wilk have a much higher power to detect departures from normality than a KS-Test.   I have been told that Shapiro-Wilk is usually the best test to use if you want to test if a distribution is normal because it has one of the highest powers to detect lack of normality, but my limited experience, it seems that Shapiro- Wilk gives me the same result as Anderson-Darling every time.   I thus have two questions:   * When does the Shapiro-Wilk test out-perform Anderson-Darling?   * Is there a uniformly most powerful lack of normality test, or, barring that possibility, a normality test that out-performs nearly all other normality tests, or is Shapiro-Wilk the best bet?


title: Is including the main effect of the covariate enough?

text: Is simply including a covariate in a model (ANCOVA) enough for variance in the DV due to it to be factored away? Or do the higher order interactions of the covariate with the other factors of the model have to be modelled as well? In some software (e.g. Statistica), you have to specifically ask what factorial effects you want to have tested, and it seems to make a huge difference in the results whether you just ask for the main effect of the covariate or also ask for its crossings with all other categorical IVs. I suppose it's up to the researcher/situation if you would then decompose any of those computed interactions (it would presumably depend on whether you consider the covariate to be 'of interest' or not), but the question is, which specific interactions do you "ask for" in the first place? Is there a trade- off between how informative an effect would be if included in the model VS the cost it would then have on inference power? Also, does the covariate (continuous) factor have the same 'status' among the other (categorical) factors, i.e. is it somehow less important to include covariate-related interactions than it is to include categorical-factor- related interactions? Or does the selection of the requested interactions simply depend on hypotheses rather than being a simple answer such as "all possible interactions should be requested"? Thanks a lot for any help!


title: Kwallis2 insignificant - need for comparisons between the groups?

text: I have a question concerning the Kruskal-Wallis test. I compared 4 groups with the kwallis2 test (in stata) and got insignificant results (the multiple comparisons are aborted). When I look at the medians of the variable in two of the groups, one is double of the other. So I did a kwallis test for these two groups and saw it was significant. How do I deal with this? Is there a reason why the multiple comparisons are insignificant that I would still find significant results when comparing two of the groups?


title: How do I measure the correlation between these two events?

text: I'm trying to measure the influence of one event over the other. Say we have two articles $A$ and $B$. We have knowledge about the users reading these two articles, so I'm trying to measure $P(B|A)$ here. Intuitively, I feel that the larger $P(B|A)$ is, the more influence the event "one has read article $A$" will influence "one will read article $B$". But again $B$ might be a very popular article s.t. $P(B)$ is almost close to 1. So here $P(B)$ is almost equal to $P(B|A)$. I think $P(B|A) - P(B)$ might be something I could possibly use here but I don't know whether it makes any sense mathematically. How should I try to measure these influences better?


title: Performing a Self-Organizing Map on multiple distance matrices in R with a better visualization

text: I have a number of distance matrices that are related to different elements. Each of these matrices represent a specific distance between these elements. I want to perform a Self-Organizing Map technique on them in R and get a Heatmap or something that could present the data well. But the SOM function does not create something that is easily interpretable. I made a sample matrix of 1000*1000 and the `SOM` function of `kohonen` library gave me this: (which I am not sure what does it mean) ![a busy cat](http://i.imgur.com/3j5nDNT.png) The code that I used for R is: `test.som <- som( data = mat , grid = somgrid(3,3,"hexagonal"))` and then I plotted the test.som. My question is how do I include more that one distance matrix in the SOM function and is there a better visualization for this data? P.S. The distance matrices are not Euclidean matrices, they should be calculated one by one.


title: What test is best for comparing diagnostic accuracy

text: I am using a group of observers who will report a certain type of X-ray test on a sample of 50 patients to look for fractures. 2 weeks later the same sample of patients will be reported again, this time using a different type of xray test. I want to find out if their is a statistical difference in the number of fractures detected by each test. Are these matched pairs? Do I use a one sample t test. Help, how do I do it


title: Question on Probability and Binomial

text: I'm doing a self-study. I am currently at part IV of the question. After several attempts, I still can't seem to figure out why $p = 0.75$ for part IV. **QUESTION SHEET** ![enter image description here](http://i.stack.imgur.com/5WeGb.jpg) **ANSWER KEY** ![enter image description here](http://i.stack.imgur.com/vL5Wh.jpg)


title: Kernel PCA for LSI using R

text: is it possible to use kernel PCA for Latent Semantic Indexing (LSI) in the same way as PCA is used? I perform LSI in R using the **prcomp** PCA function and extract the features with highest loadings from the first k components. By that i get the features describing the component best. I tried to use the **kpca** function (kernlib package) but cannot see how to access the weights of the features to a principal component. Is this possible overall when using kernel methods?


title: Applying a post hoc Wilcoxon signed rank test to a Friedman when there are different sample sizes

text: I conducted a Friedman test to related-sample group in three conditions (pre- treatment, post-treatment and 6-month post treatment). Out of 216 participants that completed the first two stages, 50 of these completed the 6-month post stage. Am I right in thinking that the Friedman test is doing the calculation on the 50 participants that completed all stages? My problem is that I then want to conduct a Wilcoxon signed ranks post hoc test to examine where the differences are. N = 216 participants as opposed to the 50 used in the Friedman test. Could anyone tell me if it is ok to use this result? Or do I have to separate the data further so the Wilcoxon test uses the 50 participants? Does this impact its validity? Any help would be greatly appreciated, Sam


title: How to visualize system log data in "phase space"?

text: First I have to tell my experience in development is extremely thin (a few bash scripts that's all) so bear with me if I can't keep up :) A little background story to begin: A few years ago during my sysadmin formation I had the chance to meet a senior security expert which showed me something I found wonderful, phase space visualization of system logs with Octave. (By "system logs" I mean logs from routers/switches/firewalls/webservers/database servers/etc...) That person used this type of visualization to detect anomalies in behavior of computer systems by mapping them "in phases [sic] space" (I don't know if I can employ the term this way.)   The result had generally the aspect of a sphere (when represented in 3D) but protruding from it were "significant events" (anomalies in this case). The same person showed my once how he could map this differently and it had the aspect of Lorenz attractors (This page has images which are very similar to what he showed me) The resulting visualization may seem to have no meanginful output given that I would plot IP addresses against time against URLs for example, remember my goal is only to detect unusual behaviors, those who depart from the general ones. **The question:** The problem I have is I lost the "formula" to plot system logs "in phases space" could you help me to find it ?   I'm not directly interested in working code, the formula in pseudocode to plot those logs would already make me happy :) Given my weak skills in math and development I'm not sure I explained clearly my point so here are some resources I found related to this subject:   * here is an example webserver access log, that's the type of logs i want to plot   * a whitepaper describing anomaly detection, it seems pretty similar to what I want to do   * the abstract of this whitepaper seems to fit the idea too but I don't have access to it PS: an article for math-guys I just stumbled upon, take it like a little present for your kindness to, at least, read my question which I think is a bit blurry due to my lack of skill: https://christopherolah.wordpress.com/2011/08/08/the-real-3d-mandelbrot-set/


title: Breusch-pagan test for regression through the origin

text: When the Breusch-pagan test in R is done on a regression model through the origin with one independent variable then the `df=0` and the `p-value < 2.2e-16`. This looks wrong and in the following link the user experience this problem, Why the Breusch-Pagan test fails if I set the intercept to zero? I have a regression model through the origin, but I have 7 indepedent variables with `df=6`. So in my case it looks ok, but i was wondering if the Breusch-pagan test can be done on regression through the origin models and can i assume the result i get is correct?


title: Weighted clustering on large spatial dataset

text: I have a very large (36k items) spatial dataset of locations of commercial landuses with their corresponding square footages. I am hoping to use the `pam()` command in R (from {cluster} package) to form clusters around a set of centers determined by other methods. I am trying to figure out how to weight the individual points such that large square footages have more attraction to other point than small square footages. My initial thought was to duplicate each point once per 1000 square feet, such that a 100,000 square foot point would be duplicated 100 times. However, I've read elsewhere that the clustering algorithms are computationally intense - the package documentation suggests using `clara()` for large datasets, but this method won't allow me to specify the medoids beforehand. Is there another method for weighted clustering? Am I perhaps going though this all wrong?


title: Visualizing 2-letter combinations

text: The answers to this question on SO returned a set of approximately 125 one- to two-letter names: http://stackoverflow.com/questions/6979630/what-1-2-letter- object-names-conflict-with-existing-r-objects                 [1] "Ad" "am" "ar" "as" "bc" "bd" "bp" "br" "BR" "bs" "by" "c"  "C"       [14] "cc" "cd" "ch" "ci" "CJ" "ck" "Cl" "cm" "cn" "cq" "cs" "Cs" "cv"      [27] "d"  "D"  "dc" "dd" "de" "df" "dg" "dn" "do" "ds" "dt" "e"  "E"       [40] "el" "ES" "F"  "FF" "fn" "gc" "gl" "go" "H"  "Hi" "hm" "I"  "ic"      [53] "id" "ID" "if" "IJ" "Im" "In" "ip" "is" "J"  "lh" "ll" "lm" "lo"      [66] "Lo" "ls" "lu" "m"  "MH" "mn" "ms" "N"  "nc" "nd" "nn" "ns" "on"      [79] "Op" "P"  "pa" "pf" "pi" "Pi" "pm" "pp" "ps" "pt" "q"  "qf" "qq"      [92] "qr" "qt" "r"  "Re" "rf" "rk" "rl" "rm" "rt" "s"  "sc" "sd" "SJ"     [105] "sn" "sp" "ss" "t"  "T"  "te" "tr" "ts" "tt" "tz" "ug" "UG" "UN"     [118] "V"  "VA" "Vd" "vi" "Vo" "w"  "W"  "y"      And R import code:               nms <- c("Ad","am","ar","as","bc","bd","bp","br","BR","bs","by","c","C","cc","cd","ch","ci","CJ","ck","Cl","cm","cn","cq","cs","Cs","cv","d","D","dc","dd","de","df","dg","dn","do","ds","dt","e","E","el","ES","F","FF","fn","gc","gl","go","H","Hi","hm","I","ic","id","ID","if","IJ","Im","In","ip","is","J","lh","ll","lm","lo","Lo","ls","lu","m","MH","mn","ms","N","nc","nd","nn","ns","on","Op","P","pa","pf","pi","Pi","pm","pp","ps","pt","q","qf","qq","qr","qt","r","Re","rf","rk","rl","rm","rt","s","sc","sd","SJ","sn","sp","ss","t","T","te","tr","ts","tt","tz","ug","UG","UN","V","VA","Vd","vi","Vo","w","W","y")      Since the point of the question was to come up with a memorable list of object names to avoid, and most humans are not so good at making sense out of a solid block of text, I would like to visualize this. Unfortunately I'm not exactly certain of the best way to do this. I had thought of something like a stem-and-leaf plot, only since there are no repeated values each "leaf" was placed in the appropriate column rather than being left justified. Or a wordcloud-style adaptation where letters are sized according to its prevalence. **How might this be most clearly and efficiently be visualized?** Visualizations which do either of the following fit in the spirit of this question:   * Primary goal: Enhance the memorizability of the set of names by revealing patterns in the data   * Alternate goal: Highlight interesting features of the set of names (e.g. which help visualize the distribution, most common letters, etc.) Answers in R are preferred, but all interesting ideas are welcome. Ignoring the single-letter names is allowed, since those are easier to just give as a separate list.


title: Assessing performance of linear model with fixed slope

text: I have a very simple linear model that I need to fit with a fixed slope (I'm interested in estimating the intercept). How would I go about describing the performance of the fit, since the usual r^2 doesn't apply. I'm reasonably handy with the math, so even a point in the right direction would be helpful. Doing it in R (lm with offset), in case there's a handy solution there


title: How to calculate an overall job satisfaction score and test group differences?

text: I am doing a project on finding out whether there is a difference in job satisfaction between employees of non-profit organisations and for-profit organisations. I also need to find whether job satisfaction varies as a function of age (categorized as below 20, 21-30 and so on), gender, and type of job (paid or volunteer). Regarding the job satisfaction, I have summed up their answers and labelled them as being satisfied, ambivalent and dissatisfied. Within the job satisfaction, I also measured their satisfaction on different aspects of their job (eg, pay, promotion) and each of these aspects summed up makes the job satisfaction score. I am so confused on how to analyze the data. I have analyzed them using an independent samples t-test but I have just remembered how my supervisor told me I could use factor analysis. But I have no experience in using factor analysis. Can anyone please help me and advice me on how I should analyze my data? @matt parker: thanks so much for your suggestion. I've only realized that I've done the wrong analysis, 1 week before the deadline! I'll try reading up on factor analysis now and _fingers crossed_ that I'll get everything done by next week! :S


title: Do hidden Markov models contain Markov chains?

text: Is it correct to say that the Hidden State Sequence in a Hidden Markov Model is a Markov Chain? Thanks


title: Significance of difference between 2 variables of Cauchy distribution

text: I've got to verify whether there is a significant difference in value market shares between 2 versions of packaging (A and B) of a given beverage: For the 'Packaging A' I've got a sample of 600 purchases on which I compute average 'Packaging A' purchase value as well as the average value of entire category purchase. Both outcomes are of normal distribution so their ratio - value share - is of the Cauchy distribition. Could anyone advice me, how to find a Cauchy distribution statistic by which I could verify the hypothesis of difference between the 2 packaging versions in value shares? (for the Packaging B the sample is 650).


title: Expectation of a function of two variables, not with respect to the joint distribution

text: I am confused by the following expectation which appears within equation (1) of the following paper http://statweb.stanford.edu/~jhf/ftp/trebst.pdf: $E_{X}[E_{Y}[L(Y,F(X))]|X]$ I am confused by the inner expectation, which is a expectation of $L(Y,F(X))$ with respect to $Y$. The result would also be $E_{X}[⋅|X]$ which doesn't make sense to me, since it would be the expectation with respect to the distribution of $X$, while conditioning on $X$.


title: Graphical pattern of short memory ARMA-type component

text: The book of "Time Series and its Applications" gives the following example, ![enter image description here](http://i.stack.imgur.com/GRljj.png) The book also includes the following statement regarding the observation from this figure ![enter image description here](http://i.stack.imgur.com/03IGu.png) I do not know what exactly does it mean for the `short memory ARMA-type component`? Especially, what should be the generic graphic pattern for this so-called `ARMA-type component`


title: I cannot calculate expected value

text: I cannot understand this answer , why we get probability of x=2,3?? - the question tell us that the play end when team won 2 games only not 3 - and why EX[x] is maximized at (d^2 E(x))/dp^2 i need help please thanks in advance ![enter image description here](http://i.stack.imgur.com/G5YtE.png)


title: Text clustering papers with Precision and Recall

text: I have created a text clustering algorithm and calculated the Precision and Recall measures for the evaluation. I am looking for papers that contain other text clustering algorithms evaluations with Precision and Recall in order to compare them. I have already found a few from Google, but I am not satisfied with the number of them. Do you have any to suggest? Update: Upon request, I update my question to include more information. I used TF-IDF for similarity and hierarchical clustering.


title: how do i tranform my ols equation into logs?

text: Hi i'm reading threw all my statistics notes and i can't find any mention of how to transform ols variables into logs. ie i can't see a equation or any method of turning a beta value into a log form. i know why we must do this but i can't work out how. also is there a different method for partial transformations ie turning the equation prices= beat0 +betaone wages +betatwo population growth into y = beta0 plus betaone wages plus log population growth


title: Keyword clustering

text: I have one million of keywords (from search queries in google), and I need to group them semantically. I have already done some research and I have found information about how to extract keywords and cluster them from a large corpus, but in my case I don't have any large documents, only those keywords. I imagine that clustering these keywords semantically is impossible (although I hope I am wrong) since I guess you need a large text to extract its meaning, and each of my keywords has a maximum of 4 or 5 words. I thought about crawling the web and getting myself a large corpus and use some of the techniques I have seen like TF-IDF and then applying a k-means algorithm, and then I could extract the keywords from those documents and its subject, and then I could compare my keywords to those extracted and cluster them accordingly... But I don't know if this would work. Could anyone tell me if my approach is correct? If so, once I have clustered the keywords from the documents I get from the web, what kinds of techniques would I need to cluster my own keywords?


title: Robust outlier detection in curve fitting with correlated errors

text: Assume I have data originating from a model $$ y_i=f(t_i)+e(t_i) $$ with $f\in C_2(\mathbb{R})$. The only thing I know about the errors is that they roughly happen to be of to different sources:   * Some more or less random noise of varying magnitude over time. The noise might just oscillate between different values or may look rather random   * Massive outliers. Those usually are correlated over a few time steps, mostly even lying on a straight line (e.g. $e(t_i)=at_i+b$ for $i=k,k+1,k+2$). In the presence of outliers, the noise usually can be ignored or is even non-existant. As long as there are no outliers involved, I am quite successful fitting B-Splines through the data. However, this approach fails if the outliers have not been removed. All fitting has to be done highly automated, thus manual analysis is no option. One Idea I have for detecting the outliers is to estimate a local measure of dispersion and declare all datapoints as outliers that are much further away form a local estimate on position than that measure. To identify my estimate on position, I have been very successful using a local median as robust estimate. A straightforward extension would be to use a local MAD as measure for dispersion. However, now my theoretical knowledge is stretched to its limits. I do not really have the background to understand what I am doing from an analytical and mathematical point of view. Thus, my question is twofold:   * Do you have a better idea on how to identify outliers? (Just drop me a thought, I will gladly do the research)   * Can you recommend literature giving me theoretical background on rubust dispresion estimation? Even better for models like the one I have sketched. P.S.: While I had quite a few lectures on statistics at university, my background is based on numerical analysis, so please be gentle.


title: Difference between Matrix Factorization and PCA

text: I'm studying Matrix Factorization (to use in Recommender Systems as link predictor) and i want to know if there is any similarity with PCA? The latent features can be compared to the eigenvectors? Thank you


title: Two sample stratified bootstrap

text: I'm interested in using bootstrapping to estimate the confidence interval of the difference between the _actual_ mean length of stay (LOS) for Hospital A and the _expected_ mean LOS at Hospital A. The expected LOS is taken from a reference population and is adjusted for severity. **Below, I've tried to roll my own code, but I suspect this isn't the correct way to deal with this situation. I'd love to use boot() with strata, and weights, but haven't found or created a working example.** Here is a minimal example of the data. Note that the mean LOS for hospital A is 2, but the expected LOS would be 1.43 (Two low severity patients expected LOS of 1 and 1 high severity patient with an expected LOS of 2.3).               hosp    severity  los         A   L   1         A   L   2         A   H   3         Ref L   1         Ref H   2         Ref H   2         Ref H   3      Below is my attempt to code this.               study <- data.frame(hospital=c(rep("A",3),rep("Ref",4)),          severity=c("L","L","H","L","H","H","H"),los=c(1:3,1,2,2,3))     h.mean <- mean(study[study$hospital=="A","los"])         o.mean <- mean(study[study$hospital=="Ref","los"])     set.seed(8)     NSAMPLES <- 10                                     # Number of resample replications     resamples <- NULL     for (sev in unique(study$severity)) {                # Loop for each severity group           hospA  <- study[study$hospital  == "A"  & study$severity==sev,]           hospRef <- study[study$hospital == "Ref" & study$severity==sev,]           resamples <- rbind(resamples,                               sapply(1:NSAMPLES, function(i) sample(hospRef$los,                           size=nrow(hospA), replace=T)))     }     r.mean <- apply(resamples, MARGIN=2, mean)     #mean of each column     q.mean <- quantile(r.mean, c(.025, .10, .25, .50, .75, .90, .975))     print(h.mean)   # Mean actual LOS for hospital A     print(q.mean)   # Percentiles for LOS for hospital A using expected LOS from reference


title: Test of symptom severity equality / symptom configuration equality

text: We have a questionnaire measuring 9 clinical symptoms. The sum scores simply adds all these symptoms up. Symptoms are ordered (0,1,2,3), and pretty strongly skewed (lots of 0s). 1) First, we want to test whether the symptom configuration deviates from the Null-Hypothesis of all symptoms being equally severe (equally common in the population). Our hypothesis is that the data do deviate from this H0. The H0 would posit that each symptom contributes with 11.1% to the sum-score of each participant. If we average each symptom over all participants, average the sum score over all participants, and then divide the sum score by each symptom mean score, we get these values: > 24.6 % 15.7 % 14.3 % 11.3 % 10.6 % 9.7 % 8.5 % 3.9 % 1.6 % As you can see, symptoms vary drastically in severity. The pattern clearly deviates from the H0 pattern that would be > 11.1 % 11.1 % 11.1 % 11.1 % 11.1 % 11.1 % 11.1 % 11.1 % 11.1 % However, we need a statistical test for this and do not know what to use (R or SPSS). 2) As a second question, we want to investigate if the symptom configuration stays stable over time. We have a second measurement point, in which we follow the same procedure to get 9 relative severity scores, one for each symptom. The scores are very similar to the ones from the first measurement point (large symptoms stay large, small symptoms stay small. Once again we wonder how we would best measure stability statistically.


title: Detrending method for nonstationary data

text: Could anybody help me with detrending data that is nonstationary? I have already made the mistake of trying to detrend it by plotting the residuals of a linear regression in excel but it was pointed out to me by 'Zach' and 'cardinal' that it is not a valid method. Unfortunately, I had used that method after many hours of internet searching about detrending data. I have searched for ARIMA models in the context of detrending a line but I only get formulaic results when what I really need is a descriptive answer. I recognize that the math behind it is important, and maybe I can work on it in time, but now I just need to detrend the line. My questions are 1. Is it possible to detrend the nonstationary data in excel. 2. If not, is there a simple free stats software I can download to work with. 3. Is there anyone who might kindly help me in achieving this small objective as I have spent several days trying to solve the problem and I haven't really achieved very much. Any help will be gratefully received.![line](http://i.stack.imgur.com/GahYp.jpg)


title: Experiment design for proportion

text: I would like to test a hypothesis that a database contains data that are correct in 95% or more of the samples. I found this Hypothesis test for a proportion experiment design that looked appropriate. I will randomly sample one-tenth of the population and mark each datum as correct or incorrect. Here is a summary of the design:   1. State the hypotheses: (Null Hypothesis: P >= 0.95, Alternative: P < 0.95)   2. Formulate an analysis plan (Significance level is 0.05)   3. Analyze sample data (calculate the σ and compute the z-score test statistic (z))   4. Interpret results (Reject the null hypothesis iff the P-value is less than the significance level (0.05).) I have a handful of questions:   1. Is this an appropriate test to use if there is no magnitude of error (only that the datum is correct or not)?   2. The StatTrek page says to stop after 10 successes and 10 failures, but to sample no more than 10% of the population. Are 10, 10, and 10 just helpful rules of thumb, or is there some other guiding principle?


title: The range of non-identically distributed binomial variables

text: Lets say we have three independent variables:   $$\eqalign{ X_{1}\sim &B(n,\frac{1}{2}+\beta) \cr X_{2}\sim &B(n,\frac{1}{2}) \cr X_{3}\sim &B(n,\frac{1}{2}-\beta). }$$ I'm looking for the distribution of $$\max(X_{1},X_{2},X_{3})-\min(X_{1},X_{2},X_{3}).$$ For sufficiently large $n$ we have $$\Pr(\max(X_{1},X_{2},X_{3})=k)\approx \Pr(X_{1}=k)$$ and similarly $$\Pr(\min(X_{1},X_{2},X_{3})=k)\approx \Pr(X_{3}=k).$$ So using those two we have: $$\eqalign{ &\Pr(\max(X_{1},X_{2},X_{3})-\min(X_{1},X_{2},X_{3})=k) \cr &\approx \Pr(X_{1}-X_{3}=k) \cr &=(\frac{1}{2}+\beta)^k (\frac{1}{2}-\beta)^{2n+k}\sum\limits_{i=k}^{n}\binom{n}{i}\binom{n}{i-k} \left(\frac{1+2\beta}{1-2\beta} \right)^{2i} }$$ but this doesn't seem to be easy to approximate... I was thinking about another approach. Binomial variables can be approximated by normal distributions and the sum of normally distributed variables is well known, so it would be easy to calculate. **The question** : Are there any inequalities that could give a better approximation? I'm aware only of the Chernoff bound but it is not very helpful here.


title: Getting an average measurement based on two raters for cases where data is missing for one rater

text: ### Context: I'm investigating behaviour in a clinical study involving children. I had both parents and teachers completing questionnaires to inform an understanding of the same underlying constructs, for example reactive aggression. At the conclusion of data collection I have parent data in all cases, n=55, and teacher data for 41 cases - therefore, I have 14 cases where I only have the parent data. For the purposes of our study it makes sense to aggregate parent and teacher observations for each case. As I have a small sample and I do have parent data for every case, deleting cases in a pairwise manner does not seem a viable option. There are several variables for which I need to do this, all of which seem to correlate well. Before aggregating data, I thought it would make sense to address the missing data issue. With regards to 'substituting' I have done some brief reading and am familiar with the basic 'within variable' substitution options, however, I thought there may be a more powerful method available where I could use: (a) the parent score we have in every case for every variable, n=55, 100% of cases. (b) our understanding of the relationship between parent and teacher scores in cases where we have both, n=41, 74.5% of cases ### Question:   * Does the general idea outlined above seem reasonable?   * What would be a good algorithm for implementing it in detail?   * How could it be implemented in SPSS?


title: How to best estimate the time remaining for a variable-length questionnaire?

text: The greatest gain of the statistics classes in both school and university seems to be that I now have an inkling of which QA site to use for this question. :) I'm a programmer and I'm making a questionnaire with a financial background. There will be many (probably over 100) questions, which will be presented to the user one-by-one, like in a wizard. Since it answering them all will be quite time consuming (likely over an hour), it would be best if the user interface could show the estimated time remaining - both for the current question and all remaining together. As there is no way of determining beforehand how long a question will take, my idea was to gather statistical data about it (especially during alpha/beta stages). The naive approach would be to just average the times taken for each question, but (thanks again to the above mentioned statistics classes) I'm aware that this isn't really the best way to do it. There are all kinds of normal distribution curves, error calculations, approximation methods, and other arcane mathematical artifacts that I could not even begin to dream about. Some more information:   * **Questions are of very varying lengths.** There are simple yes/no questions, and there are questions that will make you scratch your head. There are questions that ask to enter your contact details, and there are questions that require to meticulously fill a 30-row table. Some questions will ask you to calculate stuff. Some questions will require you to fetch some paperwork from your employer or something. Which brings up another point:   * **The whole process can be suspended/saved and resumed at a later time.** This shouldn't happen very often, but given the length of the process, and that some people don't pay attention to the pre-requirements, it can happen.   * **Some questions may not be applicable based on answers to previous questions.** For example, there might be a row of questions which are only applicable to businesses and not private persons.   * **Some questions are essentialy duplicated based on previous questions.** For example, if you entered that in the past year you worked for 4 employers, you will need to enter the same data for each of them.   * **I can detect:**     * **When the user starts/finishes a question.** Hence, I can calculate the duration.     * **When the user goes idle.** This is rather imprecise. I can only detect when there is no keyboard/mouse activity. This can be because the user has gone away, but it can also mean that he is digging through the papers looking for numbers. Or (since it will be a webpage), maybe he simply switched to another program - or another tab in the browser.     * **Which questions have been answered, which definately will need to be answered, which definately won't need to be answered, and which _might_ need to be answered.** For the last type of questions, I'm not sure if it might be possible to calculate a meaningful probability based on previous answers. Perhaps. So... any ideas on how to tackle this estimate? Remember - I'm going for maximum accuracy (or is it precision? better make it both), and would rather have better estimates than lower resource usage.


title: standard deviation set for spline cubic regression model

text: I'm attaching two sets I ran in SAS, regarding spline cubic regression models. I'm new to stats and trying my best to learn but I'm stuck here. How do I get SAS to select outliers based on two or 3 standard deviations from the predicted value using the splines? thanks               options ls=75 ps=54 center formdlim ="-";     data a;     input ID $ totalVol DIM Lame;     if dim>325 then delete;     /*if totalVol='.' then delete;     if totalVol>36 then Vol='totalVol';     if totalVol<-2 then Vol='totalVol';*/     datalines;     6   18.6    117 0     6   8.2 118 0     6   18.4    119 0     6   26.2    120 0     6   18  121 0     6   17.9    122 0     6   18.8    123 0     6   18.3    124 0     6   18.5    125 0     6   20.1    126 0     6   20.2    127 0     6   20.4    128 0     6   8.4 129 0     6   21  130 0     6   8.1 131 0     6   13  132 0     6   13.3    133 0     6   8.7 134 0     6   .   135 0     6   9.3 136 0     6   .   137 0     6   21.3    138 0     6   24.7    139 0     6   18.9    140 0     6   24.1    141 0     6   17.9    142 0     6   19.4    143 0     6   18.9    144 0     6   16.6    145 0     6   22.9    146 0     6   21.4    147 0     6   21.4    148 0     6   19.4    149 0     6   16.7    150 0     6   18.8    151 0     6   19.3    152 0     6   20.6    153 0     6   20.5    154 0     6   20.1    155 0     6   19.5    156 0     6   11.1    157 0     6   19  158 0     6   7.3 159 0     6   20.4    160 0     6   19.6    161 0     6   7   162 0     6   7.7 163 0     6   19.6    164 0     6   7.7 165 0     6   19.6    166 0     6   19.2    167 0     6   8.2 168 0     6   6.7 169 0     6   .   170 0     6   7.8 171 0     6   8.4 172 0     6   8.1 173 0     6   19.2    174 0     6   20.5    175 0     6   17.8    176 0     6   .   177 0     6   11.7    178 0     6   5.8 179 0     6   6.9 180 0     6   12.4    181 0     6   .   182 0     6   14.4    183 0     6   17.2    184 0     6   10  185 0     6   16.6    186 0     6   17.2    187 0     6   5.2 188 0     6   20  189 0     6   16.8    190 0     6   9.7 191 0     6   6.7 192 0     6   8.3 193 0     6   18.3    194 0     6   18.9    195 0     6   20.5    196 0     6   10  197 0     6   21.3    198 0     6   19.3    199 0     6   8.6 200 0     6   8.6 201 0     6   20.3    202 0     6   20.3    203 0     6   .   204 0     6   21.5    205 0     6   9.6 206 0     6   .   207 0     6   .   208 0     6   .   209 0     6   7.2 210 0     6   18.8    211 0     6   18.9    212 0     6   19.4    213 0     6   18.8    214 0     6   22.7    215 0     6   11  216 0     6   .   217 0     6   .   218 0     6   .   219 0     6   .   220 0     6   .   221 0     6   .   222 0     6   7.9 224 0     6   19.8    225 0     6   16.6    226 0     6   17.6    227 0     6   16.5    228 0     6   .   229 0     6   8.6 230 0     6   19.7    231 0     6   18.7    232 0     6   17.7    233 0     6   .   234 0     6   21.6    235 0     6   18.9    236 0     6   17.7    237 0     6   20.4    238 0     6   20.7    240 0     6   18.2    241 0     6   19  242 0     6   19  243 0     6   23.5    244 0     6   18.2    245 0     6   18.5    246 0     6   8.4 247 0     6   17.6    248 0     6   11.2    249 0     6   10.7    250 0     6   20.9    251 0     6   15.3    252 0     6   16.6    253 0     6   6.7 254 0     6   18.7    255 0     6   17.6    256 0     6   17.8    257 0     6   7.5 258 0     6   16.8    259 0     6   16.6    260 0     6   .   261 0     6   7.5 262 0     6   6.6 263 0     6   7   264 0     6   14.7    265 0     6   17.5    266 0     6   17.6    267 0     6   16.5    268 0     6   17.5    269 0     6   17.2    270 0     6   11.6    271 0     6   19.1    272 0     6   15.6    273 0     6   18  274 0     6   17.2    275 0     6   15.1    276 0     6   15.5    277 0     6   21.4    278 0     6   17.2    279 0     6   15.5    280 0     6   17.1    281 0     6   14.7    282 0     6   16.1    283 0     6   16.5    284 0     6   16.3    285 0     6   15.9    286 0     6   15.5    287 0     6   14.7    288 0     6   15.5    289 0     6   17.1    290 0     6   16.6    291 0     6   16.7    292 0     6   6.5 293 0     6   15.3    294 0     6   16.2    295 0     6   6.2 296 0     6   15.3    297 0     6   18.6    298 0     6   14.8    299 0     6   15.6    300 0     6   16.6    301 0     6   19  302 0     6   7   303 0     6   14.8    304 0     6   13  305 0     6   14.6    306 0     6   17.3    307 0     6   14.4    308 0     6   9.1 309 0     6   .   310 0     6   .   311 0     6   .   312 0     6   .   313 0     6   10.1    315 0     6   13.6    316 0     6   14.3    317 0     6   13.6    318 0     6   16  319 0     6   14.5    320 0     6   12.3    321 0     6   13.5    322 0     6   15.3    323 0     6   15.3    324 0     6   6.7 325 0     6   6   326 0     6   7.8 327 0     6   15.6    328 0     6   15.7    329 0     6   7.5 330 0     6   16.4    331 0     6   16.7    360 0     6   14  361 0     6   6.8 362 0     6   15.1    363 0     6   14.4    364 0     6   13.6    365 0     8   34.5    74  1     8   11.4    75  1     8   27.3    76  1     8   27.6    77  1     8   28.2    78  1     8   22  79  1     8   28.4    80  1     8   27.5    81  1     8   30.7    82  1     8   28.9    83  1     8   17.7    84  1     8   31.6    85  1     8   10.4    86  1     8   27.4    87  1     8   14.4    88  1     8   29.6    89  1     8   33.7    90  1     8   15.5    91  1     8   .   92  1     8   13.8    93  1     8   27.2    94  1     8   31.7    95  1     8   28.7    96  1     8   23.8    97  1     8   29.4    98  1     8   33.6    99  1     8   29.5    100 1     8   11.3    101 1     8   27.9    102 1     8   28.9    103 1     8   28.4    104 1     8   27.9    105 1     8   15.4    106 1     8   28.4    107 1     8   20.8    108 1     8   21.6    109 1     8   8.8 110 1     8   21.1    111 1     8   20.9    112 1     8   22.3    113 1     8   23.6    114 1     8   23.1    115 1     8   26.3    116 1     8   14.3    117 1     8   21.6    118 1     8   26.1    119 1     8   8.8 120 1     8   27.1    121 1     8   10.7    122 1     8   26.7    123 1     8   20.3    124 1     8   8.5 125 1     8   8.3 126 1     8   8.4 127 1     8   23.7    129 1     8   8.9 130 1     8   8.4 131 1     8   24.6    132 1     8   17.3    133 1     8   9.4 134 1     8   15.5    135 1     8   7.5 128 1     8   31.2    143 1     8   19  144 1     8   21.1    145 1     8   19.9    146 1     8   8.4 147 1     8   14.7    148 1     8   7.3 149 1     8   7.9 150 1     8   22.9    151 1     8   24.6    152 1     8   32.1    153 1     8   9.4 154 1     8   8.7 155 1     8   9.7 156 1     8   23.3    157 1     8   14.9    158 1     8   31.9    159 1     8   9.8 161 1     8   25.4    162 1     8   13.4    163 1     8   .   164 1     8   .   165 1     8   9.2 136 1     8   23.3    137 1     8   27.3    138 1     8   7.9 139 1     8   18.6    140 1     8   13.2    141 1     8   18.1    142 1     8   .   166 1     8   9.8 167 1     8   26.4    168 1     8   11.4    169 1     8   11.8    170 1     8   27  171 1     8   26  172 1     8   14.1    173 1     8   .   174 1     8   .   175 1     8   .   176 1     8   .   177 1     8   .   178 1     8   .   179 1     8   .   180 1     8   9.7 181 1     8   22.2    182 1     8   27  183 1     8   24.8    184 1     8   24.1    185 1     8   26.4    186 1     8   12  187 1     8   22.7    188 1     8   23.2    189 1     8   22  190 1     8   23  191 1     8   25.2    192 1     8   26  193 1     8   36.1    194 1     8   24  195 1     8   19.5    197 1     8   21.7    198 1     8   14.1    199 1     8   20.6    200 1     8   27.8    201 1     8   22  202 1     8   23.5    203 1     8   23.8    204 1     8   23.4    205 1     8   22.3    206 1     8   23.9    207 1     8   29.1    208 1     8   10.3    209 1     8   21.7    210 1     8   22.7    211 1     8   22.5    212 1     8   22  213 1     8   26.5    214 1     8   27.9    215 1     8   23  216 1     8   19.2    217 1     8   24.5    218 1     8   21.6    219 1     8   22.5    220 1     8   20.6    221 1     8   22.2    222 1     8   22.5    223 1     8   22.6    224 1     8   23.1    225 1     8   21.8    226 1     8   21.8    227 1     8   22.8    228 1     8   19.3    229 1     8   21  230 1     8   21.7    231 1     8   20.9    232 1     8   19  233 1     8   21.2    234 1     8   25.1    235 1     8   20.8    236 1     8   21.3    237 1     8   22.3    238 1     8   20.8    239 1     8   19.9    240 1     8   20.8    241 1     8   19.4    242 1     8   22.3    243 1     8   24.1    244 1     8   19  245 1     8   22.1    246 1     8   21.7    247 1     8   21.9    248 1     8   22.3    249 1     8   9.7 250 1     8   20.2    251 1     8   18.9    252 1     8   19.9    253 1     8   21.3    254 1     8   20.8    255 1     8   21  256 1     8   20.7    257 1     8   18.4    258 1     8   17.7    259 1     8   16.4    275 1     8   17.8    276 1     8   17.1    277 1     8   17.7    278 1     8   20.7    279 1     8   16.5    280 1     8   16.9    281 1     8   17.1    282 1     8   20.8    283 1     8   18.2    284 1     8   9   285 1     8   18.4    286 1     8   17.9    287 1     8   20.7    288 1     8   11.8    261 1     8   18.5    262 1     8   18.9    263 1     8   19  264 1     8   18.1    265 1     8   10.5    266 1     8   .   267 1     8   .   268 1     8   .   269 1     8   .   270 1     8   .   271 1     8   7.2 272 1     8   17.7    273 1     8   17.1    274 1     8   20  260 1     8   5   341 1     8   10  342 1     8   18.4    343 1     8   12.6    344 1     8   18  345 1     8   14.3    346 1     8   5.1 347 1     8   11.8    348 1     8   12  349 1     8   13.8    350 1     8   6.4 351 1     8   8.8 352 1     8   8.4 353 1     8   11.7    354 1     8   8.9 355 1     8   10  356 1     8   11.4    357 1     8   10.3    358 1     8   10.8    359 1     8   10.4    360 1     8   12.4    361 1     8   5.4 362 1     8   13.1    363 1     8   7.6 364 1     8   14.3    365 1     ;          proc print data=a ;     where ID='8';     var ID DIM Vol;     run;          proc sort data=a out=new;     where ID='8';     by ID;     run;     /*find the outliers for an identified ID*/     proc ucm data=new;     model totalVol;     autoreg;     level plot=smooth;     splinereg DIM nknots=5 degree=3 variance=0 noest;     season length=12 var=0 noest;     estimate plot=panel extradiffuse=325;     outlier maxnum=10;     forecast extradiffuse=325;     run;      and the other code I find was using the same data set but this               ods graphics on;     proc transreg data=new solve ss2 plots=(transformation obp);     model identity(totalVol) = spline(DIM / degree=3 nknots=40);          run;


title: How to compare two groups of patients with a continuous outcome?

text: I am performing a retrospective study on patients looking at the size of their nostril (continuous variable measured in millimetres) and the need for treatment which is either conservative or surgical (this is a categorical variable).   Sample size is only 15. What would be the right test to compare groups to determine if size of nostril was a significant factor in predicting method of treatment?


title: How can I determine if there's a statistically significant difference between two averages?

text: I'm writing a benchmark program in C# and Java as a first assignment for a CS class. We're supposed to write some kind of report about our methods and results from the benchmark, and I'd like to add a statistical component to mine. I've got 15 run time samples from each language, and the mean and standard deviation from each. How can I determine whether there is a statistically significant difference between them?


title: What is a propensity weighting sampling / RIM?

text: I have come across the sampling method called "Propensity Weighting Sampling/RIM", but I do not have a good idea of what these survey methods are all about. What references in the literature cover this topic?


title: Using ANOVA on percentages?

text: I have a table with four groups (4 BMI groups) as the independent variable (factor). I have a dependent variable that is "percent mother smoking in pregnancy". Is it permissible to use ANOVA for this or do I have to use chi-square or some other test?


title: OLS vs regression / classfication tree

text: I am using regression tree to find the factors effecting food insecurity. I want to know how regression tree is better than OLS in terms of heterogeneity. Can anybody help?


title: What does (SAS) Enterprise Miner's "Bucket Transformation" do?

text: In SAS's Enterprise Miner there is a "Bucket Transformation" which from what I can tell, is a term that is unique to Enterprise Miner. It takes a continuous variable and groups it to become an ordinal variable with a certain number of bins. I would like to know what it is doing and what this would be called in other software. I am attempting to do something like this in python but I am not sure how to make "optimal buckets".


title: How to prove that Manova is a special case of mixed models?

text: I am writing my master's thesis in quantile multilevel regression. My professor all of a sudden decided to change the subject of my thesis into something that could be called "quantile multilevel multivariate regression". He asked me to proof that Manova is a special case of mixed-models (saying that "someone has published something about it, I guess"). The problem is that I haven't found anything yet. We've never taken multivariate courses so I don't even know where to look at! Any kind of help will be appreciated: a link to a paper/book where this proof is showed, or if you want to write it on your own, i'll be more than happy to see it!


title: Inference with only left-censored data

text: Suppose I have a data set that is only left-censored data, ex: > <5, <5, <5, <10, <10, <10 A technique to handle left-censored data is the Kaplan Meier estimate, see page 5 of this excellent article: One problem I will run into is that the KM method relies on _some_ data to have been fully observed. For by above dataset, I have only censorships, so the KM method will return $\hat{F(t)} = 1, \forall t$. This feels wrong to be, as I feel my intuition can give a better inference. For example, looking at the above data, I know that if an individual enters the study, the have more than a 50% chance of having the event of interest occur before time 5. What statistical options do I have if all my data is left censored? ### Additional thoughts Conversely, if all my data is right-censored, I can't make a similar argument as above. The above argument works because I know $T \in [0,5]$.


title: Ranking of multiple results

text: I have a four columns data in a table that looks something like this :               Item  Result_1 Result2_ Result3 Result_4     ========================================     A       77.2    20.2    10.2      2.1     B       32.1    16.1    50.3      4.1     C       40.3    90.2    7.1       6.2      The problem I have is the following :   * Result_1 and Result_3 present values where a _higher_ score is better   * Result_2 and Result_4 present values where a _lower_ score is better Hence I am not sure how to best combine these to enable me to rank.


title: Estimation of the density/distribution

text: Let $(x_i,y_i,z_i)_{i=1,\dots,n}$ be an i.i.d. sample of $(X,Y,Z)$. How one can estimate the following object $$\int_{-\infty}^xf(\bar x,y|z)\mathrm{d}\bar x$$ where $f(x,y|z)$ is a density of $X,Y$ conditional on $Z$. We know that $F(x) = \int_{-\infty}^xf(\bar x)\mathrm{d}x$ can be estimated using $\hat F(x) = \frac{1}{n}\sum_{i=1}^n1[x_i\leq x]$. We also know that $\hat f(x,y|z) = \frac{\frac{1}{h_{xn}h_{yn}}\sum_{i=1}^nK\left(\frac{x-x_i}{h_{xn}}\right)K\left(\frac{y-y_i}{h_{yn}}\right)K\left(\frac{z-z_i}{h_{zn}}\right)}{\sum_{i=1}^nK\left(\frac{z-z_i}{h_{zn}}\right)}$ Can I construct the estimator for the above mentioned object like this? $$\frac{\frac{1}{h_{yn}}\sum_{i=1}^n1[x_i\leq x]K\left(\frac{y-y_i}{h_{yn}}\right)K\left(\frac{z-z_i}{h_{zn}}\right)}{\sum_{i=1}^nK\left(\frac{z-z_i}{h_{zn}}\right)}$$ where $K$ is a kernel function $1$ is an indicator variable, $h$ is a bandwidth and $n$ is a sample size.


title: Defense offense strategy basic

text: I am have to make this basic game engine.The offensive player has 5/6 chance to make a shot(83%).. And defense player has 4/6 chance(57.6%) to defend it.So when offensive player make a shot,how do I calculate the success rate of both players.Thank you for taking time to read this.I am starting new in this.Thanks in advance..


title: Not sure I understand how R calculates the covariance

text: Please forgive this silly question, I'm fairly new to statistics. Consider this R code:                   a = c(1,2,3,4,3,2,3,4,5,5,6,5,4,3,4,5,6,7,8,7,6,6,5,6,7,10,9)         b = c(10,9,7,6,5,6,7,8,4,6,6,5,4,5,6,5,4,5,6,7,5,4,4,5,4,3,2)         mean((a - mean(a))*(b-mean(b)))         [1] -2.42524         cov(a,b)         [1] -2.518519      why are these two values different? Are the mean and expected values not the same thing?


title: Bayesian hypothesis testing of two parameters

text: This question is about hypothesis testing in the Bayesian framework which I am new to. Suppose I have two independent Poisson models with parameters $\lambda_1$ and $\lambda_2$ such that $X \sim Pois(\lambda_1)$ and $Y \sim Pois(\lambda_2)$. $n$ observations are drawn from each of the distributions $X$ and $Y$. If I provide priors $\pi (\lambda_1)$ and $\pi (\lambda_2)$, I can get the posteriors $\pi (\lambda_1 |x)$ and $\pi (\lambda_2 |y)$. My question is how do I test the hypothesis that $\lambda_1 = \lambda_2$ against the alternative hypothesis $\lambda_1 < \lambda_2$ using a Bayesian approach? What I did was to find the 95% credible interval for $\lambda_1$ which I denoted as $C_1$. And then I set my null hypothesis to be $H_0:\lambda_2 \in C_1$ vs $H_1:\lambda_2>max (C_1)$ and test to minimize Bayes risk. Is this an appropriate way of testing the hypothesis? If not, what would be a better way? Thanks.


title: Bias Variance tradeoff from a Bayesian perspective

text: I know the general question about bias variance has been asked before. I understand the frequentist approach and the concept of model selection and the impact of bias and variance on "accuracy" of a prediction. I am looking for an intuitive explanation of bias and variance from a **Bayesian perspective.**


title: Testing for significant difference in mortality rate between multiple groups

text: How do I test for a significant difference between the mortality rate of 4 groups (each group with a different n) group 1: mortality rate = 30.9%, n=55 group 2: mr = 0%, n = 4 group 3: mr = 23.3%, n = 30 group 4: mr = 24.6%, n = 69 is it sufficient to just do a t-test between each group? seems like that would be a convoluted way to present the results.


title: How to cluster direction of traffic data?

text: I have data on many trips (same route). The trip content: `latitude`, `longitude`, and `time at bus stop`. I want to detect the trips that are in the same direction (will have to be two clusters: same direction and opposite direction). Is there an algorithm that can do this?


title: Calculate the tendency of a set of samples

text: I develop an application in which I constantly get samples of heart pulse. I defined an interval of $t$ seconds. In each $t$ seconds I have $n$ samples. In every interval, I want to calculate the tendency of those $n$ samples. For example, lets say I have $n = 5$, and I have samples with values $\\{70, 88, 95, 103, 115\\}$. I want to recognize I have a growth in the heart pulse, and I want to have some measure for the rate of grows/decrease/almost no change. I thought on two approaches for solving this problem.   1. I calculate the linear approximation for the $n$ samples using linear regression by applying least squares implemented by the normal equations. (I treat each sample as two coordinates vector with $x$ coordinate as time and $y$ coordinate as heart pulse). I get from the normal equations a linear function of the form $y = mx+b$ and my measure for the tendency is the slope, i.e. the $m$ value.   2. The second approach is to calculate the correlation (pearson's correlation) between vector $x$ and vector $y$ when $x$ is the time and $y$ is the heart pulse. I'm asking which approach do you think is better for my problem (determining the tendency of heart pulses). Or, do you have a better algorithm for solving this problem?


title: Should I take mean or median to compare short-term and long-term observation?

text: I have a finite set A of discrete values and I have a packet flow (computer networking) where each packet is assigned with a value from A. I want to determine if it is possible to predict the long-term (e.g. after a minute) characteristic (regarding A) of this packet flow by making short-term (e.g. after a few seconds) observations of the first few packets of the flow.   As a first step I want to determine a suitable short-term observation length. Initially, I wanted to gather the packets of the first few seconds of the flow, average their A-values to obtain a short-term mean A-value and compare this mean and its std. dev. with the mean and std. dev. of the long-term observation. The plan was then to increase the short-term observation duration (1/2/3/... sec.) until the short-term std. dev. is within a certain boundary. However, I realized that the distribution of the A-values from the long-term observation is highly skewed (see picture, x axis is A) and now I'm a little bit lost how I should proceed.   ![enter image description here](http://i.stack.imgur.com/f0vl1.png) I read to compare skewed distributions the median is preferred, but I don't know if I can simple switch mean with median and proceed as planned or if there are additional things that should be taken into consideration? Furthermore, do I have to show, as a prerequisite, that the short-term distribution is the same than the long-term distribution (e.g. exponential)?


title: Ambiguity with multinomial logit models

text: I have always thought that, when dealing with multinomial logistic regression, the idea was to linearly model the "logistic" functions of the probability densities of the different response categories (as explained here). I put quotation marks around "logistic" since we do not use the real logistic functions, $\log\left(\frac{\pi_j(x)}{1-\pi_j(x)}\right)$, but with denominator equal to the density of a certain "pivot" category. I then discovered the extension to a log-linear model, where the logarithms of the probability distributions, $\log(\pi_j(x))$, are directly modeled, as explained here. At the end of the day, it is still very similar to the previous assumption. However, I had problems when I discovered the existence of this alternative formulation, which is explained in the `R` package `mlogit`'s vignette (pdf). Basically, every $\log(\pi_j(x))$ is modeled with $\alpha_j+\bar{\beta}\cdot\bar{x}$, where $\alpha_j$ is an intercept which is characteristic of every response category, and the vector of linear coefficients, $\bar{\beta}$, is the same for every category. My question is not maybe a real question, but why this ambiguity?


title: Penalty value in changepoint analysis

text: I'm working with the changepoint R package in R and I understand everything but the penalty value. I know it changes the units of change in the mean, but I still don't know how to interpret it. How do I know how significant is significant? If I put in `pen.value = .20` I get 8 changepoints, but if I put in `pen.value = .30`, I only get 5 changepoints. How do I know if those 3 changepoints are significant? How should I know which value to stick with? And how do you interpret these values?


title: Finding Data in SAS Output for Linear Regression

text: How does SAS calculate t-values? For example, say we have two variables: intercept (which is always present) and temperature in a model that is regressing temperature on mortality in a certain region. t Value -1.39 (Intercept)               (B)  (Temperature)      How might one calculate (B)? I understand that symbolically if temperature is taken to be $b_1$ then we have $\frac{b_1- \beta_1}{s({b_1})}$. I assume they are testing here whether or not $\beta = 0$ so we can make the numerator just $b_1$. Where might one look in a typical SAS output to get this data?


title: Questions on Definitions and Notation (MSE, SSE, Sxx)

text: It is said that $S_{xx} = \sum_{i=1}^n(x_i−\overline x)^2 = \sum_{i=1}^n x_i^2 −n\overline x^2$. I suspect this is simple algebra but I am missing something still. How does this work? Further, Wikipedia mentions that the MSE is $\sum_i \frac{(X_i - \overline X)^2}{n-2}$. However, my text notes that SSE is $\sum_i (Y_i-\hat{Y} )^2$. However it should be the case that $MSE = \frac{SSE}{n-2}$. Can Xs and Ys be used interchangeably like this? It seems wrong to me.


title: Testing binary data

text: I have a data set that is a group of participants who accept or reject each of two devices, and I'd like to test if the two devices are accepted at different rates. Summary table like so               Overall Acceptance             Accept  Reject     X       124     20     Y       111     33      What's the best statistical test to determine if the difference between X's acceptance rate and Y's acceptance rate is significant? I'm unused to binary data so out of my depth here.


title: Confidence Interval for the Population

text: My (basic) understanding of a 95% confidence interval is that it is the interval within which the sample mean would fall 95 times should you take 100 random samples from the population. My query is this, I often see confidence intervals being used in situations that I would call descriptive, rather than inferential. Is this a proper and correct way of using them? For instance, if I measure every single washer in my factory, and plot the mean with the 95% confidence interval, what am I saying? If I retake the 'sample' 100 times, I'll get the same mean 100 times. But I'd argue it's useful to have a confidence interval because it serves a descriptive function in terms of sample size and spread of the data. Let's say I measure all the washers in my factory every month and want to use confidence intervals to see if there is a significant difference between the mean diameter between this month and the previous month. Is this concept flawed because I am looking at populations rather than samples? If I am 100% certain of the mean each month then do questions of significance cease to matter? I'm aware this may be a stupid question, links to suitable reading material accepted gratefully!


title: Can I have an Interaction term with ordinal variables in a multiple regression model?

text: My dependent variable is corruption. I want to test the effect of press freedom and democracy on corruption. The measure for press freedom varies from 1-100 and the democracy variable is ordinal scaled from 1-7. Can I make an interaction term between press freedom and democracy?


title: Property of entropy

text: When characterizing an information measure one desires to have the following 'Grouping' property (cf., Cover&Thomas, Ch.2 exercise 46) $$H(p_1, p_2,\dots, p_n)=H(p_1+p_2, p_3,\dots, p_n)+(p_1+p_2)H(\frac{p_1}{p_1+p_2},\frac{p_2}{p_1+p_2})$$ (a.k.a. recursive). An analogous Grouping axiom is employed for Renyi entropy in Jizba, Arimitzu. Can anybody give an intuitive meaning of it and why it is desired? Also in the axiomatic characterization to inference based on entropy measures one has a set of properties(or axioms)(cf.,Shore and Johnson). Does the above Grouping property have any connection with any of the axioms of entropy based **inference**? In general, is there any connection between the axioms in the axiomatic characterization of entropy(information) measures and the axioms in the axiomatic characterization of inference based on entropy measures? I know some of the connections say the symmetry property of entropy is related to the invariance property of inference, additivity of entropy is related to system independence of inference etc.


title: What could cause big differences in correlation coefficient between Pearson's and Spearman's correlation for a given dataset?

text: The Pearson's coefficient between two variables is quite high (r=.65). But when I rank the variable values and run a Spearman's correlation, the cofficient value is much lower (r=.30).   * What is the interpretation of this?


title: Probability distribution of questions in a forum

text: I would like simulate appearance of publications in a forum and I need know what is the probability distribution of new question being asked in a forum. In my first simulation I used to normal distribution, but I think that the best distribution can be exponential distribution.


title: Are these variables mutually independent?

text: I have a random variable denoted $N$. Then I have random variables denoted $X_1$, $X_2$, ..., $X_N$ distributed according to a uniform distribution. I have also random variables denoted $Y_1$, $Y_2$, ..., $Y_N$ distributed according to this same probability distribution. If we consider the variable $S_X$ such that $S_X=\sum_{i=1}^{N}X_i$ and $S_Y$ such that $S_Y=\sum_{i=1}^{N}Y_i$. Are $S_X$ and $S_Y$ mutually independent ? If we extend this problem to the case of any number $A$ of sums, for instance $S_X$, $S_Y$ and $S_Z$ when $A=3$. Are these $A$ variables mutually independent ? Thank you very much.


title: Multiple mediation analysis in R

text: I am wondering if anyone knows of a way to run a multiple mediation model in R. I know the mediation package allows for multiple simple mediation models, but I want to run one model that evaluates multiple mediation models simultaneously. I am assuming I can do this in an SEM framework (path analysis), but was wondering if anyone new of a package that computed statistics typical of mediation analysis for multiple mediators (indirect effects, Proportion of Total Effect via Mediation, etc.), and could utilize bootstrapping. I know this is a long-shot, but thought I should ask before investing time developing from scratch. UPDATE: (11/11/2013) Since asking this question a couple of years ago, I have learned to use the wonderful R package lavaan to do multiple mediation. here is example code:               model <- '     # outcome model      outcomeVar ~ c*xVar + b1*medVar1 + b2*medVar2          # mediator models     medVar1 ~ a1*xVar      medVar2 ~ a2*xVar          # indirect effects (IDE)     medVar1IDE  := a1*b1     medVar2IDE  := a2*b2     sumIDE := (a1*b1) + (a2*b2)          # total effect     total := c + (a1*b1) + (a2*b2)     medVar1 ~~ medVar2 # model correlation between mediators     '      Note that a1,a2,b1,b2, and c are labels. Then run the model:               fit <- sem(model, data=dataframe)      And look at output:               summary(fit, fit.measures=TRUE, standardize=TRUE, rsquare=TRUE)      Finally, generate bootstrap confidence intervals:               boot.fit <- parameterEstimates(fit, boot.ci.type="bca.simple")      See the lavaan website for more details: http://lavaan.ugent.be/


title: How do I decide whether to use the Wald or F-test in testing the significance of slope coefficient?

text: I have a multiple linear regression model, and need to test the significance of the slopes. My professor says that there are two ways to go about it - so which should I use? The questions in the exam would have the standard error of the regressors along with the estimated value of the regressors as well.


title: Bayesian model comparison: What is it about MCMC that makes RSS or BIC hard to use?

text: I'm trying to figure out why certain methods are used for comparing models in Bayesian statistics. DIC is often used in Bayesian model comparison. However, I'm under the impression that one could calculate likelihood and/or BIC, too. Specifically, I've been told that the calculation of likelihood could depend on 'how one marginalizes over parameters'. What does this mean? Are there other reasons why some people don't prefer BIC?


title: ANOVA and Regression give opposite results in R

text: I conducted an experiment in a factorial design: I measured light (PAR) in three herbivore treatments as well as six nutrient treatments. The experiment was blocked. I've run the linear model as follows (you can download the data from my website to replicate)               dat <- read.csv('http://www.natelemoine.com/testDat.csv')     mod1 <- lm(light ~ Nutrient*Herbivore + BlockID, dat)      The residual plots look pretty good               par(mfrow=c(2,2))     plot(mod1)      When I look at the ANOVA table, I see main effects of Nutrient and Herbivore.               anova(mod1)          Analysis of Variance Table           Response: light                          Df  Sum Sq Mean Sq F value    Pr(>F)          Nutrient             5  4.5603 0.91206  7.1198 5.152e-06 ***      Herbivore            2  2.1358 1.06791  8.3364 0.0003661 ***      BlockID              9  5.6186 0.62429  4.8734 9.663e-06 ***      Nutrient:Herbivore  10  1.7372 0.17372  1.3561 0.2058882          Residuals          153 19.5996 0.12810                            ---      Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1       However, the regression table shows non-significant main effects and significant interactions.               summary(mod1)          Call:      lm(formula = light ~ Nutrient * Herbivore + BlockID, data = dat)           Residuals:           Min       1Q   Median       3Q      Max       -0.96084 -0.19573  0.01328  0.24176  0.74200            Coefficients:                                 Estimate Std. Error t value Pr(>|t|)          (Intercept)                1.351669   0.138619   9.751  < 2e-16 ***      Nutrientb                  0.170548   0.160064   1.066  0.28833          Nutrientc                 -0.002172   0.160064  -0.014  0.98919          Nutrientd                 -0.163537   0.160064  -1.022  0.30854          Nutriente                 -0.392894   0.160064  -2.455  0.01522 *        Nutrientf                  0.137610   0.160064   0.860  0.39129          HerbivorePaired           -0.074901   0.160064  -0.468  0.64049          HerbivoreZebra            -0.036931   0.160064  -0.231  0.81784          ...      Nutrientb:HerbivorePaired  0.040539   0.226364   0.179  0.85811          Nutrientc:HerbivorePaired  0.323127   0.226364   1.427  0.15548          Nutrientd:HerbivorePaired  0.642734   0.226364   2.839  0.00513 **       Nutriente:HerbivorePaired  0.454013   0.226364   2.006  0.04665 *        Nutrientf:HerbivorePaired  0.384195   0.226364   1.697  0.09168 .        Nutrientb:HerbivoreZebra   0.064540   0.226364   0.285  0.77594          Nutrientc:HerbivoreZebra   0.279311   0.226364   1.234  0.21913          Nutrientd:HerbivoreZebra   0.536160   0.226364   2.369  0.01911 *        Nutriente:HerbivoreZebra   0.394504   0.226364   1.743  0.08338 .        Nutrientf:HerbivoreZebra   0.324598   0.226364   1.434  0.15362          ---      Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1           Residual standard error: 0.3579 on 153 degrees of freedom      Multiple R-squared:  0.4176,    Adjusted R-squared:  0.3186       F-statistic: 4.219 on 26 and 153 DF,  p-value: 8.643e-09       I know that this question has been previously asked and answered in multiple posts. In the earlier posts, the issue revolved around the different types of SS used in anova() and lm(). However, I don't think that is the issue here. First of all, the design is balanced:               with(dat, tapply(light, list(Nutrient, Herbivore), length))      Second, using the Anova() option doesn't change the anova table. This isn't a surprise because the design is balanced.               Anova(mod1, type=2)     Anova(mod1, type=3)      Changing the contrast doesn't change the results (qualitatively). I still get pretty much backwards intepretations from anova() vs. summary().               options(contrasts=c("contr.sum","contr.poly"))     mod2 <- lm(light ~ Nutrient*Herbivore + BlockID, dat)     anova(mod2)     summary(mod2)      I'm confused because everything I've read on regression not agreeing with ANOVA implicates differences in the way R uses SS for summary() and anova() functions. However, in the balanced design, the SS types are equivalent, and the results here don't change. How can I have completely opposite interpretations depending on which output I use?


title: Factor analysis on binary data and summated scales

text: We are often confronted with brand attribute associations (Yes/No) in Market Research, and more often than not we need to do Factor Analysis (we use PCA under Factor in SPSS) in order to reduce the data to something useful and interpretable. What is the best method of data reduction when you have binary data? Surrogate variables runs the risk of potentially misleading results by selecting a single variable to represent a more complex result. Factor scores do not seem to be ideal either because the data is binary. And if we are using summated scales, should we use the total or the mean, or perhaps another measure of central tendency?


title: Why does the Breusch-Pagan test fail?

text: How Breusch-Pagan can't reject the null for a series like that?               > x = c(rnorm(10), rnorm(100,sd=10), rnorm(100,sd=25))     > mod = lm(x[-1]^2~x[-210]^2)     > plot(mod$res,type='l')     > bptest(mod)                  studentized Breusch-Pagan test          data:  mod      BP = 1.1085, df = 1, p-value = 0.2924      The variance change drastically, could someone explain the reason? ![enter image description here](http://i.stack.imgur.com/Aidn1.png)


title: Sequential Monte Carlo (particle filter) with Metropolis-Hastings weighting

text: Let's say we are interested in approximating the following expectation: $$\mathbb{E}[h(x)] = \int h(x)\pi(x) dx$$ Where $h(x)$ is an arbitrary function and $\pi(x)$ is a distribution known only up to a normalizing constant. We could approximate this expectation using a Metropolis-Hastings sampler to draw samples $\\{x_t\\}_{t=1}^N$ from a Markov chain with stationary distribution $\pi(x)$, we first define the MH weight function: $$w(x,x') = \frac{\pi(x')T(x',x)}{\pi(x)T(x,x')}$$   * sample $x_0$ from arbitrary proposal   * for $t = 1...N$   * propose $x'$ by sampling from $T(x_t,\cdot)$   * if $w(x_t,x') > r$ (where $r \sim \mathcal{U}(0,1)$) then $x_{t+1} = x'$ else $x_{t+1} = x_t$   * $S \leftarrow S + h(x_{t+1})$ Giving the final approximation: $$\mathbb{E}[h(x)] \approx \frac{S}{N}$$ The question is, can we do a the same 'trick' (sampling from a Markov chain with stationary distribution $\pi$) using sequential Monte Carlo (particle filter) and use weights instead of a simple accept/reject rule. The comparison, in theory, is similar to that of ordinary rejection sampling vs. importance sampling. To do this, consider the following procedure: first sample a set of N particles: $\\{x_0^{(i)}\\}_{i=1}^N$ from arbitrary proposal, then execute the following loop L times:   * for each $x_t^{(i)}$ propose $x_{t+1}^{(i)}$ by sampling from $T(x_t^{(i)},\cdot)$   * weight each $x^{(i)}$ by $w(x_t^{(i)},x_{t+1}^{(i)})$   * $S \leftarrow S + \sum_{i=1}^N w(x_t^{(i)},x_{t+1}^{(i)})h(x_{t+1}^{(i)})$   * resample N new particles $x_{t+1}^{(i)}$ proportional to weights Giving the final approximation: $$\mathbb{E}[h(x)] \approx \frac{S}{N L}$$ This procedure is different than typical sequential importance sampling, or adaptive importance sampling, techniques in a couple of ways. First of all, the proposal distribution ($T(\cdot,\cdot)$) does not change as new samples arrive, also, it is not a proposal that is attempting to be anywhere close to the optimal IS distribution $\frac{|h(x)|\pi(x)}{Z}$. Instead, the proposal is better seen as a Markov transition kernel and must be combined with the appropriate weighting function to ensure that the transitions are consistent with the desired Markov chain. In fact, this procedure is very similar (but not the same) to running $N$ independent Markov chains. The question comes down to a couple issues. First, is this method even correct in it's stated goals, that is: if particles $\\{x_t^{(i)}\\}_{i=1}^N$ are distributed according to $\pi(\cdot)$, after one loop of the above procedure, will $\\{x_{t+1}^{(i)}\\}_{i=1}^N$ be distributed according to $\pi(\cdot)$? Also, if this is true, does this approach offer any advantages over the basic Metropolis-Hastings sampler presented first? Is anyone able to point toward any papers relating specifically to this idea: SMC to sample from stationary Markov chains? I suppose this is technically a MCMC method so, is there such thing as "weighted MCMC" or "weighted Metropolis"? Or is this similar to the idea of "waste recycling" for MCMC methods (I'm not very familiar with this)? * * * Later edits: On further thought it looks like the sequential samples are not distributed according to $\pi(\cdot)$ because the weighting function does not meet the detailed balance condition unless $\pi(x)$ happens to be uniform. This can be seen by considering symmetric transition function $T(x,x')$ $$ \begin{eqnarray*} \pi(x)w(x,x') = \pi(x')w(x'x) \\\ \pi(x)\frac{\pi(x')}{\pi(x)} = \pi(x')\frac{\pi(x)}{\pi(x')} \\\ \pi(x') = \pi(x) \end{eqnarray*} $$ This detail could be fixed by using the exact weighting function $\min(1,w(x,x'))$ and ensuring that each of the previous states is included in the weighted set with weight equal to $1-w(x,x')$. If this is done the method is very similar to just running $N$ Markov chains in parallel. These chains would not be independent per say, as there would be some complex interactions between chains (due to the combined resampling stage); I'm not able to see what the effects of this would be, if any. However, there still might be some merit to doing this because the weighted samples could still be used in the computation of the final expectation. That is, it seems this method still might achieve the goal of a rejection-free MCMC sampler. * * * More edits: I should make another distinction clear. The generalized importance sampling method covered in 14.2 of _Monte Carlo Statistical Methods_ appears to be very close to the SMC procedure I wrote out. However, they are actually quite different. First of all in this method, as with MH, the support for the proposal distribution $T(x,x')$ need not contain the support of the target proposal $\pi(x)$ (as required by Lemma 14.1), rather the support requirements fall out of the ergodicity of the simulated Markov chain. This makes $T(x,x')$ significantly different than the combination of Kernel function $K(x,x')$ and proposal $g(x)$ found in GIS. The population Monte Carlo method, however, seems to subsume the procedure I've mentioned here. The difference being I have not adapted my proposal over time where a PMC method allows a framework for doing so (among other potential advantages). So this looks to be the answer to my question, Yes you can do this and in fact, you can be even more general about your choice of transition kernel (proposal function) than you can with MH. If this is correct, I'm wondering why Metropolis-Hastings and related approaches remain so popular if they can simply be subsumed by the more general PMC method?


title: R package (relaxed lasso for Cox's proportional hazards model)

text: Is there any R package to implement relaxed lasso for Cox's proportional hazards model? Thank you.


title: How to use segmented package to fit a piecewise linear regression with one breakpoint?

text: Sorry to ask a stupid question, but I have problems using the package segmented. My linear regression is very simple, between offer and demand: > linearModel <\- lm(demand~offer) And so should be my model using "segmented": > piecewiseModel <\- segmented(lm(demand~offer), seg.Z = ~ offer, psi = NA) But I have an error message and I really cannot find a reason for this:               Error in seg.lm.fit(y, XREG, Z, PSI, weights, offs, opz) :        (Some) estimated psi out of its range      Here are my data:               demand  offer     1155    39.3     362 23.5     357 22.4     111 6.1     703 35.9     494 35.5     410 23.2     63  9.1     616 27.5     468 28.6     973 41.3     235 16.9     180 18.2     69  9     305 28.6     106 12.7     155 11.8     422 27.9     44  21.6     1008    45.9     225 11.4     321 16.6     1001    40.7     531 22.4     143 17.4     251 14.3     216 14.6     57  6.6     146 10.6     226 14.3     169 3.4     32  5.1     75  4.1     102 4.1     4   1.7     68  7.5     102 7.8     462 22.6     295 8.6     196 7.7     50  7.8     739 34.7     287 15.6     226 18.5     706 35     127 16.5     85  11.3     234 7.7     153 14.8     4   2     373 12.4     54  9.2     81  11.8     18  3.9


title: Does transpose commute through expectation?

text: Working with the generalized covariance formula for vector, $x$, I have: $E[(x-\mu)(x-\mu)^T)] = E(xx^t) - \mu E(x^T)$ But the term $E(x^T)$ doesn't make much sense to me. Does anyone have an idea why I'm getting this term with my matrix algebra?


title: Statistical significance of subsets using non-binary dependent variable

text: I have a table of customer details with demographic information and a "total customer time" that indicates how long the customer has been a customer. I need to be able to query this table in an ad-hoc way, for instance I might ask, "what is the average total customer time for all customers in Florida and with an income < 100K". The demographic information is a mix of nominal, ordinal, and ratio variables, and lots of null values, so a linear\logistic regression isn't appropriate. I'm having difficulty testing the statistical significance of subsets of this kind of analysis. I typically explore like this: What is the average total customer time for all customers?               13,452 customers, 1.7 months      then follow it up with: What is the average total customer time for all customers in Florida?               3,124 customers, 2.5 months      then maybe: What is the average total customer time for all customers in Florida with income < 100K?               2,534 customers, 2.8 months      I know how to setup a binary contingency table and use the chi-squared test when the dependent variable is binary, but here the dependent variable is an average of the ratio-level values in the subset, so I'm stuck. My colleague suggested taking the superset, looking at how many samples will be in the subset, and then pulling that many samples from the superset and calculate the average, but do it 10,000 times. Then I can use the more typical significance tests using the normal distribution. But this seems a bit ... inelegant, not to mention slow to me, and I'm not even sure its a valid approach. EDIT: To be clear, how can I test whether the change in the average of the dependent variable (total customer time) in each subset in each case is statistically significant (at some confidence level) from the average in the superset, given the change in the sample size? Any thoughts will be greatly appreciated.


title: Gelman and Rubin convergence diagnostic, how to generalise to work with vectors?

text: The Gelman and Rubin diagnostic is used to check the convergence of multiple mcmc chains run in parallel. It compares the within-chain variance to the between-chain variance, the exposition is below: **Steps (for each parameter):**   1. Run m ≥ 2 chains of length 2n from overdispersed starting values.   2. Discard the first n draws in each chain.   3. Calculate the within-chain and between-chain variance.   4. Calculate the estimated variance of the parameter as a weighted sum of the within-chain and between-chain variance.   5. Calculate the potential scale reduction factor.   6. List item I want to use this statistic but the variables I want to use it with are random vectors. Does it make sense to take the mean of the covariance matrices in this case?


title: Why does regularization of coefficient magnitude improve the generalization of linear regression?

text: What is the basic argument upon which ridge and lasso regression are based on? I went through Tikhonov regularization wiki where it was mentioned that > In many cases, tikhonov matrix is chosen as the identity matrix , giving > preference to solutions with smaller norms. In other cases, lowpass > operators (e.g., a difference operator or a weighted Fourier operator) may > be used to enforce smoothness if the underlying vector is believed to be > mostly continuous. I want to understand that why are solutions with smaller norms more appealing? Smoothness I can get but why smaller norms?


title: How do I estimate the $e_t$ from a moving average model?

text: I have an `ARIMA(0,2,1)` model. How do i estimate the $\hat{e}_t$ component of the model. I have read a whole lot of theories that confuses me the more. Is there any practical way of estimating this $\hat{e}_t$? Does `R` offer any help to that too? I know that my `ARIMA(0,2,1)` model can be written as $Y_{t} = 2Y_{t-1} - Y_{t-2} + e_{t} + \theta e_{t-1}$. I want to forecast 1-time ahead into the future. In that case, my forecast equation is given as $\hat Y_{t}(1) = 2Y_{t} -Y_{t-1} + \theta \hat e_{t}$. I know my value for $ Y_{t} =7.8$ and $ Y_{t-1} =7.8 $. I know my value of \theta as -0.6816, which i obtained from my `R` output. My problem now is, how do i determine the value for my $\hat e_{t} $ so i could find $ \hat Y_{t}(1)$? I have an `R` code that gives me all these forecasts though, but i want to know how `R` generated my first forecast and how it found the estimate for $\hat e_{t} $. Thanks for looking!


title: Time series properties

text: Has anyone any idea how one could distinguish time series according to certain properties? The only time series properties I know are stationarity/nonstationarity and homoskedasticity/heteroskedasticity. But are there any other possibilities to distinguish time series?


title: Sample simulation from mixture model in R

text: I have this mixture of a shifted binomial and discrete uniform: $$ P(R=r)=\left\\{ \pi\binom{m-1}{r-1} (1-\xi)^{r-1}\xi^{m-r}+(1-\pi){1\over m}\right\\} $$ Having fixed $\xi$, $\pi$, $m$ to be specific values, how can I generate N values for the random variable $R$? Let's say for example N = 100, 300 and 900.


title: Leave-one-out cross validation: Relatively unbiased estimate of generalization performance?

text: I have read that leave-one-out cross-validation provides a relatively _“unbiased estimate of the true generalization performance”_ (e.g. here) and that this is an advantageous property of the leave-one-out CV. However, I don't see how this follows from the properties of leave-one-out CV. Why is the **bias** of this estimator low when compared to others? ### Update: I keep investigating the topic, and I believe it has to do with the fact that this estimator is less pessimistic than, say, K-fold validation, since it uses all the data but one instance, but it would be great to read a mathematical derivation of this.


title: Should I log the quotient of 2 variables, or use the quotient of 2 logged variables?

text: I'm analyzing risk to drivers, i.e. driver deaths/distance driven. Over time distance driven increases (people drive more) while ceteris paribus driver deaths decline (vehicles are safer.) To deal with these unstable variances, should I define risk as               log(deaths/distance)      or               log(deaths)/log(distance)


title: Binomial Conditional Probability of a an event

text: ## Determining Binomial Condition Probability of a Random Sample I have a question about binomial probability involving a conditional event. This problem keeps tripping me up, because while I know how to calculate the binomial probability that a random variable is a failure, i don't know how to calculate the conditional probability of that variable. * * * My question is as follows: **70%** of the total shipments come from **factory A** , of which **10% are defective**. **30%** of the total shipments come from **factory B** , of which **5% are defective**. A random shipment comes in, and a sample of 20 pints is taken, and 1 of the pints is defective. _What is the probability that this shipment came from the Factory A?_


title: Variable selection with groups of predictors that are highly correlated

text: What variable selection approach should I consider if I have thousands of predictors with clusters that are extremely correlated? For example I might have a predictor set $X:= \\{A_1,A_2,A_3,A_4,...,A_{39},B_1,B_2,...,B_{44},C_1,C_2,...\\}$ with cardinality $|X| > 2000$. Consider the case where all $\rho(A_i,A_j)$ are very high, and similarly for $B$, $C$, .... Correlated predictors aren't correlated "naturally"; it's a result of the feature engineering process. This is because all $A_i$ are hand engineered from the same underlying data with small variations in hand-engineering methodology, e.g. I use a thinner pass band on $A_2$ than I did for $A_1$ in my denoising approach but everything else is the same. My goal is to improve out of sample accuracy in my classification model. One approach would just be to try everything: non-negative garotte, ridge, lasso, elastic nets, random subspace learning, PCA/manifold learning, least angle regression and pick the one that's best in my out of sample dataset. But specific methods that are good at dealing with the above would be appreciated. Note that my out of sample data is extensive in terms of sample size.


title: Categorical distribution in JAGS

text: How do I specify for a variable to be uniformly sampled from the discrete values $1,2,\ldots,10$ in JAGS? I guess I want to use the dcat() distribution, but I have yet to find good documentation of the parameters for this.


title: Creating a descriptive table

text: I am having trouble trying to figure out how to report categorical and continuous data in one table. I want to create a descriptive table summarizing some of the categorical and continuous data from the 3 treatment groups I had.


title: Crime data analysis: t-test or time series?

text: I need to analyse objective crime data (indexed) with a time series intervention analysis. However, I have only yearly index from 2003 to 2011. I also have subjective crime data and would like to see if there is a relation between the two different data over time. What analysis would you advise? Also, what does a t-test with this type of data show?


title: Understanding the predictions from logistic regression

text: My predictions coming from a logistic regression model (glm in R) are not bounded between 0 and 1 like I would expected. My understanding of logistic regression is that your input and model parameters are combined linearly and the response is transformed into a probability using the logit link function. Since the logit function is bounded between 0 and 1, I expected my predictions to be bounded between 0 and 1. However that's not what I see when I implement logistic regression in R:               data(iris)     iris.sub <- subset(iris, Species%in%c("versicolor","virginica"))     model    <- glm(Species ~ Sepal.Length + Sepal.Width, data = iris.sub,                      family = binomial(link = "logit"))     hist(predict(model))      ![enter image description here](http://i.stack.imgur.com/0BHU5.png) If anything the output of predict(model) looks normal to me. Can anyone explain to me why the values I get are not probabilities?


title: Scale parameter

text: I came across the scale parameter used in the logit and probit models. Does any one know what that is and what it is used for? What would go wrong if I did not use it?


title: Is there a concept called "contribution" in statistics?

text: I am reading a research report. In that report, they used some elementary statistical methods to analyze their data. There is one quantity they computed called the contribution (of a factor), but they have not provided the definition for that. I am confused. I never heard of the contribution of a factor. It seems that they use these factors to build a linear model, e.g. $y=ax+b$. Does anyone know the meaning in this context?


title: How to compile data from annual reports?

text: I am doing a study of the relationship between two items say A and B. The data consists of information printed on the annual reports from say 2000 - 2010 (10 years). For each item, i.e. A and B there are sub variables. Item A has 3 variables with 9 indicators and item B has 1 variable with 2 indicators. The data for item A are in words. I used content analysis to compute the number of times each indicator was mentioned in each report for each year. For the item B, the data are in percentages so i have 2 different percentage values for each year (10 years). I want to find the relationship (positive, negative or neutral) between these items with item A as my independent variable and item B my dependent variable. How do I go about this? I have been told to use SPSS, but I don't know what method I should use. How should I go about my analysis to determine the relationship between these two items?


title: Variable selection and logistic regression

text: I am using Matlab, I have a $600 \times 9$ matrix with each row representing the 9 features which I am trying to evaluate using logistic regression.   1. I understand that I need to perform feature scaling, but do I need to perform it on both the training and the testing set as well?   2. I have 9 features, up to which degree do I need to perform regularization... up to which higher order degree for 9 features do I need to consider?   3. How do I check which features are contributing more or less?   4. How do I divide my training and testing set, which ratio is the most ideal?


title: Why L1 norm for sparse models

text: I am reading the books about linear regression. There are some sentences about the L1 and L2 norm. I know them, just don't understand why L1 norm for sparse models. Can someone use give a simple explanation? Thanks


title: Multivariate grouping: clustering, anova, tukey

text: I have multiple variables (here: weight, horizontal diameter, price and dummy) related to different factors (here: Apple, Orange, Banana and Avocado):               Fruit   Weight      HorDiam     Price       Dummy     Apple   60      60      5       4     Apple   50      70      8       6     Orange  80      75      7       2     Orange  72      70      9       8     Banana  40      30      3       1     Banana  45      35      4       2     Banana  80      50      8       3     Avocado 100     60      13      8     Avocado 95      70      14      6      I need to test if I can group some species together: are apples and oranges significantly different? ANOVA tells me if weight (or horizontal diameter, or price) is significantly different among species. Tukey test gives me if weight of one species is significantly different from weight of another one (pairwise). Clustering seems only able to group individual observations together, not species. I can't find the appropriate test (or algorithm) to tell me if, for a single variable (weight) or for all of them (weight, horDiam and price), apples can be grouped with oranges and/or with bananas. Any suggestion? I created a _R_ code for this example:               ### CREATE TABLE     Fruit<-c("Apple","Apple","Orange","Orange","Banana","Banana","Banana","Avocado","Avocado")     Weight<-c(60,50,80,72,40,45,85,90,95)     horDiam<-c(60,70,75,70,30,35,50,60,70)     Price<-c(5,8,7,9,3,4,8,13,14)     Dummy<-c(4,6,2,8,1,2,3,8,6)     myData<-data.frame(Fruit=Fruit, Weight=Weight, horDiam=horDiam, Price=Price, Dummy=Dummy)     rownames(myData)<-c("Apple1","Apple2","Orange1","Orange2","Banana1","Banana2","Banana3","Avocado1","Avocado2")               ### ANOVA     fit.aov<-list()     summaryAOV<-list()     for (i in 1:3){       fit.aov[[i]]<-aov(myData[,i+1]~myData[,1])       summaryAOV[[i]]<-summary(fit.aov[[i]])     }               ### TUKEY     par(mfrow=c(1,3))     testTukey<-list()     mainTukey<-c("Weight", "Horiz. Diameter", "Price")     for (i in 1:3){       testTukey[[i]]<-TukeyHSD(fit.aov[[i]], conf.level = 0.95)         plot(testTukey[[i]], main=mainTukey[i])     }               ### CLUSTERING     plot( hclust(dist(myData), method="ward") )               ### CLUSTERING WITH P-VALUE     fit <- pvclust(t(myData[,-1]), method.hclust="ward", method.dist="euclidean")     plot(fit)     pvrect(fit, alpha=0.95)


title: maximum likelihood or else

text: Suppose you throw 1000 darts where each dart has 0.5 probability of scoring. For the first 500 darts each is worth 1 point, for the second 500 darts each is worth 3 points. If you score 1500 points, how many 3 point darts have you scored most likely? If I use the likelihood method, I need to maximize the following function: $$ \binom{500}{k} \binom{500}{1500-3k}$$ where $k$ ranges from $334$ to $500$. The maximum happens at $k = 398$. However, if I try to find the expectation I get a different value. From the question we can model the first dart as a binomial distribution with $B_1(500,0.5)$ and the second dart as a binomial distribution $B_2$ with same parameters. Then we need to find $$ E [B_2 | B_1 + 3B_2 = 1500]$$ which turns out to be $375$. My question is which approach should I choose given that I obtain different values for each?


title: Interpretation of marginal effect of probit model

text: I am quite new with probit models and I recently asked some questions about one of model. I was wondering of one of you could help in interpreting those results. FYI all dependent variables are logged and heteroskedasticity was taken care of using robust standard errors. I then used the "margins, dydx(*)" command to have the marginal effect. My main aim is to see the likelihood of having and institute. ![enter image description here](http://i.stack.imgur.com/IZers.png) Thanks for your time!


title: Are standard deviation, CI95%, mean, median computed in the same way for uncorrelated and correlated (time series) data?

text: My question is pretty simple. Descriptive statistics, like mean, median, sd, etc... for time series data with just 9 points are computed in same way as for uncorrelated data? Or the mathematics is totally different? Thanks