title: How to scale data in a log-lin regression

text: I am currently doing exam revision, and I am stuck on a question in one of the past exams. It asks us to data scale the following equation: $\ln W$ = 2.54 + 0.4`Educ` \+ 0.25`Exper` \- 0.025`Exper`² + 0.29`Tenure` \+ _u_. It asks us to scale _W_ by 1/1000 (convert dollars into thousands of dollars) and `Educ` from years to quarters. In class we have been taught how to scale linear and log-log models but not log-lin. The question asks us to calculate:   1. The new intercept   2. The new `Educ` coefficient   3. The new `Tenure` coefficient. From my understanding I know that $\ln W$ change will be passed onto the slope. However, I am really unsure as to how any of the four above transform. I was hoping someone might be able to explain to me how to do it. Thanks for you help and time. Edit: So as the self-study tag suggested I am going to provide what I think is the best method I have tried: So using the information that $\ln W$ was passed onto the intercept I first of all applied $\ln (W/1000)$ which I then expanded to $\ln(W) - ln(1000)$ Then I got the equation $\ln W$ = 2.54 + $\ln(1000)$ + 0.4`Educ` \+ 0.25`Exper` \- 0.025`Exper`² + 0.29`Tenure` \+ _u_. I was confused after this point because in the textbook it suggested for log- log that the slope are invariant to the changes in scale, however this is log- lin so I assumed that the scale would still apply for `Educ` & `Tenure`. Therefore for `Educ` I used the scaling of 0.4 * `w1`/`w2` : where `w1` = $1/1000$ and `w2` = $1/4$ From this I got 0.0016. Similarly by using the scale `w1` = $1/1000$ and `w4` = $1$ I got $0.00029$ for `Tenure` new coefficient. However other people in my class have informed me this was wrong (although they have a better alternative themselves), most of them arguing that the coefficients of the independent variables should not change because it has a log. Please forgive me for the formatting too. I am relatively new to this.


title: 2d histogram or 3d histogram?

text: Is the following plot a 2d histogram or a 3d histogram? On one hand it has 2 dimensions of data, but on the other, the plot itself is 3 dimensional. ![enter image description here](http://i.stack.imgur.com/fWVZy.png)


title: k-fold classification or re-run experiment?

text: I have a classification model which is slow to run. I generate test data with 3 classes (the real data has 21 at the moment and growing), and I have run the classifier and I get a confusion matrix after holding back on 80 percent (similar to the real data rate) of the labels and then testing the classifier prediction against actual. I need to test the sensitivity of the classifier to the data and I have a choice, a) I can repeatedly generate new datasets J times and classify, or b) I can take a given dataset and do k-fold on the same dataset. What are the advantages and disadvantages of each (running time should be similar enough). Academic references appreciated.


title: Help choosing the optimal time series analysis package

text: I am developing an app for time series analysis that should support the following:   * Exponential Smoothing (Holt-Winters)    * Box-Jenkins    * curve fitting (straight line, quadratic, exponential, growth)   * multiple regression   * Croston's intermittent demand model and discrete data models    * new product forecasting (Bass diffusion) Are you familiar with **open source** stat packages other than R, that supports these models? (python?)


title: How many samples should be taken to capture the mean and variance of the original set with given error?

text: I have a set $S$ of $N$ real numbers. I wanna calculate the mean $m$ variance $v$ of $S$. But since $N$ is too large, I do not want to use all of the numbers. Instead, I would like to sample $n$ numbers uniformly randomly from the $N$-number set $S$, such that mean and variance of the sampled set does not deviate too much from the original set. Say, the error I can tolerate is $e_m$, and $e_v$ respectively (which means that the mean of the sample set should fall in $(1 \pm e_m)m$, and the sample variance should fall in $(1\pm e_v)v$). How large should $n$ be?


title: How to decompose a time series with multiple seasonal components?

text: I have a time series that contains double seasonal components and I would like to decompose the series into the following time series components (trend, seasonal component 1, seasonal component 2 and irregular component). As far as I know, the STL procedure for decomposing a series in R only allows one seasonal component, so I have tried decomposing the series twice. First, by setting the frequency to be the first seasonal component using the following code:               ser = ts(data, freq=48)     dec_1 = stl(ser, s.window="per")      Then, I decomposed the irregular component of the decomposed series (`dec_1`) by setting the frequency to be the second seasonal component, such that:               ser2 = ts(dec_1$time.series[,3], freq=336)     dec_2 = stl(ser2, s.window="per")      I'm not very confident with this approach. And I would like to know if there are any other ways to decompose a series that has multiple seasonalities. Also,I have noticed that the `tbats()` function in the R forecast package allows one to fit a model to a series with multiple seasonalities however, it doesn't say how to decompose a series with it.


title: Interpretation of bootstrap question

text: I've been assigned this question as "homework": > Run a simulation study to examine the performance of the bootstrap for > obtaining 95% confidence intervals on the mean of a univariate sample of > data. > > You should look at confidence interval coverage (what proportion of times > does the confidence interval contain the true mean?) and Monte Carlo > variation (by how much do the upper and lower confidence limits vary between > simulations?). > > You may choose which bootstrap methods to evaluate. The confusing thing is that we've been given no data to use in our simulations. How would I interpret the assignment given this fact?


title: Correct variance for minimum detectable difference

text: I have a question regarding variance, paired testing and minimum detectable difference (MDD). Paired samples: $$ MDD (δ) = \sqrt{ \frac{σ^2}{n} (t_{(α/2,n-1)} + t_{(1-β, n-1)})} $$ I have a set of baseline samples ($A$) sampled at time 0. Because acquiring those samples is destructive, I’m unable to re-sample the same sample twice (similar to for example a soil sample). Therefore I cannot assume that a future re-sampling will have the same variance as the baseline sampling. To measure how big this error is I have taken a second set of samples ($B$) in close proximity to the first samples ($A$), which gives me a number ($n = 15$) of $A-B$ pairs. Sample $B$ is also taken at time 0. If I run a paired t-test on these two sample populations I fail to reject the null hypothesis ($H_0: A\neq B$). I should probably mention that both sample population $A$, $B$ and the difference ($A-B$) are normally distributed. Q1: If I want to estimate the MDD for the baseline mean, how do I integrate this sample error into the equation? Q2: Can I simply take the variance from the baseline sample and add the variance of the differences ($A-B$)? Q3: What if the samples don't reject the null hypothesis? Can I then assume that the samples are independent and use a pooled variance?


title: What is the difference between estimation and prediction?

text: For example, I have historical loss data and I am calculating extreme quantiles (Value-at-Risk or Probable Maximum Loss). The results obtained is for estimating the loss or predicting them? Where can one draw the line? I am confused. Thank you very much for help in advance.


title: Does lots of bias==underfitting, while lots of variance==overfitting?

text: From what I understand, there is a relationship between bias and underfitting; as well as variance and overfitting. Is a 'biased model' another word for an 'underfitted model'? Likewise, is a "varianced' model" another word for an 'overfitted model'?


title: how to use model.avg

text: I have been asking a number of separate questions as each are unique in themselves but connected to my learning process in running mixed effects models. I apologise if i am becoming a nuisance. Basically I learned how to run a model correctly using lme4 (an lmer) and as I had many factors I was then able to use the MuMIn package to identify those models that best explained y rather than having to run them all by hand. I would like to better understand Effect and then 95% CI using the model.avg command. I have 9 factors (including the random but not y). After using the dredge command I ran: > get.models(dd, subset=delta<4) where dd is the data.frame from the dredge command (i.e. all possible combinations of factors in the global model). I ended up with 14 models listed (out of over 200), although only the top two had a delta<2. I know that papers will often report the importance of these factors, so would it be advisable to find the Estimate and 95% CI for all the factors for all 14 models or should I just look at those factors within the top two models (ie with a delta<2)? I think the two models is what other papers have done. Could someone tell me how to do this? Do I run all models that are in the top (either 2 or 14) and then use the avg.model command? Could someone let me know what the code is for this? In addition, once I do this, would it be okay to then list my results below so that someone could help me decipher what they mean? I don't have a deep understanding of the intricacies, but it would be helpful for me to be able to further explain the magnitude of the effects of the most important factors rather than just the weights of the models themselves. I understand AICc and weights, but not the t values and such that come after each model. I appreciate everyone's patience with me in this.


title: Difference between G-test and t-test and which should be used for A/B testing?

text: The G-Test: http://en.wikipedia.org/wiki/G-test is a way to get quick estimates of a chi squared distribution, and is recommended by the author of this well-known A/B test tutorial: http://elem.com/~btilly/effective-ab- testing/ This tool assumes a normal distribution: http://www.thumbtack.com/labs/abba/ and uses difference of means to compute confidence. What is the difference between a G test and a T test? What are the benefits or downsides to using each method to measure the effectiveness of our A/B tests? I'm trying to figure out which one I should use to measure the results of my A/B test framework. Our framework has two general use cases: split the group of visitors evenly, show each one a different feature and measure their conversion on some other page (say, the sign up page); and split the group of visitors into the control group (90%) and an experimental group (10%) for a test, and measure conversions on some other page. Our website gets between 1000 and 200,000 visits per day (I am being vague on purpose to hide the true number, which doesn't change much). These visits are split with an exponential distribution across about 300 pages. Thanks, Kevin


title: Analyzing installation rates, not quite binomial. Tried bootstrapping. Is there a approximate binomial approach that would be better?

text: I'm analyzing a data set of installation rates of lighting fixture projects and want to determine the expected installation rate and the confidence interval. Projects have various numbers of fixtures intended to be installed and the most common rate is 100% followed in frequency by a handful of 0% and then a small number of rates besides 0 and 100%. The distribution of the sample data looks like this: ![enter image description here](http://i.stack.imgur.com/zvKFL.png) In many ways it seems like this data should follow a binomial distribution except for the variable number of fixtures in each project and the possibility of values other than 0 and 1. As a result of this confusion, I simply bootstrapped the average installation rates, weighted by project size, and arrived at this distribution of results. ![enter image description here](http://i.stack.imgur.com/Azhqg.png) Is this a reasonable approach, or is there a better way to treat this as an approximate binomial distribution with weights for project size, or use a non- parametric approach? Thanks in advance for any assistance...


title: Reliability standard via relative standard error for computer science or randomized algorithms

text: I am performing experiments on randomized algorithms, and want to do it scientifically sound. What relative standard error of mean (RSEM) is acceptable? I know that often RSEM<30% or RSEM<25% is used, but does that also apply for randomized algorithms? What paper/book can I cite that give a reliability standard? This would optimally be a standard for the field of computer science or randomized algorithms, but a general reliability standard would be fine, too. * * * Details: I only found   * http://www.cdc.gov/nchs/data/statnt/statnt24.pdf, which unfortunately is only for the health sector;   * http://scholar.google.de/scholar?cluster=1360212642214424308&hl=en&as_sdt=0,5 covering statistics for randomized algorithms, but it unfortunately does not mention standard errors.


title: Training a convolution neural network

text: I am currently working on a face recognition software that uses convolution neural networks to recognize faces. Based on my readings, I've gathered that a convolutional neural network has shared weights, so as to save time during training. But, how does one adapt backpropagation so it can be used in a convolution neural network. In backpropagation, one uses a formula similar to this to train the weights.               New Weight  = Old Weight +  LEARNING_RATE * 1 * Output Of InputNeuron * Delta      However, since in convolutional neural networks, the weights are shared, each weight is used with multiple neurons, so how do I decide which `Output of InputNeuron` is used? In other words, since the weights are shared, how do I decide how much to change the weights by?


title: In responses collected from online survey, how to eliminate multiple representations from same company

text: Problem statement: In responses collected from online survey, How to eliminate multiple representations from same company Assuming that I intend to get one (and only one) response from each company for analysis. In the process of survey, I sent out to a large distribution and there are multiple responses from single company. What is the best process to pick one response before starting the analysis. (e.g., Company 1 - 10 responses, Company 2 - 15 responses, Company 3,4,5 - 1 each). Unit of Analysis is Company. How to pick 1 response from Company 1 and 2 so that the analysis is not biased. Some thoughts a. Pick a response randomly and ignore the others b. Follow below strategy to narrow i. Ignore partial responses ii. Go-Thro' answers and remove not worthy responses - Like Yes for All etc iii. Choose based on responses to priority questions iv. Finally choose based on designation c. Apply some logic to group the responses into a single response Please provide the best way to handle this issue and provide references.


title: Transformation to normality for random variables with different locations

text: I have a (potentially infinite) sequence of random variables $X_i$, with $i = 1, 2, \dots$, which have the same distribution (in terms of "shape"), but different locations. I have a sample of size $N$ from only one of these distributions and I would like to find a transformation to normality that works for the whole sequence of distributions. The locations are all unknown as well the analytic form of true distributions. All I have is a sample from one element of the sequence. Notice that I need a transformation that uses the same parameters for every distribution. For example if, give my initial sample, I decide to use the Box- Cox($\lambda_1$, $\lambda_2$) transform, when I get a new sample I have to transform it using exactly the same transform (otherwise I would simply shift each sample to zero). So far I haven't found anything that works: Box-Cox requires positivity of the sample (the shifting parameter doesn't help because the location varies from sample to sample). Similar problem for the Inverse Hyperbolic Sine transform. Any suggestion is welcome! EDIT: This problem arises from an application where I have a data $y_1, \dots, y_n$, generated by a stochastic model $m(\theta_0)$, where $\theta_0$ is an unknown parameter. I'm trying to estimate $\theta_0$ by Maximum Likelihood. At each step $k$ of the optimization I'm simulating a sample $x_1, \dots, x_n$ from the model $m(\theta_k)$, and I rely on the sample being approximately normally distributed to estimate the likelihood. Unfortunately the sample often has departures from normality, which is why I'm interested in transformations. I need to use exactly the same transformation across the parameters space in order to compute the likelihood correctly. For different $\theta$s the mean of the sample changes, but shape of the distribution seems to be fairly constant.


title: Why doesn't the method of moments work when calculating the variance of the inverse gamma distribution?

text: I'm trying to calculate the variance of the inverse gamma distribution using the method of movements. According to wikipedia the variance should be: $$\sigma^2 =\frac{\beta^2}{(\alpha-1)^2(\alpha-2)}$$ Where $\alpha$ is the shape and $\beta$ is the scale of the inverse gamma distribution. When trying this out in R it works reasonably well when $\alpha$ is not to close to 2. For example:               library(MCMCpack) # for the rinvgamma function     a <- 10     b 100          # The variance according to the method of movements     b^2/((a-1)^2*(a-2))      ## 15.4321          # The variance by generating inverse gamma distributed random numbers and     # calculating the sample variance     var(rinvgamma(n=9999, shape=a, scale=b)) #     ## 15.84388      But when $\alpha$ gets close to 2 the method of movements doesn't seem to work anymore. In the following example the sample variance is much smaller than the method of movements variance:               a <- 2.2     b <- 100          # The variance according to the method of movements     b^2/((a-1)^2*(a-2))      ## 34722.22          # The variance by generating inverse gamma distributed random numbers and     # calculating the sample variance     var(rinvgamma(n=9999, shape=a, scale=b)) #      ##14479.56      Why doesn't the method of movements work? Am I doing something wrong that can be fixed or is there some other way that I can calculate the variance of an inverse gamma distribution?


title: Bootstrap confidence intervals for partitioned variances in R

text: I'm trying to determine the variance partitioning within plant drought tolerance data among hierarchical ecological levels, from species to forest sites to biome levels. I did that with the varcomp random effects model in R:               varcomp(lme(TLP_DRY~1, random=~1|Biomes/SiteNum/Species, data=d, na.action =na.omit),1)      Now, I want to bootstrap 95% confidence intervals for variance at each level, but I get incredibly weird results! Almost all of my species variance percentages become 100%, while within-species and biome variation becomes 10E-32, which looks nothing like the results for the actual dataset. **My questions:** What is happening in the bootstrap **and** What does it mean about my ability to calculate confidence intervals? My code in R for generating the bootstrap is:               matrix(NA, nrow=1000, ncol = 4)-> emptylist # define a place to put the simulated values          for(ii in 1:1000){     Data[sample(nrow(Data),100, replace = TRUE),] -> d # take a random sample the same size as the dataset with replacement          varcomp(lme(TLP_DRY~1, random=~1|Biomes/SiteNum/Species, data=d, na.action =na.omit),1) -> mod  # calculate variance partitioning           mod[1]-> emptylist[ii,1] # stick the values together in one place     mod[2]-> emptylist[ii,2]     mod[3]-> emptylist[ii,3]     mod[4]-> emptylist[ii,4]     }     quantile(emptylist[,1], c(.025, .975))  # determine the 95% confidence intervals     quantile(emptylist[,2], c(.025, .975))     quantile(emptylist[,3], c(.025, .975))     quantile(emptylist[,4], c(.025, .975))


title: Compare two survival curves for paired data

text: I want to compare two different methods for detecting status change in a survival analysis. A group of subjects is being followed for a longer period (many years), and two examination methods have been used to examine whether a status change has occurred; one method was used to examine each subject twice a year and the second method was used to examine each subject once a year. The question is if these two methods differ systematically in their ability to detect a status change. The test that I came to think of is a log rank test to see if the two method’s Kaplan-Meier curves differ. I wonder if it is a problem that the survival curves are “paired” (i.e. the two methods are used on the same subjects) when performing the log-rank test. Is it a violation of assumption in the log-rank test, or is it perhaps just an inefficient test since it doesn't account for that the two curves are related? Does anyone have a suggestion for an alternative analysis that do accounts for the dependence within the observations? * * * Maybe this is not a problem, maybe I am overthinking. Well, I don’t know the true time of status change, only the time points when the methods have detected a status change. One thought I had was to set the survival time to the midpoint of the time interval between the last examination when status change had not been detected and the examination when status change had been detected. That could compensate for the disadvantage of the method that is used for examining the subjects only once a year in contrast to the method that is used twice a year. And then construct the survival curves from these data. Thank you the paper, I’ll look into it.


title: Nonparametric test statistics for non-equal sample sizes

text: If I have two samples, with one 30-times larger than the other, which nonparametric test statistics should be considered for checking the null of equal distribution? I am using Kolmogorov-Smirnov test to check that the two samples come from the same distribution. Any mistake in this? Is there any other test statistics to be used?


title: Introduction to causal analysis

text: What are good books that introduce causal analysis? I'm thinking of an introduction that both explains the principles of causal analysis and shows how different statistical methods could be used to apply these principles.


title: How to put $$ around numbers in R?

text: **How can I put the arrays of a matrix in double $$ in R?**


title: Covariance of INID order statistics

text: In the IID case, it is known that all order statistics are positively correlated.* Thus, we know that $$\text{Cov}(X_{(i)},X_{(j)}) \geq 0.$$ Is this known in the INID (independent, non-identically distributed) case? If it is not known, how could this be proved? If it does not seem true, what would be a counter-example?   * E.g., see Bickel (1967), "Some contributions to the theory of order statistics"


title: Is there any way the adjusted $R^2$ might decrease by adding predictors?

text: Let's consider a multiple linear regression formula: $ \hat{y} = \beta_0 + \beta_1 \hat{x}_1 + \beta_2 \hat{x}_2 $ (1) which produces adjusted $R^2 = r_1$. Now I want to add to one predictor to the (1) which turns into: $ \hat{y} = \beta_0 + \beta_1 \hat{x}_1 + \beta_2 \hat{x}_2 + \beta_3 \hat{x}_3 $ (2) which produces adjusted $R^2 = r_2$. If the data fed into (1) and (2) are exactly the same, is there any way to explain $r_2 < r_1$ apart from a code bug?


title: Can you compare different clustering methods on a dataset with no ground truth by cross-validation?

text: Currently, I am trying to analyze a text document dataset that has no ground truth. I was told that you can use k-fold cross validation to compare different clustering methods. However, the examples I have seen in the past uses a ground truth. Is there a way to use k-fold means on this dataset to verify my results?


title: How to measure the period of an oscillation with least error?

text: I have a question about analyzing the data from a coupled pendulum. I have measured the amplitude $\psi(t)$ which is expected to be a beat and I want to measure the period. The ideal plot would be something like this. ![enter image description here](http://i.stack.imgur.com/cbB5V.png) I have to measure the period of the envelope and the individual components. The image is ideal, because the amplitude will be decreasing slowly with time. So **I would like to know the best method to calculate the period and its error**. I have considered these options:   * Fast Fourier Transform (FFT): The problem I see is that it's not a totally periodic function, so it might introduce more error. And I've read that the error is related to the covariance matrix, but I don't understand it.   * Measuring the distance between maxima/minimum: This would not be affected by the damping, but you would waste many points.


title: $R^2$ relative to a noiseless function

text: I am interested in computing the $R^2$ between a set of points $D_f = \\{ (x,y)\\} $ where $y = f(x)$ and a set of points $D' = \\{(x',y') \\}$ obtained adding noise to $D_f$. I don't think I can use: $$ R^2 = 1 - \frac{\sum_i (y'_i - y_i)^2}{\sum_i (y'_i - \bar{y'})^2} $$ because noise can be added to the $x$ coordinate as well. More specifically, I am interested in computing the $R^2$ to compare MIC (Maximal Information Coefficient) as in "Detecting Novel Associations in Large Data Sets" Reshef et al.


title: Multivariate ARIMA with regression

text: I have a dataset covering daily data for 3 years (3x365 rows) for multiple attributes TotalPhoneCall (main attribute that I want to predict), Christmas day, weekend, weekday, Easter, 4th_july, etc.(some are seasonal). I would like to predict TotalPhoneCall for the following month. I have to use ARIMA with regression. I may filter out unnecessary attributes if needed. How can I do this in R?


title: Any reference book for the axiom of indifference or insufficient reason

text: I need a book as a reference for the "principle of indifference" or "principle of insufficient reason". Any suggestion?


title: What helps (justification) to decide cut score/cutoff points for 5-point likert scale?

text: I am wondering what are the factors involved in deciding the cutoff value / cut score for 5-point likert scale from:               1 = not ready at all,      2 = not ready,      3 = moderately ready,      4 = ready,      5 = highly ready.       Is it proper for the cut score (in mean value) to be as such if i want it in 3 categories:               below 2 = not ready,      2 - 4 = ready,      above 4 = highly ready.      And if yes, how do I justify my decision? Are there any references or empirical ways to help decide? I was told that I could justify that the inherent meaning of the scale helps to make such decision - especially using the information embodied in the numerical value - which expert said that? Thank you.


title: Model reduction - Backward elimination

text: The rule I use for reducing covariates (trimming) is as following:   1. Look at the highest order interaction. If that is not significant, drop it. If it is significant, stop.   2. If drop the highest order interaction (because insignificant), look at the highest order polynomial. If not significant you drop it, if it is significant you keep it.   3. Continue the process. But before that, I need to pick the model with best variance-covariance structure. This step determines the highest order polynomial and interaction (which goes in the MODEL statement), as well as the variance-covariance structure (which goes in the RANDOM statement). Criteria for picking are BIC/AIC, and whether v-c structure is estimable. Assume the model with best v-c looks like:               PROC mixed covtest;         CLASS id treatment;         MODEL outcome = treatment time treatment*time timesquare treatment*timesquare                          timecubic treatment*timecubic / ddfm=kenwardroger s;         RANDOM intercept timecubic/ type=un subject=subj g gcorr s;     RUN;      So, I begin to trim applying rules above. The scenario is:   1. treatment*timecubic is not significant, I then drop it from model rerun the model w/out it.   2. Look at timecubic in the new model, it's not significant. Now the highest order interaction and polynomial corresponding to the order of the variable in RANDOM STATEMENT are excluded (timecubic). How should I proceed with this?   * Should I drop timecubic in RANDOM STATEMENT and start again with picking best c-v model, but ignoring cubic and higher order of time?   * Or, should I keep timecubic and proceed with the rest of variables in MODEL STATEMENT? If possible, please suggest some reading that may be useful. EDITED: Thanks Glen_b for suggestions.


title: Genetic distance over spatial scales

text: I have matrices of genetic distances for x number of individuals within a population and their corresponding point coordinates -one genetic distance matrix per point coordinates. I was imagining using the **Mantel test** , i.e. evaluate the significance of correlations between genetic and geographic distances. However, what I did not think of was that these two types of matrices will have different dimensions (e.g. a genetic matrix of 15 ind = 15x15, sampled at three locations = 3x3), hence the **Mantel test** will not work. I am now looking into **Moran's I** , testing for spatial autocorrelation, but it will not work with my data as it currently is. I would need one measure of variation per point coordinates. Would it be legit (never seen anyone do this) to derive a measure of variation (e.g. **var** , **sd** , **cv** ) from a single distance matrix? Since the matrix is normalized, ranging from 0-1, maybe **var** or **sd** would work? I am kinda new to analyzing spatial data, so I would appreciate any pointers or alternative tests I might be able to do. Thanks,


title: Transform moments of a random variable to fit the moments of another

text: Let's say we have the a set $(X_i,Y_i)$, $i\in I$, $I$ is an arbitray finite set of indexes, and the model $$ Y = g(X\beta)$$ Using some method, we obtain the individual first four moments of the residuals $u_i = Y_i - g(X_i, \hat{\beta})$, so we end up with the following set $$ (Y_i, X_i, u_i, m^i_1, m^i_2, m^i_3, m^i_4)$$ Now comes my question, if I select a residual $u_j$, $j\in I$, and I have a diferent set of moments $(\hat{m}^k_1, \hat{m}^k_2, \hat{m}^k_3, \hat{m}^k_4)$, $k \not \in I$. How can I transform the residual $u_j$ to a residual $u_k$ equivalent in the sense if I had a diferent distribution for the residuals in each point, with those sets of four moments, the quantile of the residual would be the same? Alternatively, could I find a transformation (linear if possible) from the moments of $u_j$ to the moments of $u_k$?


title: Which papers discuss classification or clustering of source code according to programming language?

text: My specific problem is to separate a huge archive of files containing source code and sometimes including embedded languages (apart from the main language).


title: How to rule out double causation

text: I have a model where I suspect (purely on theoretical grounds) that double causation might be an issue. How can I test this hypothesis? I.e. I have something like $Y_i = \beta_0 + \beta_1 X_i + \beta_2 W_i + u_i$. And I suspect that $X_i$ may also be causing $Y_i$. What I did is simply calculate the correlation between the covariate in question ($X_i$) and the error term in the regression ($u_i$). It is basically zero (-3.946634e-18). Can I then conclude that there is no double causation? Or is there a more formal way of doing this? Note: I don't think I can get an IV for $X_i$, so any test involving an IV seems infeasible.


title: Strange way of calculating chi-squared in Excel vs R

text: I'm looking at an excel sheet which claims to be calculating the $\chi^2$, but I don't recognise this way of doing it, and I was wondering if I'm missing something. Here is the data it is analysing:               +------------------+----------+----------+     | Total Population | Observed | Expected |     +------------------+----------+----------+     |             2000 |       42 | 32.5     |     |             2000 |       42 | 32.5     |     |             2000 |       25 | 32.5     |     |             2000 |       21 | 32.5     |     +------------------+----------+----------+      And here are the sums it does for each group in order to calculate chi square:               P = (sum of all observed)/(sum of total population) = 0.01625     A = (Observed - (Population * P)) ^2     B = Total Population * P * (1-P)     ChiSq = A/B      So for each group the $\chi^2$ is:               2.822793     2.822793     1.759359     4.136448      And the total Chi Square is: `11.54139`. However, every example I have seen of calculating the $\chi^2$ is completely different from this. I would do for each group:               chiSq = (Observed-Expected)^2 / Expected      And therefore for the example above I would get a total chi square value of `11.3538`. My question is - why in the excel sheet are they calculating $\chi^2$ in this way? Is this a recognised approach? # UPDATE My reason for wanting to know this is that I am trying to replicate these results in the R language. I am using the chisq.test function and it is not coming out with the same number as the Excel sheet. So if anyone knows how to do this approach in R it would be very helpful! # UPDATE 2 If anyone's interested, here's how I calculated it in R:               res <- matrix(c((2000-42), 42, (2000-42), 42, (2000-25), 25, (2000-21), 21), 2, 4)     chisq.test(res)


title: How to interpret the selection of a variable in a model (GLM) when spatialautocorrelation is included?

text: This is a pretty straightforward question. I am comparing outputs of two models (binomial GLM) one including environnmental-only (ENV) variables and one including environmental and spatial variables (ENV+SP). I understand now why a variable could be selected in ENV and not in ENV+SP, that is because the effect of this descriptors is only spatial dependent or because the two effects are the same and it is due to the violation of independence analysis, therefore of an increasing H0 risk (Zuur, 2009). However, I have issues understanding the reverse case that is when a variables is not selected in ENV but appears when spatial-autocorrelation is included (ENV+SP). Any help would be welcome, Thank you very much.


title: Repeated measures with no between subject factors

text: I have data set that includes:   * multiple animals normalised heart rate   * 2 trials for each animal   * 3 time points per trial I want to analyse this data by comparing mean heart rate at each time point across trials... So, I have no between subject factor and only within subject factors... This is apparently a two way within-subjects design however I have no idea how to run this in SPSS. Can anyone help? * * * **Edit** Sorry, I should have been more specific.... Animals were tested over two trials. I have taken raw heart rate and normalised it at 0.05 second intervals against the heart rate at the commencement of the each trial. From this, and based on previous literature, I have chosen 3 discrete time points in which to measure heart rate differentials between trials, 1 time point prior to the trial commencement, and two after (2 seconds prior, 2 second after and 7 seconds after). I am looking to see if trial one, at 2 second after commencement differs from all other time points in both sessions. Does this help?


title: Index function in PLM

text: I need help to perform panel regression between share return and macro economic factors. As the share returns of different companies belong to same country, only time factor make a difference. When I run the `plm` with fixed effects in R, I am unable to get a result if `index =("company","year")` and `effects="time"`, but if I change the index as year first, it works fine. Results with `index=("year", "company")` are different from `index=("company","year")`. Could you please suggest how to fix this for a better result?               > Macro.fixedIn = plm(SR~TS+GDPN+MS+IR, data= rawData, index=c("ID","Year"),                            model="within", effect="time")     Error in plm.fit(formula, data, model, effect, random.method, inst.method) :        empty model


title: What's "Relative Error" in a neural network model?

text: I'm using SPSS for building a neural network model. In the model summary there is a measure called "Relative Error". What's the formula for it? Is it related to sum of squares error?


title: Ensemble classification in MATLAB

text: I want to use ensemble classifiers for classification of 300 samples (15 positive samples and 285 negative samples, it means binary classification). I extracted 18 features from these samples, all of them are numerical, and there is some correlation between the features. I am new to MATLAB, and I tried using fitensemble but I don’t know which method to use: `AdaBoostM1`, `LogitBoost`, `GentleBoost`, `RobustBoost` , `Bag` or `Subspace`. As the numbers of features is 18, I don’t know weather boosting algorithms can help me or not. On the other hand, I have problems with the number of the learners. How many learners are suitable for this problem, and how can I get the optimal classification? I would appreciate your help.


title: How do I analyze these two sets of data

text: I am doing a research study on self efficacy and birth month (when children enter school, if they are oldest in class or youngest).   I have each of the participants scores on their self efficacy scale (rated on a 14 point likert scale) and I have ranked their birth months (starting with Sept = 1, Oct = 2, etc.). I am wondering how I can analyze this data to find statistical significance, variance, etc. Thanks!.


title: How to detect feature value combinations with outstanding win percentages

text: I am building ruby on rails (ActiveRecord) horse racing application. One of the features that I would like to include in the application is the ability to identify meaningful "Angles". For example, given the following facts:   1. Horse A has raced 50 times and has 30 Wins   2. Jockey B was riding Horse A for 25 of Horse As 30 wins Now, if Horse A is racing today and Jockey B is riding, I would want the system to   * Recognize the high win percentage Horse A has when Jockey B is riding   * Provide / Create some sort of alert of the "Angle". Ideally there would be some way statistically or otherwise to define thresholds so as to avoid a lot of false positives. Is this the sort of thing a recommendation or inference engine would be used for or is the something entirely different. Also, given what I am trying to achieve can you point me to any Gems, resources, etc that might be helpful. Thanks for your help.


title: How to increase a particular terms's weightage?

text: I am doing Text classification using LibSVM in Rapid Miner. I am using TFIDF values for processing documents. I need to Increase weightage of some terms in the documents(for eg. words in BOLD and CAPITAL letters). How can I achieve that using TFIDF values? If TFIDF is not the correct way, Can anyone suggest some ways to achieve the result. Thanks in advance Sunil


title: Variance of sample mean in a binomial distribution

text: I'm trying to derive that the variance of `phat = pq/n`. I'm unfortunately not getting that. Please tell me where I'm going wrong...               phat = x/n, where x is the # of successes in your trial and n is the number of samples     var(phat) = E[phat^2]-(E[phat])^2     = E[x^2/n^2] - p^2     = 1/n^2 * E[x^2] - p^2                  //E[x^2] seems like it should be np given that x=1 for success and 0 for failure          = np/n^2 - p^2     = p/n - p^2 = p(1-np)/n       which is not `pq/n`... arg!


title: Extracting data from bivariate vs multivariate analyses for the purposes of meta-analysis

text: I have what is a very basic question about meta-analysis. If data from individual studies about the relationship of interest are presented in both bivariate and multivariate analyses, which should I use? Would it be appropriate to calculate effect sizes based on multivariate analyses (e.g., adjusted odds ratios), or should only data from bivariate analyses be used?


title: Is it possible to assign a different cost for each misclassification in a random forest?

text: I have a vector of weights for each instance which signifies the cost of each misclassification. How can I incorporate this while training a random forest?


title: What do Lift and Gain Charts state in the context of an employee turnover model

text: So I am trying to further understand Lift and Gain charts as it applies to my employee turnover model (i.e. used CHAID in SPSS Modeler). For my data this means predicting number of people voluntarily leaving the company. I have reviewed the below references and have the basics down regarding interpretation: what is plotted on the x and y axis and the ideal curve you are looking for. I even practiced constructing my own gains and lifts charts in Excel. But all the examples I have seen thus far are for a direct mail campaign. Now I want to know what this means for my data. Does it merely mean, in the case of the gains chart that if I sample top 10% of my data I can expect 40% of terms vs sampling top 60% of my data get's 80% of terms? (please assume the 40% and 60% are the values). If so then what significance should I take away from that because I really don't get it in the context of my turnover model? References: lift-measure-in-data-mining what-is-a-lift-chart http://www2.cs.uregina.ca/~dbd/cs831/notes/lift_chart/lift_chart.html


title: Simulate a random number of sums of random variables?

text: I'm trying to solve an exercise (not homework, and it does not come with solutions) that asks to provide an algorithm to simulate a probability $P(Y<y)$ and the expected value $E[Y]$ of a random variable $Y$. Also the algorithm must have a fixed number $N>0$ of steps (number of pseudo-random $\rm Uniform(0,1)$ variables that the algorithm must generate). I understand that if I can simulate $N$ copies of $Y$ (call them $Y^{(1)}, \ldots, Y^{(N)}$), then I can estimate $P(Y<y)$ as the ratio between the number of $Y^{(i)}$'s inferior to $y$ and $N$. Similarly, $E[Y]$ can be estimated as $\frac{1}{N}\sum_{i=1}^N Y^{(i)}$. However, suppose $$Y = X_1 + \ldots + X_R,$$ where $X_1,X_2,\ldots$ is a sequence of iid variables and $R$ is a discrete uniformly distributed random variable over $\\{1,2,\ldots,n\\}$, for some fixed $n>0$. Also suppose that one $\rm Uniform(0,1)$ variable is enough to simulate one and only one $X_i$. Now, since presumably $R$ is independent of each $X_i$, what would be the appropriate way to proceed and why: simulate $R$ once, have it fixed, that is simulate $\lfloor N/r \rfloor$ copies of $Y$ ($r$ would be the simulated value of $R$), or instead use a while cycle and alternate between simulating $R$ followed by $X_1,\ldots,X_R$? Thanks.


title: Choosing between transformations in logistic regression

text: In linear regression, the transfrmations of explanatory variables is done to have maximum correlation with the dependent variable. What is the best measure of choosing between multiple transformations in logistic regression as dependent variable is binary and not continuous? The end goal is to maximize the lift (predictive power) of the model.


title: What happens when you apply SVD to a collaborative filtering problem? What is the difference between the two?

text: In Collaborative filtering, we have values that are not filled in. Suppose a user did not watch a movie then we have to put an 'na' in there. If I am going to take an SVD of this matrix, then I have to put some number in there - say 0. Now if I factorize the matrix, I have a method to find similar users (by finding out which users are closer together in the reduced dimensional space). But the predicted preference itself - for a user to an item will be zero. (because thats what we entered on the unknown columns). So I am stuck with the problem of collaborative filtering vs SVD. They seem to be sorta the same, but not quiet. What is the difference, and what happens when I apply an SVD to a collaborative filtering problem (I did, and the results seem acceptable in terms of finding nearby users, which is great, but how?). Any pointers much appreciated.


title: How to statistically test whether there is any nesting effect in the data?

text: I have a sample including all students within 52 schools which have been randomly assigned to be treatment or control (we have group randomized trial in which all students in each school is either assigned to treatment or to control ). I'd like to know whether there is any school clustering effect in this data. In other words, I'd like to compare the within-school variability of students to between school variability. One way to measure whether there is any school clustering effect is to use ICC (intra class correlation). The ICC for my data (with schools as clustering units) is 0.04. I'd like to have a statistical method helping me decide whether ICC of 0.04 is large enough to take the school clustering effect into account in my model. What would you recommend? Can F test on the ratio of the variability between schools over the variability with-in schools answer this question? Thank you for helping me on this issue.


title: Confidence interval for a coefficient with small N

text: I have a question about the confidence interval of the estimator in a multiple linear regression. Here below you can find my model:               lm(formula = GDP.per.head ~ PISA.math + Quality, data = Dataset)          Residuals:        Min     1Q Median     3Q    Max      -23954  -5958  -2253   1754  36712           Coefficients:                         Estimate     Std. Error      t value     Pr(>|t|)         (Intercept)         -82346.8       82988.5       -0.992      0.334220         PISA.math            201.8         169.5          1.190      0.249504         Quality[T.high]      29603.3       7383.0         4.010      0.000822 ***     Quality[T.medium]   22158.0        7535.9         2.940      0.008748 **      ---     Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1           Residual standard error: 12880 on 18 degrees of freedom     Multiple R-squared: 0.5798, Adjusted R-squared: 0.5098      F-statistic:  8.28 on 3 and 18 DF,  p-value: 0.001134      If I want to calculate the confidence interval at significance level 95% of the estimation / coefficient:   * Can I say that the confidence interval for the predictor "Quality high" is `29603.3 + or - 1.96 x 7383`?    * Or, because of the small sample size (n=22), do I have to consider a different quantile than 1.96? For example, should I be using the quantile for the t-distribution at 95% for 18 DF (i.e., 1.734)?


title: What is the substantive meaning of scalar invariance?

text: I understand that scalar invariance, in the context of Structural Equations Modeling (SEM), is having the intercepts for observed variables loading on the same latent variable be invariant across multiple groups. However - what does scalar invariance mean substantially? What are its implications?


title: Automating predictions from R

text: Let's say I have the following lagged data frame and am trying to predict income for 9/10 using data from 9/09.               dat = data.frame(main_date=c("09/01/2013","09/02/2013","09/03/2013","09/04/2013","09/05/2013","09/06/2013",                                  "09/07/2013","09/08/2013","09/09/2013","09/10/2013"),                      lag_date=c(NA,"09/01/2013","09/02/2013","09/03/2013","09/04/2013","09/05/2013","09/06/2013",                                   "09/07/2013","09/08/2013","09/09/2013"),                      income=c(rnorm(10)), status=c(rnorm(10)), avg_temp=c(rnorm(10)))     dat          mod = lm(income ~ status + avg_temp, data=dat)     summary(mod)          (-0.00848) + (0.4272*(0.641)) + (-0.651*(-1.274))      I have my data frame and run a linear model. Then I grab estimates and july data points for my two predictors and calculate for Y manually. However, I was wonder if R had a built in process for generating predictions of this nature or could provide insights into finding an "automated" process for performing this task in R.


title: Perfect fitting individual model in mixed model

text: Suppose I have a dataset where each individual is measured at three timepoints on a variable (we'll call it disease status). What I want to do is estimate the relationship between time and disease. Obviously a mixed model is appropriate, where each person's individual regression line is modeled then aggregated. However, I know that the relationship across timepoints is curvilinear; people tend to increase in disease activity until treated, then disease activity attenuates. Because of this, I decided to model a polynomial term. I am interested in the average shape of the function, aggregated across individuals. On the individual level, each person's regression fits perfectly (3df and 3 parameters estimated), which is usually a problem. However, across individuals it will _not_ fit perfectly (because each person has a different trajectory). Does anyone see a problem with doing this? Below is some R code to simulate the basic approach I am taking:                   #### make it reproducible     set.seed(100)              #### preallocate matrix     data = data.frame(matrix(nrow=50*3, ncol=2))     names(data) = c("disease", "timepoint")     data$ID = 1:50; data= data[order(data$ID),]              #### loop through each person and give them a disease score/timepoint     for (i in unique(data$ID)){              #### create varying timepoints         data$timepoint[data$ID==i] = sort(runif(3, 1, 6))              #### create the polynomial function with fixed effects         data$disease[data$ID==i] = data$timepoint[data$ID==i]*rnorm(1,.5,.5) + data$timepoint[data$ID==i]^2*rnorm(1,.25,.5)          }                  #### model it     require(lme4)     mod = lmer(disease~timepoint + I(timepoint^2) + (timepoint + I(timepoint^2) | ID), data=data)     mod2 = lmer(disease~timepoint + (timepoint| ID), data=data)                  ##### plot the data     x = seq(from=min(data$timepoint), to=max(data$timepoint), length.out=50)     y = fixef(mod)[1] + x*fixef(mod)[2] + x^2*fixef(mod)[3]     plot(data$timepoint, data$disease)     lines(x,y)                  #### see if polynomial terms are significant     anova(mod, mod2)                  #### look at effects     mod


title: Statistical Significance of Experiment

text: Two Groups, A & B, undergo same intervention. Group A (n=20). Group B (n=10). The only outcomes are symptomatic or asymptomatic. After the intervention, 4/20 people in group A are asymptomatic. In Group B 6/10 are asymptomatic. Is this statistically significant (i.e. p <0.05).


title: Artificial distribution to best match an actual one

text: There's a problem I am struggling to solve, if anything because I can't find the right angle from which to attack it. It's basically a fitting problem, and possibly a topological one, although I'm not sure of how to find a similar example. It goes like this: There's an experimental data set and a model to simulate the shape of the distribution, and what are to be obtained are the optimal parameters of the model so that the shape is as close to the experimental distribution as possible. The model is thus not meant to predict future data, but to learn more about the properties of the system that produces it. It is not a classic "apply curve to data" problem: we rather create an artificial data set, try to map it to the actual data set, and search for which parameters of the artificial dataset offer the best mapping. Finally consider also the further complication that y is NOT a function of x, but rather both of them are functions of a third parameter not directly represented in the dataset, and of which we know nothing in detail (at least in the case of the experimental data). To sum up the problem: there's a distribution with fixed shape and position, and a second distribution for which the shape can be modified, stretched and translated. Could you point me to the mathematical tool that would be best suited to assess the optimal parameters that make the second shape as identical as possible to the first one?


title: Automatically determine probability distribution given a data set

text: Given a dataset:               x <- c(4.9958942,5.9730174,9.8642732,11.5609671,10.1178216,6.6279774,9.2441754,9.9419299,13.4710469,6.0601435,8.2095239,7.9456672,12.7039825,7.4197810,9.5928275,8.2267352,2.8314614,11.5653497,6.0828073,11.3926117,10.5403929,14.9751607,11.7647580,8.2867261,10.0291522,7.7132033,6.3337642,14.6066222,11.3436587,11.2717791,10.8818323,8.0320657,6.7354041,9.1871676,13.4381778,7.4353197,8.9210043,10.2010750,11.9442048,11.0081195,4.3369520,13.2562675,15.9945674,8.7528248,14.4948086,14.3577443,6.7438382,9.1434984,15.4599419,13.1424011,7.0481925,7.4823108,10.5743730,6.4166006,11.8225244,8.9388744,10.3698150,10.3965596,13.5226492,16.0069239,6.1139247,11.0838351,9.1659242,7.9896031,10.7282936,14.2666492,13.6478802,10.6248561,15.3834373,11.5096033,14.5806570,10.7648690,5.3407430,7.7535042,7.1942866,9.8867927,12.7413156,10.8127809,8.1726772,8.3965665)      .. I would like to determine the most fitting probability distribution (gamma, beta, normal, exponential, poisson, chi-square, etc) with an estimation of the parameters. I am already aware of the question on the following link, where a solution is provided using R: http://stackoverflow.com/questions/2661402/given-a-set-of-random-numbers- drawn-from-a-continuous-univariate-distribution-f the best proposed solution is the following:               > library(MASS)     > fitdistr(x, 't')$loglik                                                              #$     > fitdistr(x, 'normal')$loglik                                                         #$     > fitdistr(x, 'logistic')$loglik                                                       #$     > fitdistr(x, 'weibull')$loglik                                                        #$     > fitdistr(x, 'gamma')$loglik                                                          #$     > fitdistr(x, 'lognormal')$loglik                                                      #$     > fitdistr(x, 'exponential')$loglik                                                    #$      And the distribution with the smallest loglik value is selected. However, other distrubtions such as beta distribution require the specification of some addition parameters in the fitdistr() function:                  fitdistr(x, 'beta', list(shape1 = some value, shape2= some value)).      Given that i am trying to determine the best distribution without any prior information, i don't know what the value of the parameters can possibly be for each distribution. Is there another solution that takes this requirement into account? it does not have to be in R.


title: Statistical analysis for rubric-based research

text: There are 26 participants who have participated in my research. Each participant treated with a lab module (Hands on Robotics Session). Now each participant will be evaluated using a rubric on scale of Excellent, Good, Fair and Poor. For my research i want to evaluate the following questions: Question 1 Null Hypothesis: Students learn about computational thinking (programming basics and algorithmic thinking) with the help of robotics. Alternate Hypothesis: students do not learn about computational thinking (programming basics and algorithmic thinking) with the help of robotics. To evaluate the above question, the categories i will be considering are Plan, Implementation, Testing and Precision on a scale of Excellent, Good, Fair and Poor. Question 2: Null Hypothesis: Participants effectively translate the given scenario to a flowchart Alternate Hypothesis: Participants do not effectively translate the given scenario to a flowchart I will also evaluate the question 2 on a scale of Excellent, Good, Fair and Poor. I am planning to use CHI SQUARE to test the above two hypothesis. Is CHI SQUARE a feasible option? Is Yes what should i consider for rows and columns of chi square.


title: How to combine subsets consisting of mean, variance, confidence, and number of sampled points used?

text: I have a data set that has been divided into $n$ data subsets. I am sampling from each of these subsets and getting a tuple consisting of mean, variance, confidence and number of sampled points used. **How can I combine these results?** I do not know how other than a simple function of the number of points and their averages. This wont take into account either the variance or the confidence of the score.


title: Plotting interval censored follow-up time as a line chart

text: I'm working on a survival analysis project where it would be useful to visualize everyone's followup time and event times. The data is made up of an ID, which of two possible events they had (A or B), and the time at which they had the event as well as whether or not they are interval censored. Non- censored individuals have two identical event times, t1 and t2, while censored individuals have t1 =/= t2, giving the lower and upper bound of their censored time. This looks like so (all made up):               ID, eventA, eventB, t1, t2, censored     1, 0, 1, 7, 7, 0     2, 1, 0, 5, 5, 0      3, 1, 0, 10, 10, 0     4, 0, 1, 4.5, 4.5, 0     5, 1, 0, 2, 8, 1      Where ID 5 is censored. I'd like to produce a plot like this: ![enter image description here](http://i.stack.imgur.com/zC4Yd.jpg) Essentially, make the X-axis ID, the Y axis time. Draw a line segment from 0 to t1. Then if they're censored, draw another line segment from t1 to t2. Presumably if they're uncensored they'll just overlap, which is fine. Then drop a marker at t2 for each type of event, say an X if they're eventA and O if they're eventB. I presume there's a way to do this in R, but I haven't yet really wrapped my mind around graphing in R, and most of the examples for graphs in survival analysis I'm finding are KM curves. Anyone have an idea?


title: Outliers in importance sampling

text: I'm working on a HW question in which I'm using the importance sampling method to estimate $E(X)$ where $X$ is distributed as standard Laplace. To do so, I choose my proposal density to be a standard normal. I successfully wrote my code and my estimate is very reasonable. In part of the question I've been asked that "what type of outliers in importance sampling are worrisome". I appreciate if you could guide me on that.


title: Normal distribution necessary to assess moderating and mediating effects?

text: * Is linear regression only suitable for variables with a normal   distribution?   * If so, is there an alternative nonparametric test to test mediation or moderation?


title: Significant interaction between covariate and factor in SPSS GLM

text: In testing gender difference on the relationship between variable A and B,   * A is the covariate (or independent variable)   * B is the dependent variable   * Gender is the factor As I understand it, if there is a significant interaction between the covariate and factor, then the analysis should be stopped as this violates an assumption of ANCOVA. My question is: **What next?** (I am asking this question because most text books only deal with non- significant interaction, hence there is very little guidance on what to do next if the interaction is significant, as in my case.) Does the above violation of the assumption mean that I cannot do any further statistical test? (This may be a blessing in disguise for me!) Can I draw a scatter plot of A and B with different colours for male and females and then discuss the slopes? (This will make intuitive sense to my target audience.) The goal of my project is to see if the relationship between A and B is affected by gender.


title: Logistic regression with repeated measures?

text: I would like to make a prediction for a (new) subject to have a certain outcome given the historical data and the model:               glm(outcome ~ age + treatment + history, family=binomial, ...)       however in the historical data that will be fitted by the model, I have some sort of repeated measurements on some of the subjects (and I don't know if repeated measures is the appropriate term to be used here, hence using lmer etc is doubtful); example:                 subject_ID    age    treatment    history    outcome     S_1           33      T_1         H_1        0     S_2           27      T_2         H_2        1     S_2           27      T_3         H_2        1     S_3           56      T_1         H_11       0     etc...      In this example subject_2 (S_2) has two rows because he had simultaneously two different treatments at the same time. could a logistic regression still be used or should cases like subject_2 be removed from the analysis?


title: Back-transformation and interpretation of $\log(X+1)$ estimates in multiple linear regression

text: I have performed multiple linear regression analyses with different combinations of transformed and untransformed variables--both explanatory (independent) and response (dependent) variables. All transformations were $\log_{10}(X+1)$ which seem to fit/better fit assumptions of normality. Also, I have indicator (dummy) response variables as explanatory variables. I'm trying to figure out how to interpret the regression estimates, so I would be much obliged if someone could point me toward a good web-based source of information on this, and/or answer the questions below. Thanks in advance. I am wondering: When back transforming--do I subtract the constant (1) from the the regression estimates (after raising 10 to power of the estimate), or just when reporting the mean/median for Y? In other words, does adding the constant to the response variable (before log transformation) matter as far as reporting the regression estimates for the explanatory variables? When do I subtract the constant from the explanatory variable if it is transformed? Also, for example, after constructing the regression model for a response variable which was log-transformed (x+1), my indicator (explanatory) variable estimate is:               Estimate: 0.008     SE0: 0.007     t: 1.110     P: 0.2660         with a 95% Confidence interval (in the log10 scale) of -0.0059 TO 0.0213. I do a back transformation and get: an estimate of 1.017871372 (95% CI from 0.9865 to 1.05). I interpret this as "median Rel Abnd of RESPONSE VARIABLE (which is aprox 0.04) at INDICATOR variable sites is 1.0179 times greater (95% CI = 0.9865 to 1.05) than the median Rel Abund at sites where INDICATOR variable not present, after accounting for other factors". If anyone could let me know if I'm on the right track, or how to get on the right track, that would be great.


title: How to understand if the variance is constant reading GARCH model output?

text: I'm using `GARCH()` function (tseries package) to fit garch model. I tried it with random numbers: `g <- garch(rnorm(750), order = c(1,1))` I know in this case the variance is constant BUT how to understand it reading the output of that function?               summary(g)     Call:     garch(x = rnorm(750), order = c(1, 1))          Model:     GARCH(1,1)          Residuals:          Min       1Q   Median       3Q      Max      -3.34115 -0.68029  0.07071  0.71918  3.31074           Coefficient(s):         Estimate  Std. Error  t value Pr(>|t|)     a0 9.454e-01   4.789e+00    0.197    0.843     a1 9.102e-03   4.535e-02    0.201    0.841     b1 2.448e-12   5.028e+00    0.000    1.000          Diagnostic Tests:             Jarque Bera Test          data:  Residuals      X-squared = 0.0373, df = 2, p-value = 0.9815                       Box-Ljung test          data:  Squared.Residuals      X-squared = 2e-04, df = 1, p-value = 0.9879


title: Learning to create samples from an unknown distribution

text: I am interested in new generating samples to approximate some unknown distribution X, where each new sample is a real-valued vector. The purpose it to be able to create a new (arbitrary large) stream of new samples from this approximate distribution that will be distributed in the same way, or as close as possible, to the original sampled data. Some additional points:   * I will have a large number of samples from X, e.g. in the millions, and possibly too large to fit in memory.    * The probability distribution X could be either discrete or continuous. It is likely to be multi-modal. Very extreme values are unlikely.    * If needed I can normalise the data or scale it to fit some bounds.   * Dimensionality of each sample is reasonably large (say 1000)   * Samples can be assumed to be independent   * Samples can be assumed to be almost identically distributed, although they represent a time series so it is possible that the underlying distribution may be changing very slowly. This change is unlikely to be large enough to    * Ideally I'd like the algorithm to be online, i.e. the model distribution can be updated incrementally as new real samples become available. What is the best algorithm to "learn" how to generate new samples with a probability distribution that approximates X as closely as possible?


title: Why does nobody visualise the between and within estimator?

text: Why can't I find nowhere, neither in books or help files on the internet, graphs of the "in between" estimators and the "within" estimator? Imagine you have the logwages of 10 persons and also their work experience. Why does nobody show these ten lines of this panel data in a meaningful graph? Is it correct that the within estimator would produce, in combination with the calculated 10 fixed effects, 10 lines that go as good as possible through each of the ten salary versus experience lines? And is it correct that the in between estimator would produce a regression line going through the averages of experience and salaries? But what would happen then if all averages x's are equal? In the graph below you can find 8 observed "experiences" versus "log wages", can you tell me my thoughts are correct? ![First try](http://i.stack.imgur.com/siriC.jpg)


title: X12 Seasonal adjustment: levels, differences or growth rates?

text: So far I used X12 casually for informative analysis etc. But now that I require precision - I stumbled upon a problem: What transformation of data is intended for most accurate results? I can seasonally adjust data in levels, as difference or as growth rate. Each approach yields slightly different results. In most cases deviation in between approaches is negligible but still I would love to hear some opinions. Problem is reproducible with pretty much any data, but for sake of example I suggest Consumer Price Index.


title: Relationship between r-squared and correlation coefficient

text: Let's say I have two 1-dimensional arrays, $a_1$ and $a_2$. Each contains 100 data points. $a_1$ is the actual data, and $a_2$ is the model prediction. In this case, the $R^2$ value would be: $$ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} \quad\quad\quad\quad\quad\ \ \quad\quad(1). $$ In the meantime, this would be equal to the square value of the correlation coefficient, $$ R^2 = (\text{Correlation Coefficient})^2 \quad (2). $$ Now if I swap the two: $a_2$ is the actual data, and $a_1$ is the model prediction. From equation $(2)$, because correlation coefficient does not care which comes first, the $R^2$ value would be the same. However, from equation $(1)$, $SS_{tot}=\sum_i(y_i - \bar y )^2$, the $R^2$ value will change, because the $SS_{tot}$ has changed if we switch $y$ from $a_1$ to $a_2$; in the meantime, $SS_{res}=\sum_i(f_i-\bar y)^2$ does not change. My question is: **How can these contradict each other?** **Edit** : 1) I was wondering that, will the relationship in Eqn. (2) still stand, if it is not a simple linear regression, i.e., the relationship between IV and DV are not linear (could be exponential / log)? 2) Will this relationship still stand, if the sum of the prediction errors does not equal zero?


title: One sided p value in a crossover trial

text: > A test treatment was compared with the control treatment in a cross-over > trial with 10 patients. The test treatment was superior in 8 patients and > the control was superior in other 2 patients. What is the one-sided p-value?


title: Loss specific belief propagation

text: We know standard Belief Propagation finds the parameters which maximize probability of posterior. Is there any way to use BP, for loss specific inference? For example, let's say someone wants to train his model with F1 measure.


title: ARIMA modelling

text: How can an ARIMA (2,1,0) model be estimated without attempting to calculate the coefficients of the model? What info would I need for this estimation? Issy


title: Why is it desirable to have low auto-correlation in MCMC?

text: I keep reading about the need to check for autocorrelation in MCMC. Why is it important that the autocorrelation is low? What does it measure in the context of MCMC?


title: Strange results in kriging with geoR. Is this a bug? Am I doing something silly? Or is this data set just unfortunate?

text: I am performing _kriging_ on a large number of relatively modest data sets (144 data points on a 9 x 16 grid in each data set). I was experimenting with different variogram models and other parameter tweaks to record errors, and noticed some data sets produced very large errors. Investigating these data sets revealed that in some cases the kriging predictions produce a very strange surface, and do not even pass through the original data (kriging predictions should have 0 error at locations in the source data). There isn't any error message and the same data sets work perfectly well with alternate covariance models that appear to be identical fits to the empirical variogram. Example to reproduce what I'm seeing:               # sessionInfo()     # R version 3.0.2 (2013-09-25)     # Platform: x86_64-w64-mingw32/x64 (64-bit)     #      # locale:     # [1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252        LC_MONETARY=English_United States.1252 LC_NUMERIC=C                               # [5] LC_TIME=English_United States.1252         #      # attached base packages:     # [1] stats     graphics  grDevices utils     datasets  methods   base          #      # other attached packages:     # [1] rgl_0.93.991 geoR_1.7-4   MASS_7.3-29  sp_1.0-14        #      # loaded via a namespace (and not attached):     # [1] colorspace_1.2-4   dichromat_2.0-0    digest_0.6.4       ggplot2_0.9.3.1    grid_3.0.2         gtable_0.1.2       labeling_0.2       lattice_0.20-23        # [9] munsell_0.4.2      plyr_1.8           proto_0.3-10       RandomFields_3.0.5 RColorBrewer_1.0-5 reshape2_1.2.2     scales_0.2.3       splancs_2.01-34        # [17] stringr_0.6.2      tools_3.0.2                library(geoR)     library(rgl)          ##  This data came from an inclass example, but i'm seeing the same behavior on real data sets, though i don't really know if      ##  they are caused by the same circumstances.      dat <-data.frame(x1=c(  1,  1,  1,  1,  1,  2,  2,  2,  2,      3,  3,  3,  3,  3,  4,  4,  4,  4,  4,  5,  5,      5,  5,  5,  2),     x2 = c( 1,  2,  3,  4,  5,  1,  2,  4,  5,  1,      2,  3,  4,  5,  1,  2,  3,  4,  5,  1,  2,  3,      4,  5,  3),     Zi =c(  0.150599,   0.191034,   0.228643,   0.378379,   0.38171,        0.51886,    0.53333,    0.612982,   0.700318,   1.027502,   1.046067,   1.198059,   1.287741,   1.600192,   1.757163,   1.757786,   1.913516,   1.994275,   2.083261,   2.276455,   2.294944,   2.308114,   2.100058,   1.135489,   0.580503539))          ## convert to geodata object     geodat <-as.geodata(dat)          ## compute empirical variogram and plot     evariogram <-variog(geodat)     plot(evariogram)          ## fit several models with default initial parameters.      exponentialfit <-variofit(evariogram,  cov.model = "exponential")     linearfit <-variofit(evariogram, cov.model = "linear")     sphericalfit <-variofit(evariogram, cov.model = "spherical")     gaussianfit <-variofit(evariogram, cov.model = "gaussian")          ## plot the fitted models     lines(exponentialfit, col = 'red')     lines(linearfit, col = 'black')     lines(sphericalfit, col = 'blue')     lines(gaussianfit, col  = 'green')          ## create krige.control objects from each model     ExpObj <-krige.control(obj.model = exponentialfit)     LinObj <-krige.control(obj.model = linearfit)     SpherObj <-krige.control(obj.model = sphericalfit)     GaussObj <-krige.control(obj.model = gaussianfit)          ## predictions at a single known value     krigingG <-krige.conv(geodat, locations = c(2,3), krige = GaussObj)     krigingS <-krige.conv(geodat, locations = c(2,3), krige = SpherObj)     krigingL <-krige.conv(geodat, locations = c(2,3), krige = LinObj)     krigingE <-krige.conv(geodat, locations = c(2,3), krige = ExpObj)          ##  they are all identical (as they should be since the surface should pass exactly through the known values), except the prediction using     ##  the linear variogram model. It's not just the linear model, I have observed the same behavior in many different models in the actual      ##  data sets I'm using (NOAA/ESRL computer simulated environmental variables--data available in a sqaure grid of one degree resolution)          krigingG$predict         krigingS$predict     krigingL$predict         krigingE$predict          ##  some data to plot a smooth surface of the kriging predicitons     xgrid <-seq(1, 5, .05)     ygrid <-seq(1, 5, .05)          gridLoc <-expand.grid(xgrid, ygrid)          ##  make predictions along a fine grid to see a smooth prediction surface     krigingGS <-krige.conv(geodat, locations = gridLoc, krige = GaussObj)     krigingSS <-krige.conv(geodat, locations = gridLoc, krige = SpherObj)     krigingLS <-krige.conv(geodat, locations = gridLoc, krige = LinObj)     krigingES <-krige.conv(geodat, locations = gridLoc, krige = ExpObj)          ##  create a matrix of the prediction values (required for rgl surface3d function)     zlinear <-matrix(krigingLS$predict, nrow = length(xgrid), ncol = length(ygrid))         zgaussian <-matrix(krigingGS$predict, nrow = length(xgrid), ncol = length(ygrid))     zspherical <-matrix(krigingSS$predict, nrow = length(xgrid), ncol = length(ygrid))         zexponential <-matrix(krigingES$predict, nrow = length(xgrid), ncol = length(ygrid))          ##  plot the data     plot3d(x = dat$x1, y = dat$x2, z=dat$Zi, size = 5)     ##  plot the linear model prediction surface     surface3d(xgrid, ygrid, zlinear, back = "lines", col = "red", alpha = .4)     ##  plot the gaussian model prediction surface     surface3d(xgrid, ygrid, zgaussian, back = "lines", col = "blue", alpha = .4)     ##  plot the exponential model prediction surface     surface3d(xgrid, ygrid, zexponential, back = "lines", col = "green", alpha = .4)     ##  plot the spherical model prediction surface     surface3d(xgrid, ygrid, zspherical, back = "lines", col = "yellow", alpha = .4)      If you see the same things I see, then the Gaussian, spherical and exponential surfaces are nearly identical, but the linear surface is a spiky nightmare- inducing sort of thing. Any insights? A bug? Unfortunate data? Or - say it isn't so - am I doing something silly?


title: Is there a common family of weight functions for which the center is zero?

text: One interesting formulation of the ordinary least squares model coefficient is as a weighted average slope where the weights are given by the squared differences between the $X$ values. $$ \begin{array}{c} \hat{\beta} = \frac{\sum_{i \ne j} w_{ij} \frac{Y_i - Y_j}{X_i - X_j}}{\sum_{i \ne j }w_{ij}} \\\ w_{ij} = (X_i - X_j)^2 \end{array} $$ I was wondering if you can create a generalized weighted average slope regression model by specifying a general class of weights $w^*$ that satisfy only $w_{ii} = 0$. I noticed that in smoothed density estimation, there is already a body of work on the types of weight functions to evaluate similar distances.


title: Exponent for non-linear regression (in R)?

text: I have a non-linear reglationship and I want to find the best way to determine the value for the exponent $\gamma$ in the following regression: $y = \beta x ^ \gamma$ I would preferably like to do this in R.


title: Appropriate statistics for proportions where categories are composites of independent binary variables?

text: I have data in the form of three digit binary numbers (i.e. 100, 101, etc.) scored for each individual in my samples taken from two populations I wish to compare. The three numbers represent independent states but I'm interested in comparing the occurrence of particular patterns (e.g. does 110 occur more in one population than the other?). However, the states themselves are not entirely independent because they represent _changes_ from the original state. Every member of both population is derived from a population with the same state (actually 100) and so occurrences of states different from 100 indicate that a change has occurred and any state that differs in more than one digit has undergone more than one change, meaning that the categories I'm interested in are not wholly independent (i.e. the existence of individuals with score 101 makes the existence of individuals with 111 more likely). What I want to know is whether this matters in terms of the statistical analysis? Can I compare the categories I'm interested (100, 000, 110, and anything ending in 1) using the same statistical tests I'd normally use for comparing proportions or do I need to use something different? Is the same true for the calculation of confidence intervals for the observed proportions? If it helps, the actual science is this: I'm working on phase variation in bacteria, which is a particular form of mutation. These mutations are reversible, affect particular genes and occur with very high frequency. For each gene of interest the mutation can switch between an on state and an off state. We score the current state of the genes - i.e. the genotype - as short binary values, so with three genes we might score a colony as 100, meaning the first gene is on, and the second is off and third is off. I am comparing the state of the genes in two bacterial populations, one treated, one control to see whether particular genotypes occur more often in the treated population than the control and whether the differences in the occurrence of each of these particular genotypes is significant (which would suggest that the population had undergone selection). The control (original) population is scored 100, and the interesting genotypes are 100, 110, 000, and any genotype ending with a 1 (i.e. 001, 101, etc.). Any genotype with more than one difference from the original genotype (i.e. 110, 011, etc.) must have had more than one mutation occur.


title: Calculating the likelihood of time series data when there are missing data

text: I am trying to calculate the log-likelihood of some time series data given parameter sets estimated in BUGS. I can not figure out how to handle some missing values at random points in time. For the complete data situation, such as $Y=(0.1,0.3,0.5,0.4,0.2,0.1)$, (real data is much longer) I have fitted a time series model assuming errors are normally distributed. For example, my BUGS code is something like:               for(t in 2:6){         y[t] ~ dnorm(y.mean[t], tau)         y.mean[t] <- phi0 + phi1*y[t-1]     }      i.e., the data is assumed to follow a normal distribution: $y_t \sim N(\phi_0+\phi_1 y_{t-1}, \sigma^2), 2<t<6$, where $\sigma$ is the standard deviation to the tolerance `tau` in the BUGS code. In R I can derive the log- likelihood of data, $l(y_t|\phi_0,\phi_1,\sigma,y_{t-1})=\sum_{t=2}^{t=6}P(Y_t=y_t)$ where $P(Y_t=y_t)$ is a normal probability density function, given a single MCMC sample of parameters (for example, $\phi_0=0.25$, $\phi_1=0.55$ and $\sigma=0.35$) as such:               > y <-c(0.1,0.3,0.5,0.4,0.2,0.1)     > phi0 <- 0.25     > phi1 <- 0.55     > sigma <- 0.35     >      > ymean <- phi0+phi1*y[1:5]     > ll <- sum(dnorm(y[2:6], mean = ymean , sd = sigma, log = TRUE))     > ll     [1] -0.01241878      However, I am stuck when it comes to performing the correct calculation of the log-likelihood when there are missing data, say $Y=(0.1,0.3,0.5,NA,0.2,0.1)$ and $NA$ is missing? I believe that `y[4]` has to dropped in the R code/likelihood calculation. I am not sure how (or if) to estimate `ymean[5]`, which is dependent on a missing $y_4$? BUGS of course provides a MCMC sample(s) for this missing data point, but should I use it, or do I keep the R code as is, adjusting for `NA` in the `ymean[5]` with `na.rm=TRUE` when summing over probability density functions:               > y[4]<-NA     > ymean<-phi0+phi1*y[1:5]     > ymean     [1] 0.305 0.415 0.525    NA 0.360     > ll <- sum(dnorm(y[2:6], mean = ymean , sd = sigma, log = TRUE), na.rm=TRUE)     > ll     [1] 0.08714057


title: Is there any way to define a distance metric given a Hidden Markov Model?

text: Let's say I've gotten a HMM that describes user search strings for my e-commerce website. Let's also say that I've just received a search string from a customer that doesn't have any search results. Does it make sense to ask "Based upon the HMM, what is the nearest string to this that has some likelihood above X?" Similarly, does it make sense to ask "Based upon the HMM, what string in my log of successful searches is closest to the submitted string?" And finally, given and two strings, can you find a reasonable distance metric based upon the HMM so that you can cluster the strings in your search log? And for all of the above - How? Can you point me in a good direction?


title: Percentage of binary outcome when informed of only one event

text: Suppose you have a bunch of independent time series each with binary outcomes A/B, all with the same CDF (but we don't know anything about it a-priori). We want to compute the average percentage of A outcomes, or better yet, the empirical CDF. The problem is that we only have information when the A outcome occurs. So if one time series has the following outcomes : B, B, B, A we are only informed when A occurs and when it does we know that 3 Bs occurred since the start / last A outcome. Our naïve way of doing right now is to compute the percentage of A outcomes every time information is given, but my hunch is that the "delayed" information must bias that percentage. Can someone point me in the right direction towards getting a statistically correct answer?


title: Why is a test statistic called a test statistic

text: I'm sure this is really basic -- but I want to understand the origins (etymology) of the term test statistic. Thanks.


title: Test data for randomness with repetitions

text: I have an experiment where I am trying to determine whether the answering behavior of participants can be explained as random answering. I.e. every participants will have to answer multiple questions with two options. The experimental hypothesis suggests a certain pattern, i.e. that participants will not just answer randomly. However I am unsure how to test for this. The main problem is, that I have multiple answers for each participants, so I am unsure whether I should expect these answers to be independent. If I could take them to be independent, I could easily test how likely it is for the seen data or a more extreme pattern to emerge just by chance. If I have to take into account that the answers for each participants are not independent, I have no clue how to analyse this. Is there some Analysis in SPSS which allows me to analyse this kind of data?


title: Can someone help me with the derive the MLE for the parameter in the SARIMA model?

text: My reseach topic is modeling the Inflation Process in Liberia, a SARIMA approach. I need to determine the property of the general SARIMA model and to derive the maximum likelihood estimators for the parameters in the SARIMA model.


What are the top statistic conferences to follow for applications in machine learning? I just had a discussion, regarding that some of the machine learning jargon and buzzwords are not really sound for a top level statistician. So instead of machine learning conferences I would like to follow conferences on statistics (perhaps also journals) for possible applications in machine learning. Perhaps someone could give me a short list of conferences were I could start searching?


title: ARIMA model parameter

text: For the ARIMA (0,0,1) model, some books write the equation as $$Z_t = \mu - \theta Z_{t-1}$$ whereas some books write the equation as $$Z_t = \mu + \theta Z_{t-1}$$ Why is there either a negative sign or positive sign before the moving average parameter in an ARIMA model?


title: Recursive System

text: Considering a two equation model: Equation 1               Trunk=f(Headroom)      In the first step I regress the first equation to get predicted values of Trunk. Equation 2               Price=f(Trunk_hat, Displacement)      I have performed this in Stata, however I also want a mathematical derivation of this procedure. What is the method to go about it?


title: Which statistics to use calculating prediction interval of dummy linear regression?

text: I have performed a linear regression and found a model of the form: $$ \hat{Y} = \alpha + \beta_1 x+ \delta_{high} + \delta_{low} + \epsilon\\\ $$ Where: $\beta_1$ is a continuously distributed variable $\delta_{high}$ and $\delta_{low}$ are dummy variables, each levels of 'high' or 'low' To use the model I would like to predict the most likely value and provide a confidence interval around the estimate. I am not sure how to properly calculate the standard error of the estimate. Since, for example, $\delta_{high}$ only applies in a "high" scenario and is zero otherwise I am tempted to use standard deviation/mean of only those records where the factor is actually "high" for the purpose calculating the standard error of the estimate. Therefore, should I use standard deviation and means over the whole sample or use only those records showing the dummies levels when calculating the standard error of the estimate and prediction interval?


title: How to select kernel for SVM?

text: When using SVM, we need to select a kernel. I wonder how to select a kernel. Any criteria on kernel selection?


title: Is there a method to check the homogeneity of variance of a single time series?

text: I have a data frame with prices/dates, is there a method that checks if the variance is homogeneous during all the series? I know there are many test like fligner.test, bartlett.test but I need to pass to those methods few groups....I don't have groups I have only one series and I need to know if the variance is constant. Thank you


title: Exp(B) value outside 95% Wald Confidence Interval

text: I ran a negative binomial regression on some count data using SPSS. The intercept had an Exp(B) value below the lower limit of the 95% Wald Confidence Interval. I've never seen this happen before and am curious as to what it means. This did not occur with any of the Exp(B) values for the predictors, only the intercept. Does anyone know what this means if anything?


title: Ranking Predictor Variables based on their Influence on the Dependent Variable

text: I have a data set where each example has 35 (often correlated) features and their resultant dependent variable. I want to rank these features based on their influence on the dependent variable. Is there a statistical technique to do that in R or Matlab? All variables are continuous. Please note that this is a multivariate analysis question as correlations among the features are important and may have great influence on the dependent variable.


title: Does the order of magnitude of sample values influence p-values and effect size?

text: When using an independent-samples t-test to compare two variables, do the absolute values of the samples affect the test statistic and the effect size? In other words, is it easier to obtain significance if the two variables are e.g. in the range 0-10 (e.g. number of fingers raised by one person vs another) versus if they are in the order of e.g. thousands (e.g. number of neurons that fire per minute in one brain region vs another)? And similarly, assuming normal variability for the two variables, does effect size depend on the absolute values, or does only the relative difference between the two variables (which can be the same regardless whether the variables contain small or large numbers) matter?


title: What do you call a chart that looks like a half pie chart with a needle indicating a percentage?

text: Is there a name for this data visualization? ![](http://i.stack.imgur.com/GIM3W.jpg) * * * It's almost like half a pie chart. In this case, it was animated like a speedometer you would see on a car dashboard; the divider (the needle) pointed left fully horizontal at first.


title: Simulation and change of probability

text: Let $\Theta$ be a more or less arbitrary space carrying a probability distribution $\pi(d \theta)$ on it. Suppose that for every $\theta \in \Theta$ one can simulate a random variable $X_{\theta}$ with mean $\mu_{\theta} \in \mathbb{R}$. How would you study the distribution $\tilde{\pi}$ on $\Theta$ defined through the change of probability $$\frac{d \tilde{\pi}}{d \pi}(\theta) \propto V(\mu_{\theta})$$ where $V:\mathbb{R} \to (0; +\infty)$ is given function. Ideally, I would like to simulate from $\tilde{\pi}$, which might be a little bit too ambitious. Can one construct a Markov chain with $\tilde{\pi}$ as invariant distribution, or study $\tilde{\pi}$ trough importance sampling? I have not been able to find any reference for these kind of problems. If it were possible to simulate a Bernoulli random variable with success probability $V(\mu_{\theta})$ one could use a simple accept-reject approach. In this problem, one can only simulate from $X_{\theta}$.


title: Confusion related to estimation of nugget

text: I am generating some simulated data from a multivariate gaussian distribution with a covariance matrix sigma. To add some noise, I added an identity matrix to the covariance matrix which depicts gaussian white noise. I then drew some samples from it, calculated the pairwise semivariance as $\gamma(h)=\frac{1}{2N} \sum{|x_i-x_j|^2}$ where N is the total number of observations that are h distance apart. Then I fitted a model variogram to it. I found out the nugget to be equal to 1 which makes sense because I added gaussian white noise of variance 1. However, I then realized that it is the semivariance not variance so I should have got 1/2 i.e 0.5 as nugget isn't it(approx). I am plotting not the variogram but the semi variogram so I am a bit confused. Suggestions? I am a bit confused. People are using the terms semi variogram and variogram with interchangeability. However, semi variogram is half of variogram. And people are like fitting variogram models(exponential, spherical etc) to the empirical semivariogram. So I am confused. The graph of empirical semivariogram and variogram is definitely different. Variogram is double of that of semivariogram. So if I fit the model variorum, the sill param should be different as well.


title: Do image recognition efforts always rely on machine learning and statistics?

text: This is something I've always wondered. Consider the Kinect. It takes its 3d image data and manages to recognize that a human is contained at a given boundary. Are these types of technologies exclusively machine-learning driven?


title: How to use R prcomp results for prediction?

text: I have a data.frame with 800 obs. of 40 variables, and would like to use Principle Component Analysis to improve the results of my prediction (which so far is working best with Support Vector Machine on some 15 hand-picked variables). I understand a prcomp can help me improve my predictions, but I do not know how to use the results of the prcomp function. I obtain the result:               > PCAAnalysis <- prcomp(TrainTrainingData, scale.=TRUE)     > summary(PCAAnalysis)     Importance of components:                               PC1    PC2    PC3    PC4    PC5   PC6    PC7    PC8    PC9   PC10   PC11   PC12   PC13   PC14     Standard deviation     1.7231 1.5802 1.3358 1.2542 1.1899 1.166 1.1249 1.1082 1.0888 1.0863 1.0805 1.0679 1.0568 1.0520     Proportion of Variance 0.0742 0.0624 0.0446 0.0393 0.0354 0.034 0.0316 0.0307 0.0296 0.0295 0.0292 0.0285 0.0279 0.0277     Cumulative Proportion  0.0742 0.1367 0.1813 0.2206 0.2560 0.290 0.3216 0.3523 0.3820 0.4115 0.4407 0.4692 0.4971 0.5248                              PC15   PC16   PC17   PC18  PC19   PC20   PC21   PC22   PC23   PC24   PC25   PC26   PC27   PC28     Standard deviation     1.0419 1.0283 1.0170 1.0071 1.001 0.9923 0.9819 0.9691 0.9635 0.9451 0.9427 0.9238 0.9111 0.9073     Proportion of Variance 0.0271 0.0264 0.0259 0.0254 0.025 0.0246 0.0241 0.0235 0.0232 0.0223 0.0222 0.0213 0.0208 0.0206     Cumulative Proportion  0.5519 0.5783 0.6042 0.6296 0.655 0.6792 0.7033 0.7268 0.7500 0.7723 0.7945 0.8159 0.8366 0.8572                              PC29   PC30   PC31   PC32   PC33   PC34   PC35   PC36    PC37                 PC38     Standard deviation     0.8961 0.8825 0.8759 0.8617 0.8325 0.7643 0.7238 0.6704 0.60846 0.000000000000000765     Proportion of Variance 0.0201 0.0195 0.0192 0.0186 0.0173 0.0146 0.0131 0.0112 0.00926 0.000000000000000000     Cumulative Proportion  0.8773 0.8967 0.9159 0.9345 0.9518 0.9664 0.9795 0.9907 1.00000 1.000000000000000000                                            PC39                 PC40     Standard deviation     0.000000000000000223 0.000000000000000223     Proportion of Variance 0.000000000000000000 0.000000000000000000     Cumulative Proportion  1.000000000000000000 1.000000000000000000      I thought I would obtain the parameters that are the most important to use, but I just don't find this information. All I see are Standard Deviation etc on the PCs. But how do I use this for prediction?


title: Power needed to detect an interaction

text: In my regression model, I am testing for main effects and interaction on Y as follows: X, X1, X*X1 (X1 is gender and there are 200 males and 250 females in the dataset.) The interaction is not significant but one main effect (X) is significant. My research context suggests that there should be an interaction (and the graph shows non parallel lines - see here). I understand one of the reasons for the above outcome would be lack of power. **Question: Does detection of an intereaction require more or less power than detection of main effects?** (By detection, I mean statistically significant) I am using the general linear function in SPSS.


title: Different prediction score for two SVM-based classifiers

text: As a validation study, I use two libsvm-based svm classifier against the same data set. One classifier is libsvm implementation in Rapidminer. Another classifier is Libsvm itself. Both of them assume the same parameter setting. However, the prediction score of these two classifiers are different. What might be reason to cause this kind of difference?


title: Initializing K-means clustering

text: If I have a certain dataset, how smart would it be to initialize cluster centers using means of random samples of that dataset. For example, suppose I want 5 clusters. I take 5 random samples of say, size=20% of the original dataset. Could I then take the mean of each of these 5 random samples and use those means as my 5 initial cluster centers? I don't know were I read this but I wanted to know what you guys think about the idea. Thank you in advanced.


title: Cluster randomized sampling and subseqent analysis

text: I am currently designing the sampling strategy for a prevalence survey on elderly in long term care facilities (LTCF) in my service, I used the "Limebook" from WHO as a reference. And I would like to see for comment on my design to see if I have done anything wrong here. Scenario: Let say I am doing a survey in country X, country X can be divided into 3 district A,B,C. I need to do survey in the LTCF in country, there are many types of LTCF and it can be roughly divided into "private" (P) LTCF and "non-private" (NP) LTCF. Since district A,B,C are different in size and population, so the numbers of LTCF are different in the 3 districts. And LTCFs have different number of residents. I have estimated by prevalence of disease Z, relative precision and the design effect. So the total sample size needed is 4000, with the designed effect to be 4 and cluster size to be 120. In the first stage of sampling I will determine how many clusters need to be sampled from each of the strata (strata is determined by districts X LTCF type, i.e. A-P, A-NP, B-P, B-NP, C-P, C-NP), proportional to the number of residents found in each strata.) (i.e. strata is sampled with probability proportionate to size) Let say I need to pick up 2 clusters from A-P, then I created a list of residents living in priate LTCF in district A, and then pick up 2 of them, one from LTCF-R and one from LTCF-S. Then I will sample everyone in LTCF-R and LTCF-S. (i.e. LTCF is sampled with probability proportionate to size) If LTCF-R is too small (e.g. only 30 rather than 120), its neighboring LTCF (e.g. LTCF-T also a private LTCF) will be selected and included in the LTCF-R cluster. If LTCF-R is too big (e.g. 300 rather than 120), random sample of 120 residence will be sampled. Till now I am doing something similar as the "Multi-stage cluster sampling in Nigeria 2011" mentioned in Example 5.4 in the WHO TB prevalence survey handbook. One thing that worries me quite a bit is that I need to add another LTCF into the original cluster is the residence in the cluster is not enough. This seems not quite like the scenario mentioned by Frerichs R.R in his chapter about cluster sampling. I am not sure if I can still use the formula in his chapter for calculating the disease Z prevalence of my cluster sampled data. Please feel free to comment or ask question. Thanks in advance.


What are some non-uniform random number generators not relying on uniform random number generators? Are some of them for general non-uniform distributions, or are all of them for some particular non-uniform distributions? Thanks!