title: Putting a prior on the concentration parameter in a Dirichlet process

text: _Most of this is background, skip to the end if you already know enough about Dirichlet process mixtures_. Suppose I am modeling some data as coming from a mixture of Dirichlet processes, i.e. let $F \sim \mathcal D(\alpha H)$ and conditional on $F$ assume $$Y_i \stackrel {iid}{\sim} \int f(y | \theta) F(d\theta).$$ Here $\alpha > 0$ and $\alpha H$ is the prior base measure. It turns out that if for each observation $Y_i$, if I know the associated latent $\theta_i$, the likelihood of $\alpha$ in this model is $$L(\alpha | t) \propto \frac{\alpha^t\Gamma(\alpha)}{\Gamma(\alpha + n)}$$ where $t$ is the number of distinct values of $\theta_i$ (the random measure $F$ is discrete almost surely). Escobar and West develop the following scheme for sampling $\alpha$ using a Gamma prior; first, they write $$ \pi(\alpha | t) \propto \pi(\alpha) \frac{\alpha^t\Gamma(\alpha)}{\Gamma(\alpha + n)} \propto \pi(\alpha)\alpha^{t - 1}(\alpha + n){B(\alpha + 1, n)} \\\= \pi(\alpha)\alpha^{t - 1} (\alpha + n) \int_0^1 x^\alpha(1 - x)^{n - 1} \ dx,$$ where $B(\cdot, \cdot)$ is the beta function. Then then note that if we introduce a latent parameter $X \sim \mbox{Beta}(\alpha + 1, n )$ then the likelihood has the form of a mixture of Gamma distributions and use this to write down a Gibbs sampler. **Now my question.** Why can't we just write $$ L(\alpha | t) \propto \frac{\alpha^t \Gamma(\alpha)}{\Gamma(\alpha + n)} = \frac{\alpha^t \Gamma(n)\Gamma(\alpha)}{\Gamma(\alpha + n)\Gamma(n)} = \alpha^t B(\alpha, n) \Gamma(n) \\\ \propto \alpha^t \int_0 ^ 1 x^{\alpha - 1} (1 - x)^{n - 1} \ dx, $$ and instead of using a mixture of Gamma distributions use a single Gamma distribution? If we introduce $X \sim \mbox{Beta}(\alpha, n)$ shouldn't I be able to do the same thing but without needing to use the mixture? **Edit for more details More Details:** To fill in some gaps, the argument in Escobar and West is that, letting $\alpha$ have a Gamma distribution with shape $a$ and mean $a / b$, $$\pi(\alpha | t) \propto \alpha^{a + t - 2} (\alpha + n) e^{-b\alpha} \int_0 ^ 1 x^{\alpha} (1 - x)^{n - 1} \ dx$$ and so we can introduce a latent $X$ so that $$\pi(\alpha, x | t) \propto \alpha^{a + t - 2} (\alpha + n) e^{-b\alpha}x^{\alpha}(1 - x)^{n - 1}.$$ The full conditionals are a $\mbox{Beta}(\alpha + 1, n)$ distribution for $X$ and a mixture of a $\mathcal G(a + t, b - \log(x))$ and a $\mathcal G(a + t - 1, b - \log(x))$ for $\alpha$. By the same argument, I got the same result but with $\mbox{Beta}(\alpha, n)$ for $X$ and $\mathcal G(a + t, b - \log(x))$ for $\alpha$. This seems easier to me; why don't they just do that?


title: Confusion related to regularization parameter selection by cross validation

text: I can see lots of paper mentioning they selected some parameters like regularization parameter $\lambda$ by cross validation. What do they mean by that?


title: Calculate standard deviation of data within each bar of a histogram

text: I have data with which I plot a histogram. I would like to know the standard deviation of each bar of histogram. Do you think it is an acceptable method to divide the samples into sub-samples for which I calculate the height of a bar (and so I will get a new sample of heights), and with this new sample I calculate a standard deviation? **OR** should I use an alternative method I found (i.e., bootstrapping). In that case, many new samples are created from the original one, I could get the heights for each of them and then I could get stand deviation for each height.   * What can be told about the accuracy of these standard deviations?   * Are they OK?   * Do you know a better method?


title: What problems should I watch out for when combining multiple time series?

text: Say I have a number of time series, e.g. a number of temperature records from various stations in a region. I want to get a single temperature record for the whole region with which I could describe aspects of the regional climate. The intuitive approach might be to simply take the average of all stations at each timestep, but my statistical spider-sense (which I'm definitely not well in touch with yet) tells me that this might not be so easy. In particular, I imagine that averaging over the entire region will remove some of the interesting temperature extremes, and I might have problems with dependence between close stations. What other problems might I face if I tried a strategy like this, and are there ways to overcome them, or more sensible methods of combining this kind of data? Note: Answers can be more general than the spatial example I've provided.


title: How many degrees of freedom with many subjects and many tests?

text: Lets say I run an experiment testing accuracy of hitting targets of two different sizes. I have 10 participants and 40 targets, 15 small and 25 large. I then have the number of misses for each target recorded. So I end up with 150 small target scores and 250 large target scores. I want to see if there is a difference in the number of misses on small targets vs large targets. Is the degrees of freedom based on the total number of measurements, 150 and 250? Or is it based off of the number of participants, 10?


title: Is there any PDF which satisfies the following criteria?

text: For an applied economics paper, I am looking for a 2-parameter probability distribution function that has the following properties:   * Simple, closed-form PDF $f(x)$ and CDF $F(x)$, defined on the interval $[0, M]$ (where $M$ may be infinite).   * $f(x)$ is bell-shaped, in the sense that it is convex and increasing on $[0,a]$, concave and increasing on $[a,f_{max})$, concave and decreasing on $(f_{max}, b]$, and convex and decreasing on $[b, M]$ ($a$, $f_{max}$, and $b$ need not be specified).   * The expectation of the random variable $X$ following $F(x)$ can be normalized to $1$ by eliminating one of the distribution function's parameters. The other parameter can be used to control the variance.   * Ideally, it is possible to explicitly solve an equation of the form $A + B F(x) + Cx f(x) = 0$. * * * So far, the closest distribution I could find is a Gamma distribution with appropriately normalized parameters. However, despite its simplicity, unfortunately no simple closed-form of its CDF exists. The same is true for the Log-Normal and Beta-Distribution. So my question is: **Are there any distributions which I have overlooked so far? Does anything simple and convenient exist that satisfies these criteria?**


title: Intuitive explanation of unit root

text: How would you explain intuitively what is a unit root, in the context of the unit root test? I'm thinking in ways of explaining much like I've founded in this question. The case with unit root is that I know (little, by the way) that the unit root test is used to test for stationarity in a time series, but it's just it. How would you go to explain it to the layperson, or to a person who has studied a very basic probability and statistics course? **UPDATE** I accepted whuber's answer as it is what most reflect what I asked here. But I urge everybody that came here to read Patrick's and Michael's answers also, as they are the natural "next step" in understanding the Unit Root. They use mathematics, but in a very intuitive way.


title: Random-effects anova

text: Thanks in advanced for the help: I have a continuous dependent variable $y$ and two independent categorical variables $x_1$ and $x_2$. As well, they have been sampled from different sites (site). $x_1$ has 3 levels $x_2$ has 5 levels is it correct to do:               dat <- aov(y ~ x1*x2 + Error(site))      or               dat <- lme(y ~ x1*x2, random  = ~1 | site, data=data, na.action=na.omit)      Thank you very much.


title: Same Joint Distribution, different conditional and marginal distribution

text: I have a group of samples drawn from density function $p(x,y)$, so it has the marginal density $p(x)$ and $p(y)$, and conditional density $p(x|y)$ and $p(y|x)$. In what way I can construct another group of samples, with the SAME $p(x,y)$ but the $p(x)$, $p(y)$, $p(x|y)$ and $p(y|x)$ are changed?


title: A question about hypothesis testing and maximum likelihood ratio test

text: Assume there are 2000 students, $m$ boys and the rest are girl. Now we take a sample of 5 students which contains only 1 boy. We claim that there are more girls than boys in the population. So what we want to do is to check the maximum likelihood under $H_1$ $H_0$, but the problem is I don't know how to get the maximum likelihood. Some said that it attains max when $m=400$ under $H_0$ but I don't quite get how they get.


title: Computing standard error in weighted mean estimation

text: Suppose that $w_1,w_2,\ldots,w_n$ and $x_1,x_2,...,x_n$ are each drawn _i.i.d._ from some distributions, with $w_i$ independent of $x_i$. The $w_i$ are strictly positive. You observe all the $w_i$, but not the $x_i$; rather you observe $\sum_i x_i w_i$. I am interested in estimating $\operatorname{E}\left[x\right]$ from this information. Clearly the estimator $$ \bar{x} = \frac{\sum_i w_i x_i}{\sum_i w_i} $$ is unbiased, and can be computed given the information at hand. How might I compute the standard error of this estimator? For the sub-case where $x_i$ takes only values 0 and 1, I naively tried $$ se \approx \frac{\sqrt{\bar{x}(1-\bar{x})\sum_i w_i^2}}{\sum_i w_i}, $$ basically ignoring the variability in the $w_i$, but found that this performed poorly for sample sizes smaller than around 250. (And this probably depends on the variance of the $w_i$.) It seems that maybe I don't have enough information to compute a 'better' standard error.


title: Changing the scale of a variable to 0-100

text: I have constructed a social capital index using PCA technique. This index comprises values both positive and negative. I want to transform / convert this index to 0-100 scale to make it easy to interpret. Please suggest me an easiest way to do so.


title: Use of "post-hoc" tests - do these require an ANOVA to be performed first?

text: I would like to compare multiple means using a method such as an ANOVA. This would have to be followed up with either Scheffe's test or the Games-Howell post-hoc method (my data is non-normal - to different degrees and can't all be transformed to normality using the same transformation - heteroscedastic and has greatly differing sample sizes). In their Opthalmic and Physiological Optics paper, Armstrong et al. (2000) state that these tests can only be performed after an ANOVA returns a significant result. My question is this - how can an ANOVA be appropriately applied to data such as mine, which require these post-hoc tests simply because they violate so many assumptions of the ANOVA? It seems odd to me that they should advocate the use of an ANOVA before post-hoc tests that are used in place of others because of the violations of the ANOVA assumptions. Also - on a side note, does anyone have any advice as to whether to use the above approach or to use the multcomp package available in R? Many thanks!


title: Estimating the number of times each of four pairs of dice was thrown

text: ## PROBLEM Suppose I have four dice: red (R), green (G), blue (B) and yellow (Y). I throw these dice several times. Each throw uses two of the four dice, and only the following pairs are allowed:   * RB   * RY   * GB   * GY  After each throw I note which dies rolled a 'one'. Across the four allowed pairs there are nine possible results:   1. Both R and B rolled a 'one'    2. Both R and Y rolled a 'one'   3. Both G and B rolled a 'one'   4. Both G and Y rolled a 'one'   5. Only R rolled a 'one'   6. Only G rolled a 'one'    7. Only B rolled a 'one'   8. Only Y rolled a 'one'   9. Neither of the dice rolled a 'one' As I throw the dice, I make a tally of how many times I got each of results 1 through 8. I _do not_ make a tally of how many times I got result 9. At the end of this process I give you my tallies, so that you know the number of times I got each of the first eight results. You also know the pairs that were allowed and the probability of rolling a 'one' (which is the same for all dice). However, you know neither the total number of throws I made, nor the number of times I got result 9. Using this information, your problem is to estimate the number of times I threw each of the four allowed pairs. Some things to note:   * Order does not matter; both dice are thrown together.   * Multiple allowed pairings can have the same result; for example, both RB and RY can have result 5. ## QUESTIONS   * Is the problem solvable?   * How would you solve this problem?   * What are some names for this type of problem that would be useful search terms if I want to look for more information?


title: Good strategy for multiclass classification (when there is hierarchical class structure )

text: Combining binary classifiers, I want to solve multi-class classification problem in the following setting. Suppose there is a dataset and each data is in one of four classes: A1, A2, B1 and B2. A1 and A2 are somehow similar. B1 and B2 are also somehow similar. (e.g. A1=rose, A2=iris, B1=cow, B2=horse) Intuitively, it seems a good idea to classify data in the following manner: first, split 'A1 and A2' from 'B1 and B2' then split A1 from A2 and B1 from B2. (so we make three classifiers.) I call this approach as 'Hierarchical strategy.' Of course, we can perform standard '1 vs rest strategy' or '1 vs 1 strategy' alternatively. Here are my questions:   1. Which is the best strategy from classification performance viewpoint. (I don't care about computational cost.)   2. I know the performance depends on data. Under what kind of property the 'Hierarchical strategy' is good? I'm happy if someone can formalize the problem in mathematical terms.   3. Are there research paper on this kind of considerations?


title: Significance of dummy variables in regression

text: I am using categorical variable (having three categories) as independent variable in model and found that one category is coming to be significant while another category is not coming to be significant while variable is significant at overall level to be included in model. I am not able to understand whether I should include the insignificant category in the model.


title: Gower distance and MDS: How to determine which variables count?

text: I have morphological data from two different determined groups (`It` and `Nd`), where the variables are heterogeneous (continuous, semi-qualitative, binomial). I want to know if the groups differ morphologically and if so, which variables account most for the difference between them. For this I thought using `gowdis()` {FD} and then performing a Principal Coordinate analysis (PCoA, which is equivalent to a MDS) using `cmdscale()` {MASS} for my analysis. The use of a Gower distance should allow me to use the complete data set, with all the different kind of variables. Most of the examples found, so also the R-help, concerned ecological data (like `dune` and `dune.env`). However, I have following questions:   1. Is my basic thinking correct? Somehow, I would have expected to use some kind of constrained multivariate analysis.   2. I have quite a lot of missing values, which are not only linked to a specific sample or variable (a part of my data set is from a previous study). How can I handle it? From my reading, I thought that Gower distance can handle missing values but see 3)   3. How can I manage the warning missing species scores? Because of this, I cannot plot the centroids of my variables nor create an arrow showing the impact of variables. As far as I know, this issue is linked to the missing values.


title: Substitution test for ANOVA?

text: I am working on biological dataset. Around 20000 genes of each sample are comparing with the same ones in another group in order to find significantly differentially expressed genes. I am comparing 3 different factors in 12 sample groups (3 replicates for each type). I think because of fewer amounts of replicates I don't get significant difference between the groups. Neither normal ttest nor ANOVA could find any significant differences between groups. I tried to use linear regression modelling. Again because of the bad data I couldn't fit the groups with least square model. Therefore, I used robust method. This could be used instead of ttest. I was wondering if there would be any test which fit my bad data and I can test all different factors at once?! E.x. of my dataset:               genes     group1 group1 group1  group2 group2 group2 ...     g1        5.5    5.6    5.7     6.5    6.7    6.6     g2        8.1    8.2    8.3     9.1    8.9    9.2     .     .     .      So I am comparing g1 account between group 1 and 2, and so forth... Thanks in advance! //Rozita


title: What are the potential functions of the cliques in Markov random field?

text: I have been trying to understand the representation of the joint probability density of Markov random fields in the form of factors of the potential functions. I am finding it difficult to grasp the idea of potential functions and how we are supposed to choose a potential function. Any suggestions on how I should get started. I have gone through the wiki page and some other resources, but still I couldn't grasp the essence of it.


title: How to compare loess models from related datasets?

text: I'm comparing matched data from a series of patients, looking at the effect of mental stress on cardiac conduction. I'm using the loess function to visually describe my data, and have 2 curves: one for with and one for without mental stress. I'd like to compare these curves. Is it possible to compare two loess models from different (thought related) datasets. Alternatively, could I code +/- mental stress with 1/0 and use that to get a statistical relationship between the two models? Any pointers would be super-appreciated.


title: What is the name of this continuous univariate distribution?

text: I am looking for the (or a) suitable name of a continuous univariate distribution depending on two parameters $\alpha>0$ (shape) and $\beta >0$ (scale). The support is $(0, \,\beta)$ and the survival and density functions are $$ S_X(x) = \left[1- x/\beta\right]^\alpha \qquad f_X(x) = \frac{\alpha}{\beta}\, \left[1- x/\beta\right]^{\alpha-1} $$ This is the special case of the Generalised Pareto Distribution (GPD) with location at zero and finite upper end-point $ S_{\mathrm{GPD}}(x) = \left[1 + \xi \,x/\sigma \right]^{-1/\xi} $ with $\xi <0$, but here reparametrised through $\alpha := -1/\xi$ and $\beta:=-\sigma/\xi$. Thus the exponential distribution $\xi = 0$ is the limit $\beta \rightarrow +\infty$. In Wikipedia's list of probability distributions, this is but a special case of Beta (rescaled) and Kumaraswamy's (rescaled). A comparable reparametrisation for the GPD with $\xi >0$, namely $\alpha := 1/\xi$ and $\beta:=\sigma/\xi$, leads to the Lomax distribution, so a cheap pun would suggest to use ``maxlo'' distribution.


title: way to regress a positive response variable that is also not a count variable

text: I am asked to perform a regression on a dependent variable that only takes positive value. Intuitively one can think of Poisson or negative binomial, but unfortunately the response variable that I am dealing with is not a count variable. How can I perform a regression on such response variable? any ideas? thank you


title: How to combine two data sets (level-1 and level-2 data sets) for multilevel analysis in R

text: I want to analyze data from TIMSS on mathematics. Data contains two data sets one for student level information (i.e. level-1) and another one for school level information (i.e. level-2). Data sets are provided in two different files and each file contains school ID variable. As far as I know, I can't do multilevel analysis with these two data sets and these have to be combined into one data set.Am I right? I tried to use `reshape` and `reshape2` for combining the two data set but it didn't work. In level-1 data I have SchoolID,motivation, confidence, learningApproach and math scores. Data looks like:               SchoolID studentID motivation confidence learningApproach math     100123  10012301    23  10  12        45     100123  10012302    32  5   16        67     100123  10012303    14  7   21        89     .                        .                        .                        110223  11023301    13  7   18        77     110223  11023302    24  7   11        98     110223  11023303    16  8   12        56      In level-2 data I have SchoolID and resources. Data looks like:               SchoolID    resources     100123      45.5     100223      66.7     1106303     34     110223      22.3      Could you please guide me on how to combine these two data sets in an appropriate way for using with `lme4` package? Thanks.


title: Why use a null hypothesis not predicted by theory?

text: The current question is inspired by this one: Which one is the null hypothesis? Conflict between science theory, logic and statistics? Paul Meehl attributed the first occurrence of using a null hypothesis other than that predicted by theory to Karl Pearson. Are there earlier examples of this behavior? Also, what justifications for doing so have been put forward since? Paul Meel wrote: > As I pointed out in my 1967 article, when the physicist uses a probable > error in this second way, improvement in the quality and number of > measurements leading to a lessened standard error subjects the theory to a > greater risk of falsification, because here a “significant deviation” means > a deviation from the predicted point value or curve type. That is how Karl > Pearson’s original invention of chi square at the turn of the century > worked. His idea of chi square was as an indicator of frequency discordance, > asking for example, does an observed distribution depart significantly from > the frequencies in class intervals as given by the Gaussian (or other > theoretical) function? This I call the strong use of a significance test. > **But then occurs a development in the use of chi square, at Pearson’s own > hands admittedly, in which the “theoretical” or “expected” values of cell > frequencies, rather than being positively generated by an affirmative > substantive theory generating a certain mathematical form, are instead > specified by the hypothesis that two variables are not related to one > another. So the expected values of cell tallies are provided by multiplying > the marginals on the hypothesis of independence, using the product theorem > of the probability calculus.** There is, of course, nothing wrong with the > mathematics of that procedure. But social scientists seem unaware of the > great shift methodo-logically that takes place in that reverse-direction use > of a significance test, where now the substantive theory is supported by the > achievement of significance in departing from the “empty” hypothesis that > two things are unrelated. In the strong use of a significance test, the more > precise the experiment, the more dangerous for the theory. Whereas the > social scientist’s use of chi square in a fourfold table, where H0 is that > “These things are not related,” I call the weak use. Here, getting a > significant result depends solely on the statistical power function, because > the null hypothesis is always literally false. **Source:** (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant using it. Psychological Inquiry, 1, 108-141, 173-180 **The 1967 article mentioned:** Meehl, Paul E. (1967). "Theory-Testing in Psychology and Physics: A Methodological Paradox". Philosophy of Science 34 (2): 103–115. **The Pearson paper:** Pearson, K (1904). "On the Theory of Contingency and Its Relation to Association and Normal Correlation". Drapers' Company Research Memoirs Biometric Series 1: 1–35.


title: How to find the expected Fisher information for a parameter of exponential distribution in an iid sample?

text: I am unclear to on how to find the Fisher information of $\lambda$ in the iid sample $X_1,...,X_n$. Attempt: I know that the fisher information for $X$ (not in a sample) would be: $I(\lambda) = - E \frac{d^2}{d \lambda^2} ln(f(x|\lambda)) = \frac{1}{\lambda^2}$ Where: $f_x(x|\lambda) = \lambda exp[-\lambda x]$


title: How do I handle categorical dependent variables with low level percentages in multinomial logistic regression?

text: I have a categorical dependent variable that involves four clusters of positions. In descriptive analysis, I noticed that first cluster covered about 5% of the participants, second 6%, third 60%, and fourth 29%. I have four continuous independent variables and I am thinking of using multinomial logistic regression. Do you think that low percentages in the first two clusters would be a problem? Should I omit the first two and turn the model into a binary logistic regression model predicting the probability of belonging to either the third or fourth cluster?


title: AIC & number of parameters

text: I have two questions about AICc and number of parameters. I am using AICc to select the best of three models in a logistic regression - and each model has the same number of parameters. What are some compelling arguments for this method? I have G and C statistics, but these do not allow ranking. The formula for AICc in this case is as follows for for logistic regressions:   * $K_1$ = sample size   * $K_2$ = number of parameters (including the intercept and s)   * $K_3$ = Log Likelihood of model   * $K_4$ = AICc = Akaike’s Information Criterion logistic regression Let $$K_4 = -2K_3 + 2K_2 +\frac{2K_2(K_2+1)}{(K_1-K_2-1)}$$ Additionally, for a different data set, in which the linear regression models being compared with AICc have different numbers of parameters, how might I interpret two models which have nearly equivalent AICc values: one with a single predictor variable (A), and the other model has an additional variable B (A+B)? B on its own has a significantly larger AICc value, so on its own it is one of the least likely models, but model A+B is nearly as likely as model A. Why isn't the addition of one variable, which is not a likely predictor on its own, causing a greater change in the AICc value when added to A? The formula for AICc is as follows and is used for regression models:   * $K_1$ = sample size   * $K_2$ = number of parameters (including the intercept and standard deviation)   * $K_3$ = residual standard error   * $K_4$ = AICc = corrected Akaike’s Information Criterion Let $$K_4 = K_1\log \left( \frac{K_3^2(K_1-K_2+1)}{K1} \right) + 2K_2 + \frac{2K_2(K_2+1)}{K_1-K_2-1}$$


title: Appropriate distance measures

text: I am working with hyperspectral image data in R, so I have subset an image to a region of 5000 pixels, each containing a vector 254 bands in length. I would like to cluster this data in order to try and map regions with similar surface composition. Due to differences in surface reflectance, if I plot two pixels, where for example: x=1:254, y=0:1 (reflectance) They may have very similar shape (values across all bands) but be vertically offset from one another due to the overall reflectance of the surface. For my region I have a mean spectrum, and each pixel contains a vector of 254 residual values. I can't use Euclidean distance to compare vectors, because it will change depending on the overall reflectance, so I'm not sure if there's a more appropriate measure to use that will give me a better comparison.   * This question is also posted at http://mathoverflow.net/questions/139591/appropriate-histogram-comparison-distance-measure, where it was suggested that the earth-mover distance may be a good one to try. If I do use this, then am I bringing the spectra into alignment by subtracting or adding the EM distance to one vector? If this is the case, then I lose the small differences between them?  Essentially, I would like to compare the differences in reflectance of each band to the mean, regardless of its overall reflectance offset. Additionally, and here is the embarrassing part: the original data (5000 observations for each band) has a distribution that is very slightly skewed right, and in some cases slightly bimodal. The mean spectrum may not even be an appropriate measure for this entire exercise, but I'm not sure what to do in this case. Apologies for my novice question.


title: Extracting regularization path from glmnet

text: Does glmnet provide any mechanisms to extract the regularization path from a final model? I'm using Elastic Nets (and L1) to build a binomial classifier and would like to be able to get the coefficients at each step along the path (until convergence). The `elasticnet` R package has this in the `beta` variable of the model, but it looks like `glmnet` only provides the final coefficients for each model. Any suggestions for how to go about pulling this out?


title: Comparing nonlinear regression coefficients from independent datasets

text: I performed enzyme kinetics experiments on a three independent preparations of an enzyme and produced the following three datasets which I separately fit to the Michaelis-Menten equation: $$ V= \frac{V_{max} \times S}{K_m + S} $$ I used R / nlme's nlsList function to do the fitting:               fit=nlsList(V~Vm*S/(Km+S)|prep,data=na.omit(kinetics),start=c(Vm=3.5,Km=50))      and I get some coefficient values that make sense and some reasonable predicted curves: ![V versus S plot of 3 enzymes](http://i.stack.imgur.com/0tGzz.png) how can I test for differences between the coefficients ($V_{max}$ and $K_m$) between the preparations? I think I can perform a t-test using the coefficient estimates and standard errors but I am not sure how.


title: How does one apply Kalman smoothing with irregular time steps?

text: I would like to apply Kalman smoothing to a series of data sampled at irregular time points. There is a claim on Stack Exchange that "For irregular spaced time series it's easy to construct a Kalman filter", but I haven't been able to find any literature that specifically addresses this. In my situation, I'd like to use a simple exponential covariance relationship to reflect the idea that the underlying continuous process is evolving as a linear dynamical system from which we irregularly receive samples. So: is it simply OK to apply a Kalman filter with the "predict" step using a transition model and a process noise model whose "amplitude" depend on the amount of time that has elapsed since the last measurement?


title: Sampling subsegments from discrete input ranges

text: I have an set of input ranges `{[a1, b1], [a2, b2], ...}`. Each `a` and `b` represent integer values. I have a constant "segment length" `i` that is always less than `b-a` for all `a` and `b`. I would like to uniformly sample (either with or without replacement) from the input ranges to get a set of subranges that fall within those inputs. My naive approach is to expand the inputs to a larger set:               {       [a1, a1 + i],       [a1 + 1, a1 + i + 1],       ...       [b1 - i, b1],       [a2, a2 + i],       ...       [b2 - i, b2],       ...     }      and then sample from this. This would be a pretty memory-exhaustive approach. I also need to deal with duplicate elements. Is there a smarter way to subsample?


title: Kernel regression with monotonicity constraints

text: I need to fit a bivariate data using kernel regression (local polynomial regression). It should satisfies two conditions.   * $\frac{dy}{dx_1} \geq 0$ for all $x_2$   * $\frac{dy}{dx_2} \geq 0$ for all $x_1$ How can I incorporate these two conditions into local polynomial regression?


title: Mathematics base for data mining and artificial intelligence algorithms

text: Could you give me some clarification about data mining and artificial intelligence algorithms? What mathematics base they used for? Could you give me starting point, in mathematics, to understand these types of algorithms?


title: Repeated measures ANOVA for two different groups

text: I have conducted a clinical trial using two groups with two different treatment modalities. I have measured different physical and biochemical parameters (e.g pulse, systolic blood pressure, serum sodium level etc.) in each group at different time intervals (say pre-intervention, after 1 month and after two months). Now I want to measure the effect of each treatment modality on these parameters over time and at the same time look for any significant difference existing between the two groups. Is this study design suitable for repeated measures ANOVA? How to do it then in SPSS 19? Is there any other way to do the analysis?


title: How to simulate repeated measures multivariate outcomes in R?

text: @whuber has demonstrated how to simulate multivariate outcomes ($y_1$, $y_2$, and $y_3$) for one time point. As we know, longitudinal data often occur in medical studies. My question is how to simulate repeated measures multivariate outcomes in R? For example, we repeatedly measure $y_1$, $y_2$, and $y_3$ at 5 various time points for two different treatment groups.


title: Validity of standard deviation

text: I have a total of 7 data sets that i want to compare. Each set has 2 samples only, for which i want to calculate standard deviation. Will my value of standard deviation be reliable? Or is it a a better option to compare them using only average value of the 2 samples.


title: How can I perform 10-fold cross validation by manually constructing datasets?

text: I am working in text classification in RapidMiner where, because of the nature of my problem, I cannot use the built-in k-fold cross validation strategy, so I decided to create 10 copies of my dataset manually in order to incorporate the k-fold cross validation $(k = 10)$. However I have a few confusions regarding it. I have a dataset of 4000 samples distributed in 20 categories.   1. I am thinking to incorporate 10-fold cross validation by creating 10 copies of same dataset where 90% (3600) will be used for training and 10% (400) for testing. In each of the 10 copies of dataset, the test set will be different from other datasets. In the end of the classification task I will average their results. Am I right about my interpretation of 10-fold cross validation?   2. When we say to divide data in k-folds, use k-1 folds for training, and use the last one for testing, do we mean to train the classifier for k-1 times on k-1 folds, or do we mean to train the classifier only once on k-1 folds? E.g. we have a data set of 4000 samples which is divided in 10 folds with 400 samples in each fold. By training the classifier on k-1 folds, do we mean train the classifier k-1 times (here 9) on k-1 folds (9) with 400 samples in each fold, or do we simply mean to train the classifier once on (400 × 9 samples) and test it on the last fold?


title: When are odds ratios worse/better than risk ratios as measures of the effect size of associations between categorical variables?

text: Wikipedia gives some information on this. However, I seek more detailed information on when odds ratios/risk ratios are better, and why one is better than the other in certain circumstances.


title: Comparing two non-linear biological models: differences in y vs. differences in slope

text: I'm an ecology PhD student, but unfortunately not a very mathematically sophisticated one. I am interested in ways to compare two non-linear models. I was really impressed with this answer, which covers part of my question, about looking at the absolute area between the curves. Thinking about it further, it seems to me that it might be more biologically meaningful to look at the slopes at points (x, y1) vs (x, y2) over the range of x values, since the slope is the indicator of the relationship between the predictor (lets say age) and the response (lets say body mass). I'm imagining a comparison of two populations: one in which the gestation period was long, and one in which the gestation period was short. If the relationship between age and body mass are the same for these two groups, but contingent on the birth weight (longer gestation = more mass at birth) the curves could behave in exactly the same way, but be offset by that initial difference. The area between the curves could be large, while the slopes at each x value would be the identical. I suppose that my questions are - does this make sense? If not, why not? If so, is there an obvious way to make this sort of comparison (between slopes)? Is there a way to make a comparison that takes into account both the difference in area between two curves **and** the difference in the behavior of the slopes? If I do tackle this, I will be using R. Also, bearing in mind that I'm thinking of biologically realistic models, does this make sense? Slope at any value of x seems like it could be pretty independent of the slope at the adjacent values of x, whereas area, or more precisely the absolute difference in y-values, seems like it is only going to vary so much from the difference in y-values at adjacent values of x. Using the same hypothetical example, actual body weight is only going to change so much over the course of a day, but the rate of change in body weight could vary tremendously over the same time span. Sorry for rambling, and I'm interested to hear what you think about this.


title: How do I estimate the parameters of a log-normal distribution from the sample mean and sample variance

text: Given the sample mean $\bar{x}$ and the sample variance $s^2$ of a random variable $X$, is it possible to estimate the shape $\sigma^2$ and log-scale $\mu$ of the log-normal distribution, with probability density function $$f_X(x;\mu,\sigma^2) =\frac{1}{x \sqrt{2 \pi \sigma^2}}\, \mathrm{exp}\left( {-\frac{(\ln x - \mu)^2}{2\sigma^2}}\right), \text{ if } x>0,$$ and if so, how are any resulting formulae derived?


title: Subsampling two datasets so that the new sets have similar joint prob. distribution

text: I want to subsample two equivalent (in terms of features/columns) data sets in a way that the new subsampled data sets have the same joint probability distribution. To explain it better on an example: **Example** : Assume I have salary data for software developers and construction workers available, along with their location, time of employment, experience etc. I want to compare the average salaries of the two occupations. However, the salaries are directly incomparable, since the distributions of location and experience for the two occupations is significantly different, along with a number of other features. So to make the salary numbers comparable, I would like to "force" the two data sets to have the same distribution. Is there a good/standard way to do this? The number of features I am dealing with is ~100, and there is some correlation between subsets of these features.


title: What I have to undergo testing before proceeding to time series analysis? and after?

text: I have to conduct a time series regression analysis. Before proceeding I tested for stationarity and I verified that the variables are distributed according to the normal probability distribution. After I regressed the time series model, I tested error autocorrelation and the presence of heteroskedasticity. There is something else I have to verify? Is the coefficients estimation correct?


title: Wilks's lambda distribution

text: Please help me, I don´t have any idea how to solve this problem. If $\textbf{A} \sim W_p(\mathbf{\Sigma}, m)$ and $\textbf{B} \sim W_p(\mathbf{\Sigma}, m)$ are independent Wishart matrices, show that $|\textbf{A}|/|\textbf{A}+\textbf{B}|$ has the $\Lambda(p,m,n)$ Wilk's lambda distribution. I would be very grateful if someone could guide me.


title: Evaluating the prediction ability of several alternative models

text: First off, I apologize if it seems like I'm flooding the board, but I'm new to modelling and a lot of the questions that are coming up aren't closely related to one another, so it seems better to put them in their own threads. I'm trying to predict win percentage of a particular team in a particular season in baseball. I have three independent variables. One of the independent variables is the win percentage of the team of interest in the previous season. The other two independent variables are quite similar to this one in that they are in their own right reasonable predictions. Call the target $Y$ and the independent variables $X1$, $X2$, $X3$. I've run simple linear regressions `Y ~ X1`, `Y ~ X2`, and `Y ~ X3`. I want to compare the predictive abilities of these models. Is $R^2$ a good method to do so? If I have, say, that $$R^2(\text{Y ~ X1}) > R^2(\text{Y ~ X2}) > R^2(\text{Y ~ X3})$$ is it reasonable to say that $X1$ provides the best prediction, then $X2$, then $X3$? I've read a lot of conflicting things about the value of $R^2$, so I'm not sure. Since I can't get a very coherent model by multilinear regression of $Y$ against $X1$, $X2$, and $X3$, since they are all highly correlated, I've instead decided I would try a stacked regression to combine the simple linear regressions. As far as I understand, this requires me to choose one subset of my data $S_o$ to use in performing the simple linear regressions for each predictor, then choose another, disjoint subset of my data $S_1$ to use in performing a multilinear regression for $Y$ with the simple linear regressions as my independent variables, with the restriction that the regression coefficients be non-negative. This multilinear regression is the stacked regression. What are your thoughts on using stacked regression in this situation? How do I compare the predictive ability of the stacked regression to that of the simple linear regressions? I know that $R^2$ can depend on model complexity, so I don't know how reasonable it would be to simple compare the $R^2$ value of the stacked regression to those of the simple linear regressions. Is there a reasonably simple way to figure out a range for the actual amount each of the predictions will err with some confidence? There is also the problem of deciding what proportion of my data should be in $S_o$ and what proportion should be in $S_1$ when I perform the stacked regression. Opinions? EDIT: I'm starting to think I may have misunderstood what's involved in the stack process, because after doing what I described, the resulting predictor is worse than the originals. Any ideas as to the problem?


title: Model variance and bias in cross validation

text: How do different cross-validation methods compare in terms of model variance and bias? My question is partly motivated by this answer: Number of folds for K-fold. The answer suggests that models learned with **leave-one-out** have **higher variance** than with regular **k-fold** , making **leave-one-out** a worse choice. However, my intuition tells me that in **leave-one-out** one should see relatively **lower variance** between models than in **K-fold** , since we are only shifting one data point across folds and therefore the training sets between folds substantially overalp Or going in the other direction, if K is low in **K-fold** , the training sets would be quite different across folds, and the resulting models should more likely to be different (hence higher variance). If the above argument is right, why would models learned with leave-one-out have higher variance? (and which one would be a better choice?)


title: Fit a curve (spline?) through a solid in 3D and map a value onto it

text: I have a series of xyz data in a unit cube. It forms a sort of tapered & curved cylinder. It will begin and end at 0,0,0 and 1,1,1. I want some sort of best fit curve down the 'axis' of this shape. And I need to take a new xyz point and map/project it onto this best fit curve. If I was expecting a straight line, I'd use the first principal component, but in this case I'm almost certain that I want some kind of 3d spline. I'm not after a fit to the surface. To be clear, there is no response variable; it's just xyz data points, so a model is not what I need. Can someone give me some suggestions for the name of what I want to do? I need some keywords to get me going. I'm working in `R` so if you can suggest a package even better. Alternatively, perhaps something in `Matlab` would work. But mostly I need to know what this process is called. Thanks.


title: Can bootstrap be used to replace non-parametric tests?

text: I am fairly new to statistics. The concept of bootstrapping has been confusing to me. I know that normality of the sampling distribution is required to use certain tests such as the t-test. In cases when the data are not normally distributed, by requesting "bootstrapping" in t-tests in SPSS would this circumvent the problem of non-normality? If so, is the t-statistic that is reported in the output based upon the bootstrapped sampling distribution? Also, would this be a better test compared to using non-parametric tests like Mann-Whitney or Kruskal-Wallis in cases where I have non-normal data? In situations when the data are not normal and I am using bootstrap I would not report the t-statistic: right?


title: Hypothesis testing a location shift in heteroscedastic and non-normal data

text: How do I test for a "location shift" in something like the mean or median if the shape of the distribution has changed quite significantly between groups? Often an experimental intervention seems to completely change the distribution of the data. For example, If I measure limb length in control and toxin- exposed animals, the control group sample might appear to be drawn from a symmetric, normal distribution but the experimental group might be heavy- tailed and have much more dispersion. All the statistical tests I am familiar with either assume that the data is drawn from normally-distributed populations or that the data is drawn from two populations with non-normal but identical distribution shapes. I can do something like a two-sample Kolmogorov-Smirnov test which tells me that my groups differ. But this isn't telling me anything about location, right?


title: Distribution of high and low of a random walk

text: If I have a one-dimensional random walk with position $X(\text{t})$ with $X(0)=0$ and $\text{Var}(X(1)) = 1$, and I observe $X(1) = \text{c}$, what are the distributions of the minimum and maximum values of x between times 0 and 1 conditional on c, and what is the distribution of the range over this time interval? Here is why I ask. I want to jointly simulate (min,max,range,final) for a random walk. To simulate the final value alone a normal random number generator can be used.


title: Is it appropriate to assess internal consistency on related items measured with different response formats within the same instrument?

text: In my survey, I grouped 67 unique counseling techniques that I thought were related to one another (based on theory and conceptual similarity) into six "counseling domains" using fairly intuitive, descriptive but brief labels (e.g., rapport-building or confrontation). In the first section of the survey, I asked respondents how frequently they used these techniques on a five-point scale (1=never, 5= always). The other major section of the survey included three vignettes or scenarios where key variables were changed between them were created and respondents were asked to choose which of the 67 techniques they would employ in those situations. These variables, then, are 0/1 dummies. I would like to demonstrate that the six domains are meaningful and useful constructs, and I'm thinking that reliability analysis is the way to do this. Would it be appropriate to obtain alphas (or some other measure of internal consistency) the domains for the frequency of use subsection (Likert items) **and** each of the three vignettes (dummies)? Or just the frequency of use? I don't think FA is appropriate because the factors would be based on _the frequency of use_ of the techniques, not any latent construct, but this fact may also limit what claims can be made in the reliability analysis as well. Thank you so much for any feedback!


title: Repeated measures structural equation modeling

text: I need to analyse a dataset of clinical rehabilitation data. I am interested in hypothesis-driven relationships between quantified "input" (amount of therapy) and changes in health status. Although the dataset is relatively small (n~70) we have repeated data reflecting temporal changes in both. I am familiar with non-linear mixed effects modelling in R however am interested in potential "causal" relationships between input and output here and thus am considering repeated measures applications of SEM I'd appreciate advice on which if any of the SEM packages for R (sam, lavaan, openmx?) are best suited to repeated measures data, and particularly recommendations for textbooks (is there a "Pinheiro and Bates" of the field?).


title: Tutorial on Radial Basis Function Networks?

text: I want to learn about Radial Basis Function Neural Networks, can you please suggest a good introduction or tutorial? All the introductions I found are rather short or incomplete or so.


title: Why not approach classification through regression?

text: Some material I've seen on machine learning said that it's a bad idea to approach a classification problem through regression. But I think it's always possible to do a continuous regression to fit the data and truncate the continuous prediction to yield discrete classifications. So why is it a bad idea?


title: Interpreting a quadratic logarithmic term

text: Given the regression model $y=\beta_0 + 300*ln(x_1) - 15 * (ln(x_1))^2 + \beta_3*x_2 + ... + u $ $x$ ranges from 1 000 to 30 000 (6.9 to 10.3 on a logarithmic scale). How to interpret the diminishing effect that $x_1$ has on $y$? Is it possible to interpret it for small changes in $x_1$ analog to a "normal" quadratic term, i.e. the partial effect can be calculated as $\Delta y = [\beta_1 - 2*\beta_2 * ln(x1)]$ and the maximum (turning point) as $x_{1} = - \beta_1 / (2*\beta_2)$, which would yield 300 / 30 = 10 with e^10=22026? How to give a meaningful and illustrative interpretation of the effect? For example, the effect of a 1% increase of $x_1$ or going from 20 000 to 21 000? ### Edit Thank you for the code and sorry for the delayed response. I managed to plot the marginal effect of x1 but decided to use the plot of the actual and predicted values of y against x1 while holding all other variables at their mean using the following Stata code:               reg y ln(x1) (ln(x1))^2 x2 x3 x4 x5     adjust x2 x3 x4 x5 if e(sample), gen(predict)     twoway line predict x1, sort || scatter y x1      I think this illustrates the effect of x1 on y really well and the plot is easy to understand despite the quadratic logarithmic term in the regression model.


title: Why it is often assumed Gaussian distribution?

text: Quoting from a Wikipedia article on parameter estimation for a naive Bayes classifier: "a typical assumption is that the continuous values associated with each class are distributed according to a Gaussian distribution." I understand that a Gaussian distribution is convenient for analytical reasons. However, is there any other real-world reason to make this supposition? What if the population consists of two sub-populations (smart/dumb people, large/small apples)?


title: if I have rank=3 in an cointegration process means that _ce1,_ce2,_ce3 have all to be negative and statistical significant?

text: I need some help. I have never used the cointegration process neven before and I am not used working with econometrics.I have a model with 5 explanatory quarterly variables,52 observations and the lags selection criteria proposed 4 lags (given in the choice maximum lags: 4 lags) I would like to ask about the error correction term. These are my results of VECM.![results for VECM in STATA](http://i.stack.imgur.com/Qw9Dp.png) In the johansen cointegration I had a rank 3 which means that there are 3 cointegrating relationships. Watching this video Lag selection.Cointegration.VECM. I understood that the _ce1 has to be negative and statistically significant. But that is for rank 1. If I have rank>1 all the _ces have to be negative and statistically significant? Also, in the VECM result tables do I have to check for the statistically significance of all my variables and lags?


title: Sampling from a lognormal distribution

text: Suppose we are given $\mu$ and $\sigma$ for a lognormal distribution with random variable $X$. $\mu$ is the mean of the variable's logarithm and $\sigma$ is the standard deviation of the variable's logarithm. Then if $Z$ is sampled from a standard normal distribution ($Z$ has mean 0 and std deviation 1), $$X = e^{\mu + \sigma Z}$$ Now, I am given access to a uniform random generator, and use Box-Muller to generate a sample $Z$, can I generate a sample from $X$ by applying the above formula, or will there be issues with doing this?


title: Finding the UMVUE of the variance of a gaussian with mean zero

text: Given $Z_1, ..., Z_n, \sim\mathcal{N}(0, θ^2), θ>0$. Define $X_i=|Z_i|$ and consider estimation of $\theta$ and $θ^2$ on the basis of the random sample $X_1,...X_n$. Find the uniformly minimum variance unbiased estimator (UMVUE) of $θ^2$.


title: Is this the correct way to interpret regression coefficients?

text: Say I have some data on people where I have a measure of their general health (some score out of 1000), the number of apples they eat in a year, and the number of oranges they eat in a year. Then I do a multiple regression and I get something like: $\hat Y = 442.22 + 1.1a + 0.7o$ where $a$ is the apple variable, and $o$ is the orange variable. Is it legitimate for me to then say to someone who eats 1 apple and 1 orange a year, "Your predicted health score is $442.22 + 1.1 + 0.7 = 444.02$?" I've been taught that I can only say things like "for each extra apple you eat your health is predicted to increase by 1.1 points, holding constant scores on the other variables". Did my interpretation in the previous paragraph violate this "holding constant" rule?


title: Using the Caret package is it possible to obtain Confusion Matrices for specific threshold values?

text: I've obtained a logistic regression model (via `train`) for a binary response, and I've obtained the logistic confusion matrix via `confusionMatrix` in `caret`. It gives me the logistic model confusion matrix, though I'm not sure what threshold is being used to obtain it. How do I obtain the confusion matrix for specific threshold values using `confusionMatrix` in `caret`?


title: Wilcoxon matched pairs test for my dataset?

text: I really hope someone can help me as i have been reading about this stuff and i am still really confused! I have a dataset of 180 chemical compounds with 2 data (numerical) values each - i want to see if there is a difference between these two data points for each of the compounds for the whole dataset however my data is not normally distributed so my questions are: Am i ok to use a matched paired wilcoxon rank test? Or can i get away with using a normal paired t test? I have had been reading alot online an i just want some checking i am understanding! Any additional help/resources will be appreciated! Thanks in advance :)


title: Correlation analysis for education research project

text: Good day! Our group conducted a classroom research about Social Goals Role on Student Mindset. For this study, we asked 42 college students to answer 2 sets of questionnaire. The first survey consists of 17 items (YES/NO) while the second set consists of 3 items with 6-point Likert scale. Please help me on what correlation analysis to use. Thank you in advanced!! :)


title: What are probabilistic approaches to finding the right number of clusters?

text: As per answers to this question, there are shortcomings in the heuristics of deciding on the number of clusters. A more robust approach could be _probability based clustering_ : from a probabilistic perspective, the goal of clustering is to find the most likely set of clusters given the data. Thus, we can never be "100% sure" that training instances should be placed into some cluster: they just have a certain probability of belonging to it. I wonder how if this reasoning is correct and how it would work in practice.


title: Best bandit algorithm?

text: The most well-known bandit algorithm is upper confidence bound (UCB) which popularized this class of algorithms. Since then I presume there are now better algorithms. What is the current best algorithm (in terms of either empirical performance or theoretical bounds)? Is this algorithm optimal in some sense?


title: Are growth mixture models just Gaussian mixtures applied to coefficients of polynomials fitted to time-series data?

text: Am I understanding correctly that growth mixture model is just Gaussian mixtures applied to coefficients of polynomials fitted to the time-series data? For example, we have 1000 cases, with 3 measurements each. We fit, say, a quadratic equation to each which gives us 3 extra values per case (quadratic, linear, constant). Then we fit Gaussian mixture model to those 3 coefficients, which gives us clustering of the trajectories. Is that it, or are growth mixtures something more involved?


title: Independence in classification

text: I recently came across this work: http://cvlab.epfl.ch/alumni/oezuysal/ferns.html which is a balance between the full naive Bayes assumption (full independence between the features) and absolutely no independence assumption. This seems like a very appealing idea. Is there more work like this?


title: Decision Fusion using Multiple Data Sources with different update rates and accuracy

text: If you have two data sources which report their decision at different data rates, and have different "trustworthiness" how can I combine them so that I can take advantage of both of them? I'm still relatively new to the statistics realm, but I am assuming that using data from multiple sources yields a better decision if you can combine the decisions intelligently. So, I want to take advantage of the output from both sources. To make things harder, one of the systems is providing a decision at a much higher rate than the other. Input: a probability vector from a source each time it reports a classification Output: a probability vector (ideally with a measurement of how stable the current decision is)


title: What is a kernel and what sets it apart from other functions

text: There seem to be many machine learning algorithms that rely on kernel functions. SVMs and NNs to name but two. So what is the definition of a kernel function and what are the requirements for it to be valid?


title: Hypothesis test for binomially distributed variable with large n and small p

text: Example: I want to compare the number of successes in a sample of ~10,000,000 independent bernoulli trials against a known population probability of success of ~.01. Thus, expected success count for 10,000,000 trials should be 100,000. What's the quickest hypothesis test to implement for a large n, small p? I would prefer to avoid the exact binomial test if possible as I'm trying to code an automated hypothesis test.


title: In boosting, why are the learners "weak"?

text: See also a similar question on stats.SE. In boosting algorithms such as AdaBoost and LPBoost it is known that the "weak" learners to be combined only have to perform better than chance to be useful, from Wikipedia: > The classifiers it uses can be weak (i.e., display a substantial error > rate), but as long as their performance is not random (resulting in an error > rate of 0.5 for binary classification), they will improve the final model. > Even classifiers with an error rate higher than would be expected from a > random classifier will be useful, since they will have negative coefficients > in the final linear combination of classifiers and hence behave like their > inverses.   * What are the benefits of using weak as opposed to strong learners? (e.g. why not boost with "strong" learning methods - are we more prone to overfitting?)   * Is there some sort of "optimal" strength for the weak learners? And is this related to the number of learners in the ensemble? Is there any theory to back up the answers to these questions?


title: Implications of current debate on statistical significance

text: In the past few years, various scholars have raised a detrimental problem of scientific hypothesis testing, dubbed "researcher degree of freedom," meaning that scientists have numerous choices to make during their analysis that bias towards finding with p-value < 5%. These ambiguous choices are, for example, which case to be included, which case is categorized as outlier, running numerous model specification until something shows up, do not publish null results, etc. (The paper that sparked this debate in psychology is here, see a popular Slate article and follow-up debate by Andrew Gelman here, and the Time magazine also touches on this topic here.) **First** , one clarification question: The _Time_ magazine wrote, > "A power of 0.8 means that of ten true hypotheses tested, only two will be > ruled out because their effects are not picked up in the data;" I am not sure how this fits into the definition of the power function I found in textbook, which is the probability of rejecting the null as a function of parameter $\theta$. With different $\theta$ we have different power, so I don't quite understand the above quote. **Second** , some research implications:   1. In my field of political science / economics, scholars simply use up all the country-year data available. Thus, should we not be concerned with sample fiddling here?   2. Can the problem of running multiple tests but reporting only one model be fixed simply by the fact that someone else in the discipline will re-test your paper and strike you down immediately for not having robust results? Anticipating this, scholars in my field are more likely to include a `robustness check` section, where they show that multiple model specifications does not change the result. Is this sufficient?   3. Andrew Gelman and others raise the point that no matter the data, it would be always possible to find and publish some "pattern" that isn't really there. But this should not be a concern, given the fact that any empirical "pattern" must be supported by a theory, and rival theories within a discipline will just engage in an debate / race to find which camp is able to find more "patterns" in various places. If a pattern is truly spurious, then the theory behind will be quickly struck down when there is no similar pattern in other samples / settings. Isn't this how science progresses?   4. Assuming that the current trend of journals for null result will actually flourish, is there a way for us to aggregate all the null and positive results together and make an inference on the theory that they all try to test?


title: Is a Bayesian network doing feature selection?

text: I have just started playing around and reading about Bayes Nets. Here is a snippet of code using the bnlearn package in R, which seems to be a fantastic tool.               install.packages("bnlearn")     library(bnlearn)          data <- data.frame(matrix(c("sunny","hot","high","weak","no",                      "sunny","hot","high","strong","no",                      "overcast","hot","high","weak","yes",                      "rain","mild","high","weak","yes",                      "rain","cool","normal","weak","yes",                      "rain","cool","normal","strong","no",                      "overcast","cool","normal","strong","yes",                      "sunny","mild","high","weak","no",                      "sunny","cool","normal","weak","yes",                      "rain","mild","normal","weak","yes",                      "sunny","mild","normal","strong","yes",                      "overcast","mild","high","strong","yes",                      "overcast","hot","normal","weak","yes",                      "rain","mild","high","strong","no"), byrow = TRUE,                    dimnames = list(day = c(),                      condition = c("outlook","temperature",                        "humidity","wind","playtennis")), nrow=14, ncol=5))          res = hc(data)     plot(res)      This graph that is fit using the hill climbing algorithm has two nodes that are not directed: Outlook and wind. Is it fair to say that these random variables are unrelated to the others and a sort of feature selection is occurring?


title: Grid Search for Polynomial Kernel

text: I am trying to do a grid search for a polynomial kernel. I have used the normal grid search on polynomial kernels which is wrong. I only varied $c$ and $\gamma$... but I need to vary $a$ and $b$... Is there any fast method to do grid search for polynomial kernels? Polynomial kernel: $K(x_i, x_j) = (\gamma x_i x_j + a)^b,\quad \gamma > 0$ Any suggestions or any corrections as well are welcome.


title: Probability of a relation on the uniform distribution of points over 2D space

text: Assume a set of nodes is scattered over a 2D surface $\mathcal{S}$ so that for any given $\mathcal{A} \subset \mathcal{S}$, the number of nodes inside $\mathcal{A}$ follows a Poisson distribution with parameter $|\mathcal{A}| \rho$, where $|\mathcal{A}|$ shows the area of the subset $\mathcal{A}$ and $\rho$ is the intensity of the points (average number of points per unit area). We are only interested in the points inside a given circle with radius $r$. The number of nodes inside the circle is a Poisson variable with parameter $\rho \pi r^2$. We pick two nodes from inside the circle at random. Let $d_1$ and $d_2$ show the distance of the first and the second node from the center of the circle. ![2D Distribution](http://i.stack.imgur.com/6Tt1R.png) How can I compute the probability of the event: $$ {d_1}^2 < \frac{{d_2}^2}{A(1+B{d_2}^2)} $$ where $A$ and $B$ are constants. **Edit:**   1. Assume $A > 0$ and $B > 0$.   2. I am interested in the process itself, not the points generated by the process (as whuber described in his answer below).   3. How about the case that ${d_1}^2$ and ${d_2}^2$ is replaced with ${d_1}^\alpha$ and ${d_2}^\alpha$ for $\alpha > 2$ (I guess, this modifies the problem since ${d_1}^\alpha$ and ${d_2}^\alpha$ are not uniformly distributed anymore).


title: Stationary distribution on a partition of the state space?

text: I would like to use the following type of model and am wondering if someone could let me know if this technique has been studied before (I assume it has) and where to learn more about it? The idea is simple. I have a Markov process with a possibly infinite state space S. I have a partition P of S into a finite number of sets. I would like to compute a "stationary" distribution over the "states" in P. (Bad example removed) **Example** Let S be the set of real numbers in [0,100). My Markov process is to take the current state (a real number) and add a Gaussian variable distributed with mean 50 and variance 25, and taking the result modulo 100. (E.g. -3 ---> 97.) Let P have two states: [0,50) and [50,100). I'd like to know the equivalent of the stationary distribution over P: the proportion of the time our state is in (0,50], and the proportion of the time it is in (50,100). Is this problem in general well-defined? (Of course there may be no answer for some examples.) Is there a known way to approach these problems or anywhere I can find out more? Thanks very much! (Edit/PS also let me know if I should crosspost to a different stackexchange.)


title: party vs. rpart vs. ??? for partitioning trees in R

text: It's been a while since I looked at partitioning trees. Last time I did this sort of thing, I like party in R (created by Hothorn). The idea of conditional inference via sampling makes sense to me. But rpart also had appeal. In the current application (I can't give details, but it involves trying to determine who will go to jail among a large sample of arrestees) I cannot use advanced methods like random forests, bagging, boosting etc. - I need an easily explicable rule. I would also like to have some manual control over which nodes split, as recommended in Zhang & Singer (2010) _Recursive Partitioning and Applications_. The freeware that comes with that book allows this, but is otherwise rather primitive in its user input. Any recommendations or suggestions?


title: Mixed effects regression with a positive interaction

text: I have similar findings to JDneversleep - I have 2 main effects of, language difficulties and gender. The effects of language difficulties and gender on social skills are negative. But I am getting positive coefficient for the interaction between language and gender. Could someone explain to me in terms of my variables what this means please - sorry i'm attempting stats that are way over my head :( any insight would really be appreciated!


title: Web based forecasting tools

text: We want to add a forecasting tool to our website -- it would draw off a historical database and users can choose the model (Box Jenkins, Simple Linear Regression, etc.), confidence interval and play what-if. Is anyone aware of such package we can plug into our website?


title: Determining which variables affect which measurements

text: What software (or method) can I use to determine correlations between any sets of variables and results? **Background:** I would like to analyze sets of our archived data over several years. Knowledge of the product tested often leads to some obvious correlations (ie: fuel efficiency peaks at a particular speed). Many trends are not discovered until an investigation following some issue. I'm looking to find any trends that may not be as obvious when looking at the data. Generally, I seem to find trends by making and testing a hypothesis about the data. However, the process of constructing a query, copying the results to Excel, and analyzing with Excel (sorting, formulas, statistics, graphing, etc) seems too tedious, inefficient, and slow. I'm looking for software that would take a table of data, look for trends between all sets of variables (inputs) and attributes (outputs), and display the most likely correlations.


title: Reducing the number of groups of data by joining them

text: I have one variable that attains a certain value (between 0 and 1), about 1000 observations and each belongs to one (and only one) of fifteen groups. Now I would like to reduce the number of groups to two or three by combining them. It is important that all the data within a given group in the former division will stay together after the pooling. The problem is, that the whole dataset has too large variance, so I would prefer several groups that are more focused, less varied. Example: Say you have hundreds of data on incomes of people from different EU countries. From this, you estimate average salary for each country (just an example). You would like to now create a group of "rich countries" and "poor countries" — but where do you draw the division line? Or two division lines in case of three groups (adding, say, "average countries"). Since yet again I don't know the proper statistical approach, I at least give you what my idea was: For the reduction to two, instead of trying all possible divisions, I had the idea of sorting the groups by their sample means (from $g_1$ to $g_{15}$ where means satisfy $m_1 \leq m_2 \leq \dots \leq m_{15}$) and creating pool combinations — first combination would be {[$g_1$]; [$g_2, ..., g_{15}$]}, second would be {[$g_1, g_2$]; [$g_3, ..., g_{15}]$} and so on -- this notation means that in the first case the first pool are only values from $g_1$ and the second pool are all other values. The second case the first pool are values from $g_1$ and $g_2$, the rest is in the other pool. I would then compute the p-value for sample mean equivalence test of these pool combinations. The lowest p-value of these 14 combination tests would mark the "most different" division into pools. The same could be applied to the three groups, though there would be a higher number of pool combinations and the "diversity test" would have to be constructed quite differently. * * * Feel free to ignore my scribbling... since I do not know statistics very well, I at least tried an approach that seemed most logical. Thanks again!


title: Sufficiency of two Poisson disributions

text: If $X_1,X_2$ constitute a random sample of size n=2 from a Poisson Population show that the mean of the sample is a sufficient estimator of the parameter $\lambda$ . Since the sum of Poissons is also Poisson with parameter $\lambda+\lambda=2\lambda$   Then $E[Y]=2\lambda$. where Y constitute of two Poisson distributions each with parameter $\lambda$. Do I have to show that $2\lambda$ is sufficient for $\lambda$.   I don't understand what the estimator is here?


title: Number of dependent variables greater than # IV's in multivariate hypothesis test

text: I have a multivariate regression model $\mathbf{Y}$ = $\mathbf{XB}$ + $\mathbf{E}$ where $\mathbf{Y}$ is $n \times m$. In my case $\mathbf{n < m}$, the number of columns in $\mathbf{Y}$ is greater than the number of rows. Is it still possible to do multivariate hypothesis testing with a model like this (compute Wilks Lambda, Pillai's trace etc.)? I have been getting errors from some programs and I don't see why. I can't find a mention of this in my multivariate analysis book or online anywhere, maybe I'm not googling the right terms? If computing these statistics are $\mathbf{not}$ a possibility in this circumstance, could someone explain why not? Thank you!


title: How can I determine a relation, given the raw data for three variables?

text: I have a dataset relating two variables to a result, as shown in the image below. ![image showing 2 variables related to a result](http://i.stack.imgur.com/PLoy1.jpg) In the image Variable 1 is held constant, and plotted against different values of Variable 2, and the result is shown. I have several entries for Variable 1 (though I can have as many intervals as I like. What I'm trying to do is determine an equation that gives Result as a function of Variable 1 and 2. i.e.               Result (Var1, Var2) where 0 >= Variable 1, Variable 2 <= 100      It seems like it should be a simple thing to do, but I'm not quite sure how to start.


title: Polynomial in linear regression

text: I am new to Regression and R. I know that polynomial functions are used when a regression model does not fit data (underfitting), but I want to know which degree of polynomial should be used? I also want to know that if a regression model uses multiple variables (e.g., `y ~ x1+x2+x3+x4`) then do I need a polynomial function in this model?


title: Correcting for normally distributed clock inprecision

text: I have an experiment which is executed on hundreds of computers distributed all over the world that measures the occurences of certain events. The events each depend on one another so I can order them in increasing order and then calculate the time difference. The events should be exponentially distributed but when plotting a histogram this is what I get: ![Histogram of events](http://i.stack.imgur.com/Oskbm.png) The imprecision of the clocks at the computers causes some of the events to be assigned a timestamp earlier than that of the event they depend on. I'm wondering whether the clock synchronization can be blamed for the fact that the peak of the PDF is not at 0 (that they shifted the whole thing to the right)? If the clocks differences are normally distributed, can I just assume that the effects will compensate for one another and thus just use the calculated time diff?


title: Dependent is the difference between two Likert scales: Which regression to use?

text: I asked participants in an experiment the same question once before and once after the experiment, to see the effect of the experiment on the answer. The answers were given on a 5-point scale. To me the change, so the difference between the before and the after answer to the question, is interesting. So I want to see which of multiple factors in the experiment caused the difference and accordingly want to run a regression on the DIFFERENCE in answers. The difference is always an integer, ranging from -4 to 4. Since its not continuos, I'd like to use something like multinomial logit or ordered logit, but cannot decide which one would be exactly right. Is there a 'right' regression to use? (Related, I have the problem that always a group of 5 participants took part in the same experiment. But I guess that can be solved with an FE or clusters.) edit(clarification): I have 200 observations of 40 experimental groups with 5 participants each. Dependent variables in the above scenario would be changes in the experiment-setup (same in each group but different among groups), the relation of participants in a group (personal relation, class-difference, etc.) and socio-economics of participants


title: Utilize and weight additional survey data

text: I have a household travel survey where information about travel characteristics were obtained from a sample of households. The respondents were given a diary and a GPS device to log their trips for a single day. However, for some respondents additional travel days were collected from the GPS device. Because the additional days of available data are closely correlated with respondents' diary day data, it is not correct to treat these data simply as independent data points. On the other hand, it would be wasteful and unnecessary to completely discard the additional data. So my question is the following:   1. How do I treat the additional data?    2. How do I weight this additional data? Any pointers to the literature is appreciated.


title: From half-circle to linear model

text: this is kinda homeworkish so I don't want a full solution I just want some input. I have this data set ![dataset](http://i.stack.imgur.com/ObZgv.png) And I want to transform the data (with a rbf Kernel?) in order to be able to do a simple linear ridge-classifier. I know I can do more or less the same thing using a Kernel SVM with a rbf kernel but I have to use the ridge classifier. Does anyone know how it (if?) can be done?


title: Simple regression assumptions (homoscedasticity)

text: There is a simple regression model table I was looking at in a textbook with IQ values grouped into 5 intervals and each group had an N number associated with it. There was also information given about the residuals for each group (mean and variance for the residuals). e.g For the < 75 IQ group, N = 23, Mean of residuals = -0.407 and variance = 71.288 The conclusion from the table was merely stated as "assumptions for regression have been met". I am unable to figure out what method was used to suggest if the homogeneity of variance (homoscedasticity) assumption is reasonable for the model (based on the information in the table). I'd like to know how the book arrived at its conclusion; are there plots of residuals' means/variances that indicate violations clearly? Is it like an ANOVA, where visually, one can make the simplistic estimation that if the ratio of variances exceeds a certain number, the assumption has been violated? Given a table like that, how does one proceed to test the assumptions of a regression model? Thanks!


title: Interpreting 5-way Mixed Model ANOVA

text: I'm running a 2x2x2x2x2 mixed model ANOVA (on SPSS v21) for my study and found two 4-way interactions, one 5-way interactions and a couple of 3-way interactions. Whilst I understand how to interpret a 3-way ANOVA. I'm having quite a hard time trying to interpret the 4/5-way interactions. My variables are:   * _Within-subject factors_ : lineup sex (female, male), lineup ethnicity (Asian, Caucasian)    * _Between-subject factors_ : lineup procedure (sequential, simultaneous), participant ethnicity (Asian, Caucasian), participant sex (female, male) Sample size: 552 I've gotten to the stage where I have done separate ANOVAs (splitting the data) on each factor following a 5-way or 4-way interactions obtained from the initial analysis (e.g., AxB at C1, AxB at C2 and so on) and obtained a ton of output. Some of the output had only significant main effect while others had no sig. effects. But the graph that the SPSS produced with the estimated marginal means clearly indicates a significant interaction. **E.g** : Lineup ethnicity x Participant sex at two levels of Lineup Sex. > Male Caucasians Lineup: Lineup ethnicity x Participant Sex > >   * All main effects & interaction are non. sig. > > > Female Caucasian Lineup: Lineup ethnicity x Participant Sex > >   * All main effects & interaction are non. sig. > > > But the graph indicates an interaction effect only for Female Caucasians. I've consulted with my supervisor about this and due to the time constraints, he has advised me to just compare and contrast the graphs of each ANOVA. That much I understand but I am clueless as to what to do next. Yes, I compare between the graphs but am unsure on how that would help explain the 5/4-way interactions.. Any help would be greatly appreciated!


title: Density estimation with scaled sinc-like kernels

text: Given data points $x_i$ in $\mathbb{R}^d$ with function values $f_i$, one can estimate the function at a given $x$ by $\ \ \ \ \text{f}_{est}( x ) = \frac {\sum { w_i f_i }} {\sum { w_i }}$ with $w_i = k( |x - x_i| )$.   Here $k$() is a kernel function, often Gaussian. See also Radial basis function. Consider this variant:   a) make f$_{est}$() scale-free, the same for $x_i$ at distances 1, 2, 3 $\dots$ as at 10, 20 30 $\dots$, by taking some number Nnear of the points near $x$ and scaling these Nnear distances by their average $Dav$: $\ \ \ \ wscaled_i = k( \frac {|x - x_i|} { Dav } )$ b) use the Catmull-Rom a.k.a. C-R spline kernel: ![enter image description here](http://i.stack.imgur.com/v7syf.png) The C-R spline is widely used in signal and image processing because it approximates the sinc kernel, which reconstructs band-limited signals on uniform grids perfectly. (For scattered / non-uniform points, I have no idea.) As you see, C-R is 0 at 1, negative from 1 to 2, then 0 beyond 2. Now if the scaled distances $\frac {|x - x_i|} {Dav}$ are all 1, $\sum {wscaled_i}$ will be 0; hmm. Even if they're all near 1, e.g. $\sim \mathcal{N}(1, \text{small}\ \sigma)$, that's not so hot either. Do scaled sinc-like kernels make sense for density estimation ?   Has anyone used them in practice ? (Added): Inverse distance weighting is also scale-free, property a) above.   Fwiw, changing the $\tfrac{1}{distance}$ kernel in IDW with python to the M-N spline kernel makes not much of a difference there.


title: Single sample versus multiple sample

text: I have a jar with white and black balls. Total number of balls in the jar is 100000. I want to estimate the proportion of white balls. My constraint is that the sample size for estimation should be low, lets assume 500 balls. I am debating between two approaches.   1. Draw a single sample of 500 balls, $\hat{p}$ = number of white balls divided by 500   2. Draw 10 samples of 50 balls each. Calculate the proportion of white balls in each sample, i. e., $[r_1, r_2, ..., r_{10}]$. Estimated $\hat{p}$ = average of $[r_1, r_2, ..., r_{10}]$. Which method should I use so that I am less susceptible to sampling error?


title: Propensity scores and patient comparability

text: I have two groups of patients who underwent a surgery using method A or method B. The first group are patients who were operated in 1980's and 1990's only with method A. The second group are patients operated recently with mostly method B, but also in some cases A. In addition to that, I have various variables about patients (gender, age, medical indicators, etc.) that capture the pre-operation medical history and types of symptoms that patient developed. The goal of the study is to compare the "effecitveness" of methods A and B in terms of patients' survival times after operation. Somewhat different patients (in terms of age, gender, etc.) are operated with A and B. For example, quite some more older people were operated with B than with A. I want to used propensity score matching to balance the data. My question is: Does it make sense to estimate propensity scores (method ~ age + gender + ...) and use them for creating a matched dataset for further analysis (e.g. Cox regression)? In particular, is it a problem that for patients in the first group method B was not yet available, so none of them could potentially received the alternative treatment?


title: Why is the Weibull distribution considered as a model for lifetime of a system where parts are getting worn-out?

text: My question is: how one derives a Weibull distribution for a lifetime of a system where parts are wearing out with time? Provided that this is the lifetime analysis with aging is the main application of a Weibull distribution, I was surprised not to find a derivation why specifically this distribution and not any other describe the lifetime of an aging system.


title: Does Loopy BP give the same solutions as a Gibbs sampler?

text: The literature in MCMC and LBP never refer to the fact that the two methods look (on expectation) exactly the same. To illustrate, first consider a simple Ising model, that is, a graphical model where all the variables are boolean. We can denote these variables as $\\{ X_i | X \in \\{0,1\\}, i \in (1,...,n)\\}$ and we can write the joint distribution in some factored form as: $$ P(X) = \frac{1}{Z}\prod_{\alpha} \Psi_\alpha(X_\alpha) $$ Where $\alpha$ is some subset of the variables and $\Psi_\alpha$ is a potential function over these variables. Assuming we are interested in computing the marginal distribution of a given set of variables we could run Loopy BP to compute these or we could run a Gibbs sampler to simulate from $P$ to provide samples $\\{X^{(1)},...,X^{(m)}\\}$ and compute the marginal as: $X_i = \frac{1}{m}\sum_{j=1}^m X^{(j)}_i$. For now, consider running an infinite number of Gibbs samplers at the same time, this would normally be computationally intractable since we would need to maintain an infinite number of sample sets. However, for discrete distributions we can represent these samples efficiently with a single vector. In our Ising model, for example, we can represent the sets of samples with a set of 2D vectors $\\{V_i | V \in \mathbb{R}^2, i \in (1,...,n)\\}$. The first component in the vector $V_i$ gives the proportion of the chains where $X_i = 1$ and the second component gives the proportion where $X_i = 0$ (of course we only actually need 1 dimension to represent this, but never mind). So, assuming we initialize the variables our Gibbs samplers uniformly then all of these vectors would equal $[0.5, 0.5]$. We can then continue with the regular Gibbs updates: loop through each of the variables (vectors) and re-sample them according to the neighbouring potentials (conditionals). After a burn in period these vectors will represent the set of samples which are drawn from the stationary distribution $P$, we can then compute the marginal distribution for each of the variables by **averaging** these vectors. Now, other than the averaging step, this procedure is identical to loopy BP. And, in fact, the averaging is not likely to matter since these values tend to converge in mosts cases anyway. If in fact, loopy BP is exactly the same as an "infinite" Gibbs sampler why is the literature on these two methods so different. All the Loopy BP analysis seems to be concerned with how it is minimizing the Bethe free energy while the Gibbs literature is focussed on mixing rates and the ergodicity of the Markov chain. Also, would simply averaging the messages in a Loopy BP inference procedure provide a correct estimate in cases where the updates are oscillating? Could the advancements in the Gibbs sampling literature, such as block Gibbs, be used in Loopy BP schemes to speed convergence? Lastly, if Loopy BP converges to an incorrect solution, does that imply that the Gibbs chain is non-ergodic?


title: Based on factor loadings (in factor analysis) can we give unequal weights to Likert scale items?

text: After gathering data, we compute score of any Likert (summative) scale (previously identified as factor in factor analysis) by adding up its individual item scores (and maybe dividing the sum by the number of items to get the mean score). In that calculation, we assume that every item in the scale has equal weight. However we know from the factor analysis that some of the items had greater factor loadings than the other ones comprising that scale. They are thus explaining more of the variance. **By using those factor loadings is it possible to give unequal weights to items?** For example in a 6-item scale maybe item 4 is more effective on that scale score than other items. Or, to restate my question: Although items of a Likert scale (construct) have not equal factor loadings (explaining variance of that factor) **why do researchers generally use Likert scales with equally weighted items?**


title: How would you write the model?

text: Suppose you have SAS code as follows. This is from http://www.ats.ucla.edu/stat/sas/faq/anovmix1.htm.                PROC MIXED DATA=long;       CLASS sub group trial;       MODEL dv = group trial group*trial;       REPEATED trial / SUBJECT=sub TYPE=CS;     run;       How would you write the model for this? Would it be: `dv = group + trial +group*trial + subj + error`? I am a little confused about how to write the mixed model in terms of a mathematical equation.


title: "Within transform" for logistic regression

text: I wish to model a binary outcome with controls for several thousand time fixed effects in a panel setting, without a special interest in the actual effect coefficient estimates. In variants of linear least squares regression, I could perform a "within transform": demeaning the explanatory variables and response variables within times, and estimating the coefficients on the transformed data. My question: What, if any, is the corollary of this procedure for a binary outcome setting? There is too much data to actually estimate the effects.


title: Multiple linear regression degrees of freedom

text: > The degrees of freedom in a multiple regression equals $N-k-1$, where $k$ is > the number of variables. Does $k$ include the response variable (i.e., $Y$)? For example, in the model $Y = B_0 + B_1X_1 + B_2X_2$, then does $k = 3$ (i.e., 1 df each for $Y$, $X_1$, & $X_2$)?


title: Appropriate distribution for bounded data set

text: I am designing a points-scored test. There is a limit on the maximum amount of points possible, as well as on the fewest amount of points possible. I have had a test group take the test and graphed their results, which form a lopsided bell curve or sorts. I am trying to figure out what kind of distribution would be best to describe this distribution, for the purpose of using it to calculate percentile scores for any future test takers. I would use normal, but intuitively, I feel that the highest percentiles would never be reached, since they would be beyond the upper bound of the data set. Would a truncated normal distribution or beta distribution be best? Any help, intuitive or direct, would be appreciated. I am mostly working in the R environment. ![Sample distribution of part of my data set.](http://i.stack.imgur.com/LVHnC.png)


title: Filtering outliers from geo-spatial-temporal log

text: I have downloaded my Latitude location history from Google for the time of about three years and now I'd like to, for starters, visualize where I've been. It turns out that the history contains some outliers. For example, I have been located in China or in Africa, where in fact I haven't been. Other outliers are more in the nature of back-and-forth between a location where I have in fact been and some different location a few hundred meters away. As you might imagine, the history consists of geospatial positions (latitude/longitude coordinate pairs) and time stamps. The median time interval is 60 seconds, but some intervals are much longer (phone off etc.). I am wondering if there is a "standard mechanism" ready to be used to remove outliers (locations where I haven't been). My own ideas end up in some (maybe theoretical) issues. For example, if I look at locations from step to step, I can easily tell what distance I must have travelled in which time. So I could iterate the positions from start to finish and erase points which would have required me to travel faster than humanly possible. But when comparing point n with n+1, how can I tell which one is likely to be the "true" position? And when I simply delete n+1, I will have to re-visit n in order to now look at the movement from n to n+2. And so on. So I thought, let's hear what the experts have to say. As you can tell, I'm not one. You can talk code (Python) and R to me and I have basic stats knowledge.


title: Multivariate regression and use of proportion type variables as DV in it

text: I have two very important things to know. Can anyone help? ### Question-1 If I have three categorical dependent variables and a continuous dependent variable, which are correlated (or associated), then if I conduct separate regressions with each dependent variable instead of a multivariate regression, what should actually be the basic problem? Shouldn't the change be only in the standard errors? Am I going to get correct p-values (I understand probably I will not) in both logistic regressions with the categorical DVs and in the linear regression? ### Question-2 If one of my dependent variables is in proportions (the one that is continuous as I mentioned previously), how should I analyze it? My interest is not prediction. I just want to see the effect of the independent variables. But still I am not feeling confident with linear regression. The predictions will surely give values outside the range of the proportions (0.00-1.00). Can logistic regression be an option? So that log(p/(1-p)) can be used as the dependent variable? So how do I define this log transform in SPSS?


title: Gradient of Hinge loss

text: I'm trying to implement basic gradient descent and I'm testing it with a hinge loss function i.e. $l_{\text{hinge}} = \max(0,1-y\ \boldsymbol{x}\cdot\boldsymbol{w})$. However, I'm confused about the gradient of the hinge loss. I'm under the impression that it is $$ \frac{\partial }{\partial w}l_{\text{hinge}} = \begin{cases} -y\ \boldsymbol{x} &\text{if } y\ \boldsymbol{x}\cdot\boldsymbol{w} < 1 \\\ 0&\text{if } y\ \boldsymbol{x}\cdot\boldsymbol{w} \geq 1 \end{cases} $$ But doesn't this return a matrix the same size as $\boldsymbol{x}$? I thought we were looking to return a vector of length $\boldsymbol{w}$? Clearly, I've got something confused somewhere. Can someone point in the right direction here? I've included some basic code in case my description of the task was not clear               #Run standard gradient descent     gradient_descent<-function(fw, dfw, n, lr=0.01)     {         #Date to be used         x<-t(matrix(c(1,3,6,1,4,2,1,5,4,1,6,1), nrow=3))         y<-c(1,1,-1,-1)         w<-matrix(0, nrow=ncol(x))              print(sprintf("loss: %f,x.w: %s",sum(fw(w,x,y)),paste(x%*%w, collapse=',')))         #update the weights 'n' times         for (i in 1:n)         {           w<-w-lr*dfw(w,x,y)           print(sprintf("loss: %f,x.w: %s",sum(fw(w,x,y)),paste(x%*%w,collapse=',')))         }     }     #Hinge loss     hinge<-function(w,x,y) max(1-y%*%x%*%w, 0)     d_hinge<-function(w,x,y){ dw<-t(-y%*%x); dw[y%*%x%*%w>=1]<-0; dw}     gradient_descent(hinge, d_hinge, 100, lr=0.01)      Update: While the answer below helped my understanding of the problem, the output of this algorithm is still incorrect for the given data. The loss function reduces by 0.25 each time but converge too fast and the resulting weights do not result in a good classification. Currently the output looks like               #y=1,1,-1,-1     "loss: 1.000000, x.w: 0,0,0,0"     "loss: 0.750000, x.w: 0.06,-0.1,-0.08,-0.21"     "loss: 0.500000, x.w: 0.12,-0.2,-0.16,-0.42"     "loss: 0.250000, x.w: 0.18,-0.3,-0.24,-0.63"     "loss: 0.000000, x.w: 0.24,-0.4,-0.32,-0.84"     "loss: 0.000000, x.w: 0.24,-0.4,-0.32,-0.84"     "loss: 0.000000, x.w: 0.24,-0.4,-0.32,-0.84"     ...


title: How do I get non-parametric test values (not just p) in PASW 18?

text: from what I can tell, PASW v.18 (the new version of SPSS) only gives you the p value of nonparametric tests. I am calculating Kruskal Wallis and Mann Whitney tests, and need to report the test statistic, not just p. Can someone please help? Thanks!


title: Averaged continuous Kernel Density Estimates in lieu of a discrete Kernel Density Estimate in Monte Carlo Proceedure

text: I am thinking of using this code in a Monte Carlo routine to generate Kernel Density Estimates for subsequent use in a Naive Bayes Classifier (see this earlier post). The author of the code states on the above linked page that it will "recognise that the data you have provided is perfectly discrete and since discrete data does not need smoothing, the selected bandwidth should be zero" in the case of ties. However, I do not want this but can envisage that, given the parameters I intend to use, multiple repetitions in a large Monte Carlo routine will very likely produce ties in the generated data. To prevent this I am thinking of limiting the number of iterations in the MC routine and employing checks to ensure non-discrete generated data for use in the Kernel Density Estimate code, and then repeating. Perhaps this is more clearly explained thus: instead of a 100,000 iteration MC, which is highly likely to produce unwanted ties, and then one KDE, do a 100 iteration MC and one KDE and repeat this 100 and 1 routine 1000 times. This will result in 1000 separate but similar continuous KDEs which can then be averaged to produce a single "unified" continuous KDE. Is this a valid approach? Are averaged KDEs a statistically sound methodology? The discussion section here would seem to imply that even if there are deficiencies in this approach, they wouldn't necessarily be too troublesome.