title: markov chain for prediction

text: I am using markov chain process for prediction. I created a transition matrix using training dataset .Now I want to experiment in test dataset and want to compute the accuracy of transition matrix for test dataset. How can I measure error rate of the prediction.


title: What's the difference between $p(x \mid \theta)$ and $p(x; \theta)$?

text: > **Possible Duplicate:**   >  Meaning of probability notations $P(z;d,w)$ and $P(z|d,w)$ Can you tell me the difference between $p(x \mid \theta)$ and $p(x; \theta)$? I know $p(x \mid \theta)$ is the conditional pdf, then what about $p(x; \theta)$?


title: What is the meaning of the semicolon in $f(x;\Theta)$?

text: In section 6.2, in the second paragraph of p. 335 (image below) of "Probability and statistical inference 7e" by Hogg and Tanis states: > perhaps it is known that $f(x;\Theta)=(1/\Theta)e^{x/\Theta}$ where $x$ is data and $\Theta$ is a parameter. What does "$;$" mean in this context, as opposed to "$,$" or "$|$", all three are used in different ways in the same textbook ("," and ";" are used on the same page, $|$ in the standard statement of Bayes' Theorem)? I think I understand "$,$" and "$|$" and I read $f(x,\Theta)$ as "function of data and parameters" and $f(x|\Theta)$ as "function of the data given parameters". Here is a scan of the page for more context: ![enter image description here](http://i.stack.imgur.com/poU4m.jpg)


title: Probability notation question

text: I am reading the Hidden Markov Model note http://cs229.stanford.edu/section/cs229-hmm.pdf from Stanford and have problem finding the meaning of the notation described below. What does the semicolon (;) means in below (Page 2 in above link)? $$P( z_t, z_{t-1},\ldots, z_1; A )$$ $A$ is the transition matrix. How should it be read? For example, I know $P( x | y )$ is "probability of $x$ GIVEN $y$"; $P( x=s_1 | y=o_1 )$ is "probability of $x$ being $s_1$ GIVEN that $y$ is $o_1$". Thanks in advance for any help for this possibly trivial, but surprisingly hard to find answer. **Further question on same notation issue.** In Wikipedia article on EM algorithm, They have $$L( \theta;X ) = p( X,Z | \theta )$$ In the above, X is the set of observed data, Z is the set of observed data and $\theta$ is a vector of parameters. Based on my current understanding of the notation, I will read the above as "the likelihood of theta given X is equal to the joint probability of X and Z given theta". In this context, X contains observed data, which to me appears to be random variables rather than fixed parameters. Why is semicolon ';' used here? **Update** I have just noticed a few other threads asking the same question, when these threads shown up as related threads on right pane. My apology that I couldn't find them earlier. The following two threads have elaborated answers, in agreement with the accepted answer of this thread. Meaning of probability notations $P(z;d,w)$ and $P(z|d,w)$ What is the meaning of the semicolon in $f(x;\Theta)$?


title: What model can I use to describe the following time-series?

text: I'm wondering if someone might be able to help me locate an appropriate model for the following two time-series (the cyan and blue one, the reds are rolling means). ![\[The time-series in question\]\(http://i.imgur.com/pmXerft.png\)](http://i.stack.imgur.com/LN9RP.png) I'm looking more for a general direction based on the attributes of the data rather than a "use this" answer. Otherwise I'll learn less. Though all assistance is welcome.


title: Even more with the Kolmogorov-Smirnov test with R software

text: This follows on from the previous question on differences between K-S manual test and K-S test with R. My frequency sample was               a=c(0,1,1,4,9).      Then the observed sample is                obs=c(2,3,4,4,4,4,5,5,5,5,5,5,5,5,5)      The expected sample is then               exp=c(1,1,1,2,2,2,3,3,3,4,4,4,5,5,5)      I hope you agree. First, I use `ks.test`, like another time:               ks.test(obs,exp)          data:  oss and att          D = 0.4667, p-value = 0.07626      Then, I use the ks.test the other way: The expected distribution can be the uniform. Do you agree? And then:               ks.test(obs, "punif", 0,5)          data:  obs           D = 0.6667, p-value = 3.239e-06      ### Question   * Why do the two approaches give different results?


title: Test to use to measure the association of half-deciles

text: I'm querying a large (100 million row + ) SQL Server database and doing some preliminary work on the correlation between visits to the emergency room and total healthcare costs. I ran a query and the data is returned to me in something like looks like:               personID     num_er_visits_half_decile     total_costs_half_decile     1            1                              3     2            8                              3     4            9                              6      Here a half-decile is like a quartile, but at the 5% level. So from the table we can read that person 1 is in the 5th percentile for the total number of ER visits, and is in the 15th percentile for overall total costs. Both of these variables are continuous and the sample size is ~70,000. Would this be a statistically relevant comparison - comparing the percentiles of ER visits to the percentiles of total healthcare costs? If so, what would be the best test to use in this situation? I have access to multiple types of software to perform the analysis, but I don't know what might be the most prudent test to use.


title: Using Duan Smear factor on a two-part model

text: I'm running a two-part model on a health insurance claims dataset where I predict the probability of nonzero health care costs using a logistic regression (1st part), then predict the magnitude of the log health care costs using an OLS regression (2nd part). Question: When calculating the expected health care costs = (prob. nonzero claim)x(expected claim amount)x(Duan Smear factor) what is the value for the expected claim amount for the zero $ claims? For example in the following table:                   member  observed cost  prob. of nonzero cost  expected cost  predicted cost                                  1       0              .3                     ?              (.3) x (?) x (ds factor)               2       0              .2                     ?              (.2) x (?) x (ds factor)               3       1,000          .6                     900            (.6) x (900) x (ds factor)           4       5,000          .7                     7,000          (.7) x (7,000) x (ds factor)         what is entered into the expected cost column for members 1 and 2 if the OLS regression is only applied to the nonzero cost data? Do we score the zero cost claims using the OLS regression parameters to calculate expected cost?


title: Bayesian variable selection in R

text: There are at least three R packages providing some functions to perform a Bayesian selection variable in linear Gaussian regression model: LearnBayes, mombf and BMA. I would be glad to know some opinions about which one is the best.


title: Is it possible to get high AUC while the correlation between predictor and response is very low, around 0.01?

text: My data has three continuous predictors and one binary response. I built a logistic regression model but AUC is only 0.52..it's almost like the model did nothing.. Then I calculated the correlation between the response and predictor. The correlation are all very low, around 0.01, all less than 0.05. That is to say, the predictors and response are actually independent. My question is: 1.is it possible to get high AUC while the predictors and response are almost independent 2.Since my response is binary data 0/1, is the correlation a good measure on independence? If not, what is a better way to test independence? Thank you everyone!


title: How to supplement comparing benchmark results (means) with a two-sided t-test?

text: When comparing two runs _(means)_ of different benchmark tests I use an unpaired two-sample t-test assuming equal variance. The problem I've run into is this often results in the same benchmark test result being statistically significantly different from re-runs of the same benchmark test on the same computer/device. I noticed that from run to rerun the difference can swing about `~5%`. What is the correct way to account/allow for this? ## Update (1.22.2012 @10pm) Here is a link to a test page that will run the same benchmark `30` times _(please be patient)_ and then create a histogram of the `mean` values represented as a percent of the most frequent `mean` value _(lowest`mean` value to the far-left, highest to the far right)_. Each one of the `30` benchmark runs repeats the test, collecting samples, as much as possible for `5` seconds. So it takes about `2 ½` minutes to finish all `30` runs. In this example the sample size for each of the `30` benchmark runs in Chrome will range from something like `86` to `93` and in Mobile Safari something like `47` to `53`. Even though the results are displayed in operations per second the samples are composed of the time, _in seconds_ , it takes to execute a test one time. This results in `mean` values that are **very small**. In this example, in Chrome, they are as small as `0.0001135` and `0.0001137` and `0.0001128`. When the `30` benchmarks are finished the result cells are highlighted. The `green` colored result cells are those that are the fastest and statistically indistinguishable from each other _(if this were perfect all cells would be`green`)_. Here are some screenshots of the histogram from Chrome and Mobile Safari: Chrome #1, Chrome #2, iPhone results #1, iPhone log #1, iPhone histogram #1, iPhone histogram #2 The distribution is inconsistent from test to re-test and from Chrome to Mobile Safari. This may be because the `means` are **so small**. I currently clip the `mean` values, for the histogram, to `7` decimal places. ## Update (1.23.2012 @2:30am) Here is some example data that would be used to generate a histogram:               {     // "the mean": frequency the mean was computed       "0.0001465":1,       "0.0001562":1,       "0.0001564":1,       ...     }      When I clip the `means` to `5` decimal places and rerun I get data like: Chrome #3               { "0.00012": 30 }      All 30 tests have the same `mean` when clipped to `5` decimal places. ## Update (1.25.2012 @12:40am) In my update from _1.22.2012_ I mentioned that each benchmark run collects samples for `5` seconds. Each sample is created by executing the test as many times as possible in `~50` milliseconds. Timer resolution in JS in `1ms` so to get measurement uncertainty of `~1%` I `(1/2)/0.01 = 50)`. After it's clocked I add the computed time per call _(timeTaken / numberOfCalls)_ to the sample and repeat the process collecting samples for `5` seconds.


title: How are classifications merged in an ensemble classifier?

text: How does an ensemble classifier merge the predictions of its constituent classifiers? I'm having difficulty finding a clear description. In some code examples I've found, the ensemble just averages the predictions, but I don't see how this could possible make a "better" overall accuracy. Consider the following case. An ensemble classifier is composed of 10 classifiers. One classifier is has an accuracy of 100% of the time in data subset X, and 0% all other times. All other classifiers have an accuracy of 0% in data subset X, and 100% all other times. Using an averaging formula, where classifier accuracy is ignored, the ensemble classifier would have, at best, 50% accuracy. Is this correct, or am I missing something? How can taking the average prediction from N potentially clueless classifiers possibly create a better prediction then a single classifier that's an expert in a specific domain?


title: How to define a BUGS/JAGS or WinBUGS model with a likelihood node that includes two integrals?

text: I would like to implement a BUGS/WINBUGS or JAGS model that specifies the likelihood node by marginalizing over two parameters. Is this feasible in BUGS/WINBUGS or JAGS? This is the posterior \begin{align} &P\left( g_{x} , g_{y} | \mathbf{a}\right) \propto P\left( \mathbf{a} | g_{x} , g_{y}\right) P\left( g_{x} , g_{y}\right) \end{align} and here is the likelihood \begin{align} &P\left( \mathbf{a} | g_{x} , g_{y}\right) = \int_{\boldsymbol{\omega}_{R}}\int_{\boldsymbol{\omega}_{T}}P\left( \mathbf{a} | g_{x} , g_{y} , \boldsymbol{\omega}_{R} , \boldsymbol{\omega}_{T}\right)P\left( \boldsymbol{\omega}_{R}\right) P\left( \boldsymbol{\omega}_{T}\right)\mathrm{d}\boldsymbol{\omega}_{R} \mathrm{d}\boldsymbol{\omega}_{T} \end{align} where \begin{align} \boldsymbol{\omega}_{R} \\ \boldsymbol{\omega}_{T} \\ g_{x}\\ g_{y} \\ \end{align} are all stochastic nodes


title: Specifying group covariates and cross level interactions in lmer

text: I'm new to R, and despite trying to read as much as I can about how lmer works in R, I still don't feel like I know how to correctly specify more complex models using the lmer syntax. For example, at the moment I want to use lmer for a two-level multilevel model, where the first level is features of a specific course taken by a specific student in a specific semester (with covariates like kind of course, teaching method, etc) and the second level is the student, which also has a number of covariates (e.g. ethnicity, gender, age, gpa at the beginning of the study, and score on a specific instrument). I also want to assess interactions with the teaching method, for example by including cross-level interactions between the second-level covariates ethnicity, gender, age, gpa, and the instrument score and the first-level covariate teaching method. For the sake of readability, let's limit the equation to teaching method, course type, and score. It seems to me that I have seen three different ways of doing something like what I want to do in R - they are clearly all specifying something somewhat different, but I can't figure out what the underlying math is supposed to be for each case. So, for example, in different online references and the books on MLM that I have on hand, it seems that one of these three models is recommended for what I want to do:               course_outcome ~ course_type*teaching_method + score*teaching_method + (1|student)     course_outcome ~ course_type*score + (teaching_method|student)     course_outcome ~ course_type*teaching_method + score*teaching_method + (teaching_method|student)      I'm a little confused about the differences between how lmer interprets each of these three codings - is there any chance that someone who really understands R better could possibly translate this into the basic regression equation(s) structure that would be calculated in each of these three examples? Or direct me to a reference that might more clearly explain the difference by giving the regression equations for each case? Thanks for your time!


title: Sparsity regularization for eigenvectors

text: One way to think about finding the eigenvectors of a matrix $A$ is that they are the critical points of the functional $\vec x^\top A \vec x$ subject to $\|\vec x\|_2=1$. To regularize this problem, it is easy enough to ask that we minimize $\vec x^\top A \vec x+\varepsilon\|\vec x\|_2^2$, since this corresponds to finding eigenvectors of $A+\varepsilon I$. Is there a way to replace $\|\vec x\|_2^2$ with an $L^1$ norm to encourage sparsity in $\vec x$? What sort of computation would result? Is it tractable?


title: Histogram PDF Fitting - bin Variance Poisson or not

text: So my basic query is just because you've binned and counted the data into a histogram are the error bars on the bins necessarily the Poisson $\sqrt{N}$? Longer explanation: I have an image of pixel values which represent brightness values. I am binning them into a histogram to get the PDF of the brightness distribution and then trying to fit a predicted model PDF to that. I'm fitting just using chi-square/least squares but if I use $\sqrt{N}$ as the error bars then there is basically never a "good" fit (reduced chi-squared of 3 or 4 for ~200 DOF) even for models we think should be good fits, the error bars seem too small. We only have one set of real data so I tried using one model and generating a bunch (~30,000) of simulated images drawing the pixel values from the model and randomizing the positions and making histograms for each one. I then compute the mean of each bin from these simulations and the standard deviation. The standard deviations are definitely larger than $\sqrt{N}$. And now if I take the PDF from one of those and compare it the predicted model PDF it gives a good chi squared using the errors from the simulation but again still bad if I use Poisson variance. But my advisor/boss doesn't agree/understand. He says we're just counting so there is no reason that the error bars should be larger. My idea is that they're larger/not Poisson because its not a totally random process. The number of pixels in a bin is dependent upon the underlying model and how well your particular realization/image samples that underlying distribution. And we are not using the likelihood because its difficult to interpret for goodness of fit.


title: Testing for equivalent variation within twins

text: I am interested in determining if, for a certain type of trait, there is just as much variation between twins as there are between non-related individuals. However, I am having difficulties in figuring out how I can combine equivalence testing (as covered here and here) with the kind of paired observations which come from twin studies. To be more specific, I have four set of twins, and for each of them 400,000 measurements were made (representing 400,000 different genomic loci). My hypothesis is that the general level of variation at each locus between each twin is not significantly higher than the level of variation between unrelated individuals at this same locus. While I am far from my usual comfort zone, I have come to the following partial conclusions:   * If I consider each locus separately, I cannot use any variance based methods, such as ANOVA, since I only have 2 measurements per group and it is impossible to obtain a variance without at least 3 measurements (and even then there are 0 degrees of freedom left)   * I cannot use the squared differences of measurements between twins as 4 sets of 400,000 observations, since then I lose the opportunity to evaluate the intergroup variation of the absolute measurements. Which techniques/methods/models should I be looking into to make statistical sense of such data?


title: Chances of winning - multiple lanes

text: I assume this is stats 101 level of question. If you have PC that is set to "randomly" pick a number between 1 and 100 each time a button is pushed, the chances of getting a 50 is 1 in 100, right? So if you have two PCs doing the exact same thing, and you can press a button on either, but you can only pick one PC, what are the chances of getting a 50? Also if you have one person at each PC, each pressing the respective button, what are each of their chances that one of them will get a 50?


title: Use of factor analysis + regression

text: Independent Variable: I have a survey of 50 states indicating the amount of control the state board of education has in 31 areas answered on a three point scale (1 = total control; 2 = partial control; 3 = no control). I have a solid theoretical underpinning for all 31, or to be more precise, the literature review found evidence for all 31 as being important (Study X found items 1, 4 and 7; Study Y found items 2, 9, and 11, etc.) Dependent variable: % of students graduating HS within 4 years. What I would like to do is the following:   1. Use factor analysis (SPSS) to reduce the 31 down to no more than 4 to 6 variables.   2. Using the results from 1), run a regression vs. the % of students graduating HS within 4 years. Is there some sort of step by step guide somewhere on how to do this? Thanks. **UPDATE** Ok, this is what I have done and I believe it is correct (any confirmation would be greatly appreciated) using SPSS 18:   1. In SPSS Analyze -> Dimension Reduction -> Factor   2. Descriptives: Initial Solution   3. Extraction: Method = Principal components; Analyze = Correlation matrix; Display = Unrotated factor solution and Scree plot; Extract: Based on Eigenvalue greater than 1; Maximum Iterations for Convergence = 25   4. Rotation: Method = Varimax; Display = Rotated Solution and Loading Plots; Maximum Iterations for Convergence = 25   5. Scores: Save as variables; Method = Regression; Display factor score coefficient matrix    6. Options: Exclude cases listwise; Suppress small coefficients [with] absolute value below. 10 The result are 9 saved columns (FAC1_1, FAC1_2, FAC1_3...FAC1_9) in the SPSS sheet. The Total Variance Explained -> Rotation Sums of Squared Loadings indicates that the first 5 of these explain 51.51% of the variance. ![enter image description here](http://i.stack.imgur.com/gRxfi.jpg) So, should I then go back into SPSS run a linear regression (Analyze -> Regression -> Linear) with the Dependent Variable % of students graduating HS within 4 years and the Independent Variables being FAC1_1, FAC1_2, FAC1_3, FAC1_4, and FAC1_5?


title: error on truncated rms

text: I am computing the RMS of a sample to estimate the standar error $\sigma$ of the underlying distribution (for simplicity let say a normal distribution $N[\mu$, $\sigma$]). $ \text{RMS} = \sum_{i=1}^N (x_i - \bar x)^2 / (N-1)$ I am using a truncated version of the RMS to avoid outliers in the population. I am considering only the $x_i$ inside the smallest range $I$ contanting a fraction $f$ of the total sample (for example $f=90\%$). $ \text{RMS'} = \sum_{x_i\in I} (x_i - \bar x)^2 / N'\quad\text{with}\quad N' = Nf \qquad \text{(let forget the -1)}$ Now $\text{RMS'}$ is biased, but it is quite easy to fix it, multiplying it by a constant $\alpha$ which depend only on $f$ (see here) The question is: what is the error on $\text{RMS}'$? For example I have computed the error on $\text{RMS}$ from here and using $\text{RMS} = \sqrt{s}$: $\text{error on RMS}=\text{RMS} / \sqrt{2 N}$ I expected so: $\text{error on RMS'}=\alpha \text{RMS'} / \sqrt{2 N'}$ but it seems to don't have good coverage (to cover 68% of the times the true value $\sigma$).


title: Combining many small regressions to answer question "what was most related?"

text: I have an interesting data set for which I want to answer the question: **Question** How do I rank consumption of a key resource from my data set. I am not interested in predicting, rather measuring relative consumption of a resource. Ideally I would get a coefficient for each variable and use that coefficient to show a value of consumption, so a result might be: For a Given Time Interval (i.e. 60 Minutes Ago): V1: 80 V2: 40 V3: 20 Where the V1 with 80 had twice the consumption of Y than V2. **About the data** My data set is organized in time intervals (let's say 1 second):   * **Variables** : V1, V2, ... Vn where n can range from 3 to 10,000. Vs are intermittently present for microseconds in a given second. Some Vs might be in the 10s, some might be the 10,000s. In a given 60 second period, I might only see a few Vs. I will see all n Vs if I watch for the entire time.   * **Y** : Y is a resource which intuitively is consumed by each V.  **Observations with regressions** I've run a lot of regressions and played with the data, but it's giving me weird answers.   * If I look at smaller number of rows (either random or consecutive), I get very good regression models, 90% R, good error terms.    * It seems the magic number of rows is somewhere between 60 and 300. Anything over 300, and the results are very noisy and lose predictive value.   * If I look at an entire data set, my model gets really noisy and R2 drops pretty substantially.    * The Vs time in milliseconds has very little relation to Y, so it can't be used as a proxy.   * I frequently get a few Vs that have negative coefficients. This is not usable for the result, so I would generally hide these from the report. It's likely they are being affected by a que of Vs which  **Local regression**   * It seems to me that I could be running local regressions on a set sample size, say 60 second intervals, then combining the coefficients on each of these regressions.    * Does this make sense? I researched local regressions and it seems as though these techniques make sense, but they don't care about the coefficient values.    * If it does make sense, how would I combine the coefficient values? I'm thinking a simple SUM would be best.    * Are there other statistical methods that better accomplish this goal?


title: Python module request: Spectral density estimation for multivariate time series

text: I have worked with scipy.signal.welch and spectrum.pptm to calculate power spectral density with Welch and Multitaper methods. However as far as I can see these functions are meant for one dimensional time series only. Are there any implementation for power spectral density estimation of time series with higher dimensions; in this case at each point the spectral density is represented with a matrix and the elements of matrix (except on its diagonal) are complex numbers. I would be happy if you could share your experience. For example in R the built-in method spectrum already addresses one and multidimensional case: http://stat.ethz.ch/R-manual/R-patched/library/stats/html/spec.pgram.html


title: Multiple regression with dependent variables

text: I have a dataset with 3 variables (X,Y and Z) and I want to find the best estimates for the constants a,b,c & d. I have been looking into multiregression analysis, but that does not seem to work for this more complicated formula. > Y = (a + b * Z) * X^(c + d * Z) Someone got any leads?


title: Maximum convergence rate of empirical probability

text: Given a sequence of events $$(a_1, a_2, \dots, a_n),$$ then the sequence of empirical probabilities of some event $\alpha$ is $$\left(p_1 = \frac{\sum_{i = 1}^1 [a_i = \alpha]}{1}, p_2 = \frac{\sum_{i = 1}^2 [a_i = \alpha]}{2}, \dots, p_n = \frac{\sum_{i = 1}^n [a_i = \alpha]}{n}\right).$$ Assume that this sequence converges to some value, $p^*$. If you can choose the sequence of events (i.e. which events are equal to $\alpha$ and which are not), then what is the maximum convergence rate of this sequence to $p^*$? The answer will depend on the value of $p^*$. If $p^* = 0$, then the sequence can converge immediately by never choosing event $\alpha$. Similarly, if $p^* = 1$, then the sequence can converge immediately by always choosing event $\alpha$. However, if $0 < p^* < 1$, then the sequence cannot converge immediately. So my question is, if you can choose the sequence of events $(a_i)_{i = 1}^n$ and, you want the sequence of empirical probabilities $(p_i)_{i = 1}^n$ to converge to some value $0 < p^* < 1$ as quickly as possible, then what is the maximum convergence rate? I think a simple process to achieve the maximum convergence rate is if $p_{i - 1} < p^*$ then choose event $\alpha$, otherwise choose some other event $\beta \neq \alpha$. I also think the maximum convergence rate is $1 / i$ but I'm not sure how to prove this.


title: Performing ANOVA/Regression on Percentage Data

text: I'm trying to see if there are any significant differences in 3 data sets that I have. The data is taken from around 1000 participants, each within different families who tried an experiment to see whether or not they tasted a particular flavour (salt, sour, sweet, bitter, umami) on the same part of their tongue (front, back, left, right, middle). The expected percentage of a non-related adult and child was tasting the flavour in the same place was calculated by multiplying the sum of percentages that an over-18 and under-18 would taste for one particular flavour (for example salt) would be p(under18_salt_left)*p(over18_salt_left) + p(under18_salt_right)*p(over18_salt_right) +... The percentages of related adults and children were calculated by other methods, essentially contributing as 1 if there was a match between an adult and a child. This resulted in the following: ![Analysis Results](http://i.stack.imgur.com/gdVjD.png) How should I go about in finding out if there is a significant difference between say the percentage of a father/child match and the expected probability (two unrelated participants). EDIT (07/07/2014) The format of the spreadsheet being used to calculate is shown below, where conditional IF statements give a value of "1" for each flavour if there is a match between "Father" and "Child1" given that "Family Number" is the same. ![Matching Families](http://i.stack.imgur.com/YXAUg.png)


title: Overlapping Coefficient between for two Gaussian distributions where one of them has zero variance

text: Consider the case where we have two Gaussian distributions and we are trying to find the area of overlap between them. This is also called the Overlapping Coefficient and I think it is also related to the Weitzman Measure. I am wondering about a specific limiting case when one (but not the other) of the Gaussians has a variance of zero, i.e. the PDF is a unit impulse function. In this case what would be the Overlapping Coefficient? My intuition is that it will be zero, but I am not sure.


title: How to visualize a range (min/med/max)?

text: I'm looking for a visually compelling yet immediately understandable way to visualize a range of data (min, median, max) Considerations:   1. The approach should be understandable to a wide variety of people   2. Ideally, would allow for the comparison to another set of data   3. Ideally, will work okay or both high and low N cases **What new ways can you think of to visualize this type of data**? Here are some examples: Example 1: Here is how a range of data and a comparison is displayed on Glassdoor: ![visualization of a salary range on Glassdoor](http://i.stack.imgur.com/1VL7O.png) Example 2: Here is how a range of data and a comparison is displayed on Indeed: ![visualization of a salary range on Indeed.com](http://i.stack.imgur.com/Q0Z9h.png) Example 3: Here is another very similar example from CareerBuilder: ![enter image description here](http://i.stack.imgur.com/wt74R.png) Example 4: and Trucar's visualization of a range of data (car prices paid by many users).   This is likely pushing the realm of understandability by an average audience. ![visualization of car price distribution on Trucar](http://i.stack.imgur.com/Mvyuw.png)


title: Caret and randomForest number of trees

text: I am puzzled as to why the caret package in R does not allow tuning on the number of trees (ntree) in a random forest (specifically in the randomForest package)? I cant imagine this is an oversight on the part of the package author - so there must be a reason for it? Can anyone shed light?


title: What is the name of the estimator that takes the mean of likelihood?

text: Let $X,Y$ be input and output (observed) continuous variables in $\mathbb{R}$. Let $\{y_1,...,y_n\}$ be the set of $n$ observations. Is there a name for the estimator $\hat x = \int_{x \in X} x \,p(y_1,y_2,...,y_n|x) \,dx $ (mean of likelihood)? I tried googling but I could not find its name.


title: Interpretation of coefficients in a GLM

text: After a bidirectional step wise, I have the following generalized linear model:               Y = -8.18 + 1.18 v1 - 7.32 v2      In a simple linear model (with one explanatory variable), the coefficient of my variable corresponds to the slope of my line. If the coefficient is positive, the line goes up, if it is negative, goes down. If it's positive there is a positive relation between my explanatory variable and my dependent variable and if it's negative the relation too. In a generalized linear model it is the same? In the example I give, v2 has a negative relation with my dependent variable? if there's more of v2 less of Y? I think is a silly question, but I don't get it, I appreciate the help understanding this.


title: quantile regression analysis

text: What is quantile regression? How can I use quantile regression to analyse technical efficiency in rubber farmers? I also want to use the production frontier. The response (y) variable is production. The explanatory (x) variables are tappable tree, fertilizer, chemical, total labour.


title: How to interpret output from nparcomp in R

text: I have skewed data requiring non-parametric tests. Following Kruskal Wallis tests, I want to carry out post-hoc analysis. I have chosen `nparcomp` within R because of recommendations here: https://stat.ethz.ch/pipermail/r-help/2012-January/300100.html As an example, using the ChickWeight dataset within R, the following code:               > npar <- nparcomp(weight~Diet, data=ChickWeight, type="Tukey")     > summary(npar)      Gives the following output:               #------------Nonparametric Multiple Comparisons for relative contrast effects---------#           - Alternative Hypothesis:  True relative contrast effect p is less or equal than 1/2         - Estimation Method: Global Pseudo ranks      - Type of Contrast : Tukey      - Confidence Level: 95 %      - Method = Logit - Transformation           - Estimation Method: Pairwise rankings           #---------------------------Interpretation--------------------------------------------#      p(a,b) > 1/2 : b tends to be larger than a      #-------------------------------------------------------------------------------------#           #----Data Info------------------------------------------------------------------------#        Sample Size     1      1  220     2      2  120     3      3  120     4      4  118          #----Contrast------------------------------------------------------------------------#             1  2  3 4     2 - 1 -1  1  0 0     3 - 1 -1  0  1 0     4 - 1 -1  0  0 1     3 - 2  0 -1  1 0     4 - 2  0 -1  0 1     4 - 3  0  0 -1 1                #----Analysis-----------------------------------------------------------------------#        Comparison Estimator Lower Upper   Statistic      p.Value     1 p( 1 , 2 )     0.575 0.489 0.657  2.24129062 0.1105119928     2 p( 1 , 3 )     0.628 0.540 0.708  3.70156665 0.0010425555     3 p( 1 , 4 )     0.638 0.551 0.717  3.99792977 0.0002112224     4 p( 2 , 3 )     0.560 0.464 0.652  1.59731987 0.3805179622     5 p( 2 , 4 )     0.562 0.465 0.655  1.64558875 0.3512047996     6 p( 3 , 4 )     0.497 0.401 0.592 -0.08324927 0.9999818119           #----Overall------------------------------------------------------------------------#        Quantile      p.Value     1 2.561994 0.0002112224           #-----------------------------------------------------------------------------------#       I am unsure how to interpret this output, in particular:   1.The contrast section, what do the different values of -1, 0 or 1 mean   2.The analysis section, how should the tabulated values be interpreted. Am new to stack.exchange so apologies if this would be better placed in the statistics forum Many thanks for help in Advance!


title: Poisson distribution of two independent random variables

text: Let X1 and X2 be two independent random variables. Let X1 and Y=X1+X2 have Poisson distributions with means μ1 and μ>μ1, respectively. Find the distribution of X2. I don't know how to start solving with this kind of problem. I would appreciate if someone could help me. Thanks.


title: statistical significance test on cluster analysis result

text: I've done a hierarchical clustering on a data-set composed by 33 subjects and 2 continuous variables (called V1 and V2), which produces 3 clusters. Now I'm wodering if it make any sense to perform a statistical significance test on this result. If I calculate the average value of the variable V1 (one of the two variables used to cluster my data) for each cluster, does it make any sense to compare the 3 average values to test if the differences among these 3 clusters are statistically significant or not for this particular variable? It seems that StatSoft says not. What's your opinions? If the answer is 'yes', which test should I use ? Can I use Kruskal-Wallis test to check if at least one of the clusters is different from the other clusters and then the Mann-Whitney test to analyze the specific cluster-pairs for significant differences? Thank you very much in advance.


title: Differencing a time series

text: I am looking to find the ACF of a time series, but after it is differenced. $y_t = a_1y_{t-1} + \epsilon_t , \mid a_1 \mid < 1$ I understand that to find the ACF this process needs to be stationary, but I am struggling to difference the equation due to the constant $a_1$ Any help would be greatly appreciated.


title: Data transformation for count data with many zeros

text: I have a count dataset that contains many zeros and a discrete variable that contains many zeros as well. I would like to see graphically which kind of correlation exists between these two variables. When I try in R to plot these two variables (`plot(X, Y))`) I get something that I'm not able to interpret. ![enter image description here](http://i.stack.imgur.com/cxZ4d.png) Is there any data transformation that could help me to graphically identify the relation between these two variables? I add also the histograms for the two variables, mean and sd ![enter image description here](http://i.stack.imgur.com/UdZ2z.png) In the figure below, the mean of q[,2] given q[,1] ![enter image description here](http://i.stack.imgur.com/50UCL.png)


title: Is logistic regression a non-parametric test?

text: _I recently received the following question via email. I'll post an answer below, but I was interested to hear what others thought._ > **Would you call logistic regression a non-parametric test?** My > understanding is that simply labelling a test non-parametric because its > data is not normally distributed, is insufficient. Its more to do with lack > of assumptions. logistic regression does have assumptions.


title: Is there a good specification error test against a generalized alternative?

text: Suppose I believe a sample is drawn from a population that is distributed according to some specified distributional family. I intend to estimate the parameters of the distribution using some appropriate method. However, somewhere along the way I may come to suspect that my original assumptions with respect to the functional form of the distribution may be incorrect. If my original assumption was that the distribution was normal, I knowa number of the tests for normality that I could use. But suppose I think my data has some distribution other than the normal. Are there any good tests that can be used for an arbitrary distribution against a generalized alternative (perhaps subject to some regularity conditions for it to be reliable)? For example, if I think my distribution is gamma or log-normal, are there any well-understood tests, with a literature I could find describing when they work, that would reject those choices if the data were clearly bi-modal, or strongly fat-tailed or thin-tailed relative to what is expected given a range of parameter values that is broadly consistent with the data? (I am looking forward to reaching the point where I con suggest tags. In this case, I was surprised that there is no "specification error" tag.)


title: Stouffer's Z-score method: what if we sum Z^2 instead of Z?

text: I am performing $N$ independent statistical tests with the same null hypothesis, and would like to combine the results into one p-value. After some googling it seems to me that there are two "accepted" methods: Fisher's method and Stouffer's method. My question is about Stouffer's method. For each separate test I obtain a z-score $z_i$. Under a null hypothesis, each of them is distributed with a standard normal distribution, so the sum $\Sigma z_i$ follows a normal distribution with variance $N$. Therefore Stouffer's method suggests to compute $\Sigma z_i / \sqrt{N}$, which should be normally distributed with unit variance, and then use this as a joint z-score. See https://en.wikipedia.org/wiki/Fishers_method#Relation_to_Stouffer.27s_Z- score_method. This is reasonable, but here is another approach that I came up with and that also sounds reasonable to me. As each of $z_i$ comes from a standard normal distribution, the sum of squares $S=\Sigma z^2_i$ should come from a chi- squared distribution with $N$ degrees of freedom. So one can compute $S$ and convert to a p-value using cumulative chi-squared distribution function with $N$ degrees of freedom ($p=1−X_N(S)$, where $X_N$ is the CDF). However, nowhere can I find this approach even mentioned. Is it ever used? Does it have a name? What would be advantages/disadvantages compared to Stouffer's method? Or is there a flaw in my reasoning?


title: How do SOMs reduce dimensionality of data?

text: This is a problem with which I have been grappling with for days. From my research on self-organizing maps, I know that a common feature of self organizing maps is to reduce the dimensionality of data. For example, if you had a 3x3 SOM, and an input space consisting of 50 10-Dimensional vectors, the SOM would reduce this to 50 2-Dimensional vectors. If I am creating my own SOM, where is this data? Please excuse me if my question too vague or broad. The reference vectors that are attached to each neuron in the SOM are the same dimension as the input space. The input space itself does not get reduced in dimensionality. So where is the reduced-dimension data? In other words, what data structure in relation to the self organizing map contains this data? My only guess is that this data could be found in the location of each node in the self organizing map. Thanks!


title: Hypergeometric trials - Number of trials needed to achieve a given probability

text: Given a known number of white and black balls in an urn, what is the number of trials **without replacement** required to achieve a given x probability of drawing **at least** one white ball. Which function gives me this number of trials? Thank you very much


title: Is homogeneity of variances required for changepoint analysis?

text: I want to perform a changepoint analysis with the help of the PELT method: Killick & al, Killick & Eckley I assume that the data are normally distributed before and after the changepoint. After reading these two papers I still do not know whether the assumption of equal variance before and after changepoint is required when using the PELT method ?


title: Bounding expected maximum of inner products

text: Let $x=(x_{1},\dots,x_{n})^{T}$ be $n$ dimensional random vector uniformly distributed over the $L1$ unit sphere. That is, all $x$ such that $\|x\|_{1} = 1$ have equal probability and all other $x$ have zero. Let $\{c_{i}\}_{i=1}^{n}$ be $n$ vectors such that $\|c_{i}\|_{2}=1$. How can one show that: $\mathbb{E}_{x}\{\max_{j\in[n]}|\langle c_{j},x\rangle|\} \leq \mathbb{E}_{x}\{\|x\|_{\infty}\}$


title: Why do I get exact singularity when I add a dummy variable to a multinomial logistic using R mlogit?

text: I am using the mlogit package in R to run a multinomial logistic regression on pooled discrete choice data collected using two different questionnaire formats. I want to test whether the format had a significant effect on choices. When I run the basic model I get a result. But when I run the same model with a dummy variable indicating which format the respondents saw, I get an error: "Error in solve.default(H, g[!fixed]) : Lapack routine dgesv: system is exactly singular" I was able to replicate the error using Train's Electricity dataset in the mlogit package, setting a dummy based on whether the respondent ID was odd or even:               library(mlogit)     data("Electricity", package = "mlogit")     Electr <- mlogit.data(Electricity, id = "id", choice = "choice",                            varying = 3:26, shape = "wide", sep = "")     Electr$odd.dummy <- ifelse(Electr$id %% 2 == 0, 0, 1) # As example, set dummy if ID is odd     summary(mlogit(choice ~ pf + cl + loc + wk + tod + seas | 0, data=Electr)) # Basic model     summary(mlogit(choice ~ pf + cl + loc + wk + tod + seas + odd.dummy | 0, data=Electr)) # Basic + dummy     summary(mlogit(choice ~ odd.dummy | 0, data=Electr)) # Only dummy      As with my data, the first model runs, but the second two are singular. I understand that a result will be singular if there is perfect colinearity between variables, but I don't see how this is the case here. Respondents were randomly assigned to one format or the other, and the underlying experimental design was the same in both formats, so there shouldn't be any colinearity between the dummy and the other variables. I would be grateful if someone could explain why adding the dummy leads to a singular result, and even more grateful if they could suggest a solution to avoid it.


title: How can statistical tests be categorized?

text: There are many tests in statistics. I noted that some of them may be grouped together, since they are used for similar purposes. For example   * tests of independence    * tests that compare distributions (Kolmogorov-Smirnov, Shapiro-Wilk, Chi-Square)   * tests that compare location parameters (Student t-test, ANOVA, Kruskal-Wallis, Mann-Whitney)   * tests that compare variances in groups Are there any other groups of tests with similar use? Of course, there are also many tests that are too specific and cannot be assigned to any group. Thank you in advance.


title: How to fit the coefficients of a first order linear auto-regressive function without noise in the model, and the coefficient are equal?

text: I have a simple auto-regressive function: $x_{i+1} = c - cx_{i}$ It is `linear` and `first order`. There is **no noise** in the model although there is in the data. I am using `Matlab` and have a vector (time series) of numbers that has arisen from a process governed by the above equation. I would like to obtain the value of the coefficient $c$ for the least squares fit to this data. How can this be done in Matlab by using a library function or toolbox? I have looked at the ar command, but fail to see how this can be used in my case. I consider writing instead a function to take the data and calculate the root- mean-squared error over a set of possible values for $c$ choosing the one that is smallest.


title: Does low significance prove my hypothesis wrong?

text: I have no idea about statistical testing yet I am doing a research project on a sample size of 40; it is a correlational study using Pearson's r (SPSS). My results indicate Pearson's correlations (2-tailed p) of               -.264 (.100): self efficacy     -.113 (.489): social support     -.172 (.290): happiness      My hypothesis states that these variables will be negatively correlated with stress   but it doesn't say significantly, so I think that my hypothesis is proved as it's negatively correlated. Am I thinking something which is impossible or wrong or stupid or am I right? In simple words, I cannot state that my hypothesis is proved, can I?


title: Inter-coder reliability for continuous data and 4 coders

text: I've had four students coding the frequency of certain verbal and nonverbal behaviours. In hoping to generate some sort of intercoder reliability, I've had them all code the same 10 interviews. For each of these interviews, each of the four coders have coded the number of certain verbal and nonverbal behaviours. I'm looking to determine the reliability of the coders in coding the frequency of these behaviours. It therefore is frequency data with four coders. Apologies, I hope this is a clear enough description.


title: Gaussian RBF vs. Gaussian kernel

text: What is the difference between doing linear regression with a Gaussian RBF basis and doing linear regression with a Gaussian kernel?


title: standardize two categorical variables into same unit

text: I have two categorical variables that I want to add: fruit portions = portions of fruit per day (a portion representing a handful): 1= never to 7=5 or more portions, and fruit juice frequency = how often drink fruit juice per week:1=never, 2=once a month, 3=1-2 times per week to 5=everyday. I want to add them into one category but they are in different units. Z-scores wouldn't deal with the difference I don't think, so how can I standardize them to be both either daily or weekly? thank you


title: Classify and regress at the same time

text: Is it possible to have a model that aims to classify and regress at the same time? For example if I have five independent variables and I want to use these 5 variables to predict the gender of the observation. Subsequently I let the predicted gender be the 6th independent variable and predict for income of the individual. How I train the model is by splitting the dataset into two parts. The first part is used to train the model to predict gender, income is ignored here. The second part is used to train the model to predict the income of the individual based on all the other variables including gender. Is this methodology sound? If not, why so? Thanks.


title: Degree of freedom with mixed model , using nlme package?

text: I am conducting a mixed model, but do not well understand how df is calculated, can someone help me to clarify it? I have been searching very hard on the internet but could not find the right answer. Thanks! The data in fact contains "Animal" as random effect, Breed and DFC (day from calving) and squared_DFC as fixed effect, and Red)My as the response variable. The mixed model is:               model <- lme(Red_MY~ DFC + DFC*Breed, random=~1|Animal, data= my_data)      The output:               Red_BW ~ DFC + DFC * Breed + Squared_DFC               Linear mixed-effects model fit by REML      Data: na.omit(my_data)             AIC     BIC    logLik       24045.02 24086.5 -12015.51          Random effects:      Formula: ~1 | Cow_code             (Intercept) Residual     StdDev:    7.700506 17.80507          Fixed effects: Red_BW ~ DFC + DFC * Breed + Squared_DFC                             Value Std.Error   DF   t-value p-value     (Intercept)        2.3266511 2.4810473 2688  0.937770  0.3484     Centred_DFC       -0.0232875 0.0080565 2688 -2.890523  0.0039     Breed             -1.3165288 1.1309444   80 -1.164097  0.2478     Squared_DFC       -0.0003436 0.0000345 2688 -9.957905  0.0000     Centred_DFC:Breed -0.0115884 0.0037693 2688 -3.074444  0.0021      Correlation:                        (Intr) DFC Breed  Sq_DFC     Centred_DFC        0.061                          Breed             -0.912 -0.060                   Squared_DFC       -0.183  0.002  0.009            Centred_DFC:Breed -0.063 -0.924  0.067  0.026          Standardized Within-Group Residuals:              Min           Q1          Med           Q3          Max      -4.649177035 -0.562679661 -0.004077324  0.521083911  8.448741285           Number of Observations: 2773     Number of Groups: 82      Thanks in advance!


title: Panel data: OLS assumptions

text: I am trying to run an econometric panel data (fixed effects) model with about 4000 observations (so not a small dataset). My data consists of transactions (buy and sell). The transactions are linked to a rank (1 to 5) based on the B/M ratio for the company related to the transaction. However, I am having enormous amount of trouble with the following aspect: According to the kdensity plot and the qnorm plot, the residuals does not pass the test of normality. What does this implicate? Are the p-values not to be trusted? What are the minimum requirements to a OLS regression run on panel data? ![kdensity](http://i.stack.imgur.com/Q7HzT.png) ![qnorm](http://i.stack.imgur.com/Z6Sio.png)


title: Three Events Problem Interpretation

text: So I am having trouble understanding what this problem is asking. (I am not asking someone to solve it but simply help interpret what the problem wants me to do.) The problem goes like this: > "Let $A$, $B$, and $C$ be three events. Show that exactly two of these > events will occur with probability $P(AB) + P(AC) + P(BC) - 3P(ABC)$." What I do not understand is that am I supposed to show that $P(A) + P(B) = P(AB) + P(AC) + P(BC) - 3P(ABC)$ or $P(A) + P(C) = $ SAME THING or $P(AC) = $ SAME THING or etc. I guess what is throwing me off is the "exactly two events will occur" part. I was able to show that $P(AB) + P(AC) + P(BC) - 3P(ABC) = P(ABC^c) + P(ACB^c) + P(BCA^c)$ and if this is NOT what I was supposed to do then I am completely lost.


title: Is MLE more efficient than Moment method?

text: I have got some small data sets (about 8 to 11 data points for each set), following Normal distribution. I would like to find out the 95% confidence interval of the 0.005 and 0.995 percentile of each set. Firstly, moment estimation method is employed to estimate the Normal distribution parameters, and their confidence interval is built by (mu~Normal, sigma^2~Chi-square) theorem. And find the CI of percentile by simulation. Secondly, MLE method is also employed and the parameter's CI is built by MLE~asymptotic Normal theorem. Then find the CI of percentile by simulation. As the figure shows, the MLE CI is much narrow than Moment method. We know that MLE is efficient, leading small variance and narrow CI. This understanding is consistent with our figure. But my MLE CI approach is based on asymptotic assumption, while my amount of data points is quite small. Would this (too small data amount) leads MLE's CI incorrect and worse than moment method? or it is still more efficient than moment method? Is the MLE CI too narrow to contain the 95% probability of the true value, if the amount is too small? ![enter image description here](http://i.stack.imgur.com/Mv3H1.png)


title: How to remove one of the factor from the model?

text: We have a variable "pos" in the regression which has three values : guard, forward, and center. My regression looks like y~a+factor(pos) But one of the factor's value is insignificant (i.e. forward). How can we remove it from the regression without eliminating others in R?


title: Covariance matrix in R with non-numeric variables

text: I am modeling a Gaussian distribution for a computational biology application, and I am working in the statistical package "R". In this regard, my problem is that I have to construct a covariance matrix with variables (non-numeric) and the covariance matrix is to be used in an maximizers of the likelihood function to predict the variables in the matrix. I am unable to do that because I do not have an idea of how to construct a covariance matrix with non-numeric variable in the matrix.


title: Using non linear regression model for enzyme kinetics

text: I need to choose a non linear regression model for this kind of data The experiment was a time course, for a drug, so I get phosphorylation response behaviour in time In enzyme kinetics or a binding non linear regression model, you do not take into account time, instead you plot the substrate concentration Vs Vmax or radio ligand concentration (X) Vs total binding (Y). Here I got time (X) Vs kinase phosphorylation (Y) Could you help me to get a proper model to do the fit?


title: How to find distribution of some statistics for i.i.d. normal random variables?

text: Let $Y=(Y_1,Y_2,\ldots,Y_n)^{T}$ and $Y_1,\ldots,Y_n$ are independent normal random variables with mean $0$ and variance $ \sigma^2$. Find the distribution of the following statistics and give your reasons a) $T=\displaystyle \sum_{i=1}^{6}(Y_i-\overline{Y})^2$ b) $U =\displaystyle\frac{\sqrt{5}\times Y_6}{\sqrt{Y_{1}^2+Y_{2}^2+Y_{3}^2+Y_{4}^2+Y_{5}^2}}$


title: Mean of all natural numbers?

text: Can we derive this from the summability methods (Zeta function regularization) that assigns finite values to divergent sums or products or any other?


title: Best statistical model/technique for assessing part data

text: I'm a software engineer who has been tasked with doing statistical analysis on data collected from a physical test system. The system collects data on a part by setting input conditions, reading in measured data, then changing the input conditions and doing another measurement. A typical data collection file ramps through a couple of dimensions of input data collecting output data for each. We already have a comparison routine that compares two files to one another. I now need to add the capablity to add two new compare capabilities to the program. Both are batch compare capabilities that would compare a bunch of files. In one case, all files would be compared to a control file with data collected on a known good part. We want to generate some kind of quality measure that would show how close the data sets are to the control data set. In the other mode, we assume that most of the data files are on good parts, but some are not. We need to do a compare each file to one another and then norm the result into a bell curve. I did a lot of math in school and even tutored it for 4 years in college. However statistics was never part of my formal education. For some reason my engineering program didn't think engineers needed to know statistics. I have picked up some bits and pieces here and there, but I'm not sure what type of statistical analysis is best for this task. The data set for a control file (known good part) looks something like this:               Input   Measured Out     .1  .05     .2  .06     .3  .07     etc.      A part that would test good might look something like this               Input   Measured Out     .1  .051     .2  .059     .3  .072     etc.      But a bad part might look like               Input   Measured Out     .1  .08     .2  .09     .3  .10     etc.      I need some measure that would indicate the good part compared to the control has some sort of measure of high quality, and the bad part is of low quality. Most of what I've found applies to psychological data and most of what I've learned about statistics is related to psychology (I have a bit of interest in the subject and my SO is a Psychologist who barely passed statistics in college). The nature of this sort of data is somewhat different from a psychological data set though. The data have dependencies that psychological data don't have for one thing. Can someone point me to the best statistical technique for this problem?


title: Interpreting a PCA Biplot of a time series?

text: I had a question in regards to PCA with times series data, and specifically how to possibly interpret it. Normally, PCA is used by other software that I use in relation to de-noising a data set by removing components of least eigenvalues/variance (not sure as to the precise criteria used here), and the de-noised data is passed along for other preprocessing steps. However, I decided to manually use PCA on my data and actually get a look at the results of doing so. Essentially, I have 5 variables (5 brain regions), each with 180 time points, with specific intensity values (unit is same across the 5 variables). Now, my experience with PCA is limited, but I was wondering if there are any potential interpretations that could be done with such a time series data set? From what I understand, the 4 variables (NOT the Cerebellum) are highly related to the PCA axis 1, and these 4 (RightPM, RightM, LeftPM, leftM) can be regarded as positively correlated with each other in relation to the intensity values? Confusion sets with the time points however, such as saying whether this strong positive relationship occurs at the time points plotted near the bunching of the 4 variables. I'm assuming there are issues with PCA and time series (related to autocorrelation perhaps), but I was curious if such an interpretation could be made. Any opinions on this would be great. Thank you. ![PCA Biplot of 5 variables with each having 180 timepoints](http://i.stack.imgur.com/baiN1.png)


title: Standard errors in Fixed and Random Effects

text: If the appropriate model choice is Fixed Effects in a panel study, will the inference derived from the standard errors in an Random Effects model still be valid?


title: Estimating parameters using a different method?

text: I have a probability distribution which has two parameters $a$ and $b$ I have re-parametrized the distribution such that the new distribution has two parameters $c$ and $d$ where: $c=a$ but $d = \frac{1}{b} - 1$ I can easily estimate $a$ and $b$ but I need to make inferences on $c$ and $d$, specially $d$. So my question is: If I estimate $b$ and use relationship $d= \frac{1}{b} -1$ to find out the estimated values of $d$, is this correct? Is there any information loss? Also, how can I find the standard errors and confidence intervals of $d$ if those exist for $b$?


title: Causality in Time Series

text: I am reading an article which is trying to justify the need for causal inference in their inferential framework. The thought experiment is as follows: > Suppose a statistician is asked to design a model for a simple time series > $X_1,X_2,X_3,...$ and she decides to use a Bayesian method. Assume she > collects a first observation $X_1 = x_1$. She computes the posterior > probability density function (pdf) over the parameters $\theta$ of the model > given the data using Bayes’ rule: $$p(\theta|X_1 = x_1) = \int\frac{p(X_1 = > x_1|\theta)p(\theta)}{p(X_1 = x_1|\theta')p(\theta')}, $$ > > where $p(X_1 = x_1|θ)$ is the likelihood of $x_1$ given $\theta$ and > p($\theta$) is the prior pdf of $\theta$. She can use the model to predict > the next observation by drawing a sample $x_2$ from the predictive ￼pdf: > $$p(X_2 = x_2|X_1 = x_1) = \int p(X_2 = x_2|X_1 = x_1,\theta)p(\theta|X_1 = > x_1)d\theta,$$ > > where $p(X_2 = x_2|X_1 = x_1,\theta)$ is the likelihood of $x_2$ given $x_1$ > and $\theta$. Note that $x_2$ is not drawn from $p(X_2 = x_2|X_1 > = x_1, > \theta)$. She understands that the nature of $x_2$ is very different from > $x_1$: while $x_1$ is informative and does change the belief state of the > Bayesian model, $x_2$ is non-informative and thus is a reflection of the > model’s belief state. Hence, she would never use $x_2$ to further condition > the Bayesian model. Mathematically, she seems to imply that: $$p(\theta|X_1 > =x_1,X_2 =x_2)=p(\theta|X_1 =x_1)$$ However I hardly believe that what this poor statistician should imply is: $$p(\theta|X_1 =x_1,\text{do}(X_2 =x_2))=p(\theta|X_1 =x_1)$$ Where "do(or set)" here comes from Pearl's framework of causality which can be found here and here. Now am I right about this?


title: Variance and Variograms: Sigma?

text: Can someone please walk me through this morass of σ's? There is a sigmasq in the summary output from geoR's likfit. Really this is the one I'd like to know about. I'm not clear which of all of the other σ's it is. A variance? A different parameter? nmle's variogram calls sigma "an optional numeric value used as the height of a horizontal line displayed in the plot. Can be used to represent the process standard deviation. Default is NULL, implying that no horizontal line is drawn. " I thought this was phi. geoR variofit's documentation calls it a "partial sill" and says to set a starting value. (I'm also not quite conceptually clear on what variofit is modelling differently from likfit.) Diggle & Ribiero (Model-Based Geostatistics) calls σ^2 "signal variance." Is that the same sigma? Later (p. 33), they say σ^2 is the value the (?expectation of the) vij's approach as the distance xi-xj grows, and that uncorrelated yi's have an expectation of σ^2.


title: Convolution of difference distribution doesn't conserve expectation - faulty assumption?

text: I have observations of a system which I hypothesize is defined as $T = A + G$ where $T,A, G$ are random variables. I've measured the empirical continuous distributions $T$ and $A$, and the question is, what is the distribution of $G$? Assuming $A$ and $G$ are independent, this is just the convolution integral for $T-G$, that is, since $i-(i-k)=k$, $$ P_G(G=k)=\int_{i=0}^\infty P_T(T=i)P_A(A=i-k) $$ Because the data for $T$ and $A$ are empirical distributions, I computed discrete PMFs with various discretizations and the following results don't depend much on what resolution to discretize. Inputting my data, I get a nice distribution for $G$, however, there are two things I don't understand.   1. While the discrete PMFs that approximate $A$ and $T$ sum to 1, the sum of $P_G =.543$. Do I just renormalize?   2. $E[G]=.78$ while $E[T]=1.52$ and $E[A]=1.32$. Mean of $G$ is not affected by (1). I know even for correlated variables, $E[A+B]=E[A]+E[B]$, so something must be wrong. The only assumption I've made is that $A$ and $G$ are not correlated (so the convolution applies). Is this result telling me $A$ and $G$ are correlated? What can I do to find out if they are, given what I have? I cannot observe $G$ without $A$. Edit: changed "Assuming $A$ and $G$ are not correlated" to "Assuming $A$ and $G$ are independent" per Dilip since the terminology used was incorrect.


title: Significant p-value and relative risk crossing 1

text: I am trying to convert results I am finding in scientific articles into relative risks so that I can compare studies in a meta analysis (if there is enough homogeneity in the studies). I came across a study entitled "Intervention to improve practices of prescribing appropriate medication before an operation; pre-and-post- intervention study". In the pre-intervention group, 16 out of 233 patients (so 217 with correct prescription) had an incorrect prescription and post intervention 3 out of 137 patients (134 with correct) had an incorrect prescription. In the paper, a p-value of 0.046 was given, using Fisher's exact test. I checked it using a hand calculation and got 0.049. When I put this result into an RR calculator, I get a RR of 0.32 [0.09, 1.07]. Now I am a relative stats rookie but I thought the 95% CI should not cross 1 if my p-value for comparing two proportions is below 0.05.


title: Fault causal analysis

text: I have some time-series data, which deal with various incidents / asset failures / faults somewhere in the network that have an effect on performance (we call them Key Performance Indicators). Now, I am aiming to find a correlation between a typical type of fault causing an overall outage and, therein, affecting the performance factors (targets). Can anyone help me find such a fault causal effect analysis, in terms of what methods / techniques to use and their intricacies? I have access to SPSS, MATLAB and SAS.


title: Ask a question about the offset in soft SVM?

text: If I use a balance dataset and I make the mean of dataset is 0. Can I get rid of the offset b? Is there any available simple method that for user to train a soft SVM without b? Thank you!


title: How to find the input layer and the architecture for a Neural Network

text: I'm a software developer and I'll like to learn about neural networks. At this point I've find a problem which I'll like to solve at some point. It is about electrical load forecasting. I'm looking for similar problems and it will be great if I can find some similar examples with solutions. At this point I'm having troubles in finding the right model for the RNN, and more exactly I'm struggling with the input layer. As the output I need the forecast values for each hour. Any reference to books, links resources or advices are welcome and very appreciated. This is the problem that I'll like to solve: A very small factory, use a number of equipments to produce bread. Some of them are electrical equipments which means that they consume electrical power. Knowing which equipments will run on the next day, an electricity consumptions forecast can be computed. The equipment named E.V. is a special case of equipment. The human operator completes it's values in an empirically manner in order to have a good forecast for the next day. Those values can be positive or negative. The 'on' thing from the bellow table represents the status of the machines.               +----------------------------------------------------+       |equipment name|power| 1h| 2h| 3h| 4h| 5h| 6h| 7h| 8h|       +----------------------------------------------------+       |Equipment 1   |  2MW| - | - | on| on| on| - | - | - |     |Equipment 2   |  5MW| - | - | - | on| on| on| - | - |     |Equipment 3   |  1MW| on| on| on| on| on| on| on| on|     |E.V.          |     | .1|-.1|-.1| .1|-.2| .1| .1|-.1|     +--------------+-------------------------------------+     |total/(forecast)    |1.1|0.9|2.9|8.1|7.8|6.1|1.1|0.9|     +--------------+-------------------------------------+     |real consumption    |0.9|0.9|2.7|8.2|7.9|3.1|0.8|0.7|       +--------------+-------------------------------------+      The problem is that the machines are not running at their maximal power, so it will be great if a more exactly forecast can be build. **I have data from 2 years back for every day. Also, do you think that date or week day is a good candidate for the input layer?** _I'm not very efficient in understanding an answer with a math only approach. Any example with something more appropiated with my problem is verry appreciated_


title: Standard error of mean of gamma distribution

text: I am trying to determine whether the means of two Gamma distributions are significantly different. To do this, I am trying to determine the Wald Statistic as               Z_0=(Mean_1-Mean_0)/SE(Mean_1)      and thence getting the F value using               F=Z_0^2/q      where q is the number of degrees of freedom. My first problem is determining the standard error of the mean. Is the standard error of the mean of a gamma distribution simply the standard deviation divided by the square root of the count, as with a normal distribution? I have tried to find the answer for this using               "standard error" "gamma distribution"      and found this which initially suggested that the standard error is the same as the variance and was then corrected to say that it is the same as the standard deviation (which would not take the count into account).


title: clustering in Item-based collaborative filtering

text: I read about item-based collaborative filtering in the paper from Sawar et al. I want to apply clustering on items to find the most similar items and then apply the prediction. Is this a good approach? Are there any references which use this type of item-clustering approach. thanks **References** Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item- based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web (WWW '01). ACM, New York, NY, USA, 285-295. Retrieved from http://files.grouplens.org/papers/www10_sarwar.pdf


title: Do t.test on a matrix with R

text: I have a question related to but different from this one: http://stackoverflow.com/posts/19121923/. I would like to do a t.test on a matrix of data, except that my matrix looks a bit different. I think I should use t.test with the "formula" and "data" attributes, but I'm not sure how I can reference my data correctly for this to work. My data is a frequency matrix of word frequencies in a number of different literary texts and I'm interested in finding out which words have significantly different means in two groups of texts. Each column corresponds to one text; each row corresponds to one word; the values are the relative frequency of each word in each text. Here is what my data looks like (simplified crop of my much larger matrix).                    Moliere1     Moliere2     Regnard1     Regnard2     de   3.7942024586 3.8420867113 3.6368762334 3.3582683431     vous 3.0050083472 2.091411115  3.3126585847 2.705713831     et   2.3068750949 3.0666196687 2.8615731604 3.1036129238     je   2.2765214752 2.0679121137 1.8466309557 2.4033105205      Basically, I would like to find out which words have significantly different mean frequencies in each of the two authors' texts combined, something which is a hint at the stylistic difference between the two authors. I could split the matrix in two and/or rotate it if that is of any help. I know how to work this out in OpenOffice Calc, but it would be neat to do it with R. (I use an R package called "stylo" to create the frequency matrix in the first place.)


title: Simple $\chi^2$ test question

text: I have the following question which seems extremely easy, but the way the data are set up is causing me some uncertainty: ![enter image description here](http://i.stack.imgur.com/OT0q9.png) I plan to solve this problem through finding the maximum likelihood estimate for θ=P(a sentence contains at least one Class 1 word), and then performing a Chi-Squared test. This, however, requires interpreting that the data row "# Class I Words" in fact means "# Sentences containing at least one Class I Word." This is clearly a deviation from what the label in the problem suggests, but it seems that without setting it up in such a way that each sentence can have a (0,1) outcome, a binomial distribution will not work at all. Otherwise, you could have in theory n=5 sentences (as the suggested distribution gives) but x=i≥5 Class I words in total (although this case does not actually arise in the data given). Assuming that we are only counting the number of sentences out of a group of 5 that contains at least one Class I Word does away with this issue and makes the problem workable. Any suggestions about how this question might be answered without making the assumption I have here would be greatly appreciated. Is my reasoning correct, or is there a different way to answer this question?


title: Multiple testing adjustment within versus across regressions?

text: I understand the basics behind multiplicity problems (I think at least), but there is one issue that I find hard to get my head around. Let me exemplify the problem: Assume I regress `y = a + b x1 + c x2 + d x3 + e` where `y` is the outcome variable, `a` the constant, `x1:x3` are regressors and `e` is an error term. I am interested in whether or not any of the regressors is statistically significant. After conducting the regression, I estimate another equation, namely `y = a + b x1 + c x2 + d x3 + Z + e`. Here, `Z` is a vector of confounders. I am now interested in seeing if the coefficients change and if the variables are still significant (assuming that some of them were signficant in the first step). Note that the only difference is that I have added `Z` to the regression, and I am strictly speaking not interested in `Z` itself but how the other coefficients look like after controlling for `Z`. Question: if I wanted to adjust for multiplicity, my understanding of the discussions is that I should do so across all regressions. I.e. for the above case, if I were to apply a Bonferroni adjustment (even though I know there are better ones out there), I would multiply all p-values on the regressors of interest (`x1:x3`) with 6, because I have conducted 6 independent t-tests in total. However, my intuition tells me that something is wrong here, and I feel inclined to rather do the adjustment within each regression. That is, I would multiply the p-values in the first and the second regression each with 3, since in each regression I have conducted three tests. My reasoning is that, in principle, the hypotheses changes and the regressions are very much dependent, but I lack the deep understanding of the subject of multiplicity adjustments to formalize my intuition (if it is right at all, which I am not sure it is). Hence: Would my procedure be right, or should I adjust for all tests? If my procedure is right, can someone provide a slightly better informed explanation? I highly appreciate any feedback on this! Thanks in advance.


title: Linear model or component analysis on timecourse data

text: I have some timecourse data which plotted looks like the figure below. I want to better describe the difference between the two conditions. My adviser advised :D me to use a linear model and observe a time effect (interesting!) a main condition effect (probably none) and a condition*time effect (most interesting). Now, I am wondering what this information would look like - the time course isn't gradual, and I have run similar tests with R's `lme4::lmer` and `nlme:lme` and for continuous variables (in my case time) they just output one value as if the progression is linear. Obviously in my case it is not :-/ ![enter image description here](http://i.stack.imgur.com/CpQ5X.png) I have previously worked with such data and used PCA (which indeed, does not give me just-linear progressions) - would it be a good substitute for a linear model in my case? Also, the problem with PCA is that I have no idea how to reap the added information one would get from single participants (with mixed models, I can just add the participant IDs as random variables and feed it the raw data) - is there any way to pass the entire (un-meaned) per-participant data to, say Python's `mdp.pca()` and get something reasonable out of that?


title: Confidence interval for multiple regression parameter

text: I'm given the least squares model:               Y = B0 + B1x1 + B2x2 + B3x1x2     Y = 12 -2x1 + 7x2 +5x1x2          n = 20      as well as some RSS's               > sum( lm( y ~ 1 )$residuals^2 )                                  #$ (to fix display bug)     [1] 456      > sum( lm( y ~ x1 )$residuals^2 )                                 #$     [1] 320      > sum( lm( y ~ x2 )$residuals^2 )                                 #$     [1] 360      > sum( lm( y ~ x1 + x2 )$residuals^2 )                            #$     [1] 288      > sum( lm( y ~ x1 + x2 + I(x1*x2) )$residuals^2 )                 #$     [1] 240      So, I know the least squares estimate for `B3` is 5. I did ANOVA on the full model versus the model where `B3 = 0`. I found the F statistic for `B3 = 0` to be 3.2. Now I need to find a 95% confidence interval for B3. I'm not sure where to go from here.


title: Kriging without covariance?

text: I am trying to krige monthly snowfall totals using data from weather stations and elevation. When I use a linear variogram model (set using a GUI and appears to be a good fit), the resulting layer seems overly smoothed to me. The variance layer for this result has a mean of about 13 cm. When I set my nugget and sill to zero the resulting layer matches almost perfectly at each station. The variance map for this result looks like a bunch of bulls eyes that cover the majority of the area (weather station spacing is good). The variance map mean value is 0.5 cm. Would it be wrong to set the nugget and sill like this?


title: Setting intercept to zero: Will this change both standard deviations and the error term?

text: After running a single regression with a forced zero intercept, I understand that $\beta$ (slope(s)) will change as $\alpha$ (intercept) will be set to zero. Easy. $\rho = \beta(\sigma_x / \sigma_y)$ is left in question... by way of.... $\beta = \frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(X)}$.....or......${\mathrm{Cov}(X,Y)}= \beta(\sigma_x^2)$ Given the forced change, will the $\sigma_x$ and $\sigma_y$ (std.devs) both remain as their non-forced original values, with $\rho$ (correlation) changing to match the new forced slope? What will happen to the error term given the forced zero regression? Note: Calculations confirmed in Excel via using `StdevP` & `VarP` functions in small sample group.


title: distribution of sample variance or sample coefficient of variation under gamma parent distribution

text: I am looking for anything related to the distribution of the sample variance or sample coefficient of variation (or joint with sample mean or conditional on sample mean) when the parent distribution is Gamma. Any information of reference would be greatly appreciated.


title: $\lambda$ in $\mathcal{L}_1$ penalized logistic regression and the likelihood on the training dataset

text: Define $J(w) = -l({w},\mathcal{D})+\lambda||{w}||_2^2$, then is it true that $l(\hat{{w}},\mathcal{D}_{train})$ always increase as we increase $\lambda$?


title: Is normalizing the features always good for classification?

text: I always read in books that when we do classification or machine learning tasks it's always better to normalize the features so to make them in one range like 0-1. Today I used weka to play with Iris dataset. First I just built a J48 classifier without normalizing the values, and the it made perfect performance. However when I normalized all the features to be in the range 0-1, the classifier made so much mistakes. Why is that? Shouldn't normalization be used always?


title: Why there is little improvement after including the predictors in logistic regression?

text: I'm working on a project on prediction of at-risk students using logistic regression. I'm new to this analysis, so I need some help to clarify some issues:   1. Should I code "return" student as 1 or "drop-out" student as 1 in the analysis (My purpose is to spot at-risk student)?   2. After I ran the analysis, I found almost no improvement concerning percentage of correct rate in the classification table between baseline model and model with predictors. (There is a significant difference, though.) Does this mean the predictors did not contribute much to discriminate the two groups? If my purpose is to find a parsimonious model, what standard(s) should I take?


title: How does a multivariate mixed CDF look like?

text: If I have multiple CDFs $X_1,X_2,\cdots,X_n$, (for simplicity I assume 2 : $X,Y$) and if $X$ is discrete and $Y$ is continuous, how would the joint CDF look like? I understand that:   * If they were both discrete, it would be a staircase in 2 dimensions. Similar to stacking books atop each other on a table.   * If they were both continuous, it would be a nice smooth graph coming to 1 at $(\infty,\infty)$


title: What is the Survival Function using SAS LIFEREG with lognormal distribution?

text: I am trying to plot the survival curve using output from PROC LIFEREG with a lognormal distribution. Is my formula for the survival function (prob[i], where i is time) correct? Below is my SAS code. Thanks in advance!               proc lifereg data=Data OUTEST=OUTEST;     model month*failed(0) = x1 x2 /distribution=lnormal;     OUTPUT OUT=Out XBETA=XBETA;     run;          data plot;     _P_=1;     SET OUTEST(KEEP=_SCALE_) POINT=_P_;     SET OUT;     array prob {10} p1-p10;     do i=1 to 10;     prob[i]=1-PROBNORM((LOG(i)-XBETA)/_SCALE_);     end;     run;


title: Trending algorithm for performance monitoring

text: ## Situation description I'm trying to implement a prediction (or trending) algorithm for my performance gathering system in order to see when a Linux server's resources will end (for example, free space on a storage or free memory). The result of the performance gathering process is a graph. So, I need to get on my graph something like this: ![](http://i.imgur.com/GJCyh.png) This is an example of RRDtool graph (data collected from Cacti monitoring tool). Here are 3 dashed lines that are trend lines for the disk utilization history graph. That is similar to what I need. ## What I would like to get The prediction of when the performance will reach some value (e.g. 90%). Which one of the plenty of prediction (trending) algorithms should I use in my case? ## What I've researched Holt-Winters algorithm, time series prediction. But I do not know how to use them in this particular case. May be there is other solution? ## Additional information [30,45,50,10,20,30]-> this is example of disk utilization array (in %). --------------------> Time, minutes  Critical threshold boundary is 90%. I need to know when the disk utilization will reach 90% threshold (based on the already collected history of data). Any examples in Matlab or in R are welcome.


title: How do I get the SSE for predictions using SAS?

text: > **Possible Duplicate:**   >  How to get SSE for predictions using SAS? ![enter image description here](http://i.stack.imgur.com/oeRwA.png) I'm trying to get the sum of squared errors for my predictions in SAS, but I'm not sure I'm doing it correctly. I'm not sure I fully understand the output I'm getting with my code (specifically, **stdp** ):               data tridata;     infile '\data.dat';         input x1 x2 x3 y;         proc sort data = tridata; by x3;          proc reg data = tridata;         model y=x3;         plot r. * x3;         output out = tridata2 r = resid p = pred stdp = err;     run;     quit;          /* Send your errors to a file */     data _NULL_;         file '\data-err.dat';         set tridata2;         put err;         where y eq .;     run;     quit;      This gives me a file of errors for each estimate. I imported these into Excel, squared each one, and summed them up to give me a number. Is this the correct method? Based on the description of my project, I was under the impression that I should get an SSE_test for each predicted value. See below: ![enter image description here](http://i.stack.imgur.com/jucrH.png)


title: coefficients for formula

text: This is the formula I'm using Call:               lm(formula = kg ~ ft * temp)      and the coefficients               coef(fit)               (Intercept)                    ft                  temp               ft:temp       3.02787289439677e+03 -7.68497422047300e-02 -9.10978930930598e+00  9.14330790288404e-05       Now I need this formula and the coefficients in a different program. I understand the `intercept`, `ft`, `temp` but not `ft:temp`. How does that fit in the formula? Thanks for any help


title: Normalizing SVM predications to [0,1]

text: I have trained an linear SVM which takes a pair of objects, computes features and is expected to learn a semantic similarity function between objects(we can say that it predicts whether the two objects are similar enough that they should be merged or not). The problem I am facing is that the predictions can be from $-\infty$ to $\infty$ and I need a score from [0,1] as a semantic similarity. One suggestion that I received was that use min-max normalization to normalize the scores. Is there a better way(which is more generic rather than depending on values of min and max from training data)? Please mention the assumptions also, in case your method has any. Thanks


title: How to interpret a regression that includes GDP, GDP per capita, and population

text: While `GDP per capita = GDP / population` and are obviously related, these three measurements are not perfectly collinear and can be included in OLS just fine. However, when interpreting the coefficient of `GDP per capita`, how does it make sense to say that we "keep GDP and population fixed in order to find the effect caused by GDP per capita"?


title: What extra properties does assuming errors are iid have compared to assuming errors are uncorrelated and common variance

text: In linear regression model, the means of the errors are assumed to be zero. Furthermore, we can assume either that the errors are uncorrelated and have the same variance, or even that the errors are iid. Note normality isn't assumed on the errors. What extra properties does assuming errors are iid bring to the OLS estimates, compared to assuming errors are uncorrelated and common variance? Thanks!


title: Different sufficiency condition for Goods and bads

text: I am building an underwriting model for a bank with the following construct Development Sample Window - Accounts opened before Jun'13 (i.e. have completed atleast 12 months as of Jun'14) Bad Definition - 90+ dpd delinquent within the first 12 months of opening account The general approach is to consider all the accounts which have completed 12 months of seasoning and classify the delinquent guys as bads and the non- delinquent guys as goods. But can I consider the bad accounts which have not completed 12 months seasoning? E.g. If an account opened in Aug'13 has gone bad (90+ delq) in the first 6 months itself i.e. by Feb'14 we can tag it as bad then itself rather than waiting for another 6 months. Is there a way whereby we can also include such accounts (i.e. all bad accounts opened after Jun'13) and consider the good accounts only from pre-Jun'13 (as there is a seasoning sufficiency required for good tagging) I understand that there is a 'Greedy Algorithm' to achieve the same. Can someone please elaborate how it can be applied to the problem mentioned above? P.S - I am hesitating to take bads and goods from different time frames as what characterized bad behavior in 2012 might be reasonably good behavior in recent times. Eg - If I consider online debits as a variable, the nature of variable has changed over a period of 3 years. Customers might be using online purchases much more frequently than they are now. i.e. customer spending 20000 (say) on online purchases in 2010 might be much more different and risky than customer spending 20000.


title: Weighting words based on position in text

text: I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a word or n-gram based on it's position in the global text ? I'm asking this question because I'm looking for a way to improve the score of titles, (sub)-chapter names, section names, list elements, table headers, etc.


title: Generalised (nonhomogeneous) Poisson process

text: > Define a generalised Poisson process as an arrival process that begins at > time 0 and that satisfies: > >   * The independence property: the number of arrivals during two non- > overlapping intervals are independent. >   * Small interval probabilities: > > > \begin{align*} P(X_{t+\varepsilon} - X_t \ge 2) &= o(\varepsilon) \\ > > P(X_{t+\varepsilon} - X_t = 1) &= \lambda(t)\varepsilon + > o(\varepsilon) > \\ P(X_{t+\varepsilon} - X_t = 0) &= > 1-\lambda(t)\varepsilon + > o(\varepsilon) \end{align*} > > The function $\lambda(t)$ is called the intensity function. > > Let ${X_t, t \ge 0}$ be a generalised Poisson process (and suppose > $\lambda(t)$ is continuous for simplicity). Let $p_n(t) = P(X_t = n)$ and > $m(t) = \int_0^t \lambda(u) du$ **Question (a)** Obtain $p_0(t) = P(X_t = 0)$ by showing that $p_0(t)$ solves a first order differential equation. **My Working** Define $p_0(t) = P(X_t = 0)$, then for $h > 0$, \begin{align*} p_0(t+h) & = P(X_{t+h} = 0) \\ & = P(X_t = 0, X_{t+h} - X_t = 0) \\ & = P(X_t=0)P(X_{t+h} - X_t = 0) \ \ \ \ \ \ \text{By the independence property} \\ & = p_0(t)\left[1-\lambda(t)h + o(h) \right] \ \ \ \ \ \ \text{By the small interval probabilities} \\ & = p_0(t) - p_0(t)\lambda(t)h + p_0(t)o(h) \end{align*} By rearranging, \begin{align*} \nonumber \frac{p_0(t+h) - p_0(t)}{h} & = -p_0(t)\lambda(t) + p_0(t)\left(\frac{o(h)}{h} \right) \\ \nonumber \lim_{h \rightarrow 0} \left(\frac{p_0(t+h) - p_0(t)}{h} \right) & = -p_0(t)\lambda(t) + p_0(t)\lim_{h \rightarrow 0}\left(\frac{o(h)}{h} \right) \\ \nonumber p_0'(t) & = -p_0(t)\lambda(t) \\ p_0'(t) + \lambda(t) p_0(t) & = 0 \end{align*} where the above is a separable first order linear differential equation. Solving this differential equation yields, \begin{align*} \frac{p_0'(t)}{p_0(t)} &= -\lambda(t) \\ \int \frac{p_0'(t)}{p_0(t)} dt &= -\int \lambda(t) dt \\ \ln\left(|p_0(t)|\right) + C_1 &= -\int_0^t \lambda(u) du + C_2 \end{align*} where $C_1$ and $C_2$ are the constants of integration. Note that since $p_0(t)$ is a probability, then $p_0(t) > 0$ for all $t \ge 0$. Hence, \begin{align*} \ln\left(p_0(t)\right) = -m(t) + C_3 \end{align*} Since $p_0(0) = 1$ and $m(0) = 0$, then $C_3 = 0$, so the solution is given by, \begin{align*} p_0(t) = \exp\left(-m(t)\right) \ \ \ \ \forall \ \ 0 \le t < +\infty \end{align*} **Query** My main query is when removing the absolute values from $\ln\left(|p_0(t)|\right)$ by saying "...since $p_0(t)$ is a probability, then $p_0(t) > 0$ for all $t \ge 0$." How can I actually prove $p_0(t) > 0$ for all $t \ge 0$ based on the properties of a generalised Poisson process? **Question (b)** Let $T_1$ be the time of the first arrival. Find the density of $T_1$. **My working/Query** Is my working correct here? From part (a), we know $p_0(t) = \exp\left(-m(t)\right)$. Assume that the process starts from time zero and let $T_1$ denote the time of the first arrival. The CDF of $T_1$ is given by, \begin{align*} F_{T_1}(t) = P(T_1 \le t) = 1- P(T_1 > t) = 1-P(X_t = 0) = 1-\exp(-m(t)) \end{align*} To find the PDF of $T_1$, differentiate the CDF with respect to $t$ yields, \begin{align*} f_{T_1}(t) & = m'(t)\exp(-m(t)) \\ & = \lambda(t)\exp(-m(t)) \ \ \ \ \ \ \text{By the Fundamental Theorem of Calculus} \end{align*} Thus, $f_{T_1}(t) = \lambda(t)\exp(-m(t))$ for all $0 \le t < +\infty$.


title: Feeding a layer from a deep-learnt neural network into an SVM

text: In http://jmlr.org/proceedings/papers/v32/donahue14.pdf, it is stated: > Our top-performing method (based on validation accuracy) trains a linear SVM > on DeCAF6 Can you delineate in a way hopefully tenable to relative laymen, how does one plug a layer of already trained neurons (DeCAF6 in this instance) into an SVM, both what exactly does this "integration" comprise, what is the overall flow, and some intuitions to help along the way?


title: GLMM: choosing appropriate distribution through residual tests?

text: Thank you in advance for the great forum! I have a repeated measures design: a continuous dependent variable (measure), two fixed variables (treatments) and random effects (subjects). The dependent variable is not normal and the distribution differs between treatments. I carried out GLMM (using spss), however I am unsure which distribution to choose. I compared the results from two GLMMs: GLMM with normal distribution with identity link and GLMM with normal distribution with log link. The significance of one fixed variable and the interaction of both is significant using log link and not using identity. I printed the Pearson residuals and predicted values when conducting the GLMMs and the log link residuals are (just)normal and identity link residuals are not normal. Does the normality of the residuals in the GLMM normal with log link mean that this model is appropriate? If not, what can I do to decide for the best model and distribution? Thank you!


title: Principal Component Analysis

text: I am a bit new to this. But I needed a clarification. I am creating a module in MATLAB that performs PCA on various data sets with varying number of explanatory and response variables. Depending on the data available, PCA can be performed using Covariance, Correlation or SVD. For Covariance, I first centre the data and then use the covariance matrix of this centered data. For Correlation, I first standardize the data, and then use the covariance of this matrix and for SVD, I just apply SVD to the data set. I would like to know whether these approaches produce valid results? Thanks


title: Does joint normality imply marginal normality?

text: If it is given that an $N\times1$ random vector ${\bf x} = [x_1,x_2,\ldots,x_N]^T$ has a multivariate normal (MVN) distribution, it implies that all constituent random variables $x_n; n\in[1,N]$ are jointly normal. Joint normality means that any linear combination ${\bf a}{\bf x}$ (with ${\bf a}$ a constant row vector) will be normally distributed ($\mathcal N$). Does this necessarily imply that each marginal distribution $x_n\sim\mathcal N$? It seems that $x_n\sim{\mathcal N}$ is typically the case (as the distribution is called jointly normal), but does it have to be?. It seems possible that an MVN random vector may be generated from nonGaussian RVs (e.g. using a copula with uniform marginal distributions). This is somewhat similar to the univariate case: By the central limit theorem, the sum of many independent RVs (normal or not) is normal.


title: Chi-square and repeated measure

text: I have a survey of 20 questions; answers are given on the 5-point Likert scale. Goal: to understand whether the answer on the first question affects answers on other questions. Normally speaking I would be compiling a contingency table for "the first question vs another question" and running the chi-squared test. However, here I need to compare Q1 with Q2, Q1 with Q3, ..., Q1 with Q20. Do I understand correctly that I still can apply the chi-squared test for each pair and adjust the p-values obtained? Or should I use a different statistical test altogether?


title: Looking for a regression model or other statistics models for competitions

text: We have botanical data and need to analyze this sort of scenario, in terms of students for the simplicity of explanation: We have a series of competitions or trials, each one involves two individuals at a time - each trial has two new individuals. We test for a binary score - win or lose - in the test, and want to see if attributes (variables) of the individuals affect this result. Those variables are a mix of numeric ones such as age etc., and categorical ones, such as the school the each individual comes from. So the trials are independent among them, but naturally the result of each trial depends not only on the winner's attributes but also on his opponent's. We could think of any simpler mixed effect and/or nested logistic model. Do you have any suggestions? Is it possible to to a regression analysis on this? Or any other statistics? Someone suggested Bayesian inference due to the categorical variables we have some prior data on, but we prefer not to use Bayesian methods. Thanks for the help.