title: PDF not matching histogram of synthetic ratios of independent beta

text: The PDF of the ratios of independent beta variables is described in http://www.tandfonline.com/doi/abs/10.1080/03610920008832632#.U9J02vldUcC To explore the implications, i created an implementation in python.               import numpy as np     import matplotlib.pyplot as plt     from scipy.special import hyp2f1, betaln               def ratios_beta_like(a1, a2, b1, b2, w):         if w < 0:             return -np.inf         logA = betaln(a1, b1) + betaln(a2, b2)         if w <= 1:             return (betaln(a1 + a2, b2) + (a1 - 1) * np.log(w) - logA                 + np.log(hyp2f1(a1 + a2, 1 - b1, a1 + a2 + b2, w)))         else:             return (betaln(a1 + a2, b1) + (-a2 - 1) * np.log(w) - logA                 + np.log(hyp2f1(a1 + a2, 1 - b2, a1 + a2 + b1, 1. / w)))      But it doesn't look correct to me. How have I misunderstood the paper? or is there a mistake in my implementation? * * * For example:               bins = np.exp(np.linspace(-9.21, 9.21, 101))          plt.xscale("log")     a2, b2 = a1, b1 = (1.18176, 2.75744)     plt.plot(bins, [np.exp(ratios_beta_like(a1, b1, a2, b2, w)) for w in bins])     plt.hist(np.random.beta(a1, b1, 1000000) / np.random.beta(a2, b2, 1000000),               bins, normed=True, alpha=0.5)     plt.show()      ![output](http://i.stack.imgur.com/hpeiS.png) How have I misunderstood the paper? or is there a mistake in my implementation?


title: Normality Testing - Choose the transformation that makes the data "most normal"?

text: Background: I have conducted some testing on a random sample of n=20 parts. The data is variable and I know nothing about the population statistics. I would like to use the data from this sample to make statements about the population (in particular, I would like to say with 95% confidence that the 99th percentile of the population is above some value). I have read about the importance of checking normality for this type of analysis but have also read good posts about "low power" of a normality test when the sample size is relatively small. Those posts recommend that you check "graphically" and also to transform the data as necessary even if a basic normality check indicates there isn't a compelling reason to reject the null hypothesis that the data is normal. I transformed the data using a few basic transformations (square root, inverse, etc.) just to see what it would look like. None look drastically different but they do show different "p" values. _**My question:_** **Should I choose and use a transformation with the highest "p" value** even if a (low powered) normality check of the untransformed data is not below .05? As I understand it, that transformed data set would have the lowest probability of actually "not being normal". I've attached images of the normality checks I ran on the three sets (untransformed, transformed w/ square root, transformed w/ inversion) using Anderson-Darling. The only stats tool I have available to me is Minitab. ***EDIT: The reason that I thought normality would be important is that it is typical in my industry to perform these types of reliability calculations using tables of "One-sided and Two-Sided Statistical Tolerance Limit Factors (k)" and the tables are only shown/valid for normal distributions. See, for example, "Tables for One-Sided Tolerance Limits" Industrial Quality Control, vol. XIV, no 10. You do this by calculating X +/- ks where X is sample mean, s is sample std dev, and k is from a table and is a function of desired confidence, reliability, and sample size. ![enter image description here](http://i.stack.imgur.com/WFrMQ.png) ![enter image description here](http://i.stack.imgur.com/87bDg.png) ![enter image description here](http://i.stack.imgur.com/pbxVd.png)


title: When data has a gaussian distribution, how many samples will characterise it?

text: Gaussian data distributed in a single dimension requires two parameters to characterise it (mean, variance), and rumour has it that around 30 randomly- selected samples is usually sufficient to estimate these parameters with reasonably high confidence. But what happens as the number of dimensions increases? In two dimensions (e.g. height, weight) it takes 5 parameters to specifiy a "best-fit" ellipse. In three dimensions, this rises to 9 parameters to describe an ellipsoid, and in 4-D it takes 14 parameters. I am interested to know if the number of samples required to estimate these parameters also rises at a comparable rate, at a slower rate or (please no!) at a higher rate. Better still, if there was a broadly accepted rule of thumb that suggests how many samples are required to characterise a gaussian distribution in a given number of dimensions, that would be good to know. To be more precise, suppose we want to define a symmetrical "best-fit" boundary centred at the mean point inside which we can be confident that 95% of all samples will fall. I want to know how many samples it might take to find the parameters to approximate this boundary (interval in 1-D, ellipse in 2-D, etc) with suitably high (>95%) confidence, and how that number varies as the number of dimensions increases.


title: Testing a prediction hypothesis

text: I wish to test the following Hypothesis: H = "sales growth from previous year" predicts "sales growth this year" I have the following data:               > head(x)       Location Week Sales_Y1 Sales_Y2 Sales_Y3  Y2_vs_Y1  Y3_vs_Y2     1      1_1    5 24924.50 21665.76 23510.49 0.8692556 1.0851449     2      1_1    6 46039.49 37887.17 36988.49 0.8229277 0.9762801     3      1_1    7 41595.55 46845.87 54060.10 1.1262231 1.1539993     4      1_1    8 19403.54 19363.83 20124.22 0.9979535 1.0392686     5      1_1    9 21827.90 20327.61 20113.03 0.9312673 0.9894439     6      1_1   10 21043.39 21280.40 21140.07 1.0112629 0.9934057      So my Hypothesis is that Y2_vs_Y1 predicts Y3_vs_Y2. The simplest form of this hypothesis could be > Y3_vs_Y2 = [ Y2_vs_Y1 + or - 5%] How can I test this hypothesis?


title: Zero-mean correlation

text: In the following blog post: here, the author goes and calculates the alternative correlation measure for two time series. Essentially, calculates the correlation between two series by giving the mean of each series 0. I tried reproducing it, but am not getting close results. Here is the code:               setInternet2(TRUE)     con = gzcon(url('http://www.systematicportfolio.com/sit.gz', 'rb'))     source(con)     close(con)          load.packages('quantmod')        tickers = spl('SPY,GLD')          data <- new.env()     getSymbols(tickers, src = 'yahoo', from = '1970-01-01', env = data, auto.assign = T)     for(i in ls(data)) data[[i]] = adjustOHLC(data[[i]], use.Adjusted=T)             bt.prep(data, align='remove.na', dates='1994::')               prices = data$prices["2012:10::",] #use dates in line with post     ret<-na.omit(prices/mlag(prices)-1)     prices<-cumprod(1+ret)     sma<-bt.apply.matrix(prices,SMA,40) #moving Average          osc<-na.omit((prices - sma) / sma) #percent deviation from MA          x<-osc[,1]     y<-osc[,2]          mean.x<-0    #0 #mean(x)     mean.y<-0    #0 #mean(y)          sd.x<-sd(x)     sd.y<-sd(y)          sum(((x-mean.x)/(sd.x))*((y-mean.y)/(sd.y))) / (nrow(x)-1) #equation for correlation      In the blog post, the author showed 2 example results. One used a MA length of 20 days and another used an MA length of 40 days. I am using the following correlation formula: ![enter image description here](http://i.stack.imgur.com/HmNUg.gif) I get values of -17 and -1.8 for 20 day MA and 40 day MA respectively. This differs from-9 and -54 from what the author used. Any ideas as to what I am doing incorrectly?


title: Probability over $50$-day ranges to predict results on one day

text: The title is a bit complex, so I'll just post the question. This is in preparation for a final I'm going to take. > Consider the total number of cars that a car dealership might sell over $50$ > randomly selected (contiguous) days. Assume that the expected value of the > number of cars sold over such a random sample of $50$ days is $62$ cars and > that the standard deviation is $7$ cars. > > The question I'm having trouble with is: What can you say about the > probability that the dealership sells more than $1$ car on a randomly > selected day? I answered that you can't say anything, because all you know is the probability for a sample of $50$ days, which means all the cars could theoretically be sold on one day, or they could be sold evenly over the course of $50$ days. Apparently, this answer is incomplete. I can't figure out what you could possibly gather from the information given; any help would be greatly appreciated. Thanks.


title: How does adding a "lives" mechanic change average picks?

text: I'm tuning a slot machine app and need to make sure the mini-game is appropriately balanced. The mini-game is a grid of 30 treasure chests, 6 of which are traps. The user continues to open chests (and earn prizes) until they hit a trap. What I'd like to do is add a "lives" mechanic where hitting a trap depletes a life instead of ending the game outright. When user is out of lives, the mini- game ends. Here's the equation I'm using to determine the return %: > $(return \ \text{%}) = (\text{%} \ chance \ of \ triggering) * (avg \ > reward) * (avg \ num \ of \ picks)$ Here's the table I'm using to derive average number of picks:               Pick #     Probability     Expected       1          20%             0.20       2          16.55%          0.33       3          13.60%          0.41       4          11.08%          0.44       5          8.95%           0.45     6          7.16%           0.43     7          5.67%           0.40     8          4.44%           0.35     9          3.43%           0.31     10         2.61%           0.26     11         1.96%           0.22     12         1.44%           0.17     13         1.04%           0.14     14         0.74%           0.10     15         0.51%           0.08     16         0.34%           0.05     17         0.22%           0.04     18         0.13%           0.02     19         0.08%           0.01     20         0.04%           0.01     21         0.02%           0.00     22         0.01%           0.00     23         0.00%           0.00     24         0.00%           0.00     25         0.00%           0.00     Total      100.00%         4.43      Here's the equation I'm using to determine probability: > $(probability) = \frac{PERMUT(24, \ Pick \ \text{#} - 1) \ * \ (num \ of \ > traps)}{PERMUT(30, \ Pick \ \text{#})}$ How does adding a "lives" mechanic change these equations?


title: What is the mathematical difference between using a un-informative prior and a frequentist approach?

text: Un-informative priors are preferred in instances where bias is not acceptable (ie. courtrooms, etc.) However, it seems to me that it would just make sense to use a frequentist approach instead. Why does the Bayesian approach even have a non-informative prior? Thanks!


title: How to create weight

text: Background: - I have a country with 8 states - Each state has different population size, some much larger than others - I've recorded the number of campaign respondents from each state Problem: - I'd like to compare the respondents between each state - But one might argue that the larger states will always win because they have a larger population to begin with - So how do I create weights which will account for population size so that I can adjust respondent numbers in a more meaningful way? Thanks.


title: Difference between lowess and regression plot

text: Why there is difference between lowess smoother plot and regression plot in the following plot? ![enter image description here](http://i.stack.imgur.com/z5g3A.jpg) I run packages: "datasets" "utils" "graphics" "stats" "methods" in RStudio.


title: How to compare forecasting methods?

text: I have several intermittent data. Based on those data, I would like to compare several forecasting methods (Exponential Smoothing, Moving Average, Croston, and Syntetos-Boylan), and decide whether Croston or Syntetos Boylan is better than SES or MA for intermittent data or not. The measure I'd like to compare is Mean Absolute Rate or Mean Squared Rate proposed by Kourentzes (2014) instead of usual MAPE, MSE measure, at every level of \alpha$ smoothing parameter, assuming that the smoothing parameter used for Inter demand Interval and demand size in Croston and Syntetos boylan is the same. My question is: 1. Considering that for every data, there is the possibility that the value of optimal alpha is different for each smoothing methods, therefore a value of alpha in a method may minimize the the MAR or MSR and may not in other method, such that that one method may be better than other method for that value of alpha and may not in other method. How does one solve this kind of problem? my current solution is to compare the two graph of MAR for every level of alpha between two different method. my expectation is that the two different method will show different characteristics when profile analysis is done.   2. Is there any better solution, like experimental design? I am rather confused in how to design the experiments. the observation is those several data, the level is smoothing parameter alpha, treatment is those methods. and the value is the MAR. is it viable? and logical to do? The hypothesis is whether there is differences in each treatment at every level of alpha or not. and I doubt the linearity assumption is fulfilled here. Edit: Anyway, I do not think this is viable as research question. The fact that the error measure is scale dependent (if my definition of scale dependent are right) made the study of this problem is quite problematic, as there is no way to compare the different methods of forecasting.


title: Fractional Factorial Design

text: In a $2^5$ design, it is believed that only the main effects $(A,B,C,D, E)$ and $AB,AC$ interaction effects are non-zero. I need to construct a fractional factorial with minimum number of runs which can be used to estimate all the main effects and $AB,AC$ interaction effects .   * i hoped this be a $2^{5-2}$ fractional factorial and to construct the design :   * First, i wrote down the basic design, which consists of the $8$ runsfor a full $2^{5-2}=2^3$ design in $A,B,C$   * Then the two factors $D$ an $E$ are added by associatingtheir plus and minus levels with the plus and minus signs of the interaction $BC$ and $ABC$.   * The complete defining relation for this design is $$I=BCD=ABCE=ADE$$ Table01: Construction of the $2^{5-2}$ Design with the Generators $I=BCD=ABCE=ADE$ ![enter image description here](http://i.stack.imgur.com/VH3QP.png) But is it `Principal Fraction`? If not, how can i construct Principal Fraction in this way? If i write down the data of Table01(where the data is in plus and minus notation) in treatment combination , this is $$ \begin{array}{|c|} \hline d\\ ade\\ be\\ ab\\ ce\\ ac\\ bcd\\ abcde\\ \hline \end{array} $$ ### With this data how can i estimate effect $A,B,C$? There aren't treatment combinations $a,b,c$ in the fraction. Table02. Alias structure for the $2^{5-2}$ Design with $I=BCD=ABCE=ADE$ $$ \begin{array}{C} A=ABCD=BCE=DE\\ B=CD=ACE=ABDE\\ C=BD=ABE=ACDE\\ D=BC=ABCDE=AE\\ E=BCDE=ABC=AD\\ AB=ACD=CE=BDE\\ AC=ABD=BE=CDE\\ \end{array} $$ ### The alias structure determines which effects are confounded with each other. But does it play role in construction of the design? If so,how? And the final question is if the situation were as following: In a $2^5$ design, it is believed that only the main effects $(A,B,C,D, E)$ and $AB,ABD$ interaction effects are non-zero. I need to construct a fractional factorial with minimum number of runs which can be used to estimate all the main effects and $AB,ABD$ interaction effects .   * Will it be a $2^{5-2}$ design ? As I need the $ABD$ interaction to be estimated and my basic design, $2^{5-2}=2^3$, has only effect $A,B,C$ .


title: Histograms With No Final Interval Upper Limit

text: If you have the following histogram, how do you calculate the mean if you do not know the upper limit of the final interval:               Range   Freq.     20-30    5     30-40    2     40+      4


title: Low $R^2$ value in social science or education research

text: I'm looking for paper(s) that talk about "why low $R^2$ value is acceptable in social science or education research". Please point me to the right journal if you know one.


title: R program- question on the mod function

text: I would like to multiply x with two different values based on the mod function.               if (x mod 2=0)      then x=x*2.15     else if (x mod 2>0)      then x=x*3.15      My sample is               x <- sample((10:30),size=15,replace=T)      I need do this in R programming function.


title: Student t-distribution

text: If $X_i$, $i =1,...,n$ all follow a normal distribution $N(\mu,\sigma^2)$, and are independent, does $\frac{\sqrt{n}\cdot(\frac{1}{n}\cdot \sum X_i - \mu)}{\sqrt{(\frac{1}{n}\cdot \sum (X_i-\mu)^2)}}$ follow a student t-distribution with n degrees of freedom? I am aware of the classic formula with the n-1 degrees of freedom t-distribution. I just want to understand if this formula is true too.


title: Correct Analysis for determining impact of input parameters on data

text: I work for a manufacturing company that collects times (man-hours) for tasks. The tasks are similar in concept from one product to another, but the characteristics of the product can vary (like size, voltage, etc). It has been something like ten years since I've had to do any heavy lifting in terms of data analysis... I'm trying to take our dataset (about four years' worth of data) coupled with some parameters about each of the products and use it to ultimately be able to input a product's characteristics and be able to predict the output in hours. Can anyone help me get started with this? I'm an EE by background and have worked with neural networks and linear regression in the past, but it's been something like 10 years, so I've forgotten it all. Thanks so very much, Timothy


title: Compare Cox with ANN

text: I am trying to compare Cox model with ANN by R but I have some problems.   1. the result of package `coxph` is hazard but the result of package `nnt` is survival time   2. How to create labels in ROC curve in `rocr` package.   3. How to calculate concordance indexes by which package.


title: how to discard values that are far from center of cluster in mixture model

text: I am trying to fit a bivariate cluster model with X and Y. What I would like to do is discard (make not clustered / un-grouped) that are far from the cluster center (for example $\mu$ + 2*standard deviation) at each direction.               X2 <- c(rnorm(150, 10, 1),rnorm(50, 10, 5),  rnorm(150, 25,1),         rnorm(50, 25, 20), rnorm(200,35,1),  rnorm(200,80,1), rpois(50,30))     Y2 <- c(rnorm(800, 30, 2), rpois(50,30))     df <- cbind (X2, Y2)     plot(df, pch = 20, col = "gray40", ylim = c(15,45), xlim = c(-10, 90))      ![enter image description here](http://i.stack.imgur.com/xuEP2.jpg) I am expecting 4 clusters here and I would like to group only those inside ellipse that has high probability (say with in $\mu$ + 2*standard deviation in both X and Y) and make rest un-grouped. Here is what tried with mixture model:               require(mixtools)     out <- mvnormalmixEM(df, lambda = NULL, mu = NULL, sigma = NULL, k = 4,      arbmean = TRUE, arbvar = TRUE,                 epsilon = 1e-08,  maxit = 10000, verb = FALSE)     plot(out, density = TRUE, alpha = c(0.01, 0.05, 0.10, 0.12, 0.15),       marginal = TRUE)      I do not know how to put such threshold here. ![enter image description here](http://i.stack.imgur.com/9PbSF.jpg)


title: When using Nonmetric Multidimensional Scaling, is there an explanatory metric similar to loadings in PCA?

text: As a beginner to MDS, here is my thought process: Given a data set of environmental factors that may effect a certain sites, when I run a PCA on each site I get a list of principal components. If I want to understand which environmental factors are driving each of the principal components, ie:               PCA1 <- Mostly Oxygen and Salinity driven     PCA2 <- Mostly Temperature driven      I look at the loadings of the PCA results to find out which of the original environmental factors are weighting each of the principal components. When I run a NMMDS on a similar set of data, my result is a plot in which the sites are grouped so that more dissimilar sites are further apart. While I can make an educated guess as to what is causing the grouping in an MSD plot, is there something analogous to the loadings of a PCA that explains which environmental variables may be playing the largest roles in the resulting plot? Or a way to quantify the input of each environmental variable to the final structure of the plot?


title: which distributions have high "Fat Tail Ratio"?

text: For a distribution $X$, assuming it's "normalized" so that $E\{X\}=0$ and $Var\{X\}=1$. Given significance level $\alpha$, define "Fat Tail Ratio" as $$FTR(X,\alpha) = z_{1-\alpha/2} - z_{\alpha/2} $$ where $z_{\alpha/2}$ and $z_{1-\alpha/2}$ means $P\{x<z_{\alpha/2}\} = \alpha/2$, and $P\{x<z_{1-\alpha/2}\}=1-\alpha/2$. Which distributions have the high FTR? For example, assuming $\alpha=5\%$... For Normal Distribution, $FTR(N,5\%) = 2*1.96 = 3.92$. For Student t-distribution, according to wiki page, let $X = \sigma T$, for $v>2$, $Var\{X\} = \sigma^2 \frac{\nu}{\nu-2}$. So if $\nu=4$, $Var\{X\} = 1$ => $\sigma=\frac{1}{\sqrt{2}}$. Hence $FTR(t(\nu=4), 5\%) = 2 * 2.776 * \sigma = 3.926 $. I'm wondering if there are other distributions or distribution families, commonly used, has even higher "Fat Tail Ratio"?


title: Power in proteomics?

text: Grants often require power analysis to support a proposed population size. In proteomics (and most -omics), there are 100's to 1000's of features/variables measured across 10's of samples (maybe 100s, but unlikely). Also, it is known that some of these measurement units (e.g., spectral counts of proteins) are not normally distributed and so we will use non-parametric test for analysis. I have seen the power of a population size determined assuming a single measurement and assuming a t-test, but I don't think this is completely correct. Another problem with spectral counts specifically is that each of the 100's of features is on very different scales with vastly different errors (larger values have less error). [ This problem is nicely described in the limit fold change model, Mutch et al., 2002 ] What would be the appropriate way to determine the power of a proposed population size given some assumptions of FDR and an acceptable fold-change? Using the tool here I was able to determine given the following: 300 genes 3 false positives 1.4 fold-differences 0.8 desired power 0.7 stdev requires a sample size per group of 49. This was handy since I am proposing a 50v50 design, know that 1.4 fold-change is pretty accepted, 1% FDR is fine, and I will probably measure 300 proteins in this experiment. This problem of power size calculation will continue to occur, so it would be nice to have a referenced approach in place. **EDIT:** I read where a colleague proposed to model spectral counts from negative binominal distributions using the likelihood function followed by a Wald test. Basically uses prelim data to get protein variance estimates and then calculate detectable fold changes between groups for each quantile. There is also an FDR (alpha) input. So, given a >80% power and set sample size, they can determine detectable fold-changes for 25% lowest variance, 50% smaller variance, and 25% highest variance. Problem is that I don't know how they did this. Not sure if sharing this approach will help anyone with a possible answer.


title: Is there a conjugate prior for Gumbel-distributed data?

text: I am trying to fit a Gumbel distribution to a set of N i.i.d. data points. I was wondering whether there is a conjugate prior to infer the two parameters of the Gumbel pdf.


title: How to rescale  two questionnaires that use slightly different response scales in order to make them comparable?

text: I used a -4 to +4 Likert scale in one set of questionnaires and -3 to +3 for the same questionnaires, but gathered respondents in a different way. I want to rescale from -4 to +4 down to -3 to +3 for the second group. How do I do this and secondly is there a referenced/accepted procedure for doing this?


title: Where is a good place to find survey results?

text: Sites like eMarketer offer general survey results about internet usage. Who else has a big set of survey results, or regularly releases them?   * Preferably marketing research focused. Thanks!


title: What are the advantages of using mutual information over Pearson correlation in network inference?

text: Does mutual information discriminate against fold change differences?


title: How do you convey the beauty of the Central Limit Theorem to a non-statistician?

text: My father is a math enthusiast, but not interested in statistics much. It would be neat to _try_ to illustrate some of the wonderful bits of statistics, and the CLT is a prime candidate. How would you convey the mathematical beauty and impact of the central limit theorem to a non-statistician?


title: Best method for comparing multiple ranging measures

text: I've been asked to give some advice for some clinicians who are comparing two different methods of blood pressure measurement. I suggested to them that we should proceed with a two-one-sided-test technique to determine equivalence of the two techniques. Unfortunately I have now learned that the clinicians have multiple measurements of blood pressure by each of the two methods and that the blood pressure can be quite variable within each patient during the period of observation (they are theatre cases). Is it possible to use some multiple regression technique to perform an equivalence test? Can I simply use confidence intervals to determine variability between the two techniques whilst accounting for inter-patient variability if I use the patients as factors in the regression model? Sorry it's an amateur question, but despite the Masters degree, I still feel like quite the amateur!


title: Best Way to Aggregate and Analyze Data

text: Having just recently started teaching myself Machine Learning and Data Analysis I'm finding myself hitting a brick wall on the need for creating and querying large sets of data. I would like to take data I've been aggregating in my professional and personal life and analyze it but I'm uncertain of the best way to do the following: 1) How should I be storing this data? Excel? SQL? ?? 2) What is a good way for a beginner to begin trying to analyze this data? I am a professional computer programmer so the complexity is not in writing programs but more or less specific to the domain of data analysis. EDIT: Apologies for my vagueness, when you first start learning about something it's hard to know what you don't know, ya know? ;) Having said that, my aim is to apply this to two main topics: 1) Software team metrics (think Agile velocity, quantifying risk, likelihood of a successfully completed iteration given x number of story points) 2) Machine learning (ex. system exceptions have occurred in a given set of modules what is the likelihood that a module will throw an exception in the field, how much will that cost, what can the data tell me about key modules to improve that will get me the best bang for my buck, predict what portion of the system the user will want to use next in order to start loading data, etc)


title: How to calculate the "exact confidence interval" for relative risk?

text: I am working on some MRSA data and need to calculate the relative risk of a group of hospitals compared with the remaining hospital. My colleagues throws me an excel with a formula inside to calculate the "exact confidence interval of relative risk", I can do the calculation without difficulties, but I have no idea on how and why this formula is used for do such calculation. I have attached the excel file here for your reference. Can anyone show me a reference on the rationale of the calculation? Article from textbooks will be fine to me. Thanks!


title: Expectation of length of a confidence interval for a proportion

text: a) For a given significance-level $\alpha$, if we find $\mathop{\mathbb E}(b-a)$ such that $\mathbb P(a \leq X \leq b) \geq 1−\alpha$ for all possible values of $a$ and $b$, from the point of view of interpretability, what extra interpretability does the expected length buy us? Say, if I had observed proportions $ p_{1} ,p_{2},p_{2} ...p_{n}$ over different sample sizes and modeled them as binomial proportions and calculated confidence intervals (CIs) for each of them: b) Would comparing any pair of expected lengths of these confidence intervals be analogous to the expected length of the confidence interval around the difference of binomial proportions? c) How does the expected length of the CI compare with the standard error of the estimator- considering the intervals to be symmetric around the estimator?


title: How much sub-questions impact one ordinal question in a survey

text: I have a variable NPS which is an item on a survey, the answer format is ordinal, 0 to 10. Then there are sub-questions on the same survey, some are ordinal, some are categorical. I would like to determine which sub-question or which group of sub-questions are the best predictors of NPS. The goal is to be able to prioritize which area(s) to focus on in order to improve NPS over time. What would be some appropriate methods to identify and verify such a relationship?


title: Remove duplicates from training set for classification

text: Let us say I have a bunch of rows for a classification problem: $$X_1, ... X_N, Y$$ Where $X_1, ..., X_N$ are the features/predictors and $Y$ is the class the row’s feature combination belongs to. Many feature combination and their classes are repeated in the dataset, which I am using to fit a classifier. I am just wondering if it is acceptable to remove duplicates (I basically perform a `group by X1 ... XN Y` in SQL)? Thanks. PS: This is for a binary presence only dataset where the class priors are quite skewed


title: When to use Fisher and Neyman-Pearson framework?

text: I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought. My question is, ignoring philosophical objections for a moment; when should we use the Fisher-approach of statistical modelling and when should be use the Neyman-Pearson method of significance levels etcetera? Is there a practical way of deciding which viewpoint to endorse in any given practical problem?


title: Understanding the plm package - Why aren't my standard errors correct?

text: For fun, I tried to replicate the results of Petersen (2009) who deals with the correct estimation of standard errors in finance panel data sets. In a nutshell, he estimates the following standard regression for a panel data set: $$ Y_{it} = X_{it} \beta + \epsilon_{it} $$ where $\epsilon_{it} = \gamma_i + \eta_{it}$ and $x_{it} = \mu_{i} + \nu_{it}$. Hence, both the residual and the independent variable have a firm- specific component. Petersen goes on to show that this results in biased standard errors when applying the standard OLS. For example, he shows in table 1 of his paper that if both the residual volatility and the variable volatility are driven by 50% by a firm-specific component, the true standard errors are nearly twice as large as the ones given by OLS. He shows that in a MCS and I reproduced those results in R, as you can see from the code below. Naturally, I asked myself how I would compute the correct standard errors in R and the package of choice seemed to be `plm`. However, I just don't get the correct results out of it and I don't know what I miss. Here is my code:               library(plm)     runMCS <- function(runs, nrN, nrT, fracFirmX, fracFirmEps, sd_X, sd_eps, beta) {            betas    <- numeric(runs)       se_betas <- numeric(runs)       panel_betas    <- numeric(runs)       se_panel_betas <- numeric(runs)            for (i in 1:runs) {              #Model epsilon, X, and Y         eps <- rep(rnorm(nrN,                           mean=0,                           sd = sd_eps * sqrt(fracFirmEps)),                     each=nrT) +                       rnorm(nrN * nrT, mean=0, sd = sd_eps * sqrt(1-fracFirmEps))         X   <- rep(rnorm(nrN,                           mean=0,                           sd = sd_X   * sqrt(fracFirmX)),                       each=nrT) +                       rnorm(nrN * nrT, mean=0, sd = sd_X   * sqrt(1-fracFirmX))         Y   <- beta * X + eps              #Compute regression (OLS)         reg <- summary(lm(Y ~ X))              #Save results         betas[i]    <- reg$coef[2, 1]         se_betas[i] <- reg$coef[2, 2]              #Try plm         df <- data.frame(Firm = rep(1:nrN, each=nrT),                          Time = rep(1:nrT, times=nrN),                          Y = Y,                          X = X)         preg <- summary(plm(Y ~ X, data=df, index=c("Firm", "Time"), effect="individual", model="within")) #within is fixed effects         panel_betas[i]    <- preg$coef[1, 1]         se_panel_betas[i] <- preg$coef[1, 2]       }            return(c(avg_beta = mean(betas),                 true_se = sd(betas),                 avg_se = mean(se_betas),                 avg_clustered = mean(panel_betas),                se_clustered = mean(se_panel_betas)))          }     MCS_50_50 <- runMCS(50, 500, 10, 0.5, 0.5, 1, 2, 1)     MCS_50_50          avg_beta       true_se        avg_se avg_clustered  se_clustered         1.00503955    0.06020203    0.02825567    1.00433092    0.02985546      Note that I only run the simulation 50 times here because the plm function slows it down considerably. So basically, it makes virtually no difference if I call `lm` or `plm`. I'm pretty confident that I set the `index` and `model` option correct after reading the vignette of the package. However, I must miss something here! Interestingly, the package also has the `fixef` function and if I call that on one run, I get something like this:               summary(fixef(plm(Y ~ X, data=df, index=c("Firm", "Time"), effect="individual", model="within")))     1      13.60377     0.44112    30.8391 < 2.2e-16 ***     2    -830.74707     0.44136 -1882.2236 < 2.2e-16 ***     3    -326.96042     0.44137  -740.7840 < 2.2e-16 ***     4     169.16463     0.44246   382.3287 < 2.2e-16 ***     ...      I'm not quite sure how to interpret those results, but here, I get considerably larger standard errors for each firm separately. If I would average those, I would end up with something above 0.44 which is considerably closer to the true standard errors, but still not right. So, again a very long question from me, sorry for that ;-) Note that I did check answers before and I found this interesting link. The white paper that is referred to in the answer is interestingly the same person that implemented the solution on Petersen's webpage. So I'm pretty sure that I could get the correct standard errors by implementing Mahmood Arai's solution. But I'm looking for an already implemented and therefore safe option and I just wonder why that plm function does not work.


title: Why does the supremum of the Brownian bridge have the Kolmogorov–Smirnov distribution?

text: The Kolmogorov–Smirnov distribution is known from the Kolmogorov–Smirnov test. However, it is also the distribution of the supremum of the Brownian bridge. Since this is far from obvious (to me), I would like to ask you for an intuitive explanation of this coincidence. References are also welcome.


title: Fitting a continuous result to categorical predictors semiparametrically

text: Suppose one has a relatively large number of observations, each of which consists of a continuous result and a small number (2 or perhaps 3) of categorical variables, each of which has a large cardinality (up to 10k or so). So, for instance, one might wish to predict income given a person's university and chosen industry (this is just an example). Right, now I am essentially fitting a model that looks like `log(income) ~ 0 + university + industry` in R notation, so each university and each industry has an associated "score", but there really is no reason to think the effect is additive, or that taking the log of income is the best transformation. Still, it seems to work (somewhat). Another idea I would like to try out is to express `log(income)` as a sum of all possible polynomial powers (up to total degree 4 or so) of the associated scores - this would require a lot of regularization, I suppose. However, I suppose what I would really like is just to fit something like `g(income) ~ f(score.industry,score.university)` and simultaneously fit `g,f,score.industry,score.university` where the latter two variables are actually vectors. Does anyone have any insight as to how this might be done?


title: Intuition behind power law distribution

text: I know that the pdf of a power law distribution is $$ p(x) = \frac{\alpha-1}{x_{\text{min}}} \left(\frac{x}{x_{\text{min}}} \right)^{-\alpha}$$ But what does it intuitively mean if, for example, stock prices follow a power law distribution? Does this mean that losses can be very high but infrequent?


title: Data mining papers/examples

text: I am looking for some medium to long length papers/websites/etc about data mining, specifically where one dataset is explored in depth from data preparation through final model. I am particularly interested in discussions about application of machine learning algos and also basic data modeling. An example would be Luis Torgo's book 'Data Mining with R'. Any suggestions would be appreciated.


title: How do I specify priors for angle parameters in BUGS/JAGS?

text: I am writing a hierarchical BUGS model that involves both linear and angle variables. I want the hyper-parameters to be normally distributed, which is straight-forward for the linear variables, but I'm not sure what to do about the angle variables. I would gladly use a von Mises distribution or a truncated normal, but neither is available in the BUGS language. My angle variables range from 0 to pi. Here is my model:               model {             # parameters        for ( i in 1:P ) {           Y[i] ~ dnorm(Yhat[i], eachtau[plot[i]])           Yhat[i] <- C[plot[i]] + a[plot[i]] * exp(-v[plot[i]] * year[i]) * cos(w[plot[i]] * year[i] - phi[plot[i]])        }             # hyperparameters        for ( i in 1:M ) {           C[i] ~ dnorm(mu.C, tau.C)           a[i] ~ dnorm(mu.z, tau.z)           v[i] ~ dnorm(mu.v, tau.v)                # what do we do with these angles?!           w[i] ~ ?           phi[i] ~ ?                eachtau[i] ~ dgamma(0.001, 0.001)        }             # priors        mu.C ~ dnorm(0.0, 0.001)        tau.C ~ dgamma(0.001, 0.001)        mu.z ~ dnorm(0.0, 0.001)        tau.z ~ dgamma(0.001, 0.001)        mu.v ~ dnorm(0.0, 0.001)        tau.v ~ dgamma(0.001, 0.001)     }      If it's important, I'm actually using JAGS (rather than OpenBUGS or WinBUGS), using the rjags package in R, but I believe the model syntax is the same for both. What's the simplest way to model the angle variables with something like a normal in BUGS?


title: How to choose significance level for a large data set?

text: I am working with a data set having N around 200,000. In regressions, I am seeing very small significance values << 0.001 associated with very small effect sizes, e.g. r=0.028. What I'd like to know is, is there a principled way of deciding an appropriate significance threshold in relation to the sample size? Are there any other important considerations about interpreting effect size with such a large sample?


title: Probability density function (pdf) of normal sample variance ($S^2$)

text: I need to know the formula for the pdf of $S^2$. I know this: $$ \frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{n-1} \>, $$ but I want to state the correct formula for the pdf of $S^2$, not $(n-1)S^2/\sigma^2$. Any thoughts?


title: Statistical equivalence in the birthday paradox

text: I was reading a Wikipedia article on the birthday paradox and stumbled upon the following statement: > ...the pairings in a group of 23 people are not statistically equivalent to > 253 pairs chosen independently... Could you explain what does it mean and why does it matter in this case? * * * Here is the quotation in its original context: > Although the pairings in a group of 23 people are not statistically > equivalent to 253 pairs chosen independently, the birthday paradox becomes > less surprising if a group is thought of in terms of the number of possible > pairs, rather than as the number of individuals.


title: Examples of "Hard" Expectation Maximization other than clustering?

text: Are there examples of learning algorithms(other than k-means clustering) which fit the paradigm of Hard-EM? By hard EM, I mean the variant described in here.


title: If I take n-standard deviations of my data what would be my confidence level with the estimate?

text: I have some measurement data of estimates vs actuals (of some metric $m$). For every such data I also have the ratio of $\frac{actuals}{estimates} = r \space (say)$ - if this ratio is greater than 1 it implies underestimation (and overestimation if less than one). Given I have substantial data, about 100 or so data points, I calculate the mean $\mu$ and standard deviation $\sigma$, ($\forall \space r_i| i=1,...,100$ from above). Based on the data above I can say that historically derived confidence range of my estimate is $\mu \pm n\sigma$ - where 'n' is the number of standard deviations I wish to consider. Question - what can I say about my _confidence level_ for the 'next estimate' for $n=1,...,6$? Example: Let's say $\mu = 1.16$ and $\sigma = 0.38$ (data points are {$r_i|i = 1,...,100$}). For $n = 1$ the upper limit $u = 1.53$ and lower limit $v = 0.78$ (i.e., $\mu \pm n\sigma$) Now, say I make an estimate of 100 for $m$ and using the historical data with one-standard deviation as above I can say the following about a most likely estimate:   * The estimate should be revised to $116$ (i.e., $100 \mu$) due to underestimation on 'average'   * The value is quite likely to be within $78$ and $153$ ($100u$ and $100v$) So for $1\sigma$ (i.e., $n=1$) what is confidence level of the estimate? Is it simply just a ratio of $\frac{numbers Within Range}{Total No. of Elements}$? If I now make $n=2$ what can I say about the confidence level of the estimate for $m$? Basically if I have historically derived confidence ranges ($u, v$ and mean $\mu$) can I peg some sort of a percentage probability (or confidence level) of the likelihood value for the estimate of $m$? How would I do it? Should I restrict the max value of $n$ to 6? (why so?) PS: For those in the know, this is the cone of uncertainty data calculation for a particular metric. And I'd like to know the confidence value that I can associate with my estimate as per historic data.


I found this: ![enter image description here](http://i.stack.imgur.com/WTaPr.png) http://www2.cs.uregina.ca/~dbd/cs831/notes/confusion_matrix/confusion_matrix.html defining TP,FP,TN, FN as: ![enter image description here](http://i.stack.imgur.com/x6sxM.png) while on the other hand I often find it in such form: ![enter image description here](http://i.stack.imgur.com/8Z991.png) http://www.cs.cornell.edu/courses/cs578/2003fa/performance_measures.pdf Which notation for confusion matrix is more accurate?


title: Overlap in time series training sets

text: I have a time series prediction problem where the aim is to forecast the average value of $y_t$ over the next $T$ periods, given all the information available up to point $t$. For example, I want to forecast $$\bar{y}_t = \frac{1}{T}\sum_{k=1}^T y_{t+k}$$ as a function of a bunch of other variables $x_t$ which are available at time $t$. When building a training set from the data, I could ensure that I have no overlapping responses by considering $$t = 0, T, 2T, 3T, \dots$$ However, I feel that this may not be making best use of the available data, and result in models with a lot of variance. An alternative is to use overlapping responses, for example $$t=0, \tfrac{1}{2}T, T, \tfrac{3}{2}T, \dots$$ but I worry that this may create a lot of bias in the trained model. Are there known results about how using overlapping data affects the bias/variance tradeoff? Is there a "best" level of overlap to use?


title: Log transformation with negative values

text: This may for sure be a stupid question but suppose a model \begin{equation} y = \alpha \times r^\gamma \times \varepsilon \end{equation} the parameters $\alpha$ and $\gamma$ are of interst for me (and $\varepsilon$ is the standard error term). so my first intention was do take the $log()$ of the function. however, i know that the **$y$ may be negative**. any hints for this situation?


title: Correct approach to assessing correlation between events in time domain?

text: I am doing some research comparing low frequency oscillations in heart rate with palpated/observed phenomena. I am using a heart rate monitor to collect data and simultaneously asking an observer/palpator to note the occurrence of (possibly related?) phenomena and recording the timings of these events. The following graph represents the product of my number crunching so far (its taken a while to get there!) http://s129.photobucket.com/user/ogzsam/media/correlation.png.html In this graph the blue line represents low frequency oscillations in heart rate; the coloured vertical lines represent the event markers for the palpatory events reported by the observers. The x axis is just elapsed time in seconds, the y axis is, well i'm not sure really what that is yet, but it is the x axis I am interested in! I am trying to assess whether there is a statistically significant correlation between the maxima and minima, or peaks and troughs, of the blue waveform, with the coloured vertical markers in time. Do the palpatory markers correspond (as they appear to) with the peaks and troughs of the wave to a greater extent than that which we would expect by chance alone. (For those who are interested the low frequency waveform was extracted from beat-to-beat heart rate data. This was Fast Fourier Transformed to yield a power spectral density plot showing the relative power of constituent oscillatory frequencies. The desired frequency range was then filtered for by replacing all other data with 0s, then this was Inverse fourier transformed and plotted against time to give this blue waveform. This is why I am not sure what the unit on the y axis represents: I understand how to make excel do an FFT, but the maths behind it still hurts my brain!) I am following a piece of published research (Nelson et al 2001 ref below) which made similar measurements, and followed a similar approach, but their statistical approach was simply to pair the palpatory markers with peaks/troughs of the waves, and then to do a paired t test using the values on the x axis, to assess for significance. This seems wrong to me on many fronts - the points on the x axis are not normally distributed; pairing palpatory markers with peaks or troughs does not take account of how many peaks and troughs [I]do no[/I]t correspond with any marker. etc etc. My approach so far has been to work out the average wavelength/frequency of the peaks and troughs in the blue waveform, or the average interval between adjacent maxima and minima, and to see if a marker falls with 10% of this interval, away from a peak or a trough. This at least enables me to designate a marker a 'hit' or a 'miss'. It also enables me to see what percentage of markers 'hit' and peak/trough, and what percentage of peaks/troughs hit a marker. This is descriptively interesting but as far as I can see, it doesn't take into account how many of them might hit due to chance alone. For this reason a colleague suggested that I derive a 'mean waveform' taking a maxima or minima as a start point and simply adding the mean interval between peaks/trough to derive notional maxima and minima. I could then see how many of the time markers 'hit' or 'missed' these notional mean peaks and troughs. From this I could then make a contingency table comparing expected hits and misses with observed hits and misses and come up with a test statistic. I am unclear whether this would constitute and Odds Ratio or a Chi square test but i think they are fairly similar? I have been reading up on Bland Altman plots as I was looking into reliability of different methods of measuring the same phenomena hoping this would yield something, but I understand that this relies on having paired data. I think to do this anyway I would need to be looking at the [I]intervals between[/I] events (peaks/troughs or palpation markers) and I can't help thinking that this is the direction I should be going in but I have just not wrapped my head around it yet. I guess if I take the time interval between events I can at least assess for distribution, variance etc. and i have the feeling that this might enable me to use Pearson's product moment correlation coefficient, but confess I have not wrapped my little mind around this yet either. I will update if I get any further with this Here's the paper who's approach (minus the stats!) I am roughly copying: Nelson, K. E., Sergueef, N., Lipinski, C. M., Chapman, A. R. & Glonek, T. (2001) Cranial rhythmic impulse related to the Traube-Hering-Mayer oscillation: comparing laser-Doppler flowmetry and palpation. The Journal of the American Osteopathic Association, 101 (3): p.163–173.


title: About interpretation of the results of quantile regression

text: After applying quantile regression with t=0.5,0.6 on the data set WBC( Wisconsin Breast Cancer)with 678 observations and 9 independent variables($inp_1,inp_2,...inp_9$) and 1 dependent variable(op) I have got the following results for beta values.               | t         |  0.5          |  0.6      |     | b1        |  0.002641     |  0        |      | b2        |  0.045746     |  0.01     |      | b3        |  0.005282     |  0        |      | b4        |  0.004397     | -0.002    |      | b5        |  0.002641     |  0.004    |      | b6        |  0.065807     |  0.1111   |      | b7        |  0.005282     |  0.002    |      | b8        |  0.031394     |  0        |      | b9        |  0.004993     |  0        |      | intercept | -0.181388     | -0.009    |       How to interpret the above beta coefficients and what do they mean exactly?.   * t=0.5 means are we considering first 50% of the total data?   * t=0.6 means are we considering the first 60% of the total data? can we write a equation like $y=intercept+\sum_{i=1}^{9}b_i*inp_i$ as in Linear Regression to calculate the predicted output of y or not? If we are taking into consideration 5 quantiles of data ,Does it mean that we are dividing data it into 5 parts??which variables i have to consider if the data is to be divided into 5 parts? _and_ I have got 5 equations for 5 quantiles, what exactly do each equation represent? Can I write single equation for the data set as in mean regression by combining the 5 equations of each quantile ?


title: Bootstrap p.value, Wilcoxon rank sum or other test?

text: Imagine I have a network, and in this network I have n nodes of interest. I calculated the average shortest path between these nodes. I want to see if they are statistically significantly close to each other. Therefore I select n nodes from the network randomly, and work out the average shortest path length. I repeat this 1000 times, so I have my asp for my nodes of interest a vector of 1000 asps for randomly selected nodes How can I compare the two? Just count how many times the asp of my nodes interest < randomly selected nodes?, and divide by 1000 to get a p.value? Or should I compare the asp of the nodes of interest with the vector of 1000 random asps using e.g. t.test or wilcoxon rank sum? Thanks


title: Probability that random variable B is greater than random variable A

text: I would like to find the answer to the following question. A casino offers games with one of the following 21 p’s [0; 0.05; 0.10 .... 0.90; 0.95; 1] and one of the following 21 v’s (values) [0; 5; 10 ... 90, 95, 100]. The person taking part in the game either wins the v or he gets nothing. In sequence A, 98 games are offered. The total expected value (ev) of the 98 games is: 1785.75 euro. I assumed that the variance (var) could be computed by summing: ev * (v-ev). Thus, a game with a p of 0.5 and a v of 80 has an ev of 40 and a var of 1600. The total var is 31862.94. In sequence B, 98 other games with different p’s and v’s are offered. The total ev is 1276 euro and the total var is 48585.75. I assume that both binomial probability distributions can be approximated with a normal distribution with means of 1785.75 and 1276 and sd’s of 178.5019 and 220.4218 (square root of the variance). I would like to know the probability based on the normal approximations that sequence B leads to more v than sequence A. The difference between the means is 509,75 euro which is between 2 and 3 sd’s. I am not looking for very exact answers. I would just like to know how the probability that V(B) > V (A) can be approximated.


title: When is it appropriate to use PCA?

text: My research is a about clustering a huge data set. Right now I'm not using any technique for feature selection, because I only need 3 attributes from each row. I hope I'm making myself clear, lets say I have 1000 rows of data and in each row there are at least 15 attributes (field with different category, e.g. IP add, timestamp, numbers .. etc.) From this 15 attributes I'm only using 3 attributes. Do I need to use PCA? I'm clustering when there is a perfect match. I'm still reading about PCA, it is talking about dimensional in the data, how to determine the need to use PCA. Will the PCA technique make my cluster more accurate?


title: How are the numbers of modes of marginal and joint distributions related?

text: Can a unimodal multivariate distribution have a multimodal marginal distribution? If all marginal distributions are unimodal, can the multivariate distribution be multimodal?


title: Estimating the number of streets in $CITY?

text: I'm not sure if this is the right place to ask these kinds of questions. I want to estimate the number of streets there are in a certain city. Any advice? How would you begin? (To be precise, I want to calculate the number of streets of Valladolid, but I suppose that in order to calculate it I must have into consideration things like: area, maybe population or density ... so the calculation can be extrapolable depending on some variables, at least in part).


title: Factor Significance for Factor Model

text: I am working with a given factor model of the form $$y=B x +\epsilon$$ where $x$ is a random vector in $R^M$, $B$ is an $N\times M$ matrix of factor loadings, $y$ is a random vector in $R^N$ (with $N \gg M$) and $\epsilon$ is a vector of mean-zero innovations, independent of $x$. For simplicity, the innovations are normally distributed and pairwise independent. My question is relatively simple to state: How can I test whether a factor is significant, in the same sense in which a predictor is significant in linear regression? **Clarification** : I should perhaps have emphasized that I am working with a **given** model, and I have to assess the predictive value of each factor of the model. In other terms, is there a simple way to assess whether dropping a factor from a model would result in a loss of predictive power, _without_ comparing two different models with and without the factor? In the latter case, there is a great deal of literature on model selection, based on ML, GMM etc. (The rotational invariance of factor models doesn't play an essential role, btw).


title: What are the differences of the Student's T-Test, KS-Test and the MWW-Test?

text: I have got a question that What are the differences of the Student's T-Test, Kolmogorov-Smirnov Test (KS- Test), and the Mann-Whitney-Wilcoxon Test (MWW-Test)? In the case of the T-test, the null hypothesis is μ1 = μ2, indicating that the mean of feature values for class 1 is the same as the mean of the feature values for class 2. In the case of the KS-test, the null hypothesis is cdf(1) = cdf(2), meaning that feature values from both classes have an identical cumulative distribution. Both tests determine if the observed differences are statistically significant and return a score representing the probability that the null hypothesis is true (From Levner 2005). How about the MWW-Test? Are there any assumptions of the distribution of the data for these tests? When we want to test if there is significant difference between paired variables, which one or ones we should use? Please also provide reference papers/links when you answer the questions. Thanks very much. A.


title: Does the Central Limit Theorem allow one to create confidence intervals from a web traffic dataset?

text: I'm a journalist turned developer who hobbies in APIs and analysis of web traffic. I've always enjoyed learning about stats but as I learned, I learned that I have misapplied some basic concepts in the past. I now know a bit better, and know to doublecheck my ideas with those are smarter than me -- I'm hoping to find a great answer to whether my idea is going to work. A while back I was using traffic data to do basic summary statistics and standard deviation as a basic measure of variability. Where I went wrong was that I tried to create a confidence interval -- wrong because I presumed my dataset was Gaussian when in fact I now believe it to resemble something more like a Power Law. So I could sample the data and create standard deviations of the data all day but and they wouldn't be good models. Recently I've been thinking about this problem. Given a limited set of data, how can I try to model the traffic for a website in question? I have built an API on top of a wonderful dataset, and that dataset is used in all kinds of pages around an actively used site. My idea: use the API access pattern as a small sample of my larger data set. If you're following, what I'd do is basically treat each datarequest as a sample, maybe grouping requests by hour and logging total, standard deviation, number of samples n, etc. Then take use standard deviation of those samples to create the model, under the assumption that the Central Limit Theorem says that distribution will be normal. As I understand it, even though my dataset is not normal the statistics I derive from that dataset should be. Is that the case? If so, can I create a confidence interval from that data?


title: Assumption to make on data so that we can have an approximation of the final number of representatives

text: Let $t$ be a distance threshold (user-defined parameter). $X$ is a dataset containing some $p$-dimensional data-points. $L$ is a list of representative- data collected from $X$. Given the following simple algorithm:               L = []     for each x in X:        d = min dist(x, c), for c in L        if d > t:           add x to L      What is the weakest thing we can assume on the data, so that we would be able to determine the complexity in terms of the final size of $L$ (i.e. $|L|$)?


title: Why vertical distances?

text: Why does OLS estimation involve taking vertical deviations of the points to the line rather than horizontal distances?


title: Can cross validation be used for causal inference?

text: In all contexts I am familiar with cross-validation it is solely used with the goal of increasing predictive accuracy. Can the logic of cross validation be extended in estimating the unbiased relationships between variables? While this paper by Richard Berk demonstrates the use of a hold out sample for parameter selection in the "final" regression model (and demonstrates why step-wise parameter selection is not a good idea), I still don't see how that exactly ensures unbiased estimates of the effect X has on Y any more so than choosing a model based on logic and prior knowledge of the subject. I ask that people cite examples in which one used a hold-out sample to aid in causal inference, or general essays that may help my understanding. I also don't doubt my conception of cross validation is naive, and so if it is say so. It seems offhand the use of a hold out sample would be amenable to causal inference, but I do not know of any work that does this or how they would do this. Citation for the Berk Paper: Statistical Inference After Model Selection by: Richard Berk, Lawrence Brown, Linda Zhao Journal of Quantitative Criminology, Vol. 26, No. 2. (1 June 2010), pp. 217-236. PDF version here This question on exploratory data analysis in small sample studies by chl prompted this question.


title: "Fraction of response interested in" -- what does it mean?

text: I am doing a sample size calculation for a survey and am using the formula found at the bottom of this page: http://www.raosoft.com/samplesize.html In the formula, r is 'the fraction of responses that you are interested'. What is this exactly? I thought it would be the percentage of the population I want to look at, but this is essentially the sample size I need to examine, which is the outcome answer, so that can't be right. How do you get a value for r? Thanks!


title: Detecting clusters in a binary sequence

text: I have a binary sequence such as `11111011011110101100000000000100101011011111101111100000000000011010100000010000000011101111` Where clusters of mostly 1's are followed by a larger number of zeros, like in the picture below (black stands for 1): ![enter image description here](http://i.stack.imgur.com/i0UdS.jpg) I would like to apply a technique (preferably in R or in Python) where I can automatically detect these clusters of 1's, and produce spans (denoted as red lines in the image). I know one could do this with a threshold, i.e. saying that two clusters must be seperated by at least _n_ 0's to be clusters, but I wonder if there are other established methods which do not use _predefined_ thresholds. Any idea?


title: Is it true that $P((-z \leq Z \leq z) = \alpha = 2(P(Z \leq z) = \alpha/2)$ by symmetry?

text: I have a question: > Is it true that $$P((-z \leq Z \leq z) = \alpha = 2(P(Z \leq z) = > \alpha/2)$$ by symmetry? So you really only need to find one value of $z$?


title: Is there a parameter that is calculated as a range to mean ratio?

text: I was wondering whether it's me mis-remembering something or is there a dimensionless measure of dispersion calculated as: range/(mean or median). Where range is a difference between max and min values. P.S. Could this be called "relative range"?


title: Semi-heavy tailed distribution?

text: The generalized hyperbolic distribution is said to have semi-heavy tails. I know, that heavy tails means, that the tails are not exponentially bounded, or it has heavier tails than the normal distribution. The german wikipedia connects this to the excess function and knows a medium-tailed distribution. I looked at google books and google, but I could not find a good source, which explains what semi-heavy tail means and especially what is the difference to heavy-tailed and why should it be more appropriate for financial data than the heavy-tailed. I thought heavy-tailed is appropriate for financial data, since we can observe this behaviour empirically.


title: Comparing OLS and ML through log likelihood value

text: The log-like likelihood values that are computed when I do a regression (by for instance eviews), are they comparable for different estimation techniques, specifically OLS and Maximum Likelihood? My intuition is that they are comparable when ML is done with a normal distribution, because I think that then the likelihood function should be the same as that of OLS (because OLS assumes normality of the disturbances). However, when ML is not done with a normal distribution, I think that they cannot be compared. Is this correct? Or can they be for instance compared asymptotically in some cases maybe?


title: Scale a matrix to predefined stddev and mean

text: Today i stumbled upon an filtering algorithm which as final step it says that it > scale the output matrix so it will have the same stddev and mean as the > input matrix. Is that possible?


title: Probability of vector similarity wrt dimensionality

text: I'm trying to show that the probability that two vectors are similar to each other increases as the number of dimensions decreases. I define similarity between two vectors $a = [a_1, a_2, ..., a_n]$ and $b = [b_1, b_2, ... b_n]$ as each dimension in vector $a$ being in a vicinity of its corresponding dimension in vector $b$. Intuitively this seems to make sense to me because if the vectors have less dimensions, then there is less room for variability, but I'm trying to be more formal than that. What I have so far is that the probability of two vectors being similar can be calculated as $$ p(a \sim b) = p(a_1 \sim b_1)p(a_2 \sim b_2) \ldots p(a_n \sim b_n) $$ and since all probabilities are lower than one, the higher the value of $n$, the lower the probability of the vectors being similar. I don't think this accounts for a possible statistical dependence between the values of the dimensions in one vector. Am I making sense here? Can you give me some ideas on how to support/reject this argument or point me in the direction of some useful literature? **Edit:**   1. **Regarding the probability distribution** : the distribution is not known. If somewone wants to assume one and use it as an example, I would still be interested in the answer. If there is at least one for which what I describe is valid and at least one for which it isn't, I would be even more interested.   2. **Regarding the definition of vicinity** : when I say that $a_i$ is in the vicinity of $b_i$ I mean that $a_i$ is in the interval defined by $[b_i - \epsilon_i , b_i + \epsilon_i]$ for $i = 1, 2, \ldots, n$ and $\epsilon_i$ arbitrarily small. I'm afraid I cannot be more specific than that. Again, further assumptions are also welcome.


title: How to deal with collinearity in lme with categorical IV with > 2 levels

text: I'm analysing data from our experiment. We had participants in 4 groups, each participant was measured 4 times. We measured cortisol in saliva, so it leads us to the linear mixed models, because the individual cortisol levels have different slopes. I have fitted following model:               lmer1 <- lmer(Cortisol ~ group*measurement + (1|id), data=df)      I used treatment codig for both categorical variables, because we are interested in differences between 1st measurement in first group with other measurements. My problem is, that I get strong correlations between factor levels and I'm not sure, how to solve it. Contrast coding would be one solution, but it would answer different question (as I said, we want to compare differences between 1st group,1st measurement and all the others). This is my correlation matrix for fixed effect from lmer method (lme4 package):                         (Intr) group2 group3 groupP msrmn2 msrmn3 msrmn4 grp2:2 grp3:2 grpP:2 grp2:3 grp3:3 grpP:3 grp2:4 grp3:4     group2      -0.770                                                                                group3      -0.650  0.500                                                                                groupP      -0.557  0.429  0.362                                                                           measuremnt2 -0.602  0.464  0.391  0.335                                                                    measuremnt3 -0.598  0.460  0.388  0.333  0.521                                                             measuremnt4 -0.602  0.464  0.391  0.335  0.524  0.521                                                      grp2:msrmn2  0.461 -0.600 -0.299 -0.257 -0.765 -0.398 -0.401                                               grp3:msrmn2  0.390 -0.300 -0.589 -0.217 -0.647 -0.337 -0.339  0.495                                        grpP:msrmn2  0.329 -0.253 -0.214 -0.578 -0.546 -0.284 -0.287  0.418  0.353                                 grp2:msrmn3  0.461 -0.599 -0.300 -0.257 -0.402 -0.772 -0.402  0.519  0.260  0.220                          grp3:msrmn3  0.383 -0.295 -0.579 -0.213 -0.333 -0.641 -0.333  0.255  0.501  0.182  0.495                   grpP:msrmn3  0.333 -0.256 -0.216 -0.585 -0.290 -0.557 -0.290  0.222  0.188  0.499  0.430  0.357            grp2:msrmn4  0.462 -0.598 -0.300 -0.257 -0.402 -0.399 -0.767  0.518  0.260  0.220  0.518  0.256  0.223       grp3:msrmn4  0.390 -0.300 -0.589 -0.217 -0.339 -0.337 -0.647  0.260  0.510  0.185  0.260  0.501  0.188  0.496     grpP:msrmn4  0.329 -0.253 -0.214 -0.578 -0.287 -0.284 -0.546  0.219  0.185  0.493  0.220  0.182  0.499  0.419  0.353      Do you have suggestions about how to solve this (reduce collinearity/ignore it)?


title: Map a normal distribution N(x,s) to an ordinal response variable

text: I am working on a simulation. I am going to extract a series of normally distributed values from a distribution $N(x,s)$ whose mean $x$ and variance $s$ is known. I want to pretend that such distribution $N(x,s)$ is a latent causal variable that determinates a response ordinal variable $O$ which is in a known range (i.e. $[1,10]$). All other characteristics of $O$ (value distribution) are to be determined on the basis of $N$. What function could I use to map $N$ values to $O$? Since this is just a demonstrative simulation of an algorithm working on ordinal data (this mapping is not the core of the simulation, but it is part of the generative data process) and I am not oriented toward any particular psychometric theory, I am looking for some advice for a simple mapping function, though it would be better if based on some psychometric theory.


title: Derivation of the Bivariate normal distribution using change-of-variable technique

text: ![enter image description here](http://i.stack.imgur.com/kRbNV.jpg) I am not familiar with the change-of-variable technique that the question refers to. Does anyone have an idea what is meant and how one should go about doing it?


title: What function of distance for the questionnaire data?

text: I have data from questionnaire from school. First question is study program (only 2 programs) and next 35 questions are various questions (influence of friends etc.) Possible answers for 35 questions are "definitely yes", "mostly yes", "mostly no" and "definitely no". Some of data contains missing values. I want to do hierarchical clustering in R (hclust). First of all I merged answers "definitely yes" with "mostly yes" and "mostly no" and "definitely no". So now I have "yes", "no" and missing values. **For "yes" I assigned value 1, for "no" -1 and for missing values 0.** My questions are: **Is my categorization of data good?** **What function of distance is suitable for this type of data?** hclust in R usually use Euclidean and Manhattan distance, but I think these distances are not suitable for my type of data (because of missing values). Thanks


title: Understanding summations in COV formula for time series

text: I am looking through Time Series Analysis: With Applications in R (my first exposure to time series) and refreshing summations. _I._ When given the following rule: COV[$\sum_{i=1}^{m} c_{i}Y_{t_{i}},\sum_{j=1}^{n} d_{j}Y_{s_{j}}$] = $\sum_{i=1}{m}\sum_{j=1}^{n}c_{i}d_{j}Cov(Y_{t_{i}},Y_{s_{j}})$ and then encounter COV{$\frac{e_{t}+e_{t-1}}{2},\frac{e_{t-1}+e_{t-2}}{2}$} in order to use the rule, do you need to re-write the summations so that they match the rule, like this: COV[$\sum_{i=0}^{1}\frac{1}{2}e_{t-i},\sum_{j=1}^{2}\frac{1}{2}e_{t-j} $] or is there a different way to think about these summation? Another example: _II._ Given that Var[$\sum_{i=1}^{n}c_{i}Y_{t_{i}}] =\sum_{i=1}^{n}c^{2}Var(Y_{t_{i}})+ 2\sum_{i=2}^{n}\sum_{j=1}^{i-1}c_{i}c_{j}Cov(Y_{t_{i}},Y_{t_{j}})$ to compute Var[$\frac{e_{t}+e_{t-1}}{2}$] how would you set this up? _III._ What do you do in II. when the summation in the last portion starts at i=2? My method from I. (trying to rewrite the summation to fit the problem) seems to breaks down. How should these be considered?


title: How to quantify moderation effect in an intervention study with multiple time points?

text: I want to measure a moderation effect between 3 continuous variable in an intervention study. I am using scores from self-report measures of masculinity, emotionality, and psychological distress. The idea is that emotionality will be manipulated via an experimental intervention designed to increase emotionality, and then emotionality will moderate the relationship between masculinity and distress. I hypothesize that as emotionality increases due to the intervention, the relationship between masculinity and distress decreases or is no longer significant. There will be two groups (treatment vs. control) and 3 time points (pre-, post-, and follow-up). I am most interested in measuring this moderation as variables change between time 1 and 2, and time 2 and time 3. Should I be using a mixed effects model, using treatment as fixed and time as random effect and use change scores on emotionality between time points? Distress would be the outcome variable. I am feeling confused by how to best measure this moderation considering it involves change across time points and deciding on if I should use change scores. Any help would be much appreciated in terms of conceptualizing this model!


title: How to test for a significant difference between several dependent correlation coefficients?

text: I have a group of correlation coefficients (more than two). They are all dependent on one variable A in the form of r_A1, r_A2, r_A3....r_Ak, where 1, 2 ...k denote other variables; they all have the same sample size. My question is: what statistics should I use when I want to know whether any one of the correlation coefficients is any different to any of the others? I know if there are only two dependent correlation coefficients, this can be easily compared using most statistical tools, but this is a multiple correlations test. I know that a Chi-square test can be used to compare equality of several correlation coefficients (see http://home.ubalt.edu/ntsbarsh/business-stat/otherapplets/MultiCorr.htm for an example), but to my knowledge this approach is for testing the difference between INDEPENDENT correlation coefficients. So I am wondering whether there is any approach that is equivalent to Fisher's least significant difference that can be used to make comparisons among several dependent correlation coefficients? EDIT: thanks @russ-lenth for your answer. In general, I found that CIs computed from lsmeans are larger than those computed using Fisher's Z method. Here's an example of the CIs that I get through the lsmeans function:                rep.meas      lsmean         SE df   lower.CL  upper.CL      M1        0.76914236 0.13325688 23  0.4934795 1.0448052      M2        0.82346705 0.11830361 23  0.5787374 1.0681967      M3        0.89294217 0.09386717 23  0.6987631 1.0871212      M4       -0.09985512 0.20747224 23 -0.5290441 0.3293339      M5        0.56183690 0.17249315 23  0.2050076 0.9186662      M6        0.79086279 0.12760947 23  0.5268825 1.0548431      M7        0.14667681 0.20625924 23 -0.2800029 0.5733566      Take M1 whose r = 0.769 as an example: the width of the CI from lsmeans is (1.0448-0.4935) 0.5513. The width of the CI computed from Fisher’s Z is (0.8948-0.5302) 0.3646, which is much smaller than the former. Is the difference between the widths of the two confidence intervals too large?


title: When is beta distribution bell-shaped or concave?

text: Is there some restriction to parameters $( \alpha , \beta)$ that makes the beta distribution concave down? Bell-shaped like e.g. a normal? For example, the cases in purple and black, but not the red green or blue cases: ![enter image description here](http://i.stack.imgur.com/lnZk2.png)


title: Sample Weights for classification using Gradient-Boosted trees?

text: How can "weights" be given to different samples according to their relative importance while using Gradient boosted decision trees for classification? How does the `R` package `gbm` do this?


title: What is the "scale" parameter in "continuous autoregressive model" in cts package?

text: I am trying to use the "car" command in "cts package" in R program and I see the "scale" parameter there. I wonder whether this can be assumed to be equivalent to time intervals for time series forecasting. For example, the code is like this:               car(x, y=NULL, scale = 1.5, order = 3, ctrl=car_control())      and the official cts package explanation is the following: > scale: The kappa value referred to in the paper by Belcher et a. (1994).We > now recommend selection of kappa along with the model order by using AIC. > Also, it is suggested to choose kappa close to 2pi times 1/mean.delta > (reciprocal of the mean time between observations), though it is a good idea > to explore somewhat lower and higher values to see whether the spectrum > estimates were sensitive to this choice. Choosing kappa lower increases the > risk of trying to estimate the spectrum beyond the effective Nyquist > frequency of the data - though this does depend on the distribution of > intersample times. Can anyone have some ideas, please..?


title: statistical careers where you get to move around?

text: I'm about to get my PhD. My background is in engineering and applied mathematics. My research in statistical & machine learning methods for biological applications (without being too specific). I love mathematics and doing research, but I'm astoundingly terrible at politics and I think I have 0 chance of success in academia. I've looked into (and done internships in) corporate engineering & data-science positions, and it's not for me. I've also taught and, oh boy, I am not a teacher. I'd really love work where I got to move around a lot (not staring at a computer all day), and where I got to work closely with a small team. I don't mean a career where I had to travel a lot to different cities necessarily, but rather one where I just physically have to go places and talk to people and maybe work with physical objects on occasion. Do you have any suggestions for career paths where I might combine these desires with my love of mathematics?


title: Evaluation of classifier using ROC curve in the presence of rare events

text: For binary classification problems where the data is highly imbalanced, i.e. much more negative samples than positive samples, it is recommended to evaluate the performance of a classifier using the ROC curve because it does not depend on the actual ratio between positive and negative class (see e.g. He et al). Yet, I recently came across an article that stated that rare events are usually "better predicted" when looking at the ROC curve. Unfortunately, I did not save the article and was unable to find it again so far. Therefore, I decided to ask here if someone could point me to a paper that demonstrates this behaviour or can give an explanation where this comes from. My follow-up questions would then be, what the preferred way is to evaluate a classifier under these circumstances.


title: Metropolis Sampling and invalid states

text: I have a short question about Monte Carlo integration with Metropolis sampling. I have a continuous state space, but only certain parts of this state space are valid. It is possible that the transition function can suggest a move to an invalid part of the state space, and the jump should be obviously rejected. However, I'm not sure whether this invalid sample should be counted; i.e. if N samples are taken and one is rejected from an invalid part of state space, should I divide by N-1 in the estimate. Thanks


title: Detecting Clusters of "similar" source codes

text: Assume I have 400 students (that's in a big university) that have to do a computer science project, and that they have to work alone (no group of students). An example of project could be let "implementing a fast fourier transform algorithm in fortran" (I know, that doesn't sound sexy but that makes my question simpler). I am the correcter and I want to sent routines to check if there are groups of student that have proposed implementation that are "too similar to be truly independently written". This is unsupervised search for clusters. I think the question is more about which attributes to use rather than the which clustering algorithm to use. The first thing I would do is a letter by letter histogram. Ideally, since cheaters are smarter than that, I would eventually try well chosen random permutations of letters to see if a good matching of letter's histogram (with permutation) exists. Also that those not explore the structure of the code, only the marginal distribution of letters... what solution do you have ? are there existing software or packages dedicated to that problem ? (actually in my old days computer science teachers claimed they had that type of tool, but I now suspect that they had something very simple) I guess lawyer from software developments have that type of issues also (not with 1000 students, but with 2 large codes... which makes things harder) ?


title: Why does my naive Bayes classifier only give me probabilities near 0?

text: I'm building a text classifier using the naive Bayes formula. I'm still early in the development, but I already see a problem with my technique and I was wondering if you guys would have idea that would help me solve this problem. What I want to do is score texts to order them from the more likely to be in class A to the less likely to be. I only have one class and I want to find the likelyhood that a text is in it. The problem is I only get prediction really near zero (1,068E-12 for exemple). the reason is most words have a probability of being in class A inferior to 0.5. Even if I have words with probabilities > 0.5, theses probabilities are farther from 1 then the probabilities <0.5 are farther from 0. So when I choose the N words with the probabilities the farthest from .05 I usually get only (or at least more) probabilities <0.5. And so the more words I use (N) the more the probability is near 0. Is there some optimization I could implement that would help with this problem (For now I don't even remove stop words, but I plan to)? Or is a Bayes classifier a bad choice for my problem?


title: Maximum likelihood estimator, exact distribution

text: $$\frac{e^{-(y-θx)^2/2x^2) -x/λ}}{(λ\sqrt{2πx^2})}$$ This is the joint distribution function. a)I have to find the marginal function of $X$ and $Y|X$. Now the $X\sim \text{Exp}(1/λ)$ and $Y|X\sim N(θx;x^2)$ b)Then, find the MLE for $λ$ and $θ$.   For $λ$ we have: $\cal{L}= \prod \left[e^{(\frac{-x}{\lambda})}/\lambda\right] =[1/λ]^n e^{(-∑ 〖x/λ〗)}$ $\log(\cal{L})= -\log\lambda^n-(\sum x)/\lambda $ $\frac{dl}{d\lambda}= -n/\lambda+\sum x/\lambda^2 =0$ $\hat \lambda =\sum x/n $ and for $θ$: $\cal{L}=\prod [\frac{1}{\sqrt{2\pi}}\frac{1}{x} e^{(〖-(y-\theta x)〗^2/ 2x^2 )} ] $ $=1/\prod x\cdot e^{\sum \frac{〖-(y-θx)〗^2}{2x^2}}$ $\log(\cal{L})=-\sum \log x - \sum y^2/〖2x〗^2 + \sum \theta y/x- \theta^2/2$ I derive $\theta$ and resolve $\hat\theta = \sum y/x$ c) Are they unbiased? $E(\hat{\lambda})=\lambda$. For $\theta$ do I have to use the law of iterated expectation? I have the same question for the variance of $\theta$. We know that for large n we have asymptotically unbiased mle. d)Exact distribution of $\lambda$ and $\theta$? Asymptotically the distribution for $\theta$ and $\lambda$ is normal, but is it possible to find the exact distribution for theta? (for $\lambda$ it is a $\text{gamma}(n,n/\lambda)$, right?) I have to try with the transformation law, given that x is a function of y. I have many doubts here!


title: Cosslett (1983)

text: Has anyone worked with semi parametric methods like Cosslett (1983) or Ichimura or Klein-Spady for binary outcomes? I was wondering if anyone has any R code that he/she could share?


title: MANOVA vs repeated measues ANOVA example

text: I have the following situation (4 independent treatment groups, each group is tested for 3 dependent variables) and was wondering what would be the best to do:   1. What is the best test to do? I have always done MANOVA with post hoc for each dependent variable. Now someone said it should be repeated measures ANOVA due to sample size and limit the comparisons to only 3.   2. How do I determine if we have too many comparisons and thus have to reduce the p value?   3. Are about 7 samples per treatment group enough?


title: Using particle swarm optimization (PSO) as classifier

text: I know PSO is intended for optimization, but I saw in an article that survey techniques of ML to the Intrusion detection problem (KDD 99 dataset): > Chen et al. [55] demonstrated a ‘‘divide-and-conquer’’ approach to > incrementally learning a classiﬁcation rule set using a standard PSO > algorithm. This algorithm starts with a full training set. One run of the > PSO is expected to produce the best classiﬁer, which is added to the rule > set. Meanwhile, data covered by this classiﬁer is deleted from the training > dataset. This process is repeated until the training dataset is empty. May someone clarify those lines to me? (the article of Chen et al. "PSO BASED APPROACH TO RULE LEARNING, 2007" isn't available to me).


title: Give an example about to conditional expectations with respect to a sigma field

text: I have problem with exercise, I didn't solve. Give an example on $\Omega=\{a,b,c\}$ in which $$E(E(X|F_1)|F_2)\neq E(E(X|F_2)|F_1)$$ Thanks very much for your help.


title: How to normalize data of unknown distribution

text: I am trying to find the most appropriate characteristic distribution of repeated measurements data of a certain type. Essentially, in my branch of geology, we often use radiometric dating of minerals from samples (chunks of rock) in order to find out how long ago an event happened (the rock cooled below a threshold temperature). Typically, several (3-10) measurements will be made from each sample. Then, the mean $\mu$ and standard deviation $\sigma$ are taken. This is geology, so the cooling ages of the samples can scale from $10^5$ to $10^9$ years, depending on the situation. However, I have reason to believe that the measurements are not Gaussian: 'Outliers', either declared arbitrarily, or through some criterion such as Peirce's criterion [Ross, 2003] or Dixon's Q-test [Dean and Dixon, 1951], are fairly common (say, 1 in 30) and these are almost always older, indicating that these measurements are characteristically skewed right. There are well- understood reasons for this having to do with mineralogical impurities. ![Mean vs. median sample age.  Red line indicates mean=median.  Note older means caused by skewed measurements.](http://i.stack.imgur.com/ltnWO.png) Therefore, if I can find a better distribution, that incorporates fat tails and skew, I think that we can construct more meaningful location and scale parameters, and not have to dispense of outliers so quickly. I.e. if it can be shown that these types of measurements are lognormal, or log-Laplacian, or whatever, then more appropriate measures of maximum likelihood can be used than $\mu$ and $\sigma$, which are non-robust and maybe biased in the case of systematically right-skewed data. I am wondering what the best way to do this is. So far, I have a database with about 600 samples, and 2-10 (or so) replicate measurements per sample. I have tried normalizing the samples by dividing each by the mean or the median, and then looking at histograms of the normalized data. This produces reasonable results, and seems to indicate that the data is sort of characteristically log-Laplacian: ![enter image description here](http://i.stack.imgur.com/jfct2.png) However, I'm not sure if this is the appropriate way of going about it, or if there are caveats that I am unaware of that may be biasing my results so they look like this. Does anyone have experience with this sort of thing, and know of best practices?


title: What is the correct process of finding the point estimators for the following situation?

text: Let $X_1,X_2,...,X_n$ be a random sample from a uniform distribution on $(\mu-\sqrt 3\sigma,\mu+\sqrt3\sigma)$. Here the unknown parameters are two, namely $\mu$ and $\sigma$, which are the population mean and standard deviation. Find the point estimator of $\mu$ and $\sigma$. I have tried to do that by the Method-of-Moments(MOM). The procedure is, $M\prime_j=\mu\prime_j(\theta_1,\theta_2,...,\theta_k); j=1,2,...,k$ where $M\prime_j$ is the $j^{th}$ sample moment about zero & $M\prime_j=\frac{1}{n}\sum_{i=1}^n X_i^j$ & $\mu\prime_j$ is the $j^{th}$ moment about zero ,ie, $j^{th}$ raw moment. Now, $M\prime_1=\mu\prime_1=\mu\prime_1(\mu,\sigma)=\mu$ And $M\prime_1=\frac{1}{n}\sum_{i=1}^n X_i=\bar X$ Again, $M\prime_2=\mu\prime_2=\mu\prime_2(\mu,\sigma)=\sigma^2+\mu^2$ $\Rightarrow M\prime_2=\sigma^2+\mu^2$ $\Rightarrow \sigma^2=M\prime_2-\mu^2$ $\Rightarrow \sigma=\sqrt{\frac{1}{n}\sum_{i=1}^n(X_i-\bar X^2)}$ see http://math.stackexchange.com/questions/416581 Hence Method-of-Moment estimators are $\bar X$ for $\mu$ and $\sqrt{\frac{1}{n}\sum_{i=1}^n(X_i-\bar X^2)}$ for $\sigma$. But the procedure seems illogical to me for the following reason: $\bullet$ I haven't considered the pdf of uniform density. so this procedure is also applicable for normal density. Then where is the difference? _What is the correct process of finding the point estimators for the above situation?_


title: Point estimator

text: When the point estimator under consideration has a pdf, then $P[T=\tau(\theta)]=0 $ where $\tau(.)$ is some function of parameter $\theta$ $T$ is an estimator of $\tau(\theta)$ But I did many exercises to find point estimators of the parameters of density functions. For example the two point estimators of the mean, $\mu$ and the variance $\sigma^2$ of a normal density are $\bar X$ and $\frac{n}{n-1}S^2$ respectively. But for the following statement, are those probabilities necessarily zero? "When the point estimator under consideration has a pdf, then $P[T=\tau(\theta)]=0$"


title: Power analysis for minimum sample size for 6 continuous predictors and 1 dichotomous criterion

text: My study has 6 continuous predictor variables and a dichotomous criterion variable. The IRB wants me to provide a power analysis, which I take to mean how I decided on number of participants to recruit. **How do I use a power analysis to compute minimum number of participants needed?** All of the calculators (including G*Power 3) require only one predictor variable, so I'm lost.


title: The formula for covariance in terms of joint cdf

text: I want to show that $$\newcommand{\cov}{\operatorname{cov}}\newcommand{\d}{\mathrm{d}}\cov(x,y) = \iint (F_{X,Y}(x,y) - F_X(x)F_Y(y))\,\d x\,\d y$$ However, I have no idea how to start. I know that $$\cov(x,y) = \iint (x-\mu_X)(y-\mu_Y)f_{X,Y}(x,y)\,\d x\,\d y = \iint xyf_{X,Y}(x,y)\,\d x\,\d y - \mu_x \mu_y$$ where $\mu_x = \int x f_X(x)\,\d x$ and $\mu_y = \int y f_Y(y)\,\d y$. However, I have no idea how to proceed. I would be very grateful for at least a small hint.


title: Estimating the number of Twitter users that drive to work

text: How would I go about estimating the number of active Twitter users that drive to work each morning worldwide? What data would I need to know, and how would I approach this problem using those data points?


title: Geometric Interpretation of Softmax Regression

text: I'm writing a series of blog posts on the basics of machine learning, just for fun, mostly to validate my understanding of Andrew Ng's class. As I'm currently studying generalized linear models (GLMs), my method so far is to generate a small 2D dataset for each regression algo, and apply batch gradient descent on the corresponding error function, to train the parameters. I use Python tools to try to illustrate and interpret the results in an intuitive way. Following my first post on linear regression: http://cjauvin.blogspot.ca/2013/10/linear-regression-101.html I've been able to build my logistic regression post in such a way as to emphasize the geometric interpretation of the trained $\theta$ parameters, i.e. show that they correspond to the parameters of the equation of the 2D decision boundary in the general form, $ax + by + c = 0$: http://cjauvin.blogspot.ca/2013/10/logistic-regression-101.html Next I've been trying to do the same with softmax regression (work in progress, not yet posted): http://nbviewer.ipython.org/6904092 and everything seems fine (i.e. the negative log-likelihood is getting minimized, as well as the classification error, as my notebook graphs show) but I run into difficulties when I try to interpret the $\theta$ parameters in a geometric way, as I did with logistic regression: the resulting decision lines don't make sense (as the last graph of my notebook shows). I have many doubts: does it first make sense to try to interpret those parameters in that way? Or perhaps there's a bug in my training algo? Or something else? **Update (requiring no external reading)** : If I train a logistic regression model on 2D data, the resulting three components of $\theta$ can be interpreted as the parameters of the decision line equation, in general form $(\theta_0x + \theta_1y + \theta_3 = 0)$, which might yield, when plotted, something like ![enter image description here](http://i.stack.imgur.com/b2wFs.png) If I extend this reasoning to the trained parameters of a 3-class softmax regression, the 9 components should correspond to 3 general form equations. However, when I plot them, as below, they don't seem like decision lines, and I'm wondering if it simply makes sense to interpret them geometrically like that. And if not, is there another intuitive way they can be interpreted? ![enter image description here](http://i.stack.imgur.com/YB2Yi.png)


title: Probability $X$ is less than mean

text: Let $X$ be a random variable with expected value $\mu$. I need an upper bound on $\Pr( X < \mu )$. All of the bounds I could find deal with something like $\Pr( X < \mu - a)$ and become trivial if $a = 0$. In the particular case I am interested in $X$ is a finite sum of independent random variables, i.e. $X = \sum_i x_i$, $x_i \in \{0, 1 \}$, where $Pr( x_i = 1) = p_i, Pr( x_i = 0) = 1 - p_i$. Additionally, I know that $\mu \ge 1$; I don't know if that makes a difference. Any help would be appreciated.


title: Do concepts in probability help us understand when an analogical inference is acceptable?

text: I'm a philosophy undergraduate. I've recently read about analogical inferences. When making inferences by analogy, we observe that many of the properties of the analogues are the same, then infer that the rest of the properties are probably the same. For example, an analogy between sets of the vital properties of two plants:   * Premise 1: a tree needs roots in the ground, sun, leaves, and water to live;    * Premise 2: a weed needs roots in the ground, sun, and leaves to live,    * Conclusion 1: a weed probably needs water to live too. I also read that the more properties one can predicate of both analogues, the stronger the analogy between them. I don't know enough about probability and defeasible inference to discern whether that claim is accurate. To me, it seems as though what warrants inductive inferences (on which statisticians are experts) warrants analogical inference: If we regard the set of vital properties of each plant as a population, and regard each vital property of each plant as members of that population, and regard the set of vital properties mentioned in the premises of the analogy as a sample, then at first blush it seems that we could justify the inference that 'if both plants have all the vital properties in the sample, then both plants probably have the same vital properties.' But if that's an acceptable inference, what makes it acceptable? Are there concepts in probability that can help explain why that's the case? Thank you.


title: Closed form Karhunen-Loeve/PCA expansion for gaussian/squared-exponential covariance

text: The Gaussian, or squared exponential covariance is $k_{SE}(s,t) = \exp \left\{ -\frac{1}{2l} (s - t)^2 \right\}$. It is a common covariance function used in Gaussian processes. The Karhunen-Loeve expansion is an orthonormal decomposition of sample paths of a Gaussian process. If $g(t)$ is a sample path from a Gaussian process with mean 0 and covariance $k(s,t)$, then $g(t) = \sum_{i=1}^\infty \xi_i f_i (t)$ where the eigenfunctions $f_i(t)$ are deterministic functions determined by $k$ and eigenvalues $\xi_i$ are the standard normals. My question is, **does there exist a closed form expression for the $f_i$ corresponding to $k_{SE}$?** According to [1], closed form expressions for $f_i$ are known for exponential covariances ($k(s,t) = \exp \left\{ -|s - t| \right\}$), band-limited stationary processes (finite sums of trigonometric functions), and Brownian motion. [1] Huang, S. P. and Quek, S. T. and Phoon, K. K., Convergence study of the truncated Karhunen–Loeve expansion for simulation of stochastic processes, International Journal for Numerical Methods in Engineering (2001), http://dx.doi.org/10.1002/nme.255


title: What does "a testing procedure is valid" mean?

text: How is "a testing procedure is valid" defined? For example, in Wikipedia > Fisher's exact test is a statistical significance test used in the analysis > of contingency tables. Although in practice it is employed when sample sizes > are small, it is **valid** for all sample sizes. Thanks and regards!


title: Can you do change of variables with quantiles? i.e. from quantile for the CDF $F(x)$ , get quantile for the CDF $F(x)^{\kappa}$?

text: I have a random variable $Z$ with some CDF $F(z)$ and quantile $Q(p) \equiv F^{-1}(p)$. I have $Q(p)$ in closed-form but not $F(z)$. Create a new random variable $\hat{Z}$ defined so that the CDF $\hat{F}(z) \equiv F(z)^{\kappa}$ for some $\kappa > 0$. (it sounds like this is not necessarily the CDF of $Z^{\kappa}$, but that doesn't bother me. **Question:** Can I get a quantile function for the random variable with CDF $\hat{Z}$, call it $\hat{Q}(p)$, from some transformation of the original quantile function $Q(p)$ I have in closed form?