title: Check if a certain procedure is random or not

text: I am not a statistician, but I have a personal project and I need a help. I will simplify the project a lot to make it more clear and easier to understand. Then in base of your answers, I can fit it to my real project. Let's say that I have a pot with many balls inside with a number on the balls. For a week, every half an hour, I choose a few balls (each time a different number of them) and note down the number on them. Then I repeat the same procedure for many weeks. I will end up with a dataset like this:               Monday 00:00 - [15,45,24,57,437,23,89,2,42,.....]     Monday 00:30 - [12,345,643,64,23,4,64,754,......]     ....     Tuesday 00:00 - [24,2,57,865,3,6,8,655,86,.....]     ....     Sunday 23:30 - [14,543,64,32,57,43,768,.....]      The higher the number on the ball, the better one. So, I want to check if there is a certain datetime in the week that may have the higher numbers or it is a totally random procedure. As I said, it is a simplified version. Picking a ball from a pot is certainly random. Maybe a Chi-square is right choice? But can I fit my data to this option? My first thought was to create a graph (bar graph maybe) where the x-axis would be the datetime of the week and the y-axis the sum or the average of the numbers. But this will end up to another problem. If one day has only mid numbers and another days has a lot of high and low numbers, will end up the same bar. Obviously, it isn't. May I have your ideas here?


title: How i make an ANCOVA in R

text: I want to make an ANCOVA analysis of data density per plant epiphytes. At first, I would like to know if there is any difference in plant density between two slopes one N and one S, but I have other data such as altitude, canopy openness and height of the host plant. I know that my covariate would have to be the two slopes (N and S), I built this model that runs in R and although I have no idea if it performance good. Also I would like to know the difference if I use the sign "+" or "*".               model1 <- aov(density~slope+altitude+canopy+height)     summary(model1)     model1


title: Which model for count data over time?

text: I am currently assessing some results I have from a model I applied on a corpus of text data I have mined. My problem is that my professor have told me to use a certain method, and I do not really know how to attack this problem the most sensible way. The main idea is to assess if there is a relationship between months and/or years on my response variable. My data has the following nature: > DV: Binary (Event or non event) > > IV1: Month where event occurred/not occurred > > IV2: Year where event occured/not occurred               I have a total of 431.000 observations      As it is now I have transformed my data so that I have count data for each time period instead of a binary DV. I also did a logit transformation on defined as ln(DV/(1-IV3)), in order to sort out the effect of activity. > DV(Event): Events in a time period given IV1 and IV2 > > IV1(Year): Year 1995 to 2012 > > IV2(Month): Jan to Dec within IV1 > > IV3(Activity): Events + Non-events in time period given IV1 and IV2 > (Neutralized)               I have a total of 176 observations      Right now my model is defined as: `DV = b1*IV1 + b2*IV2 + error`but I am struggling alot with the intuition of whether this make sense. I have been looking into poisson models and zero inflated models, but until now the most sensible I think is to do the logit transformation in order to neutralize activity and the look at year and month as factors in a normal linear regression, but then again I am thinking that I might as well use a logistic regression. Does any of you know how to handle such a case? I am using R, SPSS and rapidminer.


title: Is a rank statistic always distribution free?

text: For an iid sample $X_1, \dots, X_n$, is the rank statistic distribution free? I.e. for each $k$ s.t $1 \leq k \leq n$, will the distribution of the rank of $X_k$ not depend on the distribution of $X_i$? I think yes. Suppose $X_i$ has a cdf $F$ which admits an inverse $F^{-1}$, then $F^{-1}(X_1), \dots, F^{-1}(X_n)$ will still be iid but with the uniform distribution over $[0,1]$. Since $F$ is invertible, both $F$ and $F^{-1}$ are strictly increasing, so the rank of $X_k$ and the rank of $F^{-1}(X_k)$ will be the same. So we can assume the distribution of $X_i$ to be the uniform distribution over $[0,1]$, and thus the rank of $X_k$ doesn't depend on the distribution of $X_i$. If $F$ is not invertible, the ties might change when going from $X_1, \dots, X_n$ to $F^{-1}(X_1), \dots, F^{-1}(X_n)$. We cannot say the distribution of the rank statistic doesn't depend on the distribution of $X_i$? Thanks and regards!


title: Computing CDF of multivariate Gaussian from eigendecomposition

text: I have a multivariate Gaussian for a set of data, and I'd like to compute the confidence interval for that data sample. In hopes of finding an elegant solution, I did an eigendecomposition and transformed the data into the basis defined by the eigenvectors where the eigenvalues are the variance of that dimension. What's nice about this basis is there is 0 covariance, and each axis is "independent". I compute the CDF of each Gaussian independently and from that determine the confidence interval for each dimension, but now I run into the problem of how to combine them. In some sense, if these de-correlated dimensions are independent, I ought to multiply the probabilities, but it doesn't make sense to me to multiply the confidence intervals. I thought about averaging them, but that also doesn't make sense because if one dimension has a confidence interval of $\sim1$, then averaging doesn't take into account the exponential nature of this math. The last thing I considered was an L2 norm, but that doesn't work at all. Anyway, I'd appreciate some help figuring this out. Specifically, how can I determine the multivariate Gaussian CDF from the CDFs of the individual de- correlated Gaussian dimensions? Or confidence interval instead of CDF – basically the same information I am interested in. I posted this in the Math forum before I discovered this one. I hope that's kosher:


title: Sample size calculation for multivariate problems

text: I am interested in multivariate investigations. I have been trying to learn about designing such experiments where there is one dependent variable (a class/group) and many independent variables that ideally would help discriminate between the groups. However, the 'basic' issue I am running into is the problem of sample size calculation. I have been pouring through literature to see if there is a consensus how to calculate sample size for a multivariate investigation. Does anyone here know of a method to accomplish this task? I am primarily familiar with the R program but I am also competent with SAS and the Python programming language. Is anyone familiar with this task?


title: Is the beta distribution really better than the normal distribution for testing the difference of two proportions?

text: I'm working at an online agency, where we run a lot of AB testing in order to test differences in proportion between two groups (test vs. control). Standard practice in the industry for testing difference of proportions is either based on the normal distribution or chi-squared. Chi base $\lambda$ tests tend to use a lot of data, which you don't always have, while normal distribution tests are problematic, since proportions are bound by $0$ and $1$, unlike the normal approximation. I claimed to my colleagues that a test that uses the beta distribution should always over perform both normal / chi options, since its built for proportions. Here is my R code to perform the test, this test shows that test over performed the control group (95%):               library(ggplot2)     number_of_success_test      <-   46     number_of_success_controll  <-   33     number_of_failures_test     <- 2643     number_of_failures_controll <- 2579     test1 <- rbeta(100000, number_of_success_test,     number_of_failures_test,     ncp=0)     test2 <- rbeta(100000, number_of_success_controll, number_of_failures_controll, ncp=0)     test  <- data.frame(test1, test2)     quantile(test2, 0.95)     g <- ggplot(data=test, aes(x=test1)) +                             geom_density(color="red",  bindwidth=0.0000001) +                             geom_density(aes(x=test2), bindwidth=0.0000001) +                             geom_vline(xintercept=quantile(test2, 0.95)) +                             geom_vline(xintercept=quantile(test1, 0.5),                 color="red")     g + xlab("CR") + geom_text(label="95 pecentile - control group",                                 x=quantile(test2, 0.95), y=15000) +                       geom_text(label="50 pecentile - test group",                                 x=quantile(test1, 0.5), y=12000, color="red")      Am I right? Is it really always better to use beta distribution over chi / normal distribution, when dealing with difference in proportions? (Also, is my approach in the R code right?)


title: What data cleaning to do for logit regression with only dummies?

text: Does anyone know what exact data cleaning steps one need to undertake in order to clean data for a logit regression (not a logistic regression)? I have only time variables, meaning year and month, as my independent variables, and I am using R. A logit regression is simply a normal linear regression where the DV have been transformed with the following formula: > `logit(y) = ln(y/(1-y)` for An example: > 3 of 12 people gets cured from taking a pill in period 3 -> > `ln(0.25/(1-0.25)` > > 5 of 25 people gets cured taking a pill in period 5 -> `ln(0.20/(1-0.20)` One can use the logit transformation if you have ratios and in many papers and books it is closely related to the logistic regression.


title: Population standard deviation for a binomial(?) distribution?

text: I have been confused by two separate questions (Stock & Watson - introduction to econometrics ch.3), using different values for standard errors. **The first:** In a survey of 400 voters, 215 respond to vote for the incumbent, 185 for the challenger. Let p denote the fraction of all likely voters who preferred the incumbent at the time of the survey and $\hat p$ be the fraction of survey respondents that prefer the incumbent. _Now for the variance it is given by $\hat p(1-\hat p)/n$ and when calculating the $SE(\hat p)$ we have to take the square root of the variance to get $0.0249$, and I am fine with this._ **The second question:** In a given population 11% of voters are African American. A survey using a random sample of 600 landline telephone numbers finds 8% African Americans. Is there evidence the survey is biased? _Now when calculating the t-statistic we use the null hypothesis with $p=0.11$, but it then states that $se(\hat p)=\hat p(1-\hat p)/n$._ Why do we no longer have to take the square root of the value above to find the standard error? I imagine it must be to do with knowing the population variance?


title: Response Rate in Research

text: I want to ask about response rate. I have distributed 2500 posts. However, I have received back around 30. Then, I sent 200 posts as reminders with appealing letter. I have received one or two only. The total usable questionnaires: 21. After a while, I determined that I need to send by email. Therefore, I sent over 5,000 emails with around 6-time reminders. over 500 people have opened the questionnaire, but not all of them responded. Maybe people who showed an intention to reply by answering few questions are around 250. The usable ones around 165. Is there any problem with response rate? The informants were owner-manager of small and medium manufacturing enterprises. In addition, I have tested the homogeneity among the both groups (posts and emails) they could emerged except in DV they show significant difference? 1. The question can I emerge the both groups even response rates of posts is too low at the first place? 2. If yes (I can merge), what should I do as only both groups show difference regarding DV (not IVs, not demographic variables) Thank you


title: How should I color scheme a heat map based off data below?

text: I have subdivided the globe into 100 sq mile bins and then collected how many tweets were sent with a geolocation within each bin. At first I color coded each bin on the map based off a standardized value of (tweetInBin)/(maxTweetInAnyBin). This produced only one "hot spot" with all the other places being almost uniform in color. ![http://i.stack.imgur.com/4gJms.png \(Sadly I don't have enough rep yet to actually embed the photo\)](http://i.stack.imgur.com/kx5xr.png) As you can see from the percentile chart, the 100th percentile is so large in population relative to the other bins, that standardizing by the max val destroys any meaningful representation. My question to you guys is how do I color code my data so that I can have a meaningful heat map. I was toying with the idea of linearizing the data by color coding based off of percentiles instead of based off of population. Basically the 100th percentile would get 100% intensity on the color scale, while the 50th percentile would get 50% intensity on the color scale and so on. My only gripe with this method is that it marginalizes the actual difference in populations much like taking the median marginalizes the outliers. How do I handle the outliers while still conveying population information?


title: Repeated measures within factors settings for G*Power power calculation

text: I'm trying to perform a power calculation with G*Power. There are two quite important options, the meaning of which is not clear to me: "Number of groups" - what is this? I have a 2x2 repeated measures factorial design. Does this mean there are four groups, or, as the rest of the internet suggests, is this box for between-subjects factors? If it's for between- subjects factors, I assume this is 1 for a repeated measures factorial. "Number of measurements" - also not entirely clear. Is this the number of data points I collected per participant? What if they've been averaged before they go into the ANOVA? Or is this the number of within-subjects conditions? So 4 for my experiment? Any guidance on the meaning of these settings much appreciated - as far as I can tell these are undocumented.


title: Applying Bayes: Estimating a bimodal distribution

text: I'm trying to estimate a bunch of bimodal distributions, i.e. two means and two standard deviations, based on a variable number of inputs. If no input is present, a constant value should be returned. From my anecdotal knowledge of what Bayes is about, this should be exactly it, no? Adapt a prior based on incoming evidence. My question is, though, how would I go about that in detail? Maybe I'm just missing a good tutorial somehow.


title: testing correlated data from 1:1 line

text: I have correlated data points and they are fitted with a trend line. I wanted to know how far these points lie away from an arbitrary 1:1 line (45 degree line with data of equal magnitude). Is there any test to find how significant it is away from this 1:1 line? Thank you


title: Updating a Dirichlet distribution with partial data

text: I've got some categorical data where each observation has multiple attributes, and I want to make a probabilistic model of this using Dirichlet distributions. For example, in the two dimensional case I would have two attributes A, B having n, m distinct values respectively, and then I would have a Dirichlet distribution with nm parameters. However, I also want this model to be able to handle incomplete data for a given observation. For example, I may be able to specify A for an observation but say nothing about B. I want to be able to update my model based on this data but keep within the realm of Dirichlet distributions. For example, I don't want to go into hyperdirichlet distributions because the number of parameters I'd have to keep hold of will explode.


title: Bandits without exploitation: finding the best items with incomplete information

text: I'm trying to analyze a general game. This is probably well-known, in which case pointers to relevant literature would suffice (but explanation would not be declined!). If it's not standard, of course any help would be appreciated, even if short of a full answer. Suppose I have N objects (say, 100) with nontrivial priors $v_1,\ldots,v_N$ representing my best guess as to their values. I have time to perform $m$ trials (say, 1000) on any of the objects, and the tests can be adaptive. I want to choose the top $k$ (say, 3) objects in order. Each trial improves my estimate of the true value of one of the objects. But unlike the multi-armed bandit problem, there is no exploration-vs-exploitation tradeoff as such: all that matters is the list at the end of the $m$ trials. You can think of the cost function as (value of item #1 - value of item selected as #1) + ... + (value of item #$k$ - value of item selected as #$k$), so getting close is sufficient. The priors are fairly accurate (certainly a good guide on where to search) but are subject to unknown systemic errors. The trials are unbiased but have random error. Fortunately, the random errors are the same for all trials, regardless of the object, so that seems at least somewhat statistically controllable. I'm very much at an information-gathering point here; assume what you must to answer the question and I'll try to make it fit. :)


title: Predicting count data with random forest

text: Can a Random Forest be trained to appropriately predict count data? How would this proceed? I have quite a extensive range of values so classification doesn't really make sense. If I would use regression would I simply truncate the results? I'm quite lost here. Any ideas?


title: Regression with averages and collinearity

text: I am stuck with the definition of a regression model regarding the inclusion of a constant averaged by the dependent variable. I have a dataset `M=(id, x1,x2,x3,x4)` where:               id: is the id of the product      x1: is a continuous variable denoting price (0 to 999 USD)      x2: is a categorical variable denoting whether the product belongs to category A     x3: is a categorical variable denoting whether the product belongs to category B     x4: is a discrete variable having the customer rating with values V=(1,2,3,4,5)       Through a transformation for each product `id` I generate the averages for `x4` as `AVG(x4)_{i}` where `i={1,2,3,4,5}`. Now my dataset becomes               M=(id, x1, x2, x3, AVG(x4)_{1}, AVG(x4)_{2}, AVG(x4)_{3}, AVG(x4)_{4}, AVG(x4)_{5})      I want to regress `AVG(x4)_{i}` on `x1, x2, x3`, however because `x1` from theory has a direct interaction with the dependent variable, I want to regress the model:               AVG(x4)_{i} = b1*x1+b2*x2+b3*x3+b4*(AVG(x4)_{i}*x1)+C+e      for each value of `i`.   * Is that definition of the model correct?    * Should `b4` dropped due to collinearity?


title: Minimizing the expected loss

text: I was wondering about the motivation behind the following definition of expected loss: $$E[L] = \sum_{k} \sum_{j} \int_{R_{j}} L_{kj} p(x, C_{k})dx$$ where $L_{kj}$ is the loss matrix, in which $j$ is the predicted class and $k$ the true class, $R_{j}$ is the decision region corresponding to the $j$ class and $x$ is an input vector. For the sake of concreteness, let's assume we have only two regions $R_{1}$ and $R_{2}$ and that elements contained in $R_{1}$ and $R_{2}$ belong to class $C_{1}$ and $C_{2}$, respectively. For example, an element $x_{i}$ in region $R_{2}$ will contribute with the term: $$L_{12}p(x_{i}, C_{1}) + L_{22}p(x_{i}, C_{2})$$ but $L_{22}$ probably is $0$ because the loss associated to predicting class $C_{2}$ when the true class is $C_{2}$ is what we want. I understand that we want to minimize $E[L]$, so every time we predict the class incorrectly, we are increasing $E[L]$ according to $L_{kj}$ but why are we multiplying, in this example, the term $p(x_{i}, C_{1})$ or in general, $p(x, C_{k})$? Simplifying, for every assignment of $x$ to the class $j$, we want to minimize: $$\sum_{k} L_{kj}p(C_{k}|x)$$ but the question remains, why p(C_{k}|x)? By the way, I can see that the expectation requires a probability but I can't see why to choose the probability of the true class given $x$. Regards


title: Peak detection in a ucontroller using FFT

text: A project I am working on currently is to create a heart beat sensor using a finger plethysmograph. It returns a signal to my ucontoller that only includes positive peaks. For example, the ideal signal it will sent to my ucontroller would be               1 1 1 5 1 1 1 5 1 1 1      Where the numbers represent the amplitudes. There ideally will be no negative components to the signal. My goal is to detect these peaks and count the number of milliseconds between the peaks and use that to calculate the bpm accurately. I could use a slope detection method but it would probably only be accurate for an ideal signal with no noise. For the real implementation and not so ideal case, I was thinking that I could use an FFT. Theoretically the highest bpm of a human heart is approximately 4Hz, my nyquist rate would be 8Hz. With a 16 point FFT, each bin would contain real and imaginary parts in sections of 8/16 Hz. I could calculate the magnitude for each bin. Im guessing the bin with the highest magnitude would be frequency of the peaks?? Is my approach correct?? Can I only detect differences of .5Hz using a 16 point FFT? Also, do I use the FFT in a window shift method or only perform the FFT after 16 points of a signal??


title: Choosing a multiclass strategy

text: What would be the best approach to choosing the multi-class strategy (MSVM, OVA, ECOC) for classifying a large number of classes with limited examples of each class? Another two factors that define the solution, in my case:   * The probability of prediction is an important factor that decides the next steps in my solution.   * The size (i.e. size of the classifier object on disk which can later be loaded in memory and used for prediction), of the fitted classifier (since I need the solution to be portable without providing it as a service). I do not need to analyze the classifier(s) any more than getting the category with reasonable probability.    UPDATE: The problem is basically that of text classification.


title: Continued fraction representation of the multivariate normal distribution

text: Can anybody help me with a continued fraction representation of the multivariate normal distribution? Such a representation is well-known for the univariate case; see, for example, > C-I. C. Lee. On Laplace continued fraction for the normal integral. _Ann. > Inst. Statist. Math._ , 44(1):107–120, March 1992. Does such a continued fraction representation also exist for the multivariate case? (I followed the suggestion by posting the same question on math stackexchange, see here)


title: Feature selection and parameter tuning with caret for random forest

text: I have data with a few thousand features and I want to do recursive feature selection (RFE) to remove uninformative ones. I do this with caret and RFE. However, I started thinking, if I want to get the best regression fit (random forest, for example), when should I perform parameter tuning (`mtry` for RF)? That is, as I understand caret trains RF repeatedly on different feature subsets with a fixed mtry. I suppose the optimum `mtry` should be found after the feature selection is finished, but will the `mtry` value that caret uses influence the selected subset of features? Using caret with low `mtry` is much faster, of course. Hope someone can explain this to me.


title: Special method for forecasting on time-series clusters in R?

text: I'm doing a project related to identifying sales dynamics. My database contains 26 weeks after launching the product (so 26 time-series observations equally spaced in time). ![http://imageshack.com/a/img18/5628/l5qg.jpg](http://i.stack.imgur.com/Dquwy.jpg) ![http://imageshack.com/a/img34/8953/yh6i.jpg](http://i.stack.imgur.com/8Dh2C.jpg) I used two methods of time-series clustering to see which patterns dominate in different groups (clustering by `units_sold_that_week`). The first method is based on k-medoids and the second one connected with clustering by parameters of growth models. My next step is to make forecasts based on these clusters. Is there any special method for forecasting based on time-series clusters? In my project, I have to combine the topic of clustering and forecasting on clusters. I am running my analyses in R, so I would be grateful for any suggestions regarding R procedures. Please note that I am relatively new to time series analysis so any clarity you could provide, on R or any package you could recommend that would help accomplish this task efficiently, would be appreciated.


title: How to model to improve the room usage efficiency based on motion sensor history

text: To reduce the confusion, I changed my application from traffic to meeting room, so this application is about modeling a meeting room efficiency , the data collection is built by placing a kind of motion sensor at the center of room for detecting and counting the motion activities. So my data are time- series based for the motion activities. The model I want to build is for the purpose of optimizing the time length for showing the people the status of room being occupied or not. If a sensor receives motion event at the moment, in order to decide how long to show the occupied status, the application would look back the history of motion events, (say last 5 minutes) In case if counting 20 consecutive motion events in the last 5 minutes, then it means there was high density of activities, so I would assume the room was busy and will be busy in the next consecutive minutes, then make the occupied status time stay longer time (5 minutes) before it turns to free status If in cases that in the last 5 minutes, only 2 events being monitored, then it means the room might be not busy, then make this occupied status time stay only 1 minute. So far I'm only concerned about what kind of statistical model should I use to predict the how long the occupied time length based on the latest 5 minutes'motion history?


title: Predicting time-to-event given a third-party estimate that may be adjusted over time

text: I've been doing some work on modeling aspects of a sales process, where each in-progress sale is referred to as an Opportunity. I'm trying to develop a model to predict when an Opportunity will close (doesn't matter if it's lost or won). A given Opportunity has several descriptive characteristics, such as the estimated amount of the sale, the sales rep, etc. In addition, each Opportunity has an expected close date. This expected close date (i.e. expected number of days to close) is obviously a valuable predictor for the actual close date. In my current model, I use this original expected # of days until close along with several descriptive attributes as inputs to a quantile regression forest algorithm to get quantiles for the actual time to close. This current approach works ok, but it only takes in to account the ORIGINAL estimate for the time to close (generated by the sales rep). In practice, over the lifecycle of the Opportunity, this estimate may be changed several times by the sales rep who is working on the sale. I'm hoping someone has advice on how I can model this whole process where an Opportunity has an original time to close estimate which is progressively updated over time. Obviously the model will be used every day in order to update the predictions for Opportunities that have had their close date estimated updated. Ultimately, I only care about getting quantiles for the actual time to close.


title: NA or "noResponse" in categorical variables

text: I am learning R and data analysis methods, so I apologise if my question is too basic. I have a data sample of 4K+ records of sociologic survey. There are a few demographic variables and a lot of yes/no type categorical questions. There is a binary variable I am trying to predict based on the categorical questions and/or demographics. A lot of the categorical questions have no responses. I would like to understand what would ramifications of treating the blank fields as NA or setting them to a value of "noResponse". My understanding is that if the fields are NA, R will disregard them from calculations when creating a model. If I set them to "noResponse" R will use the fields in calculations and this will signify the choice by respondent not to answer. It likely would be good to know if missingness pattern is MCAR, MAR or MNAR, but given the number of variables I don't see a way of determining this apart from guesstimate. All I know is that people didn't have to answer any of the questions apart from the binary dependent variable.


title: Linear Discriminant Analysis matrix dimension

text: I am trying to implement Linear Discriminant Analysis for face recognition. I have 3 classes and each classes have 10 image each. The dimension of matrix in class A, B and C is 10*500 .So each row will represent an image If I find the mean matrix of each class I am getting dimension of 1*500. That is I will be adding the row and divide by 10. Global mean matrix of all classes I am getting 1*500 dimension. Within Scatter Matrix Sw= The dimension of Matrix is 10*10 Matrix. Between Scatter Matrix Sb= The dimension of Matrix is 1*1. Next Step is I have to find Inverse(Sw)*Sb. But the matrix dimension is totally different. I know I am doing some mistake but I don't know where? Could you please help me to solve this problem? Can you please tell me how the dimension of the matrix should be?. thank you


title: Permutation test of meta-analysis : correlation coefficient

text: I am going to estimate False Discovery Rate using permutation tests. To my knowledge, several R packages are applicable for multiple testing. I have several independent datasets for meta-analysis. I computed a Pearson correlation coefficient, converted it into a Fisher-Z score and then calculated the mean effect size of each gene pair from several independent datasets. Actually, in each dataset, there are two different subgroups - healthy subjects and patients. In this co-expression network, I analyzed the gene expression profiles of the _patient_ group only. Here is my question. > What should I do to estimate FDR using permutation test with creating random > and independent shuffling gene expression values of all genes in each > dataset to break the inter-gene relationships while keeping intact the > expression mean and standard deviation of the genes in every dataset? Unfortunately, I have no idea how to do it. I am still new to statistics and as well as R. I would appreciate it if you could answer the question with corresponding code I could run.


title: Response variable bounded by dependent variable

text: I have a problem: y~x1+x2+x3+x4 where the response variable y is always more than or equal to the explanatory variable x1 (for e.g. x1 represents the income of one person in a couple and y is the overall income of the couple). It seems that normal linear regression is inadequate in this case. What kind of regression models should I be looking into?


title: How would one use the pi-lambda theorem in determining that a CDF completely defines the distribution of a random variable?

text: The pi-lamda theorem states that if $Q$ is a pi-system contained in a lambda system $L$, then the sigma-algebra generated by $Q$ is also contained in $L$. We can use this to show that the CDF completely determines the distribution of a random variable, but I do not understand the connection.


title: how to test most frequent variable between groups?

text: I have different groups of dolphins, some with offspring, some without. I'll count how many types of vocalizations were emitted by each group, and want to find out if there is a statistical difference between groups with and without offspring. My dataset is something like:               Group_number    Vocal1  Vocal2  Offspring     Gr01    5   3   0     Gr02    7   3   0     Gr03    4   4   0     Gr04    1   6   0     Gr05    7   9   0     Gr06    6   2   1     Gr07    2   4   1     Gr08    2   6   1     Gr09    9   7   1     Gr10    8   8   1      I have to do this one vocalization type at a time (I have some tens of types)? Or can I do it all in a single run (like a multiple linear regression)? Someone told me to do a chi-square, but am not sure how to do it, or if it's appropriate. T-student, ANOVA, MANOVA, GLM, what's the pros and cons of each method, supposing my data is normally distributed? And what if it's not? Sorry if it seems a silly and vague question, but maybe someone could point me to a simple tutorial or something. Thanks in advance!


title: Looking for the 'Elbow' in data

text: Subitization is the rapid, accurate enumeration of low-numerosity displays, distinguished from counting by a sharp non-linearity in the plot of response times. Below is a representative plot, from Watson, D. G., Maylor, E. A., & Bruce, L. A. M. (2007). Notice that mean enumeration times for displays 1-3 increases roughly linearly, but mean enumeration time for 4 does not follow the linear trend. Some research suggests that the subitization 'limit' is dependent on task conditions and participant working memory. ![enter image description here](http://i.stack.imgur.com/krvVw.gif) I'm looking for a way to test where the elbow is, with the ultimate goal of identifying what a participant's subitization limit is. Currently, my best idea is to do something like repeated polynomial contrasts. Basically, I would test for a quadratic trend in numerosities 1-3, then in numerosities 1-4, etc. I would want to say that I have passed the subitization limit when the quadratic trend becomes significant (adjusting for repeated tests). That's about the limits of my statistical savvy, though, so I can't evaluate this idea too well. Thoughts? Thanks in advance.


title: Question about Eigenfaces (PCA applied to facial images)

text: I've been reading up a bit on eigenfaces. I think I understand the basic concept of it - vectorize a set of facial images then reduce the dimensionality of the images using PCA. What I don't really understand it the visualization of the lower-dimensional representation of the images. In the facial images, the number of dimensions is the number of pixels so if you reduce the dimensionality of an image, you reduce the number of pixels. But then how do you visualize this image? Is it just a much smaller version of the full-dimensional original? The examples that I have seen do not look like this. Or do you alternatively make each pixel bigger so that the overall image is the same size as the original?


title: Find the difference between two vectors and the mean of the result

text: So, with the help of someone here in stats i could solve most of my problems. Now i'm stuck at this part: I have the below code, which worked very well if this values were fixed.               immediate.amput.below.EV     watchful.wait.EV      Now what they are is vectors. They represent a vector with [n_psa] size. What i need to do is, i need iterate for n_psa values, i.e for 100 values and find the difference between each immediate.amput.below.EV[1] - watchful.wait.EV[1] and so on, save the results only as positive numbers, even if the output is a negative number in a new vector, and find the mean of the new vector. This is what i used to do.               DELTA_COST <- max(immediate.amput.below.EV,watchful.wait.EV) - min(immediate.amput.below.EV,watchful.wait.EV)     mean(DELTA_COST)


title: Confidence interval for geometric mean

text: As title, is there anything like this? I know how to calculate CI for arithmetic mean, but how about geometric mean? Thanks.


title: are the percentage differences significantly different from oneanother?

text: morning, say i have 5 wheat varieties undergoing 2 treatment. when counting the seeds of the replicates the percentage difference between the means of the 2 treatments is 10% for variety 1, 30% for 2, 50% for 3, 80% for 4 and 90% for variety 5. how could i demonstrate that var 1 has a significantly greater % decline than var, 3 but not var. 2 for example? any hep would be much appreciated. many thanks


title: How to do logistic regression subset selection?

text: I am fitting a binomial family glm in R, and I have a whole troupe of explanatory variables, and I need to find the best (R-squared as a measure is fine). Short of writing a script to loop through random different combinations of the explanatory variables and then recording which performs the best, I really dont know what to do. And the `leaps` function from package **leaps** does not seem to do logistic regression. Any help or suggestions would be greatly appreciated Leendert


title: Variable type in survey analysis

text: I have a survey analysis data in which responses for various independent questions range from 'Poor' to 'Excellent'. Belown is the scorecard for these responses:               Excellent 100     Very good 90     Good      75     Fair      25     Poor       0      There were also some blanks present in the survey data which I was asked to replace with the mean of the entire column. For example, if I had 5 responses for say 'Quality of food'as 0, 25, 75, **blank** , 100 then I had to replace the blank with 50 (average of the 4 responses). My dependent variable, however does not contain any blanks. My questions are as follows:   1. Can I still treat the dependent variables as Ordinal   2. What type of regression is best suited for this kind of data, Linear or Ordinal Logistic. As per my understanding, linear regression would fail in this case Any help would be greatly appreciated. Thanks.


title: Correlation analysis and correcting $p$-values for multiple testing

text: I am investigating the effect of a drug on EEG and cognition in epilepsy patients. We have done EEG and neuropsychological (NP) tests twice, before and after medical treatment. Some of parameters in EEG and NP tests were significantly altered after drug treatment (done by Wilcoxon rank sum test, as the distribution was not normal). Now, I would like to calculate correlation between change in EEG parameters and change in NP tests. There are 7 parameters in EEG parameters and 27 in NP tests. I would like to run correlation analysis on every pair and see if any pair is significantly correlated. My questions are:   1. Should I use nonparametric correlation, say Spearman, to calculate $\rho$? (as values of EEG and NP paramaters are not normal)   2. Since there are 7 parameters in EEG and 27 parameters in NP tests, there should be 7*27 number of testing, which requires correction of p value for multiple comparisons. Which method should I use?  I am looking for not-so-conservative method. I have seen from another posting that I can use permutation test for multiple comparisons. Could someone explain how I can do this?


title: How to calculate regularization parameter in ridge regression given degrees of freedom and input matrix?

text: Let A be $n \times p$ matrix of independent variables and B be the corresponding $n \times 1$ matrix of the dependent values. In ridge regression, we define a parameter $\lambda$ so that: $\beta=(A^\mathrm{T}A+\lambda I)^{-1}A^\mathrm{T}B$ . Now let [u s v]=svd(A) and $d_{i}=i^{th}$ diagonal entry of 's'. we define degrees of freedom (df)= $\sum_{i=1}^{n} \frac{(d_{i})^2}{(d_{i})^2+\lambda}$ . Ridge regression shrinks the coefficients of low-variance components and hence the parameter $\lambda$ controls the degrees of freedom.So for $\lambda=0$, which is the case of normal regression, df=n, and hence all the independent variables will be considered. The problem I am facing is to find the value of $\lambda$ given 'df' and the matrix 's'. I have tried to re-arrange the above equation but was not getting a closed form solution. Please provide any helpful pointers.


title: Why am I getting a 10-15% type I error rate for a 2 x 2 ANOVA?

text: While testing type I error rate using R, I found that I am getting higher than 5% false positives with a 2-way ANOVA. However, I thought one of the reasons for using an ANOVA (as opposed to multiple tests) was to control the type I error. So, what am I doing wrong? Here is the R code:               generate.dat <- function(N=30)       # Generate data with 2 factors.       # N is number of subjects per group.     {       # id is the subject id, we use N*4 because there are four groups.       # group is the full name of the group, A1, A2, B1, B2       # value is the measured/observed result in our experiment, again N*4.       # the mean value is 0 for all groups, i.e. there are no effects of group.       dat <- data.frame(id=factor(1:(N*4)),                         group=factor(c(                               rep(c("A1", "A2"), each=N),                               rep(c("B1", "B2"), each=N))),                         value=rnorm(N*4))            # split group into separate factors       dat$factor1 <- factor(substring(dat$group, 1, 1))       dat$factor2 <- factor(substring(dat$group, 2, 2))            dat     }          # e.g.     # dat <- generate.dat(3)     # print(dat)     #   id group       value factor1 factor2     # 1   1    A1  0.42385602       A       1     # 2   2    A1 -0.34829466       A       1     # 3   3    A1 -1.40946883       A       1     # 4   4    A2 -0.09177423       A       2     # 5   5    A2 -1.26614034       A       2     # 6   6    A2 -1.18024188       A       2     # 7   7    B1 -0.86129559       B       1     # 8   8    B1 -1.30517594       B       1     # 9   9    B1  0.50221849       B       1     # 10 10    B2  0.42755864       B       2     # 11 11    B2 -0.03262990       B       2     # 12 12    B2  0.85593000       B       2          do.anova <- function(dat)       # Perform a 2 way between-subjects anova on dat.       # Return the lowest p value obtained.       # (We are interested whether there is at least one significant effect       # in this particular data set).     {       results <- aov(value ~ factor1 * factor2, data=dat)       min( summary(results)[[1]]$`Pr(>F)`[1:3] )     }               # Peform 1000 replications:     # each time generate some data and get the lowest p value     # from the resulting anova.     out <- replicate(1000, do.anova(generate.dat()))          # How many times out of the 1000 was there a significant effect?     sum(out < .05) / 1000 # about 14%      I doubt there is really a 14% chance of finding an effect, so what am I doing wrong? Many thanks.


title: Detecting Outliers in Count Data

text: I have what I naively thought to be a fairly straight forward problem that involves outlier detection for many different sets of count data. Specifically, I want to determine if one or more values in a series of count data is higher or lower than expected relative to the rest of the counts in the distribution. The confounding factor is that I need to do this for 3,500 distributions and it is likely some of them will fit a zero inflated overdispersed poisson, while others may best fit a negative binomial or ZINB, while still others may be normally distributed. For this reason, simple Z-scores or plotting of the distribution are not appropriate for much of the dataset. Here is an example of the count data for which I want to detect outliers.               counts1=[1 1 1 0 2 1 1 0 0 1 1 1 1 1 0 0 0 0 1 2 1 1 2 1 1 1 1 0 0 1 0 1 1 1 1 0               0 0 0 0 1 2 1 1 1 1 1 1 0 1 1 2 0 0 0 1 0 1 2 1 1 0 2 1 1 1 0 0 1 0 0 0               2 0 1 1 0 2 1 0 1 1 0 0 2 1 0 1 1 1 1 2 0 3]     counts2=[0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0               0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0               0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0               1 1 0 0 0]     counts3=[14 13 14 14 14 14 13 14 14 14 14 14 15 14 14 14 14 14 14 15 14 13 14 14               15 12 13 17 13 14 14 14 14 15 14 14 13 14 13 14 14 14 14 13 14 14 14 15               15 14 14 14 14 14 15 14 1414 14 15 14 14 14 14 14 14 14 14 14 14 14 14 13 16]     counts4=[0 3 1.......]     and so on up to counts3500.      Initially I thought I would need to write a loop in Python or R that would apply a set of models to each distribution and select the best fitting model according to AIC or other (maybe the fitdistrplus in R?). I could then ask what were extremes for the given distribution (the counts that fall in the tails e.g. would a count of "4" be an outlier in the counts1 distribution above?). However, I am not sure this is a valid strategy, and it occurred to me there may be a simple methodology for determining outliers in count data of which I was not aware. I have searched extensively and found nothing that seems appropriate for my problem given the number of distributions I want to look at. My ultimate goal is to detect significant increases or decreases in a count for each distribution of counts, using the most statistically appropriate methodology.


title: Recognition of simple patterns and prediction

text: I have been doing supervised learning and classification with multilayer perceptron for some time. But now I need to use unsupervised learning to infer the presence of a pattern and I would need some help. It is not a hard problem by itself and I can solve it by supervised learning (by showing "examples" of the possible patterns), but I want my algorithm to be able work without giving it pattern samples to learn from. Imagine that I have nine tiles presented as a 3x3 square, numbered from 1 to 9. They light out one at a time in a certain pattern unknown to the user. The pattern could be a sequence of anything from 2 tiles (say 3-4-3-4-3-4...) to a much longer sequence (say **2** -5-3-2-6-4-6-1- **2** -5-3-2-6-4-6-1 etc). I need an algorithm able to infer the pattern (or "regularity") and then use this knowledge to do an educated guess of what should be the next tile to light up. It doesn't need to learn forever though. It could be: "If you think you have a pattern and it repeats itself at least 10 times, presume you've found it". I was reading scholarly articles on the subject but I think I need some ideas to narrow my researches first. Thanks!


title: Ljung-Box Statistic

text: Suppose I fit an AR(2) model to a dataset and get the diagnostics. If the Ljung-Box Statistic is significant for all lags for a time series model what is the interpretation of that? Does that mean that the model is not a good fit? Other than this, the ACF of the residuals indicate that they are stationary and normal.


title: How to correctly apply the Fama&French with momentum to estimate the degree of outperformance?

text: I have a question about the use of the Fama & French factors. I’m currently examining possible differences in abnormal earnings between to different type of real estate portfolios. To counter Roll’s (1977) critique on the use of Beta to estimate a portfolio’s degree of outperformance, I will include a few approaches. Two different portfolio constructions are used:   1. the value;    2. and equally weighted Four different market proxies are used:   1. CRSP value weighted index,    2. CRSP / Ziman Equity value weighted index,    3. a small cap index,    4. and S&P 500 index. My first question is:   **Is it possible to use the (value weighted) FF factors with momentum to estimate the outperformance of the two different portfolios, since one is equally weighted while the other is value weighted?** Another thing is the four different benchmarks. The first three returns are based on the NYSE, AMEX, and NASDAQ stocks, same as the four factors downloaded from Kenneth K French data library. However this will not be the case for the S&P 500 benchmark. So, my second question is:   **Is it still useful to use these four factors or do I need to manually calculate them with data from the S &P 500 stock?**


title: Naive Bayes without model

text: I have the following scenario: I have two "states". I measure variables $n$ that are affected by the state. A state of 0 is the background state and in this case I expect each variable $n_i$ to be drawn from a normal distribution $N_i$~$(\mu_i,\sigma_i)$ for which I already estimated the parameters. When the state is 1, I expect the variables to have a higher value than what you would expect from the background value, but I don't have any knowledge as to what exact distribution they will follow. I would like to calculate, given an observation of the variables, how likely I am to be in state 1 relative to state 0. It seems like Naive Bayes would be the correct solution, as I expect the variables to be mostly independent. I can calculate the probabilities relevant to state 0, e.g. probability of the observation given state 0. However, how do I calculate the probability of the data given state 1 if I do not know the distribution? Another thought I had was that perhaps the approach should be different, for example calculate a z-score for each variable, and test that distribution. I guess it would be centered at zero for state 0 and higher at state 1. How would I test that? T-test? Any thoughts are welcome.


title: Interpreting significance of Cragg-Donald F-Statistic for weak instruments

text: I have a first-stage F value of 9 for a model with 1 instrument and 1 endogenous variables, the mechanical rule of thumb of 10 would say my instruments are weak. However, I am reading the 2005 paper by Stock and Yogo, who have tabulated the critical values and I don't understand the interpretation of the following in the note to the table: the desired maximal size (r) of a 5% Wald test of β = β0 Also, would Anderson CC , Cragg-Donald, or KP tests for under identification be of any use in a just identified model with only one endogenous variable so there is no matrix rank to test as such. Also could anybody suggest any other tests?


title: Determine minimum number of observations to be checked manually

text: I'm trying to classify transactions (say credit card transactions) as good or bad without looking at it manually. (The only way to figure out whether a transaction is good or bad is by manually inspecting all the details.) I've devised certain rules to tag transactions that would not require manual inspection. For e.g., say I have 100 transactions being processed everyday and based on the rules, 60 get through without manual inspection. Now out of those 60 transactions how many do I need to check manually and which test to perform to see whether this rule is working?


title: Obtaining base level with margeff & gologit2 in Stata

text: I would be extremely glad if someone could help me with this. I'm using the `gologit2` generalized logistic regression/ partial proportional odds model for ordinal dependent variables. The dependent variable describes firm size and is a categorical variable with three groups, small, medium and large. I used `margeff` to get the average marginal effect coefficients (see below), but I'm having difficulties in understanding how I can get base level that the three dummies (firm types) below are compared to (e.g. -.1368403 + base = probability that a given firm type is of y size). Is the base simply the mean of that value when y = 0 (small)? What about if I use the at(means) option in `margeff`? Cheers!               . margeff, dummies(Iprsv* \ Ipscl* \ Icountry* \ Inace*)          Average partial effects after gologit2     y  = Pr(size04)           --------------------------------------------------------------------------------       variable |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]     ---------------+----------------------------------------------------------------     Small      |     Iprsvinn_1 |  -.1368403   .0136474   -10.03   0.000    -.1635887   -.1100918     Iprsvinn_2 |  -.0130166   .0185622    -0.70   0.483    -.0493979    .0233646     Iprsvinn_3 |  -.0665578   .0193864    -3.43   0.001    -.1045544   -.0285611


title: High kurtosis, skewness and outliers

text: Currently I am working on my master this which is about excess returns (Sharpe ratio) of Asian REITs. I just transformed all the data in variables which are ready to use in SPSS. In the panel data there are some missing value due to firms which didn’t had a listning in e.g. 2002 but do in 2005. In this case I left the identifiers (cusip codes) but have a space for missing data. Before I perform a fixed effect regression (LSDV method) I explore the data for normality, skewness and kurtosis. This is where it where I get confused, because some of the variables show high kurtosis (38.024) and skewness (5.480). I did transformed the by taking the ln of all variables. QQ plots and histograms did improve but not all. But is it wise of me to even took the LN, since some of the values are negative. Taking the LN does result in an empty values. What else can I do to improve skewness and kurtosis? Square root can’t also be taken of negative numbers and if I am correct, if one variable is adjusted, all variables in the data set have to be adjusted. When analyzing for outliers I use the ‘outlier labeling rule’ suggested by Tukey et al (1986). As in line with what the paper recommends, the G value is set on 2.2. But this result in a awfull large amount of outliers. Although ouliers are not represntive, I am afraid that I have to delete (in some cases) more than 20 obeservations, while the average amount per variable is about 800. I could ease the ‘G’ value. Is this correct? Thanks in advance Henk


title: Correlation in linear regression

text: I once heard the following statements about the "correlation" in the context of regression. > correlation only measures linear relationships > > $\text{corr}(X,Y)=0$ does not mean the variables are not related! My question is why can't correlation measure non-linear relationships? Secondly, what does $\text{corr}(X,Y)=0$ tell us?


title: How to sample from the distribution of a Gaussian scale parameter

text: I would like to be able to sample the standard deviation of a multidimensional Gaussian distribution of dimension $n$; that is, given some $\phi$, I would like to sample $P(\sigma | \phi) \propto \frac{1}{\sigma^n} e^{-\frac{\phi}{2\sigma^2}}$ For high $n$, this distribution is sharply peaked; for my purposes $n$ will be on the order of 1000-3000. What is an ``efficient'' method of obtaining a single sample from this distribution? (A single sample as $\phi$ will change between each sample.)


title: Do AR models with GARCH errors have a positive spectral density?

text: Can someone please help me to verify that AR models with GARCH errors have a positive spectral density and are bounded?


title: References on deterministc augmented variable for Gibbs sampling

text: According to Wikipedia, > It is also possible to extend Gibbs sampling in various ways. ... It is ... > possible to incorporate variables that are not random variables, but whose > value is deterministically computed from other variables. What is this technique called? What are some good references on it?


title: Multiple-sample z-test for proportions

text: Let's say I want to do a multiple-sample z-test for proportions. Is there any way to rank them? How would I compare more than two samples at once?


title: Exponentially weighted moving linear regression

text: I have a problem where I need to calculate linear regression as samples come in. Is there a formula that I can use to get the exponentially weighted moving linear regression? Not sure if that's what you would call it though.


title: Does it make sense for a partial correlation to be larger than a zero-order correlation?

text: This is probably demonstrating a fundamental lack of understanding of how partial correlations work. I have 3 variables, x,y,z. When I control for z, the correlation between x and y increases over the correlation between x and y when z was not controlled for. Does this make sense? I tend to think that when one controls for the effect of a 3rd variable, the correlation should decreases. Thank you for your help!


title: Finding the column index by its name in R

text: In a data frame, I would like to get the column's index by name. For example:               x <- data.frame(foo=c('a','b','c'),bar=c(4,5,6),quux=c(4,5,6))      I want to know the column index for "bar". I came up with the following but it seems inelegant. Is there a more straightforward builtin that I am missing?               seq(1,length(names(x)))[names(x) == "bar"]     [1] 2


title: Where do data mining and computer vision converge?

text: I am traditionally a computer vision guy, and have started doing some self- learning in data mining. I'm having a hard time finding places where these 2 fields converge, as I'd like to get a self-study project that relates the two fields. where are these areas two used together?


title: Hypothesis tests on 2-dimensional panel data w/ 1 characteristic?

text: I have a 2-dimensional panel data set with one characteristic looking along these lines:               individual | year | BMI     1            2005   25.3     1            2006   25.4     1            2007   24.1     2            2005   22.9     2            2006   22.1     2            2007   21.1     ...      I also have annual values for another external variable V for the years 2005 - 2007, and I'd like to investigate whether there's a correlation between people's BMI and V. In particular, I have a hypothesis that the true correlation is > 0 that I want to test. My question is, can I test this by constructing a "time series" that pairs each BMI value with the value of V for the given year and use Pearson's r for the hypothesis test and correlation coefficients? Why or why not? If I do that, are the results at all useful? If not, what are the alternatives that are rather simple to perform? I'll clarify what I mean by constructing the time series. Assume V is 1.1, 2.2 and 3.3 for 2005 - 2007, respectively. The constructed "time series" would then be as follows (BMI values from above):               BMI  | V     25.3   1.1     25.4   2.2     24.1   3.3     22.9   1.1     22.1   2.2     21.1   3.3      A related question that I found (unfortunately it didn't give me all answers I was looking for): Are short time series worth modelling? Thanks in advance!


title: What are some useful guidelines for GBM parameters?

text: What are some useful guidelines for testing parameters (i.e. interaction depth, minchild, sample rate, etc.) using GBM? Let's say I have 70-100 features, a population of 200,000 and I intend to test interaction depth of 3 and 4. Clearly I need to do some testing to see what combination of parameters holds up best out-of-sample. Any suggestions on how to approach this test design?


title: What data from MATLAB's svmstruct are needed for classification in a different language?

text: As the title already states Iam wondering what data exactly are needed from the MATLAB `svmstruct` to be able to classify a new instance outside of MATLAB, e.g. in python or c++. I certainly need:   * the `SupportVectors` themselves (for calculating the distance using the kernel function).    * The kernel function (including parameter[s]) ported / rewritten   * The bias Now I am uncertain about `ScaleData`, `Alpha` and `SupportVectorindices`... are they needed? Do I lack some base understanding here?


title: How can one do "two series" Fourier analyses in R?

text: In Statistica one can do a "two series"/bivariate/cross spectrum Fourier analysis to examine the coherency, gain, and phase spectrum across a pair of signals. It is probably a failing on my part, but my attempts to use those search terms to come up with similar values from R have come up blank. What functions/packages should I review in more detail? Can you provide some simple examples?


title: Optimal case/control ratio in a case-control study

text: What is the optimal case/control ratio in a case-control study? Why do most textbooks or monographs suggest that it is more than 1? Can it be less than 1 (what are the drawbacks?)? Thank you.


title: Zipf-Mandelbrot distribution citation

text: I am using the Zipf-Mandelbrot distribution in my work according to its definition on wikipedia: http://en.wikipedia.org/wiki/Zipf%E2%80%93Mandelbrot_law. I would like to obtain the citation of the paper or the book that introduce it.


title: Selecting best model based on linear, quadratic and cubic fit of data

text: I have a Java code that performs a linear regression on a set of data using the Gauss-Jordan elimination. It calculates a linear, quadratic and cubic functions using the least squares method. My problem is choosing a function out of the three that best models my data. This is ofcourse, without plotting the curve. So lets say I have a set of data `{x:1,2,3,4} {y:3,8,910}`, and I get a linear, quadratic and cubic function for it using the least squares method. How do I choose which function is the right model for my data?


title: How to account for lag in a simple regression in R?

text: I'm trying to do a regression with data over time, and where I suspect there may be a lag component in the relationship between my dependent and independent variables. I've actually found some data online here that provides a good minimal example               sales<-data.frame(Quarter=1:8, Sales=c(16850, 12010, 14740, 13890, 12950, 15640, 14960, 13630), Newspaper=c(1000, 500, 2000, 1000, 1000, 500, 1000, 500), TV=c(500, 500, 500, 1000, 500, 1000, 1000, 1500), Online=c(1500, 500, 500, 1000, 500, 1000, 1000, 500))     lm(data=sales, Sales~Newspaper+TV+Online)      So the data I found is sales figures per quarter and matching advertising expenditure, but I would expect there to be a lag between advertising and purchases. How can I model that in this data ?


title: Analyzing "Reverberation"

text: I'm trying to measure the reverberation so to speak of a variable - and its effect on itself in the subsequent units of time and its effect on other variables. For example `var1` will have an event where it generates 20 units on day one. I'd like to analyze what happens to `var1` on day two, day three, and so on. But also what happens to `var 2` on day one, day two, and so on. My measurements look like this: `var1: 20,19,17....`, `var2: 15,15,16....`, What is the best way to go about this? (any helpful R packages would also be great) * * * EDIT: Here's what I mean by reverberation in my case. A TV ad is shown on Monday. I can measure the impact of that, but how do I model it over the next few days. Also, when that ad runs on Monday, presumably, there will be more internet hits as well (which I can measure). How can I model the relationship to the TV ad and how long the effect lasts. I was thinking maybe survival analysis or something along those lines. But I'm familiar with survival analysis at the wikipedia level along. * * * Thanks


title: Convergence of identically distributed normal random variables

text: I had this example in my machine learning lecture. > Let $X_2,\ldots,X_n$ be identically distributed (but not independent) copies > of $X_1$ drawn from $\mathcal N(0,1)$. Then $X_n$ converges to $Y = -X_1$ as > $n \rightarrow \infty$. There isn't any explanation for why this sequence of random variables converges to $Y$. Convergence in the sense $\lim_{n \rightarrow \infty} F_{X_n}(t)$ where $F_{X_n}$ is the CDF of $X_n$. Can anyone help me out?


title: Formulating and fitting a nonlinear model that looks like a product of linear models

text: I've got a dataset that I'd like to model, and it seems like the best model would be of the form: $y = (\beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3) \cdot (\beta_4 x_4 + \beta_5 x_5 + \beta_6 x_6) + \epsilon$. The $\beta_4$ through $\beta_6$ are independent of each other and have a linear effect on $y$ once modulated by the $\beta_1$ through $\beta_3$, which are independent of each other but scale the effect of the other predictors. I would expect the residuals of the above model to be fairly normal. If I want to fit this model in R, I can use a nonlinear model fitting routine such as `nls()`, it seems. But is there a better approach? This looks like the sort of model that might be a special case with a name and an optimized fitting approach. Is it?


title: A measure to describe the distribution of a dendrogram

text: Could anyone suggest some statistical measures to describe the distribution of a dendrogram? If I have two dendrograms, how could can I quantify their structural differences?


title: "Multiple response" analysis of arrest records

text: **Background:** I'm a junior researcher at an institute dealing with regional issues, particularly involving drug policy. Almost two years ago, one of our senior researchers began collecting arrest data about a nearby large city. He had been transcribing newspaper police blotter by hand until I joined a year and change ago and convinced him to switch to automated data collection. We're now ready to start analyzing this dataset. It contains about 20k arrest records with virtually no missing or invalid values (I'd estimate that there are less than 100 such records, and some of those are just parser errors). These records include full name and home address of the arrested person, arresting officers, exact charges, etc. (we've even found SSNs in the data). At this point, we're just exploring the data, but we expect to find differences in number and severity of charges according to race, arrest location, and home location. **Question** The dataset has two variables that, to my understanding, are categorical multiple-response questions. "Arresting Officer" is two columns, one of which always has a value and one of which may have an a value (for the second officer present at the arrest). "Charges" is five columns, each containing a single charge and in no particular order. To complicate the structure further, a person arrested on more than five charges will be issued multiple records. We messed around with the above dataset for a few hours so far and have gotten as far as being able to obtain useful counts and percentages in SPSS by using the Multiple Response Set feature and plugging that into Custom Tables. This is a fine start, but we'd like to move on to more detailed analysis at some point. Unfortunately, neither of us are aware of the recommended (or anti- recommended) methods for analyzing a categorical multiple-response question. This also applies to our desire to eventually group specific charges (e.g., "possession of controlled substances < 4 oz.") into broader categories such as 'violent crimes' or 'drug crimes'. Note that I'm not looking for just a snap answer here, as we don't have a deadline. I would be glad to do readings, so feel free to point me in the direction of tutorials, textbooks, and so on. I'm also not particularly attached to SPSS - it's just what my coworker is used to. If there are clear disadvantages to using it for this type of problem, I don't mind learning something new.


title: Application of Bayes rule - relation between a pair of words

text: I do not have a statistical background but I'm trying to understand bayes theorem using a practical example. In my example I plan on using bayes theorem to calculate how often two words occur next to each other. I essentially have some large text and I plan to use the formula before: Likelihood "this" is followed by "is" = (Probability the word "this" is followed by "is") * (Probability of the word "this") / (Probability the word "is") And I would calculate the probability of a word as follows: P(word) = (# number of times "word" occurs) / (total # of words) And lastly, after I calculate this on my initial dataset, I plan on updating my likelihood as follows: Likelihood = (old Likelihood) * (new likelihood) Does this sound correct? I would really appreciate any feedback.


title: Is there any software to draw area-proportional Venn diagrams when a∩b∩c= 0?

text: I would like to draw area-proportional Venn diagrams with 3 sets of data with any free online software. I tried with eularAPE software but it doesn't work because my 3-set of data do not overlap. My data are: a =45, b=20, c=12, a∩b =9, a∩c =6, b∩c= 1, a∩b∩c= 0. Is it possible to generate proportional Venn diagrams when a∩b∩c= 0?


title: Recurrent vs Hopfield neural networks

text: What is the difference between recurrent neural networks and Hopfield networks, or are they same?


title: How to combine covariate-adjusted and unadjusted standardised mean differences in meta-analysis?

text: I have completed an meta-analysis of final scores (comparison of means at post test); however, i have noticed that despite being randomised, differences at baseline are present and are biasing the findings. Can I combine co-variate adjusted means (where reported) and final scores means in one meta-synthesis?


title: standardising sub-data sets, such that the whole dataset is standardised

text: Consider a data set $X$ made up of smaller subsets: $X=A \cup B \cup C$, with $A,B,C$ disjoint data sets. Eg: $A=\{1.0, -1.0, 0\}$, $B=\{5.0, -7.0, 2.0\}$, $C=\{1.5, -5.0, 8.0\}$ Is is possible to transform (scale, shift) $A,B,C$ such that each is individually standardised to have mean=0, and std dev=1.0, and so that $X$ also has mean=0, std dev=1.0?


title: Comparing 6 week mean to current weeks actual

text: I am trying to catch changes in customers sales from week to week using R.               Historical.stats <- ddply(Historical, .(Company,Customer), summarise, mean= mean(Sales), lowerlimit = quantile(Sales,.05) ,higherlimit = quantile(Sales,.95), pval=t.test(Sales,data=Historical)$p.value)      My first attempt at doing this was to use the Quantile and then merge by Current Week data by the Company and Customer. I then would check if the sales this week were above or below the lower and higher quantiles.               dat <- merge(CurrentWeek, Historical.stats, by = c("Company","Customer"))     dat$plot <- ifelse((dat$Sales< dat$lowerlimit | dat$Sales> dat$higherlimit) ,1 ,0)      After doing more research would it be best to use something like the Levine test, a standard t.test or some type of ANOVA method to do this type of anomaly detection. I understand I won't be able to take into account seasonality or holidays.               Example Data     Company   Customer   Week   Sales     Company A Customer 1 05/07/2014 100     Company A Customer 2 05/07/2014 50     Company B Customer 1 05/07/2014 200     Company B Customer 2 05/07/2014 150     Company A Customer 1 05/14/2014 20     Company A Customer 2 05/14/2014 20     Company B Customer 1 05/14/2014 100     Company B Customer 2 05/14/2014 50


title: How to assess information content / mutual information between two sets of variables ?

text: Am testing data from a clinical device, data was extracted using two complementary methods               Method 1: Existing method, 17 variables      Method 2: New method, 13 variables       I need to quantitatively and visually explain that the **information content / mutual information** of both methods are the same. I have correlation data and **some of the variables are correlated and some are not**. I am looking for an intuitive way to explain that irrespective of differences in number of variables between the data extraction methods and direction of correlation the information captured by methods are comparable. Please let me know your thoughts, thanks in advance !


title: $\chi^2$ fitting with correlated errors

text: I have been doing some fitting using the fairly standard expression, $$\chi^2 = \sum_i \frac{(y_{i} - F(x_{i}, \theta))^2}{\sigma_{i}^{2}}$$ where y is my measured data, $\sigma$ is the experimental uncertainty and $\theta$ is the parameter I'm estimating. I know that in reality my data are not independent and that I can have correlated errors. Is there a standard way of developing a more detailed model taking into account errors and correlated errors? I have found some hints from high energy physics such as http://www.desy.de/~blobel/banff.pdf .


title: Shape of distribution of histogram

text: If we have the following histogram: ![enter image description here](http://i.stack.imgur.com/HxSwZ.jpg) And, want to describe it based on one of the following options, which one do you think is correct? Why?   1. positively skewed    2. symmetric    3. none of the above    4. negatively skewed My initial suggestion would be "3. none of the above". What do you think? Thanks.


title: Model selection and cross-validation: The right way

text: There are numerous threads in CrossValidated on the topic of model selection and cross validation. Here are a few:   * Internal vs external cross-validation and model selection   * @DikranMarsupial's top answer to Feature selection and cross-validation However, the answers to those threads are fairly generic and mostly highlight the issues with particular approaches to cross validation and model selection. To make things as **concrete as possible** , say for example that we are working with an SVM with an RBF kernel: $K(x, x' ) = (\gamma \, \vert x - x'\vert)^2$, and that I have a dataset of features **X** and labels **y** , and that I want to   1. **Find the best possible values of my model ($\gamma$ and $C$** )   2. **Train the SVM with my dataset (for final deployment)**   3. **Estimate the generalization error and the uncertainty (variance) around this error** To do so, I would personally do a grid search, e.g. I try every possible combination of $C$ and $\gamma$. For simplicity, we can assume the following ranges:   * $C \in \{10, 100, 1000\}$   * $\gamma \in \{0.1, 0.2, 0.5, 1.0\}$ More specifically, using my full dataset I do the following:   1. For every ($C$,$\gamma$) pair, I do repeated iterations (e.g. 100 random repetitions) of $K$-fold cross validation (e.g. $K=10$), on my dataset, i.e. I train my SVM on $K-1$ folds and evaluate the error on the fold left, iterating through all $K$ folds. Overall, I collect 100 x 10 = 1000 test errors.   2. For each such ($C$,$\gamma$) pair, I compute the mean and the variance of those 1000 test errors $\mu_M, \sigma_M$. Now I want to choose the best model (the best kernel parameters) that I would use to train my final SVM on the full dataset. My understanding is that choosing the model that had the **lowest error mean and variance** $\mu_M$ and $\sigma_M$ would be the right choice, and that this model's $\mu_M$ are $\sigma_M$ are my best estimates of the model's generalization error bias and variance when training with the full dataset. BUT, after reading the answers in the threads above, I am getting the impression that this method for choosing the best SVM for deployment and/or for estimating its error (generalization performance), is flawed, and that there are better ways of choosing the best SVM and reporting its error. If so, what are they? I am looking for a concrete answer please. Sticking to this problem, how specifically can I choose the **best model** and **properly estimate its generalization error**?


title: Why test for both correlation and groups differences using the same variable?

text: A paper asks four research questions which deal with the measurement of latent trait _x_ , a continuous variable which is acquired through learning or practice, and the speed with which the knowledge or skill can be accessed and used in a large group of participants ($n=500$). The first two research questions are:   1. Is there a difference in response times between groups with different _trait x_?   2. What is the correlation between response times and _trait x_? Subjects are assigned to the groups referenced in RQ1 _post hoc_ based on their performance on the _trait x_ test. It seems to me that if there is a correlation between response times and _trait x_ , then one could _always_ find a grouping such that there will be a statistically significant difference between the groups. So, my questions are:   1. Can anyone explain to me why both questions need to be asked?    2. Is there some fundamental statistical fallacy going on here or am I being overly simplistic in my thinking? ### edit The trait could be any psychometrically measured latent trait, but I have often seen these two questions appearing simultaneously in a variety of papers where something has been acquired through learning or practice such as in education or applied linguistics. In this example, it could be something trivial such as the ability to correctly pick the name of a famous person in a picture from a list of possible names. From what I understand, one of the main problems with the first question is the arbitrary discretisation of a continuous variable (trait _x_ ) which results in information loss and possible bias through the arbitrary bins/thresholds which are created. Is this correct? Are there other problems happening here? ### real life example I've been hesitant to give an _actual_ example because then replies usually focus on the specifics whereas I was trying to generalise, but here is one. There are two research questions:   1. What is the size of the vocabulary of university juniors and how good is their reading comprehension?   2. Do university students’ vocabulary knowledge and content knowledge influence their reading comprehension? The researcher administers test of reading comprehension, vocabulary size, and content knowledge. Ignoring the double-barrelled RQ1, so far, so good. But here is the part I can't understand (which is similar to my more abstract question above): The analysis includes: a correlation analysis               Variables               Vocabulary knowledge Content knowledge         ---------------------   -------------------- -----------------     Reading comprehension   .70**                .41**                     Vocabulary knowledge                         .22**                          **p < .01 (two-tailed)        a comparison of groups based on vocabulary size test performance                                  Reading comprehension      Content knowledge                        Mean   SD    t        df   Mean   SD    t     df     -----------------  -------------------------  ----------------------     Above (n = 83)     20.34  6.18  12.25**  244  30.10  4.31  2.06  192     Below (n = 163)    11.23  5.15                28.82  5.12          **p ≦ .01                                  and a multiple regression               Model        Sum of squares  df    Mean square  F            ----------   --------------  ----  -----------  --------     Regression   6719.39         2     154.91       154.91**     Residual     5270.35         243   21.67             Total        11989.74        245           Note. R2 = .56     **p < .01          I simply can not figure out how the second test (a _t_ test) is even justified because the groups are created from a variable which has already been shown to correlate with the DVs in the _t_ tests. Isn't that a foregone conclusion? _note:_ I realise that the study is not particularly good and that given a multiple regression, both the _t_ test and the correlation are not even relevant here, but this is simply an _example_ of the phenomenon I'm asking about in the question. That is, if a correlation has already been established between two variables, does it make sense to then arbitrarily bin one of the variables into groups and test for a difference between groups. I see variations on this theme quite often and I can't figure out why it's justified.


title: Univariate clustering for longitudinal cohort

text: We have screening information on thousands of patients followed for several years. We also have their cancer outcomes, whether or not such cancers were identified by screening or were otherwise symptomatic. Exogenous to that question is the question of how people screen. It's a tough question, in general, since there are several factors determining how/why/when people screen. Relevant to that is the idea of predicting _when_ a given individual will screen conditional upon their screening history. Usually, rates are the only measure used to characterize such patterns. That is done by calculating a total risk time denominator. But it may be more complex than that. As we know, processes which determine interarrival times can be quite complex indeed. Whether they are dependent, or approximately Markov is important. Additionally, we would like to know what the shape of the density of interarrival times is and whether there are common distinguishing characteristics, such as bimodality in those data. However a parametric approach can be quite limited indeed. And structurally testing assumptions while building sophisticated enough models to determine differences seems like it would greatly increase the type 1 error. Is there a general Markov model or unsupervised learning approach that can be used to identify clusters of times series that are distinct from one another in a more general fashion than rates?


title: Robust estimator of mean for skewed data

text: For heavy-tailed symmetric data, a trimmed mean or other robust estimator of the mean could be a better estimator of the mean than the sample mean. The trimmed mean will be biased for a skewed distribution. What is a robust estimator of the mean that is not biased for a skewed distribution?


title: ANOVA analysis with only mean and standard deviation

text: This is probably a silly question, but how can I run ANOVA using only the mean, standard deviation, and the sample size? i.e.               Group  mean    sd    n      1:    200      5    5        2:    100     22    5      3:    150     14    5      I am using Statsoft Statistica software. I know how to run it using the spreadsheet with all the data, but I have never done this using only the information listed above. Any information would be greatly appreciated.


title: graphical methods / deep architectures for collaborative filtering

text: Having read "Restricted Boltzmann Machines for Collaborative Filtering" (Salakhutdinov et. al. 2007), I'm wondering if there has been any follow-up work on applying graphical and/or deep architectures for recommendation engines. The paper suggests some interesting possible extensions, such as using stacked RBMs instead of single-layer ones. However, I didn't find any follow-up papers by Salakhutdinov's publication page.


title: What do I take as gamma's argument?

text: I have collected wind speed data for over a year. I want to calculate the mean wind speed using the formula in which the mean speed is the function of the two weibull parameters and the gamma function. How do I evaluate the gamma function in this instant? what do I take as the argument?


title: How can I perform a meta-analysis on variables which are not normally distributed?

text: I am working on a project to estimate a radiation protection value called DDREF which is the ratio of two regression coefficients. Specifically: $DDREF$ $=$ $a/B$ where $E$ ~ $a*D+B*D^2$ $D$ is dose and $E$ is some outcome, like survival. $a$ and $B$ are coefficients determined by regression. I have independent estimates of DDREF from many studies and I would like to combine them into a single estimate. But I'm not sure how to do this because the ratio of two regression coefficients is not normally distributed. Does anyone know how to perform a meta analysis on variables that are ratio distributed? Or perhaps its better to ask the more general question how do I apply a meta-analysis to produce a central estimate from variables which are not normally distributed?


title: Softmax regression bias and prior probabilities for unequal classes

text: I'm using Softmax regression for a multi-class classification problem. I don't have equal prior probabilities for each of the classes. I know from Logistic Regression (softmax regression with 2 classes) that the prior probabilities of the classes is implicitly added to the bias ($\log(p_0/p_1)$). Usually what I do is to manually remove this term from the bias. My question is, what is the corresponding term in softmax regression bias? Thanks.


title: coin toss p value

text: When you toss a coin you get 8 heads and 2 tails. How would you analyze whether a coin is fair? What is the p-value? The p-value is easy to calculate and it is 56/1024. However I couldn't solve the second part. In addition, more coins are added to this experiment. Now you have 10 coins. You toss each coin 10 times (100 tosses in total) and observe results. Would you modify your approach to the the way you test the fairness of coins?


title: PCA with same cases in different periods of time

text: I am new user of principal component analysis (PCA) and I have a big doubt. I have 32 observations with 45 variables and I know that I cannot use the simple PCA for this analysis (n < p). However, I want to use this 32 cases (individuals) for one year (e.g. 2008) and then I would like to create a table with the same 32 cases but for 2009, 2010, 2011 and 2012. So, is it possible to make a good analysis of PCA doing that? and also, can I improve or remake the model every year only adding the new 32 cases for each year? or what can I do?


title: Analysis of spatial data over time and space

text: I have a data set having year-wise monthly average of minimum and maximum temperatures of 32 stations around the country since 1948. The latitude and longitude of the stations are given as well. I have been asked to dig deep into the data set and analyze the much I can. The main objective is to understand the trend and possibly predict the future values. I have little experience to work with such data sets. I usually like to use R. I have three questions in particular. 1) I can see that the latitude and longitude of an station are given to be 25.72 and 89.26 respectively. The corresponding Y and X coordinates are calculated as 844822.3 and 426172.9. I don't understand how they have calculated this. Can anyone explain? 2) What model should I fit to the data to get a good prediction of future values? Since I have the data since 1948, I might use a time series model, but here I have the latitude and longitude information too. So is there any model that can use these information too? Is there any particular package in R that will be useful for me? 3) Can I use R to find interesting plots mapping my findings or different summary statistics in map of the country? Some 2D or 3D plots will be excellent for me. Many thanks for your kind response in advance.


title: Box-Ljung degrees of freedom for residuals of advanced GARCH-like model

text: Diagnostic checks of ARMA-GARCH residuals via Box-Ljung tests. I am not sure if I use the correct degrees of freedom. An arbitrary example: ARMA(0,1)-eGARCH(2,1)-SkewedStudent's-t. Output:                       Estimate  Std. Error    t value Pr(>|t|)     mu      0.000418    0.000115    3.62471 0.000289     ma1     0.013034    0.009898    1.31689 0.187877     omega  -0.155739    0.007117  -21.88184 0.000000     alpha1 -0.177460    0.017078  -10.39099 0.000000     alpha2  0.110598    0.016937    6.52996 0.000000     beta1   0.982730    0.000796 1233.83974 0.000000     gamma1  0.018616    0.025403    0.73284 0.463655     gamma2  0.153550    0.026689    5.75323 0.000000     skew    0.918887    0.014489   63.41908 0.000000     shape   9.399434    0.795150   11.82095 0.000000      Assuming that I keep all parameters in the model (also insignificant ones) I would test for the autocorrelation in the standardized residuals for some lags (e.c. 5,10) using a Box-Ljung test with "lag-1" degrees of freedom (because of only a single MA-lag). I would further test for autocorrelation in the squared standardized residuals for some lags (e.c. 5,10) using a Box-Ljung test with "lag-3" degrees of freedom (2 ARCH-lags and 1 GARCH-lag which sums up to 3). Are the df used the correct ones? Or do I have to consider all the 5 parameters of the eGARCH model and the 2 parameters for the distribution? If i fix some insignificant parameters to be zero, does this affect the df? Only recommendations for simple cases such as GARCH(1,1) with normal innovations have been found in the literature. Many thanks!!!


title: How to deal with survey question with multiple response?

text: I have a dataset asking people whether they have been to a certain places (e.g. A, B, C, D), and they can make more than one choice, then a specimen is taken from their nose to see if they are infected with some disease. I need to find out the relative risk of getting infected for one going to a certain place, I can only think of logistic regression right now, is there any other suggestions? Thanks.


title: Inconsistency between Chi-sq and CI Estimation using Wald test

text: I am trying to calculate the relative risk for being tested as positive for people aged >25 or <=25, and here is the result.               $data                  Negative Positive Total       >25        115       11   126       <=25       117        3   120       Total      232       14   246          $measure           risk ratio with 95% C.I.             estimate      lower    upper       >25  1.0000000         NA       NA       <=25 0.2863636 0.08189025 1.001391          $p.value           two-sided            midp.exact fisher.exact chi.square       >25          NA           NA         NA       <=25 0.03815237   0.05147711 0.03500415          $correction     [1] FALSE          attr(,"method")     [1] "Unconditional MLE & normal approximation (Wald) CI"      One thing that puzzled me a lot is that why I get a statistically significant result from chi.square (I don't use fisher's exact test because NONE OF THE expected value is small than 5), but the risk ratio includes 1, how can I explain? Thanks. ### CORRECTION Sorry for the mistake, I rechecked the data and NONE of the expected cell is smaller than 5


title: How to calculate user-category affinities, given user-item scores for categorized items?

text: I have 2 million ( _user_ , _item_ , _category_ , _score_ ) tuples, where users give 0/1 scores to items that belong to categories. I want to convert these to a set of ( _user_ , _category_ , _affinity_ ) tuples, using the items linking a user and a category to calculate that affinity. Two possibilities:   * Proportion of positive scores to total scores: $a_{uc} = \frac{\sum{scores}}{|scores|}$   * Bayesian average: $b_{uc} = \frac{C\,\bar{a}_{uc}\,+\,\sum{scores}}{C\,+\,|scores|}$, where $C$ is the mean or median number of item scores per ( _user_ , _category_ ) pair. I'm leaning towards Bayesian average, since it deals better with sparse data -- a single positive vote shouldn't result in a higher affinity than nine positive and one negative -- but I could use some published research on this problem to justify Bayesian average specifically, or to find other possible measurements. Are there any terms I should be looking for that deal with this problem?


title: Correlation between two variables of unequal size

text: In a problem I am working on, I have two random variables, X and Y. I need to figure out how closely correlated the two of them are, but they are of different dimensions. The rank of the row space of X is 4350, and the rank of the row space of Y is substantially larger, in the tens of thousands. Both X and Y have the same number of columns. I need a measure of correlation between the two variables, and Pearson's r requires X and Y to have equal dimension (at least R requires the two r.v.'s to be). Do I have any hope of doing a correlation between these two, or should I find some way of pruning off observations from Y?


title: SPSS mediational analysis help

text: I'm doing a psychology project and I'm struggling with statistics (SPSS) especially on meditational analysis. I have searched all over my notes from my bachelor degree but there is nothing on meditational analyses whatsoever. There were hardly any steps shown, only results are provided in all the journals I read from web of science. I’m also confused about inputting variables (there are loads of variables). I sought help from my department but the help given will be taken into consideration when we are graded.


title: Parameters estimation of non-stationary random process using different runs

text: Let $X(t)$ be a non-stationary continuous time-dependent random process with a known model but unknown parameters. I'd like to know if it's possible to estimate the parameters of $X(t)$ not by using different observations of the same run at different instants $t$, but rather by observing different runs at different instants $t$. Is it as straightforward? It doesn't seem too obvious to me. Is there any property I can invoke? The random process in question is the time integral of a Ornstein-Uhlenbeck process, if that matters.


title: KLDIV Kullback-Leibler or Jensen-Shannon divergence between two distributions

text: let consider following probability distributions               X   P                Q           Kullback                      Kullback_divergence     1   0.526810511     0.6557  -0.166341349                        11.97922287     2   0.213080233     0.0357  0.54919316           3   0.02600177     0.8491   -0.130769473             4   0.088595233     0.934   -0.301057372             5   0.546739097    0.6787   -0.170539661             6   0.28479873     0.7577   -0.40204582          7   0.082491531     0.7431  -0.261600315             8   0.106326487   0.3922    -0.200222185             9   0.92608417    0.6555    0.461696987          10  0.866939299   0.1712    2.028853203          11  0.122989593     0.706   -0.310072991             12  0.782494583   0.0318    3.615893457          13  0.030182806     0.2769  -0.09651152          14  0.580645161   0.0462    2.120337307          15  0.632129887     0.0971  1.708443428          16  0.118564409     0.8235  -0.331517665             17  0.98791467     0.6948   0.501651889          18  0.139591662     0.3171  -0.165238155             19  0.668691061   0.9502    -0.338953731             20  0.839289529   0.0344    3.868023675           i would like to know what this number shows us?does this express information about second distribution?or what is distance between two distribution,as i know if it far from $1$,then it means that they are really different,but which is good and real?please i need your advice's


title: Moment generating function if the PDF is $f_z= Cz^{k-1}(1-\frac{z}{d})^bF(-a+k+1,b;b+1;1-\frac{z}{d})$

text: Let $z$ a random variable with PDF : $f_z= Cz^{k-1}(1-\frac{z}{d})^bF(-a+k+1,b;b+1;1-\frac{z}{d})$, where $0\leq z \leq d$, $F$ is the Hypergeometric function, $k$ is a positive integer, $-a+k+1 >0$, $C$ is a constant and $a ,b$ are 2 real positive numbers. Please I need to compute a closed form of the Moment generating function of $z$.


title: Jeffries Matusita distance for 14 variables

text: I wish to perform Jeffries-Matusita distance on 14 spectral bands. Is there anyone who can help with how it is done in R? Thank you.


title: The 'best' model selected with AICc have lower $R^2$ -square than the full/global model

text: I have used the R lme function (nlme package) to construct linear mixed models, with a single random effect (as a random intercept) and a varIdent variance structure on a fixed effect (that is a factor). The data is from a commercial fishery and I am looking to see which are the most important variables that need to be reported for this fishery. I have 3 biological variables (total length, weight, proportion of females) and two operational variables (effort and season) (unfortunately I can't post my data as it's confidential and I haven't managed to reproduce my problem with dummy data). I have used the AICc to select the 'best' models from a list of a priori models and I am now calculating the R-squared values from each of the 'best' models (using the code from https://github.com/jslefche/rsquared.glmer/blob/master/rsquaredglmm.R from the blog post http://jonlefcheck.net/2013/03/13/r2-for-linear-mixed-effects- models/ and taken from Nakagawa and Schielzeth (2013) r2 for GLMMs). I have 6 models with a AICc < 10 and I've used that as my cut off as it gives me a balanced number of models to the calculate the relative importance of each variable of interest using the sum Akaike's weights (the purpose of the modelling). However, the R-squared values are lower for my 'best' models (Akaike weights between 0.5-0.8; marginal $R^2$ = 0.48) than the model with all my fixed effects (Akaike weights 0.00; marginal $R2^$ = 0.89). I've re-run the model selection using BIC and I get the same problem. My question is: Are most 'best' models the models that have a lower $R^2$ value (and lower AICc) or have higher R2 values but higher AICc values?


title: Kolmogorov-Smirnov of two heart rate samples

text: I'm accumulating a lot of vital signs from emergency department patients, and would like to assess if different features yield significantly different distributions. Eg. for heart rate I have 888.424 measurements (stored in all_hr) for all patients and 321.357 measurements for geriatric patients (stored in g_hr). Summary stats yield:               summary(all_hr$hr)      Min. 1st Qu.  Median    Mean 3rd Qu.    Max.      2.00   74.00   87.00   88.56  101.00  242.00           summary(g_hr$hr)     Min. 1st Qu.  Median    Mean 3rd Qu.    Max.      2.00   72.00   85.00   87.77  100.00  207.00       When plotting the ecdfs for these samples I get: ![ecdfs for all patients and geriatric patients](http://i.stack.imgur.com/RSwC9.png) I'm trying to comprehend how to use the Kolgomorov-Smirnov ks.test() function in R for this:               ks.test(all_hr$hr,g_hr$hr,alternative="two.sided")          Two-sample Kolmogorov-Smirnov test          data:  all_hr$hr and g_hr$hr      D = 0.0289, p-value < 2.2e-16     alternative hypothesis: two-sided           Warning:     In ks.test(all_hr$hr, g_hr$hr, alternative = "two.sided") :     p-values will be approximate in the presence of ties      So as far as I understand this test, the hypothesis that samples are from different distributions is true? Now, I've also attempted a similar approach, but grouping patients by speciality ward instead. E.g. here is the ecdfs two randomly drawn 6.000 sample distributions for neurological and gastro-internal patients: ![enter image description here](http://i.stack.imgur.com/PqV3k.png) Yet, when running ks.test on these two samples, I also get the same result               ks.test(N.vs.hr.rs$hr,S.vs.hr.rs$hr,alternative="two.sided")          Two-sample Kolmogorov-Smirnov test          data:  N.vs.hr.rs$hr and S.vs.hr.rs$hr      D = 0.6307, p-value < 2.2e-16     alternative hypothesis: two-sided           Warninf:     In ks.test(N.vs.hr.rs$hr, S.vs.hr.rs$hr, alternative = "two.sided") :     p-values will be approximate in the presence of ties      So, in summary...is this test a valid way of asserting differences in distributions? And should I have no quarrels with the fact that two very dissimilar ecdf plots yield the same ks.test results?