title: Improving the SVM classification of diabetes

text: I am using SVM to predict diabetes. I am using the BRFSS data set for this purpose. The data set has the dimensions of $432607 \times 136$ and is skewed. The percentage of `Y`s in the target variable is $11\%$ while the `N`s constitute the remaining $89\%$. I am using only `15` out of `136` independent variables from the data set. One of the reasons for reducing the data set was to have more training samples when rows containing `NA`s are omitted. These `15` variables were selected after running statistical methods such as random trees, logistic regression and finding out which variables are significant from the resulting models. For example, after running logistic regression we used `p-value` to order the most significant variables. Is my method of doing variable selection correct? Any suggestions to is greatly welcome. The following is my `R` implementation.               library(e1071) # Support Vector Machines          #--------------------------------------------------------------------     # read brfss file (huge 135 MB file)     #--------------------------------------------------------------------     y <- read.csv("http://www.hofroe.net/stat579/brfss%2009/brfss-2009-clean.csv")     indicator <- c("DIABETE2", "GENHLTH", "PERSDOC2", "SEX", "FLUSHOT3", "PNEUVAC3",          "X_RFHYPE5", "X_RFCHOL", "RACE2", "X_SMOKER3", "X_AGE_G", "X_BMI4CAT",          "X_INCOMG", "X_RFDRHV3", "X_RFDRHV3", "X_STATE");     target <- "DIABETE2";     diabetes <- y[, indicator];          #--------------------------------------------------------------------     # recode DIABETE2     #--------------------------------------------------------------------     x <- diabetes$DIABETE2;     x[x > 1]  <- 'N';     x[x != 'N']  <- 'Y';     diabetes$DIABETE2 <- x;      rm(x);          #--------------------------------------------------------------------     # remove NA     #--------------------------------------------------------------------     x <- na.omit(diabetes);     diabetes <- x;     rm(x);          #--------------------------------------------------------------------     # reproducible research      #--------------------------------------------------------------------     set.seed(1612);     nsamples <- 1000;      sample.diabetes <- diabetes[sample(nrow(diabetes), nsamples), ];           #--------------------------------------------------------------------     # split the dataset into training and test     #--------------------------------------------------------------------     ratio <- 0.7;     train.samples <- ratio*nsamples;     train.rows <- c(sample(nrow(sample.diabetes), trunc(train.samples)));          train.set  <- sample.diabetes[train.rows, ];     test.set   <- sample.diabetes[-train.rows, ];          train.result <- train.set[ , which(names(train.set) == target)];     test.result  <- test.set[ , which(names(test.set) == target)];          #--------------------------------------------------------------------     # SVM      #--------------------------------------------------------------------     formula <- as.formula(factor(DIABETE2) ~ . );     svm.tune <- tune.svm(formula, data = train.set,          gamma = 10^(-3:0), cost = 10^(-1:1));     svm.model <- svm(formula, data = train.set,          kernel = "linear",          gamma = svm.tune$best.parameters$gamma,          cost  = svm.tune$best.parameters$cost);          #--------------------------------------------------------------------     # Confusion matrix     #--------------------------------------------------------------------     train.pred <- predict(svm.model, train.set);     test.pred  <- predict(svm.model, test.set);     svm.table <- table(pred = test.pred, true = test.result);     print(svm.table);      I ran with $1000$ (training = $700$ and test = $300$) samples since it is faster in my laptop. The confusion matrix for the test data ($300$ samples) I get is quite bad.                   true     pred   N   Y        N 262  38        Y   0   0      I need to improve my prediction for the `Y` class. In fact, I need to be as accurate as possible with `Y` even if I perform poorly with `N`. Any suggestions to improve the accuracy of classification would be greatly appreciated.


title: How to read RMSE and MAD in a stock prediction model

text: I am working on a stock prediction model which predicts next day Open High Low and Close stock prices, my question is how to use RMSE and MAD to measure the error or accuracy for the entire model say 500 stocks? Or to clarify is the concept of RMSE and MAD useful only when looking at a single stock prediction error ? Does it make sense to average all RMSE or MAD for all 500 stocks ? Thanks, Deepesh


title: Correlated Bernoulli random variables

text: I have about $50$ Bernoulli random variables $X_i$ whose joint distribution is unknown, but I can generate a sample of size on the order of $10^4$. They are not independent, but I think the dependence is rather weak. I would like to get some idea of ${\rm Pr}({\rm all}\ X_i = 1)$, which I expect to be considerably smaller than $10^{-4}$, maybe around $10^{-6}$ (so it's unlikely that the sample will contain outcomes with all $X_i = 1$). If I assumed they were independent, I could estimate their means and multiply these estimates. What can I do that takes correlations into account?


title: What's the best way to measure accuracy/significance of a Stock Prediction Model?

text: I'm looking for insights on how to test the accuracy of a model I built to predict next days stock price - Open, High, Low and Close next day prices. Is there a unique indicator of accuracy or standard approach that should be used to represent the accuracy of this system ? This is a predictive model for prices, not a trading system. Input to the model is the historic time series - end of day stock prices-Open, High, Low, Closing. I've run some initial stats by calculating the ratio of predicted prices to the actual end of day prices i.e if PC represents predicted close price for next day I am simply taking the ratio of PC / C where C is the actual close price for the predicted day and representing it as a %. Have researched some forecasting system models online and understand that the common approach is to use Mean Absolute Percent Error, Mean Absolute Deviation, R-Squared, Correlation, Cumulative Forecast Error. As an example my prediction model is giving me values for Correlation and R-Squared 0.995 0.9924 0.9912 0.9885 & 0.9905 0.983 0.981 0.976 respectively for the O H L C predictions..would these be considered good values ? Any suggestions on what would be the best approach?


title: Interpolation problem

text: We have the following solar meteorology parameters through satellite data with resolution of 100 km$^2$ (10 km x 10 km). Insolation on horizontal surface Diffuse radiation on horizontal surface Direct normal radiation Like this we can have about 4-5 more parameters which are associated with Solar Power generation. These same parameters are also measured at 23 ground stations. We also have same parameter data which are taken at ground station at respective long-latitude locations. Our objective will be to estimate all those solar parameters for every 1 km x 1 km area. By interpolating satellite data and ground station data in given vicinity area with best possible accuracy. My question is Which is best interpolation technique we can use: please provide some references and the software which can be used.


title: determining probability of spatial pattern metric greater than some instance

text: Let's say I have a grid of cells from a satellite image that take on the value of 0 or 1, with 1 being forest. There are many spatial pattern metrics that measure things about this pattern such as some dimension of forest fragmentation. A common one is the join-count statistic which is simple enough that its expectation and variance are calculable for a random allocation process. This allows one to estimate the probability of a join-count with an equal or greater value than that observed, given a random process. One only needs to know the relationship among grid cells (are they spatially connected/neighbors or not) and the probability of any cells taking on a specific value (e.g. the probability that a cell is forest and probability that it is not). More complex spatial patterns do not typically have a standardized version. For instance, the perimeter of all forest patches can be divided by the surface area of all patches to calculate a perimeter-area ratio. The farther from one large circle (the minimum) then the more fragmented. It would be easy to calculate the probability of any given spatial pattern (they would all be the same, I believe, if they contain the same number of forested/non-forested cells), however I am interested in knowing the expected perimeter-area ratio and variance so that measures such as this can be standardized. Are there any texts someone can suggest? I have read quite extensively within landscape ecology and related fields, but I do not believe this has been addressed, so I need a math text or even area of study to research of which I am not currently aware. A simple approach would be Monte Carlo simulation, but this would be very computationally intensive for even moderately large collections of grid cells. Further, the patterns observed are typically quite rare in comparison to a random process and so you could allow the simulation to run quite a while and I believe never fill in the tails of the distribution created by it so as to get a good handle on the probability of observing a pattern with the same value or greater. Any guidance related to obtaining my objective is appreciated. The primary motivation of this is not to test "straw man" null hypotheses of random allocation but instead to be able to compare the patterns from two areas while removing some of the confounding due to differences in the number of cells allocated to each category.


title: Bayesian alternative or complement to the Student t-test

text: I was just recently using the Student t-Test to check whether values from two samples could have an identical mean or not. I was wondering whether there is a complementary technique in bayesian statistics that does a similar thing. Especially I am wondering about the $P$ values that I obtain as a result (http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html). With a sample size of 2 in my two samples, I figure that the resulting p value would probably fluctuate quite a bit if I added some more measurements or take some away.


title: Divison of multi class into binary class using libsvm (not classification)

text: I am working on multi class imbalance data.I want to divide my classes based on one verus one method and then perform sampling techinque How can I do that I just want to divide the classes not classification. Is it possible in libsvm


title: About sigma algebra (proof of Borel field)

text: Suppose that the sample space S of some experiment is finite. Show that the collection of all subsets of S satisfies the three conditions required to be called the collection of events:   1. $S \in \beta $   2. $ if \; A \in \beta, \; A^\complement\in\beta$   3. $if A_{1},A_{2},\cdots,A_{n}\in\beta,\; A_{1}\cup A_{2}\cdots\cup A_{n}\in\beta$ How can I proove axioms above? I.e., how can I show that the collection of all subsets of S satisfies the three conditions required to be called the collection of events. A detailed explanation with Borel field and its full proof will do help me improve my statistical insight and skills.


title: How to quantify correlation stability?

text: In many financial models we are interested in measuring the correlation between variables, returns etc. However, research shows that during crises times we observe "Correlation Breaks" where previously un-correlated variables become correlated. What is the best way to quantify "stability of correlation" by examining the historic time series of the variables on which I am interested? Would some kind of bootstrapping be a good way to start examining the sample distribution of the correlation? Is there any other method that I could apply?


title: Can I use "left eye" and "right eye" in my sample as two different subjects?

text: My data is as follows. I have two groups of patients. Patients in each group had a different type of eye surgery. 5 variables were measured on patients in each group. I want to compare those variables between the two groups using a permutation test or MANOVA. The eye on which the surgery was done does not really matter in the analysis. However, Patient 2 in group A for example, had the surgery on both eyes and therefore has those 5 variables measured twice, one time on each eye. Can I consider patient 2 Left and patient 2 Right as two different observations? Same for patient 31 in group B. $$ \begin{array} \hline \text{Patient} & \text{Surgery type} & \text{Side} & \text{V1}& \ldots & V5\\\ 1 & \text{A} & \text{Left} & 91 & \ldots & 22\\\ 2 & \text{A} & \text{Left} & 87 & \ldots & 19\\\ 2 & \text{A} & \text{Right} & 90 & \ldots & 23\\\ . & . & . &\\\ 31 & \text{B} & \text{Left} & 90 & \ldots & 17\\\ 31 & \text{B} & \text{Right} & 88 & \ldots & 19\\\ 32 & \text{B} & \text{Right} & 91 & \ldots & 24\\\ . & . & . &\\\ \hline \end{array} $$ Thank you


title: Help with creating a formula for "diversity coefficient"

text: I am creating a small webapp and a part of it is generating a number (I like to call it diversity coefficient) that would represent how "diverse" someones listening habits are. The problem is - I am not sure how to create a valid formula for what I want. I will try my best to explain what information I have, and what I need as a result, and hopefully someone will be able to provide me with a solution. I guess it's nothing hard or complicated, but I am not that deep into math, so I am not sure how to get what I want (tried playing with different formula ideas for a few day, but could not get what I wanted)... What I do is - I gather information from someones lastfm profile, which is a website that lets you track the music you listen to, and also lets you use all the data that they have stored on their servers to create something with that data (you can see a sample of a profile and what it is about here if you like, just to better understand what it's all about). So let's say you listen to a certain band, and you listen to a number of their songs. The data that I have is the following:   * total count of how many times you listened to a certain band    * number of different songs (from that band) that you listened to   * total count of how many times you listened to each of that different songs For example:   * someone listened to a band called FooBar for 1012 times   * he listened to 17 different songs from that band   * he listened to the first of that 17 songs for 100 times, the second one of that 17 for 110 times, the third one 120 times, the fourth one 170 times, the fifth one 500 times and the remaining 12 songs only once per song. What I would like to have as a result is a number that ranges from 0-100. That number would represent if a person listens to all the songs evenly or if he just listens to one or two song from a band all the time, and all the other song only once or twice and then forgets about them... A perfect 100 score would be for a person that listened to FooBar for 500 times, 5 different song, each one 100 times... That is as even as it gets. A score of 0 (or some very low score) would be for a person that listened to FooBar for 500 times, 5 different songs - one of them for 496 times, and the remaining four songs only once per song. I think that it is some kind of "density" that I want, but how to get to that number - I am not sure.. And that is where I need your help. I would be happy to provide any additional info to help you help me. The thing looks like this (if you take a look at these bars each one representing a different song (and its share of total count), that is why I say "density"): ![bars sample](http://i.stack.imgur.com/IoOer.jpg) In this example it is clear that Dredg is more evenly spread than Deftones, and should have a higher score (in my case the score is almost the same which is not good, it's just something that remained from my experimenting with the formula). Thanks! PS - feel free to edit tags, I really have no clue what to put there..


title: Practical definition of a UMP test?

text: I'm trying to make sense of these two statements about UMP (uniformly most powerful) tests:   1. If $g(t\mid\theta)~$ is a UMP then $~g(t\mid\theta_1)>k g(t\mid\theta_0)~\forall~t\in C$ and $g(t\mid\theta_1)<k g(t\mid\theta_0)~\forall~t\in C^c$   2. For $\mathbf X~i.i.d.~f(x\mid\theta): \theta\in\Omega\subset\mathcal R$ if $f(x\mid\theta)$ has an MLR in $T(x)$ and any $k$, a test that rejects $H_0 \iff T>k~$ is a UMP test of size $\alpha$ with $\alpha=P_{\theta_0}(T>k)$ Why are these statements not tautological? How are they not the definition of _any_ one-tailed statistical test? What would be an example of a one-tailed statistical test that isn't a UMP? Moreover, what is the relationship between LRT and UMP tests? I'm reviewing old exams where sometimes a question asks for an LRT and sometimes a UMP... Aren't all simple LRT tests UMP? Thanks.


title: Residuals from glm model with log link function

text: In the following example:               x <- rnorm(100)     y <- rpois(100, exp(1+x))+10     g1 <- glm(y ~ x, family = Gamma(link = "log"))          par(mfrow = c(2,2))     plot(y~x)     plot(y,resid(g1))     plot(y,resid(g1$lm))      I am trying to look at the pattern in the predictor variable, i.e. if the model is 'correct' there should be no patterns left in the residuals when plotted against the predictor. From the example shown, which residual am I suppose to use, `resid(g1)` or `resid(g1$lm)`, or something else? Same goes when using a `gam`, is that `resid(g1$lm)`, `resid(g1$gam)`? I find this very confusing.


title: What does the abbreviation "p.e." mean?

text: I came across a paper that uses the abbreviation "p.e.": Khatri and Madria, The Von Mises-Fisher Matrix Distribution in Orientation Statistics. 1976. It's in Section 7 on page 105. I'm including a short excerpt: > Using the density function of $K^{\frac{1}{2}}XX'K^{\frac{1}{2}} = S$ given > in Khatri (1975a), we find that the p.e. $X$ given $XX'=I_n$ is What does the abbreviation "p.e." mean in this sentence?


title: Is it true that if $ \epsilon_t \sim^{iid} (0,1) $, then $ E(\epsilon_t^{2}\epsilon_{t-j}^{2}) = 1 $?

text: Under the GARCH($m$,$s$) model, it can be shown that $ E(\eta_t\eta_{t-j}) = E[(a_t^{2}-\sigma_t^{2})(a_{t-j}^{2}-\sigma_{t-j}^{2})] = 0 $. In my proof attempt, I came across $ E(\epsilon_t^{2}\epsilon_{t-j}^{2}) = 1 $ where $ \epsilon_t \sim^{iid} (0,1) $. Is it indeed true that if $ \epsilon_t \sim^{iid} (0,1) $, then $ E(\epsilon_t^{2}\epsilon_{t-j}^{2}) = 1 $?


title: Hypothesis test and Neyman-Pearson framework

text: > Let $X_{1},..,X_{n}$ be a sample from normal distribution with mean > ${\theta}$ and variance 1. Let $H_{0}:{\theta}={\theta}_{0}$ and > $H_{1}:{\theta}={\theta}_{1}$. Use the Neyman-Pearson lemma to show that the > best test for $H_{0}$ vs $H_{1}$ has a critical region of the form > mean$(X_{1},..,X_{n}){\geq}k$. I have done this part. The next question is: > Let ${\theta}_{0}=0$ and ${\theta}_{1}=1$. Find the values of $n$ and $k$ so > that the power function of the test ${\phi}$ has values > ${\phi}{\theta}_{0}=0.01$ and ${\phi}{\theta}_{1}=0.9$. Write down the > corresponding decision rule. My work thus far: Let $Y= \text{mean}(X_{1},..,X_{n})$. we have $P(Y{\geq}k|{\theta}_{0})=0.01$. Similarly for ${\theta}_{1}$. But how do I go from here?


title: Rank correlation statistics comparison

text: I am trying to understand the relative behavior of the following rank correlation statistics:   1. Spearman coefficient   2. Kendall Tau / Concordance percentage   3. Normalized Gini coefficient (area under curve of percentage captured versus percentage observations)   4. Normalized Area under ROC curve (for binary classifiers) I don't believe any of these are functionally related to the others. The accepted answer here references this paper and Spearman and Kendall are highly correlated (as one would expect). Are there good intuitions behind/discussions of relative (across datasets) or absolute (for a given dataset) differences for (any pair of) these measures?


title: Sum of normally distributed random variables expressed with respect to one component

text: A rather elementary question: If we have: $X = X_1 + X_2 + ... + X_n$ and $X_i \sim N(0,\sigma^2)$ and independent, can we express $X$ as $\sqrt n X_i$ for any $i = 1,...,n$ ? I think since all the components $X_i$ follow the normal distribution and we are summing $n$ of them up, the resulting rv is $X \sim N(0, n\sigma^2)$ and therefor we can use $X = \sqrt n X_i$. Is this reasoning correct? If yes, under what condition it can be false?


title: Impact of data dimensionality on computation complexity of SVM?

text: What is the impact of data dimensionality on computation complexity of SVM? I found on the literature that the complexity of SVM is $O(N^3)$, where $N$ is the number of training examples. If the number of dimension (e.g., $D$) does not impact the training time why it s better to reduce the dimensionality for training high-dimension dataset? Is it just to avoid overfitting? BTW, I m using SVDD (support vector data description) from RRTools.


title: Can time-invariant variables cause autocorrelation?

text: I am running a pooled OLS regression and Random effects regression. I have tested for autocorrelation for both methods. In the pooled OLS model I find serial correlation but for the RE model I find no serial correlation. How is this possible? Is it possible that the unobserved time invariant variables that are removed in RE model are causing serial correlation in the pooled OLS regression?


title: Permuting the formula argument to Hmisc:aregImpute

text: In Frank Harrell's RMS Short Course today, I became aware that multiple imputation with `Hmisc:aregImpute` is not invariant to the ordering of terms in its formula argument, and that one therefore ought to check results of imputation under permutations of these terms. Here's a simple function that generates the necessary permutations:               perms <- function(n, formula) {         rhs <- attr(terms(formula), 'term.labels')         formulae <- list()         for(i in 1:n){             formulae[[i]] <- as.formula(paste("~", paste(sample(rhs), collapse="+")))         }         formulae     }      For example:               > perms(5, ~ age + sex + bp + death + mi)     [[1]]     ~mi + age + bp + sex + death     <environment: 0x10dc9cbb8>          [[2]]     ~bp + mi + sex + age + death     <environment: 0x10dc9cbb8>          [[3]]     ~mi + bp + age + death + sex     <environment: 0x10dc9cbb8>          [[4]]     ~age + death + bp + sex + mi     <environment: 0x10dc9cbb8>          [[5]]     ~bp + age + death + sex + mi     <environment: 0x10dc9cbb8>          >       This could be used with `lapply` as follows:               as <- lapply(perms(5, ~ age + sex + bp + death + mi)                  , aregImpute, data=mydata, n.impute=10)      But I wonder whether it might make more sense to incorporate this into aregImpute itself.


title: Injective uniform distribution on an n-sphere

text: For an application I'm working on, I need to go from some uniformly distributed variables to an uniform distribution on an n-sphere. The standard way to do this seems to be choose (n+1) normal distributed variables, and then divide by the norm. However, I need to be able to, given a point on the sphere, extract the uniform values I would have used to get this point. There the standard method fails, at it is not injective. I think I have devised a way to do this, however, but would like someone with more statistical knowledge than me to confirm this. Or if anyone knows of an easier way to do it, that's also appreciated. Now, here's the idea: Let $x_1, \ldots, x_n$ be cartesian coordinates for $\mathbb{R}^n$, and choose spherical coordinates given as in the Wikipedia article for an n-sphere, i.e. $\begin{align} x_1 &= r\cos\phi_1 \\\ x_2 &= r\sin\phi_1\cos\phi_2\\\ x_3 &= r\sin\phi_1\sin\phi_2\cos\phi_3\\\ &\qquad\vdots\\\ x_{n-1} &= r\sin\phi_1\ldots\sin\phi_{n-2}\cos\phi_{n-1}\\\ x_n &= r\sin\phi_1\ldots\sin\phi_{n-1} \end{align}$ The volume element of an (n-1)-sphere of radius r is $\mathrm{d}V = r^{n-1}sin^{n-2}\phi_1\sin^{n-3}\phi_2\ldots\sin\phi_{n-2} \mathrm{d}\phi_1\mathrm{d}\phi_2\ldots\mathrm{d}\phi_{n-1}.$ If we compute $\mathrm{d}x_i$ and $\mathrm{d}x_1\mathrm{d}x_2\ldots\mathrm{d}x_{n-2}$ I think we get that $\mathrm{d}V = (-1)^nr\mathrm{d}x_1\ldots\mathrm{d}x_{n-2}\mathrm{d}\phi_{n-1}.$ I am then of the belief (and this is what I want confirmed by someone "in the know") that to choose points uniformly on the sphere, we can choose points $x_1, \ldots, x_{n-2}$ uniformly on the ball $x_1^2+\ldots x_{n-2}^2 \leq r^2$ and choose $\phi_{n-1}$ uniformly from $[0,2\pi)$. If we can, we have reduced the problem to finding points uniformly on a (n-2)-ball. This can be done by choosing points uniformly on the boundary and multiplying by $u^{1/(n-2)}$, where $u$ is uniformly chosen from $[0,1]$. Choosing points uniformly on the boundary is the same as choosing points uniformly on an $n-3$-sphere, and we're back to the original problem, but now 2 dimensions lower. We can work our way down to either the 1-sphere or the 1-ball, where we can choose a point uniformly with no problem.


title: Showing independence

text: $X_{1}$ and $X_{2}$ are independent, chi-square distributed random variables. I've managed to show that $Z=X_{1}+X_{2}$ is independent of $\frac{X_{1}}{Z}$. How do I conclude then that $Z=X_{1}+X_{2}$ is independent of $(\frac{X_{1}}{Z},\frac{X_{2}}{Z})$?


title: Consistency of estimator

text: Regression: Wage=b0+b1collegegrad, where collegegrad is a dummy variable. Suppose you want to estimate the wage ratio between college graduates and non- college graduates. Is the estimator theta=b0/b1 consistent? My thinking is that if we could increase our sample size to infinity, we would cover the entire population and thus get the true ratio, so the estimator is consistent. Am I correct, or am I missing something?