title: Bootstrap of parameter estimates and confidence intervals (hurdle model)

text: Think you can help with this. I´ve run a set of candidate hurdle models on insect abundance data with pscl package for R. These models had an abundance part with a truncated negative binomial distribution; and an ocurrence part with binomial distribution. I wish to calculate the model parameter estimates and its CI by bootstrap so as to assess if parameter estimates for each explanatory variable are different from 0. I think functions of mostly used packages do not allow hurdle type objects, so I suppose that the only option is to "make" one... I understand a little bit of R programming/code, but I'm not good enough! Definitely, I need some help! thanks in advance, Maria


title: How to estimate model with both linear and exponential parameters?

text: I have a theoretical growth function that can be perturbed by events, and I'd like to estimate the growth parameters as well as the perturbation, and the rate of falloff after that perturbation. I'm thinking of using a logistic function to model the effect of the event and the falloff of that effect (if any). To ground this, $x$ is time, and $t$ is the time the event occurs. Before time $t$, or if the event never occurs, we have a simple linear regression. After the event occurs, I model the contribution of the event with magnitude controlled by $\beta_2$ and rate of falloff by $\beta_3$. $y_i=\left\{x_{i}<t:\beta_{0}+\beta_1x_i+\epsilon_i,x_i>t:\beta_0+\beta_1x_i+2\beta_2\frac{1}{\left(1+e^{\beta_3\left(x_i-t\right)}\right)}+\epsilon_i\right\}$ ( _edited to add the error term_ ) Here's a Desmos graph if it helps. I'm really not sure how to estimate parameters for this model in any of the stats packages I'm familiar with in R. Do I need to turn to Bayesian methods?


title: Randomisation for 2 site clinical study

text: If I want to conduct a randomised study in 2 sites comparing two clinical treatment arms, and every site is expected to include 60-80 patients, is it acceptable to randomise every new patient with simple coin toss (one centralised scheme for all patients, regardless of site) or is some stratified randomisation more appropriate (stratified per center)? I have tried to understand urn randomisation and wonder if it would be a better model. If so, can you suggest an easy way of doing so? I have found RANDI3 online (http://dschrimpf.github.com/randi3/) but I have no idea whether I can 'trust' its output. Any advice welcome


title: Building a Dictionary Using PCA / SVD and Regenerating the Signals

text: I have the following problem and I know I can solve it using PCA / SVD, I just can get to write it properly. I have many sensors, each produces M = 3500 samples of a signal.   The signal of each sensor is a little different from the other. I have N = 256 Sensors. Now, I want to build a dictionary of those 256 Signals in a way that when I get a new 3500 Samples from a different sensor I will be able to calculate its coefficients and keep the only.   I want this signal to have K = 16 Parameters. Later, when I want to regenerate an approximation of this new signal, all I need is to multiply those coefficients by Dictionary / Base matrix. I know how to do it an arbitrary Base $ H $.   For instance, Using Fourier Series I can Have $ H \in {R}^{16x3500} $.   Where each row of $ H $ is a 3500 samples of an Harmonic Function with a given frequency.   This yields a dictionary which is given by the Pseudo Inverse $ {\left({H}^{T}H \right)}^{-1}{H}^{T} $.   Namely, If I get new 3500 samples given by $ {x}_{new} $, The coefficients are given by $ {f}_{new} = {\left({H}^{T}H \right)}^{-1}{H}^{T} x $.   The approximation of $ {x}_{new} $ is given by $ {\hat{x}}_{new} = H {f}_{new} $. Now, how can I get the optimal base for that using the PCA / SVD?   How do I get the new coefficients?   How do I approximate the "New Data"? Thank You.


title: Uncertainty propagation in linear interpolation

text: How do I calculate the uncertainties in linearly interpolated values from a given tabulated function? * * * I am just coming back into the fold after a bit of a hiatus, and am having trouble jumpstarting my brain. I have a tabulated function $(x_i, y_i, e_i)$, where $x_i$ is on an irregular grid and $e_i$ is the uncertainty in the value of the $y_i$. The datapoints a uncorrelated. For various reasons, I need to carry out a linear interpolation to put these values onto a regular grid. I have no problems in calculating the interpolated values, but I can't seem to find a way to calculate the uncertainty in the interpolated value. The methods I remember fail to reproduce the tabulated uncertainties (if $x=x_i$), and seem generally clunky. So, my question:   How do I calculate the uncertainties in linearly interpolated values from a given tabulated function?   Bonus: How to calculate covariances too... (If you need it to illustrate your point) Some sample data:               xi   yi  ei      x   y  e     1.0  7.0 2       1.0      1.9  8.1 3       2.0     3.1  9.0 4  -->  3.0     4.0 10.3 5       4.0     4.9 11.0 6       5.0     6.2 12.5 7       6.0      * * * Edit: The real data and analysis: I have x-ray diffraction patterns collected from the same sample with the detector in two (or more) positions, and as such, the irregular x grid for both datasets is different. I want to add the two (or more) patterns together. The purpose of the addition is to obtain higher intensity, and thus, a higher confidence in the final intensities. The data will then be modelled by the Rietveld method. In each dataset, there is ~20000 points, of which I've reproduced an indicative subset. What would be the best method to put all datasets onto a common grid to enable the intensities to be added (taking into account the measurement uncertainties)? Note: Most (all) implementations of the Rietveld method assume that the intensities are uncorrelated, which is why I'd also like to be able to calculate the correlations that I'm introducing. At this point in time, I'm going to ignore intensity correlation in my analyses, but I would like to minimise it if possible.               x1             y1            e1              x2         y2           e2           x        y         e     26.699982   257.611938  16.819771       26.699972   206.448425  15.027238       26.7             26.703737   220.754547  15.610549       26.703726   231.364349  15.949701       26.70375             26.707493   222.360214  15.620609       26.707479   246.007446  16.460543       26.7075          26.711248   237.475983  16.173454       26.711233   228.180939  15.793293       26.71125             26.715004   240.830673  16.305054       26.714987   232.759888  16.082907       26.715           26.71876    223.493912  15.632176       26.718739   261.551666  17.050766       26.71875             26.722515   262.328674  17.005249       26.722492   225.026382  15.707023       26.7225          26.726271   228.291 15.732837       26.726246   231.585678  16.034586       26.72625             26.730026   248.281403  16.418236       26.73   241.725464  16.333258       26.73            26.733782   231.120026  15.864223       26.733753   231.883682  16.055712       26.73375             26.737537   239.188919  16.191217       26.737507   242.005142  16.454927       26.7375          26.741293   239.229172  15.99611        26.741261   255.124924  16.882158       26.74125             26.745049   202.675842  14.854559       26.745012   262.2164    17.161072       26.745           26.748804   239.148224  16.121639       26.748766   244.202606  16.573368       26.74875             26.752562   191.03067   14.338655       26.75252    254.840759  16.829659       26.7525          26.756317   236.928757  16.135286       26.756273   232.351868  16.019405       26.75625             26.760073   220.250137  15.574051       26.760027   255.123459  16.750168       26.76            26.763828   237.463058  16.071577       26.763781   232.629074  16.038975       26.76375             26.767584   226.210571  15.687851       26.767534   243.375717  16.346483       26.7675          26.771339   258.652649  16.857044       26.771288   225.64296   15.716341       26.77125             26.775095   254.923645  16.736721       26.775042   237.500946  16.209282       26.775           26.778852   278.621552  17.502522       26.778795   236.081726  16.144192       26.77875             26.782608   209.142868  15.011824       26.782549   247.611633  16.537083       26.7825          26.786364   258.206909  16.859592       26.786303   211.21933   15.235976       26.78625             26.790119   229.081314  15.720976       26.790056   246.538696  16.564161       26.79            26.793875   257.760193  16.797421       26.79381    234.845978  16.230515       26.79375             26.797632   233.179077  16.043262       26.797564   239.260361  16.297928       26.7975          26.801388   241.209442  16.297544       26.801317   262.272491  17.26512        26.80125             26.805143   240.669388  16.293869       26.805071   240.155869  16.292097       26.805           26.808899   287.645905  17.957613       26.808825   262.359009  16.943207       26.80875             26.812656   244.147873  16.433184       26.812578   259.774536  16.899658       26.8125          26.816412   255.581009  16.748405       26.816332   216.269989  15.321012       26.81625             26.820168   243.794861  16.375322       26.820087   238.458344  16.207962       26.82            26.823923   220.18042   15.498717       26.823841   216.359161  15.257962       26.82375             26.827681   238.572983  16.115116       26.827595   232.603851  16.037193       26.8275          26.831436   240.484207  16.147575       26.831348   264.862213  17.206285       26.83125             26.835192   261.019531  16.854525       26.835102   245.935654  16.422411       26.835           26.838949   232.71814   15.875105       26.838856   230.69754   16.041531       26.83875             26.842705   244.266739  16.374409       26.842609   245.167053  16.469751       26.8425          26.84646    270.787384  17.307497       26.846365   253.443481  16.735092       26.84625             26.850216   282.155518  17.636786       26.850119   277.531006  17.620699       26.85            26.853973   265.355927  17.173996       26.853872   259.03952   16.980762       26.85375             26.857729   257.106873  16.916601       26.857626   242.682373  16.365713       26.8575          26.861486   287.789215  17.998287       26.86138    255.16008   16.818199       26.86125             26.865242   261.124512  16.957371       26.865135   252.704788  16.718046       26.865           26.868998   257.830231  16.834381       26.868889   243.115891  16.262362       26.86875             26.872755   276.640381  17.562687       26.872643   253.842941  16.795219       26.8725          26.876511   262.03244   16.95348        26.876396   231.254578  15.941954       26.87625             26.880266   268.417633  17.215582       26.880152   234.510742  16.206791       26.88            26.884024   281.486755  17.655949       26.883905   272.454926  17.481216       26.88375             26.887779   235.749084  15.985992       26.887659   286.539307  17.886707       26.8875          26.891537   235.417145  15.996305       26.891413   249.974762  16.565758       26.89125             26.895292   242.421432  16.314156       26.895168   264.592438  17.155731       26.895           26.899048   263.415894  17.077515       26.898922   264.821533  17.236376       26.89875             26.902805   248.896179  16.558359       26.902676   287.547363  17.982763       26.9025          26.906561   254.223572  16.722179       26.906431   250.62146   16.74309        26.90625             26.910318   264.850677  17.205517       26.910185   252.432877  16.799511       26.91            26.914074   268.862457  17.212996       26.913939   288.753265  17.934757       26.91375             26.917831   284.015381  17.787331       26.917694   266.463898  17.182816       26.9175          26.921587   285.391663  17.782124       26.921448   262.388794  17.172642       26.92125             26.925344   256.088654  16.847933       26.925203   282.458008  17.656187       26.925           26.9291 278.193542  17.568636       26.928957   264.536438  17.119614       26.92875             26.932858   261.607147  16.925268       26.932711   273.392395  17.510523       26.9325          26.936613   293.501556  18.081835       26.936466   241.179169  16.363207       26.93625             26.940371   281.078217  17.598589       26.94022    296.551788  18.337276       26.94            26.944126   277.295685  17.448036       26.943975   264.610352  17.222281       26.94375             26.947884   275.656372  17.435543       26.947729   268.674225  17.29664        26.9475          26.951639   280.630981  17.632053       26.951485   267.253937  17.170872       26.95125             26.955397   310.859436  18.560068       26.955238   295.029663  18.20949        26.955           26.959152   309.720245  18.579161       26.958992   284.24057   17.89658        26.95875             26.96291    254.522202  16.709709       26.962748   319.454041  18.907776       26.9625          26.966667   328.244995  19.204264       26.966501   272.877869  17.509052       26.96625             26.970423   301.559174  18.193857       26.970257   292.79657   18.161213       26.97            26.97418    286.889801  17.815968       26.97401    311.167603  18.728683       26.97375             26.977936   340.261688  19.487011       26.977766   299.764618  18.262012       26.9775          26.981693   296.298218  18.075247       26.98152    344.325958  19.784254       26.98125             26.985451   301.822418  18.299801       26.985275   322.565887  19.127018       26.985           26.989206   304.588318  18.381588       26.989031   288.446136  18.008892       26.98875             26.992964   302.583008  18.287107       26.992785   292.30899   18.098988       26.9925          26.996719   309.682159  18.487858       26.99654    310.9328    18.74452        26.99625             27.000477   296.918793  18.144386       27.000294   290.236755  18.092058       27           27.004234   328.477325  19.130945       27.004049   341.49704   19.674849       27.00375             27.00799    305.255676  18.363239       27.007803   324.09375   19.100769       27.0075          27.011747   331.636841  19.262068       27.011559   349.302277  19.932892       27.01125             27.015505   298.813782  18.232843       27.015314   340.791534  19.633039       27.015           27.01926    377.82962   20.703222       27.019068   333.971771  19.401537       27.01875             27.023018   337.940399  19.406948       27.022823   321.540894  18.975531       27.0225          27.026775   315.944 18.782553       27.026579   351.45636   19.887695       27.02625             27.030531   329.575897  19.110327       27.030333   332.281525  19.417547       27.03            27.034288   395.013153  21.12104        27.034088   354.798248  20.02552        27.03375             27.038046   376.693756  20.584587       27.037844   355.299011  20.054609       27.0375          27.041801   377.918793  20.545406       27.041597   371.191742  20.603811       27.04125             27.045559   379.556 20.583424       27.045353   378.1474    20.748695       27.045           27.049316   353.320923  19.883623       27.049109   359.688446  20.224152       27.04875             27.053074   384.733154  20.791872       27.052862   367.917542  20.361286       27.0525          27.056829   381.299683  20.57424        27.056618   381.987885  20.719294       27.05625             27.060587   394.974365  21.1723     27.060373   396.737946  21.296721       27.06            27.064344   410.550293  21.496117       27.064127   367.995148  20.421124       27.06375             27.068102   398.530121  21.26141        27.067883   382.928436  20.908066       27.0675          27.071857   432.425446  22.100449       27.071638   421.565887  22.011845       27.07125             27.075615   467.759308  23.259808       27.075394   462.320862  23.155806       27.075           27.079372   455.897797  22.65077        27.079149   435.147186  22.45047        27.07875             27.08313    450.400574  22.790157       27.082903   429.493591  22.332462       27.0825          27.086887   452.224335  22.708344       27.086658   487.865631  23.960541       27.08625             27.090643   457.796967  22.896755       27.090414   466.144714  23.226973       27.09            27.0944 525.631104  24.670202       27.09417    475.891571  23.476278       27.09375             27.098158   503.192535  24.212145       27.097923   507.236694  24.462635       27.0975          27.101915   517.572388  24.4188     27.101679   488.699615  23.682343       27.10125             27.105673   496.762482  23.964514       27.105434   486.923187  23.739136       27.105           27.109428   532.518188  24.957394       27.10919    541.927673  25.270697       27.10875             27.113186   533.502869  24.934303       27.112946   509.530853  24.190039       27.1125          27.116943   628.999268  27.041285       27.116701   575.867737  26.024132       27.11625             27.120701   587.537964  26.205851       27.120457   573.836609  25.906281       27.12            27.124458   641.46051   27.382612       27.12421    622.094727  27.276428       27.12375             27.128216   598.144653  26.295834       27.127966   594.514221  26.573986       27.1275          27.131973   646.872864  27.515984       27.131721   682.880066  28.333942       27.13125             27.135731   685.346069  28.378246       27.135477   715.428162  29.170425       27.135           27.139486   705.060852  28.710655       27.139233   701.500671  29.095297       27.13875             27.143244   742.444946  29.799131       27.142988   743.038147  29.763405       27.1425          27.147001   739.384155  29.611042       27.146744   742.684509  29.910776       27.14625             27.150759   756.785889  30.014313       27.150499   813.66803   31.310173       27.15            27.154516   811.989807  31.013382       27.154255   857.058044  32.132923       27.15375             27.158274   862.963013  32.065449       27.15801    805.47345   31.155424       27.1575          27.162031   859.406433  32.093735       27.161766   875.511963  32.665203       27.16125             27.165789   895.645935  32.995667       27.165522   915.463867  33.07769        27.165           27.169546   969.672363  34.077866       27.169277   996.215515  34.755962       27.16875             27.173304   1032.02478  35.365906       27.173033   996.287354  34.484081       27.1725          27.177061   1101.285034 36.482513       27.176788   1097.823242 36.574898       27.17625             27.180819   1089.692505 36.372944       27.180544   1172.005981 38.15443        27.18            27.184576   1233.372925 39.165882       27.184299   1213.508423 38.770805       27.18375             27.188334   1286.500488 40.216381       27.188055   1316.095947 40.561798       27.1875          27.192091   1351.022949 41.322506       27.191811   1444.984009 42.750305       27.19125             27.195848   1410.998779 42.263809       27.195566   1474.066528 43.096645       27.195           27.199606   1605.868286 45.268463       27.199322   1485.535522 43.290718       27.19875             27.203363   1670.843506 46.438526       27.203077   1600.005127 44.899033       27.2025          27.207121   1852.570068 48.931038       27.206833   1712.454712 46.573402       27.20625             27.210878   1949.015503 50.506145       27.210588   1849.947876 48.792233       27.21            27.214636   1991.649902 51.212296       27.214344   2024.669678 51.460781       27.21375             27.218393   2165.216553 53.421818       27.2181 2088.17041  51.730793       27.2175          27.222151   2337.598877 55.876877       27.221855   2346.668457 55.977787       27.22125             27.225908   2473.485596 57.56147        27.225611   2412.149414 56.592995       27.225           27.229666   2797.586426 61.769291       27.229368   2696.092773 60.324032       27.22875             27.233423   2991.792969 64.599525       27.233124   2909.843506 63.37001        27.2325          27.237181   3118.806885 65.786011       27.236879   3074.491699 65.597015       27.23625             27.240938   3504.134521 70.3414     27.240635   3375.719727 68.693611       27.24            27.244696   3891.235352 75.364098       27.24439    3758.651611 73.219566       27.24375             27.248453   4244.597168 79.501091       27.248146   4025.467773 76.340263       27.2475          27.252211   4531.950195 82.043549       27.251902   4374.370605 80.199883       27.25125             27.255968   4912.705566 85.931183       27.255659   4829.439941 84.864685       27.255           27.259726   5583.304688 93.659035       27.259415   5185.368652 88.253494       27.25875             27.263483   5801.212402 94.912148       27.26317    5837.462402 95.271805       27.2625          27.267242   6527.44873  101.581688      27.266926   6164.637207 97.746201       27.26625             27.271  7149.898438 107.501686      27.270681   6996.065918 105.987755      27.27            27.274757   8024.799805 115.727562      27.274439   7717.926758 113.278625      27.27375             27.278515   8782.509766 122.771156      27.278194   8513.098633 120.068474      27.2775          27.282272   9776.30957  130.636856      27.28195    9582.835938 129.320023      27.28125             27.28603    10684.06055 138.121933      27.285706   10629.82422 138.458023      27.285           27.289787   12160.97461 150.254181      27.289463   11498.96484 145.095963      27.28875             27.293545   13230.85352 159.008743      27.293219   13194.12207 157.515976      27.2925          27.297304   14882.88086 170.719299      27.296974   14356.03418 166.620728      27.29625             27.301062   16311.61523 180.528198      27.30073    15988.86719 178.297455      27.3             27.304819   18470.22266 196.541138      27.304487   17430.61523 188.006866      27.30375             27.308577   18905.74023 199.37529       27.308243   18763.6582  196.852402      27.3075          27.312334   19843.95898 206.008362      27.311998   19107.41602 199.18042       27.31125             27.316092   19446.24805 202.544922      27.315756   19581.70898 202.482391      27.315           27.319851   18449.27148 195.371994      27.319511   18548.38477 196.500046      27.31875             27.323608   17099.03516 185.706924      27.323267   17395.10156 188.679016      27.3225          27.327366   15468.32129 174.745087      27.327023   15864.07031 177.236633      27.32625             27.331123   13734.60449 161.846298      27.33078    14618.6084  169.099823      27.33            27.334881   12631.53125 152.229172      27.334536   12794.08008 155.524338      27.33375             27.33864    11271.24707 142.581253      27.338293   11787.24023 148.055161      27.3375          27.342398   10512.77539 136.916977      27.342049   10371.50879 135.465714      27.34125             27.346155   9237.021484 126.318268      27.345804   9684.5  129.716415      27.345           27.349913   8405.818359 119.341942      27.349562   8767.294922 122.610458      27.34875             27.353672   7818.925781 113.856117      27.353317   8091.36377  115.909096      27.3525          27.35743    7115.508301 108.37809       27.357073   7147.862305 107.73645       27.35625             27.361187   6415.062012 100.436485      27.36083    6645.275391 103.097359      27.36            27.364944   5803.602051 95.109955       27.364586   5926.142578 95.61367        27.36375             27.368704   5287.605957 89.370728       27.368343   5482.560059 91.383064       27.3675          27.372461   4909.004395 85.33609        27.372099   4844.45459  85.257843       27.37125             27.376219   4367.385742 79.370476       27.375854   4488.047852 81.663086       27.375           27.379976   3993.688721 75.518738       27.379612   4184.442383 78.276047       27.37875             27.383736   3681.97876  72.029762       27.383368   3660.422119 72.288017       27.3825          27.387493   3425.320801 69.451126       27.387125   3500.459961 70.925537       27.38625             27.391251   3095.907471 65.610405       27.390881   3144.00415  66.149261       27.39            27.39501    2904.301758 63.035709       27.394638   2957.160156 64.107269       27.39375             27.398767   2597.830811 59.3172     27.398394   2628.188721 59.741573       27.3975          27.402525   2454.955566 57.157295       27.402151   2406.833252 56.985153       27.40125             27.406284   2210.20874  54.052696       27.405907   2295.345947 55.378689       27.405           27.410042   2074.173096 52.329163       27.409664   2135.306641 52.890289       27.40875             27.413799   1886.714233 49.559029       27.41342    1919.739868 50.086132       27.4125          27.417559   1770.544434 47.881283       27.417177   1816.12146  48.338406       27.41625             27.421316   1557.283813 44.360882       27.420933   1684.497192 46.274689       27.42            27.425074   1503.644043 43.386501       27.42469    1542.882446 44.166553       27.42375             27.428833   1377.763306 41.332214       27.428446   1567.353882 44.635925       27.4275          27.43259    1353.757812 40.70454        27.432203   1396.887451 41.758747       27.43125             27.436348   1170.140991 37.721062       27.435959   1336.477661 40.869045       27.435           27.440107   1180.836548 38.067421       27.439716   1239.929199 39.100754       27.43875             27.443865   1147.78894  37.471424       27.443474   1160.866455 37.903599       27.4425          27.447622   1058.838013 35.651024       27.447229   1054.965942 35.629913       27.44625             27.451382   1044.922241 35.50663        27.450987   1013.544983 35.041023       27.45            27.455139   981.188721  34.571899       27.454742   927.535461  33.372013       27.45375             27.458899   875.704956  32.246819       27.4585 905.684937  32.86861        27.4575          27.462656   865.433899  32.310036       27.462257   917.381287  33.185966       27.46125             27.466413   885.555725  32.773746       27.466013   823.479797  31.281075       27.465           27.470173   834.432922  31.544697       27.46977    781.178345  30.347902       27.46875             27.47393    731.929443  29.260817       27.473526   759.411682  29.847685       27.4725          27.47769    698.259766  28.603771       27.477283   708.03717   28.836737       27.47625             27.481447   687.848328  28.403528       27.481041   705.581177  28.773079       27.48            27.485207   678.07135   28.106058       27.484797   652.00238   27.51049        27.48375             27.488964   610.292725  26.522755       27.488554   651.322815  27.606476       27.4875          27.492722   631.037537  27.175158       27.492311   624.520813  26.990871       27.49125             27.496481   523.445618  24.470995       27.496067   583.280762  26.098219       27.495           27.500238   582.989319  25.97337        27.499825   522.271545  24.460474       27.49875             27.503998   552.380798  25.282471       27.503582   555.14447   25.34499        27.5025          27.507755   547.555786  25.190542       27.507338   488.443329  23.549656       27.50625             27.511515   517.328491  24.430441       27.511095   532.32251   24.760672       27.51            27.515272   575.333679  25.930967       27.514853   497.951385  23.879543       27.51375             27.519032   512.204834  24.226938       27.518608   512.114502  24.152409       27.5175          27.522789   495.360992  23.918774       27.522366   456.453064  22.703745       27.52125             27.526548   466.525665  23.121805       27.526123   510.131897  24.172394       27.525           27.530306   457.033813  22.80759        27.529881   466.042755  23.023245       27.52875             27.534065   459.896362  22.758112       27.533636   419.527649  21.644688       27.5325          27.537823   461.733063  22.827629       27.537394   447.970917  22.637915       27.53625             27.541582   452.128998  22.628756       27.541151   387.796875  20.855265       27.54            27.54534    412.326538  21.644251       27.544909   446.473572  22.585011       27.54375             27.549099   398.742493  21.326605       27.548664   415.603424  21.639071       27.5475          27.552856   424.974548  22.037888       27.552422   384.086853  20.945688       27.55125             27.556616   410.101379  21.549911       27.556179   393.997864  20.959766       27.555           27.560373   384.349487  20.824251       27.559937   414.126709  21.79455        27.55875             27.564133   406.186279  21.521414       27.563694   361.580261  20.164911       27.5625          27.56789    360.696869  20.141962       27.56745    374.833771  20.588844       27.56625             27.57165    360.720215  20.143303       27.571207   357.429138  20.093407       27.57            27.575407   351.623444  19.897425       27.574965   374.868317  20.701504       27.57375             27.579166   408.509186  21.463638       27.578722   371.973999  20.481779       27.5775          27.582924   335.177002  19.386414       27.582479   334.185669  19.414314       27.58125             27.586683   342.521362  19.56319        27.586235   366.800964  20.464724       27.585           27.590443   346.306213  19.587896       27.589993   331.174438  19.351027       27.58875             27.5942 331.486481  19.224245       27.59375    334.341888  19.308029       27.5925          27.59796    338.926971  19.40839        27.597507   349.915955  19.882914       27.59625             27.601717   291.329834  17.975086       27.601265   352.604065  19.898081       27.6             27.605476   358.433655  20.067392       27.605022   310.955994  18.565992       27.60375             27.609236   334.920197  19.429049       27.60878    303.003632  18.373362       27.6075          27.612993   287.238342  17.838184       27.612537   323.746307  19.020924       27.61125             27.616753   306.155426  18.448715       27.616293   319.403442  18.8167     27.615           27.62051    313.259705  18.827635       27.62005    305.238556  18.543093       27.61875             27.624269   298.106781  18.218935       27.623808   326.727875  19.142216       27.6225          27.628029   336.71228   19.448822       27.627565   308.32016   18.582729       27.62625             27.631786   325.233154  18.965395       27.631323   286.17688   17.770529       27.63            27.635546   311.634796  18.637335       27.63508    294.259491  18.037977       27.63375             27.639303   304.706024  18.388884       27.638838   295.109406  18.122005       27.6375          27.643063   305.301941  18.395889       27.642595   277.055817  17.6217     27.64125             27.646822   313.338776  18.772058       27.646353   289.719177  17.842659       27.645           27.650579   298.848236  18.174519       27.65011    310.301422  18.55541        27.64875             27.654339   265.391693  16.953909       27.653868   267.216339  17.073452       27.6525          27.658098   272.78064   17.280085       27.657625   304.284027  18.332893       27.65625             27.661856   305.482086  18.407038       27.661383   275.859741  17.511847       27.66            27.665615   296.386871  18.080801       27.66514    272.589233  17.457758       27.66375             27.669374   260.02832   16.916584       27.668898   271.612976  17.425808       27.6675          27.673132   310.00769   18.44916        27.672655   284.807159  17.838223       27.67125             27.676891   293.63327   17.998568       27.676413   282.919739  17.716915       27.675           27.680651   295.854095  18.017424       27.68017    301.875092  18.394066       27.67875             27.684408   272.622864  17.36412        27.683928   276.695435  17.471575       27.6825          27.688168   276.579803  17.340225       27.687685   276.434875  17.517595       27.68625             27.691927   270.084076  17.135855       27.691442   283.377045  17.777632       27.69            27.695684   254.886475  16.669485       27.6952 265.695068  17.099983       27.69375             27.699444   265.606781  17.094158       27.698957   263.489441  17.05014        27.6975          27.703203   281.023895  17.533472       27.702715   252.295532  16.62472        27.70125             27.706961   285.609985  17.703798       27.706472   277.086731  17.465652       27.705           27.71072    298.809174  18.112206       27.71023    280.543213  17.689232       27.70875             27.714479   260.353058  16.87396        27.713987   250.574188  16.573614       27.7125          27.718237   276.885193  17.45262        27.717745   259.199097  16.861282       27.71625             27.721996   236.178192  16.015791       27.721502   301.490356  18.400581       27.72            27.725756   247.770172  16.481607       27.72526    261.563324  17.117252       27.72375             27.729515   276.269745  17.475348       27.729017   262.837189  17.039047       27.7275          27.733273   240.565948  16.153196       27.732775   259.085907  16.885988       27.73125             27.737032   281.739594  17.579294       27.736532   313.839417  18.863436       27.735           27.740791   273.610931  17.302837       27.74029    260.170776  16.926085       27.73875             27.744551   268.404846  17.246708       27.744049   283.201477  17.642088       27.7425          27.748308   254.902618  16.767962       27.747807   245.388168  16.384956       27.74625             27.752068   268.374359  17.212734       27.751564   274.828735  17.444706       27.75            27.755827   282.476166  17.657354       27.755322   235.657654  16.148746       27.75375             27.759586   257.19696   16.759798       27.759079   264.412659  17.079205       27.7575          27.763344   253.514648  16.641918       27.762836   287.414459  17.849409       27.76125             27.767103   285.195831  17.738724       27.766594   265.933838  17.179968       27.765           27.770863   272.149536  17.270281       27.770351   246.45018   16.558067       27.76875             27.774622   220.818451  15.475125       27.774111   261.313141  17.002283       27.7725          27.778379   258.381226  16.838909       27.777868   240.765503  16.233316       27.77625             27.782139   225.21434   15.685786       27.781626   256.711212  16.857025       27.78            27.785898   231.654419  15.935867       27.785383   265.196808  17.131144       27.78375             27.789658   234.376343  16.058842       27.789141   292.645447  18.058426       27.7875          27.793417   261.384705  16.942513       27.792898   246.895554  16.454943       27.79125             27.797174   251.88031   16.629318       27.796658   233.284836  15.94837        27.795           27.800934   244.516464  16.358343       27.800415   225.843765  15.661814       27.79875             27.804693   263.893402  17.044815       27.804173   233.660446  16.008617       27.8025          27.808453   247.052444  16.432692       27.80793    220.580353  15.527525       27.80625             27.812212   236.569031  16.07646        27.811687   245.471848  16.292435       27.81            27.815969   253.395264  16.601618       27.815447   241.774445  16.269558       27.81375             27.819729   263.030884  16.892704       27.819204   263.268127  17.067692       27.8175          27.823488   223.847458  15.520136       27.822962   260.296112  16.966854       27.82125             27.827248   211.959488  15.183031       27.826719   268.057983  17.255955       27.825           27.831007   247.711182  16.510641       27.830479   253.73201   16.721733       27.82875             27.834764   217.858749  15.366168       27.834236   237.723114  16.293709       27.8325          27.838524   244.396072  16.350092       27.837994   244.308258  16.311089       27.83625             27.842283   234.730118  16.015654       27.841751   242.190506  16.331738       27.84            27.846043   227.770309  15.7642     27.84551    230.195328  15.799245       27.84375             27.849802   228.575867  15.821275       27.849268   240.061096  16.118477       27.8475          27.853561   263.276672  17.068258       27.853025   206.348511  14.841624       27.85125             27.857321   250.560776  16.572704       27.856783   206.536362  14.8906     27.855           27.861078   239.218353  16.159765       27.860542   235.019913  15.935363       27.85875             27.864838   247.305298  16.384516       27.8643 236.672928  16.083691       27.8625          27.868597   252.726212  16.524702       27.868057   261.304108  16.969328       27.86625             27.872356   235.50386   15.968966       27.871817   231.001572  15.855899       27.87            27.876116   215.917114  15.295437       27.875574   247.365158  16.487011       27.87375             27.879875   213.492035  15.25972        27.879332   245.987518  16.360123       27.8775          27.883635   229.914215  15.745885       27.883091   253.841843  16.696445       27.88125             27.887392   229.421692  15.846799       27.886848   213.708725  15.311213       27.885           27.891151   236.662079  16.150654       27.890606   241.196442  16.296644       27.88875             27.894911   213.46492   15.187263       27.894365   239.802856  16.267569       27.8925          27.89867    237.516998  16.176313       27.898123   220.771957  15.506441       27.89625             27.90243    234.038055  16.001099       27.90188    242.23735   16.235064       27.9             27.906189   220.551498  15.560532       27.90564    206.81221   14.946399       27.90375             27.909948   250.02037   16.568855       27.909397   247.377335  16.324753       27.9075          27.913708   243.975861  16.354435       27.913155   228.919266  15.743267       27.91125             27.917467   239.123169  16.220346       27.916914   249.639938  16.543022       27.915           27.921225   243.559311  16.292747       27.920671   238.668793  16.155157       27.91875             27.924984   235.045685  16.003975       27.924429   244.874649  16.2843     27.9225          27.928743   235.891724  15.962676       27.928188   214.899094  15.256536       27.92625             27.932503   219.885284  15.408209       27.931946   254.30809   16.793606       27.93            27.936262   260.085693  16.952793       27.935705   211.037003  15.115447       27.93375             27.940022   242.985474  16.353039       27.939463   219.61763   15.5637     27.9375          27.943781   239.24585   16.262785       27.94322    211.325806  15.280083       27.94125             27.94754    231.566666  15.963982       27.94698    220.791534  15.542738       27.945           27.9513 260.656036  16.990906       27.950737   231.810532  15.878859       27.94875             27.955059   262.265228  17.098459       27.954496   227.132797  15.75332        27.9525          27.958818   215.683334  15.313498       27.958254   231.602097  15.932183       27.95625             27.962578   220.570541  15.561907       27.962013   264.902496  17.04768        27.96            27.966337   254.470688  16.804611       27.965771   232.680267  15.906054       27.96375             27.970097   244.5896    16.264879       27.969528   230.648224  15.933537       27.9675          27.973854   245.162506  16.36952        27.973288   225.606033  15.679212       27.97125             27.977613   234.010452  15.999166       27.977045   256.237213  16.792482       27.975           27.981373   213.115234  15.127      27.980804   228.330688  15.838371       27.97875             27.985132   219.218826  15.570371       27.984562   248.798157  16.584879       27.9825          27.988892   210.154175  15.121605       27.988321   208.729065  15.124132       27.98625             27.992651   197.702744  14.699856       27.992079   220.896011  15.620786       27.99            27.99641    234.040497  16.069506       27.995838   227.315704  15.766306       27.99375             28.00017    211.508118  15.221255       27.999596   224.617126  15.747621       27.9975                          28.003355   236.759277  16.157446       28.00125


title: Completely random or structured selection of receipients for an A/B mailing

text: and apologies if this is an overly simple question. I am doing an A/B mailing (where we send out two different solicitation letters). It's going out to past donors so we have some information on them and I have a master list of about 18,000 eligible donors. The point is to see which letter produces more and/or greater donations. Should I generate the two groups completely randomly, or should I sort them by amount given in the past year and manually assign them to two groups taking care that each group had the same distribution of people with various donation histories (same number of 'high donors', 'medium donors' and 'low donors'). What is the best way to create two equivalent groups so that examining the amounts donated later will have meaning ?


title: Correlated explanatory variables where both are significant

text: I am running a multiple linear regression using SPSS to test the effect of ethnicity and ethnic/racial attitudes or perceptions on political predispositions. One model, as an example, looks like this: DV: 'support for democracy' (1-7 scale) Controls: age, gender, education etc. Predictors: Ethnicity, and various attitude/perception vars My question is what to do with these attitude/perception variables when they are correlated; i.e. whether to leave all of them in the model, or remove one or more. A particular case is the pair of variables 'common national culture' ('Bolivians share many common values that unite us as a nation; 1-7 disagree- agree scale) and 'strength of national identification' ('To what extent do you identify as a Bolivian citizen?'; 1-7 scale). As you might expect, 'common national culture' and 'strength of national ID' are correlated (Pearson coefficient=.268 and is significant). But they also both have significant coefficients in the multiple regression, and adjusted R-squared for the model decreases substantially when either one is removed. In this case - i.e. when both correlated vars have significant coefficients -, should both be kept in? Many thanks in advance. VIF is between 1.2-1.4 for each. Condition indices have values of around 12, but there are 15 variables in the model, so perhaps this is not so remarkable? Looking through the variance proportions, however, neither have any values >0.2


title: Integrate first derivation - area under curve of first derivation

text: I want to calculate the area under the first derivate. Can I do that with splinefun()?               Year <- c(1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969)     Population <- c(4835, 4970, 5085, 5160, 5310, 5260, 5235, 5255, 5235, 5210, 5175)     sample <- data.frame(Year, Population)     sample     plot(sample, type='l')     spline <- splinefun(sample$Year, sample$Population)     plot(sample$Year, spline(sample$Year, deriv=1), type='l', axes=T)      Or is there another way to calculate the area under the first derivate?


title: Analysing a CV with machine learning

text: What would you use to categorise or rank a Resume/CV with machine learning? I would like to make particular CVs stand out to recruiters.


title: how can calculate $E(X^Y|X+Y=1)$

text: let $X,Y$ are two independent random variables Bernoulli with probability $P$. how can calculate $E(X^Y|X+Y=1)$


title: Understanding the difference between Supervised and unsupervised learning?

text: I have been reading about the **Supervised** and **Unsupervised learning**. What I came to know through this link is that in case of Supervised learning you have a set of input and a set of labels which are then used in the training phase whereas in case Unsupervised learning there are no labels. Now my basic question is:   1. Since there are no labels available in case of Unsupervised learning so does that mean that there is no training phase in unsupervised learning?    2. If there is a training phase in Unsupervised learning and we do not use any labels then what is the importance of training phase and how is it performed since we do not have any error or reward signal to evaluate a potential solution in training phase? Isn't it equal to the normal testing phase where we have the set of data but no labels and we predict the labels?    3. I was reading about a **Nonparametric bayesian modeling** method and the introduction said that-"The method is unsupervised and has no free parameter that require tuning". In this what is the meaning of no free parameter that require tuning? Is it that there is no requirement of training phase?  Please bear with my lack of understanding about these topics as I am new to this and would love if you could provide some explanation to clear out my doubts.


title: How to improve results when using sampling in skewed binary classification?

text: I am using a data set with **18 features** with True/False output (Related to mobile ad targeting). True values occurs only **0.4 %** of the time. So, I have used sampling to keep the ratio of True and False classes nearly 1:1. I have used logistic regression using python's scikit learn. However, now also I get an **f-score of about 0.57**. What could be the possible reasons for this and how to overcome these problems. P.S. : I have posted all the details in another question related to overcoming the skewness in the data set on crossvalidated: How to deal with a skewed class in binary classification having many features?.


title: LCA number of parameters & degrees of freedom

text: I have a series of physicians' claims submissions. I would like to perform cluster analysis as an exploratory tool to find patterns in how physicians bill based on things like Revenue Codes, Procedure Codes, etc. The data are all polytomous, and from my basic understanding, a latent class algorithm is appropriate for this kind of data. I am trying my hand at some of R's cluster packages, & specifically `poLCA` & `mclust` for this analysis. I'm getting alerts after running a test model on a sample of the data using `poLCA`.               > library(poLCA)     > # Example data structure - actual test data has 200 rows:     > df <- structure(list(RevCd = c(274L, 320L, 320L, 450L, 450L, 450L,      636L, 636L, 636L, 450L, 450L, 450L, 301L, 305L, 450L, 450L, 352L,      301L, 300L, 636L, 301L, 450L, 636L, 636L, 307L, 450L, 300L, 300L,      301L, 301L), PlaceofSvc = c(23L, 23L, 23L, 23L, 23L, 23L, 23L,      23L, 23L, 23L, 23L, 23L, 23L, 23L, 23L, 23L, 23L, 23L, 23L, 23L,      23L, 23L, 23L, 23L, 23L, 23L, 23L, 23L, 23L, 23L), TypOfSvc = c(51L,      51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L,      51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L, 51L,      51L, 51L, 51L), FundType = c(3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,      3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L,      3L, 3L, 3L, 3L, 3L, 3L), ProcCd2 = c(1747L, 656L, 656L, 1375L,      1376L, 1439L, 1623L, 1645L, 1662L, 176L, 1374L, 1376L, 958L,      1032L, 1368L, 1374L, 707L, 960L, 347L, 1662L, 859L, 1375L, 1654L,      1783L, 882L, 1440L, 332L, 332L, 946L, 946L)), .Names = c("RevCd",      "PlaceofSvc", "TypOfSvc", "FundType", "ProcCd2"), row.names = c(1137L,      1138L, 1139L, 1140L, 1141L, 1142L, 1143L, 1144L, 1145L, 1146L,      1147L, 1945L, 1946L, 1947L, 1948L, 1949L, 1950L, 1951L, 1952L,      1953L, 1954L, 1955L, 1956L, 1957L, 1958L, 1959L, 2265L, 2266L,      2267L, 2268L), class = "data.frame")          > clust <- poLCA(cbind(RevCd, PlaceofSvc, TypOfSvc, FundType, ProcCd2)~1, df, nclass = 3)          =========================================================      Fit for 3 latent classes:      =========================================================      number of observations: 200      number of estimated parameters: 7769      residual degrees of freedom: -7569      maximum log-likelihood: -1060.778           AIC(3): 17659.56     BIC(3): 43284.18     G^2(3): 559.9219 (Likelihood ratio/deviance statistic)      X^2(3): 33852.85 (Chi-square goodness of fit)           ALERT: number of parameters estimated ( 7769 ) exceeds number of observations ( 200 )           ALERT: negative degrees of freedom; respecify model       My novice assumption is that I need to run a greater number of iterations before I can get results that are robust? e.g. "...it is essential to run poLCA multiple times until you can be reasonably certain that you have found the parameter estimates that produce the global maximum likelihood solution." (http://www.sscnet.ucla.edu/polisci/faculty/lewis/pdf/poLCA-JSS-final.pdf). Alternatively, perhaps certain variables, particularly CPT & Revenue Codes, have too many unique values, and that I need to aggregate these variables into higher level categories to reduce the number of parameters? When I run the model using package `mclust`, which optimizes the model based on BIC, I don't get any such alert.               > library(mclust)     > clustBIC <- mclustBIC(df)     > summary(clustBIC, data = df)          classification table:           1   2           141  59            best BIC values:             VEV,2     VEV,3     EEV,3            -4562.286 -4706.190 -5655.783      If anyone can shed a bit of light on the above alerts, it would be much appreciated. I was also planning on using the script found in the `poLCA` documentation to run multiple iterations of the model until the log-likelihood is maximized. However it's computationally intensive and I'm afraid the process will crash before I have a chance to post this. Sorry in advance if I've missed something obvious here; I'm new to cluster analysis.


title: On the use of weighted correlations in aggregated survey data

text: I am analyzing data from two surveys that I merged together:   * School staff survey, for years 2005-06 and 2007-08   * School students survey, for years 2005-06 through 2008-09 For both of these data sets, I have observations (at the student or staff level) from 3 different school districts, each having representative samples per year within their distinct school district. For analysis, I combined the student data into two 2-year periods (2005-07 and 2007-09). Then I then I 'ddply'-ed each data set to obtain percentages of staff or students that responded to questions according to cutoffs (e.g., whether they answered in the affirmative, "Agreed", or whether the student marked that they used alcohol, etc.). So when I merged the staff and student level data sets together, the school is the unit of analysis, and I only have 1 observation per school per 2-year time periods (given that the school wasn't missing data for a given time period). My goal is to estimate associations between staff and student responses. So far, my plan was to obtain Pearson correlation coefficients between all the variables (as they're all continuous responses representing percentages) for each school district separately from each other (as this eliminates the generalizability assumption for the other districts in this data set). To do this, I would average the district data over the two years anyway to get just one observation per school. **Questions:**   1. Is this an appropriate analysis plan? Is there some other method I may use that could provide me better inference or power?   2. If my plan is appropriate, should I obtain weighted correlations based on school's enrollment (as there are more smaller schools than large that would be contributing disproportionally to the correlation coefficients)? I have asked the data administrator about this, and he mentioned that the main factors that determine the necessity for weighting my data is whether or not I think school size affects the degree of correlation and whether my interpretation will be at the student or school level. I think my interpretation will be at the school level (e.g., "a school with this percentage of staff answering this way is correlated to this percentage of students responding this way...").


title: What does it mean if all the coefficient estimates in a lasso regression converge to zero?

text: I attempted to run lasso on a 12 X 52 matrix (11 predictors) using this MATLAB function http://www.mathworks.com.au/help/stats/lasso.html. I found that the results converged to zero. How should I interpret this? In the interests of full disclosure, I'd like to state that this isn't what happens when I include my full set of predictors. However, it does happen when I remove some of the predictors that (subjectively) seem to be the better ones from my data set before running lasso.


title: Online algorithm for mean absolute deviation and large data set

text: I have a little problem that is making me freaking out. I have to write procedure for an online acquisition process of a multivariate time series. At every time interval (for example 1 second), I get a new sample, which is basically a floating point vector of size N. The operation I need to do is a little bit tricky:   1. For each new sample, I compute the percentuals for that sample (by normalizing the vector so that the elements will sum to 1).   2. I calculate the average percentuals vector in the same way, but using the past values.   3. For each past value, I compute the absolute deviation of the percentuals vector related to that sample with the global average percentuals vector computed at step 2. This way, the absolute deviation is always a number between 0 (when the vector is equal to the average vector) and 2 (when it is totaly different).   4. Using the average of the deviations for all the previous samples, I compute the mean absolute deviation, which is again a number between 0 and 2.   5. I use the mean absolute deviation to detect if a new sample is compatible with the other samples (by comparing its absolute deviation with the mean absolute deviation of the whole set computed at step 4). Since every time a new sample is collected the global average changes (and so the mean absolute deviation changes as well), is there a way to compute this value without scanning the whole data set multiple times? (one time for computing the global average percentuals, and one time for collecting the absolute deviations). Ok, I know it's absolutely easy to calculate the global averages without scanning the whole set, since I just have to use a temporary vector for storing the sum of each dimension, but what about the mean absolute deviation? Its calculation includes the `abs()` operator, so I need to access to all the past data! Thanks for your help.


title: Is it possible to accumulate a set of statistics that describes a large number of samples such that I can then produce a boxplot?

text: I must clarify immediately that I am a practicing software developer, not a statistician, and that my college stats class was a _very_ long time ago… That said, I would like to know if there is a method for accumulating a set of descriptive statistics that could then be used to produce a boxplot, that does not entail storing a bunch of individual samples? What I am trying to do is produce a graphical summary of queue service times within a complex multi-queue process. I have in the past used a package called tnftools that allowed large samples to be accumulated and then post-processed into a nice graph of response times and outliers… But tnftools are not available for my current platform. Ideally I would like to be able to accumulate a set of descriptive statistics "on the fly" as the process runs, and then extract the data for analysis on demand. But I cannot simply have the process accumulate samples as the memory / IO involved in doing so would have an unacceptable impact on the performance of the system.


title: What algorithm does ward.D in hclust() implement if it is not Ward's criteria?

text: > The one used by option "ward.D" (equivalent to the only Ward option "ward" > in R versions <= 3.0.3) does not implement Ward's (1963) clustering > criterion, whereas option "ward.D2" implements that criterion (Murtagh and > Legendre 2013). (http://stat.ethz.ch/R-manual/R-patched/library/stats/html/hclust.html) Apparently ward.D does not implement Ward's criteria propperly. Nonetheless it seems to do a good job regarding the clusterings it produces. What does method="ward.D" implement if it is not Ward's criteria? * * * http://adn.biol.umontreal.ca/~numericalecology/Reprints/Murtagh_Legendre_J_Classif_2013.pdf


title: Understanding the calculation of the independent samples Cohen's d from t & df

text: It seems that for calculation of Cohen's $d$ (an estimate of effect size), you should use the difference in means (or in one-sample case, from baseline), divided by $SD$, if that information is available: $$ \frac{\bar X_1 - \bar X_2}{SD} $$ However, when you only have the $t$-statistic and the degrees of freedom ($\rm df$) available, what is the formula for Cohen's $d$? In the one- sample / paired-sample case, it's intuitive how to derive $d$ from $t$, since $t$ is calculated as the difference in means over standard error: $$ t = \frac{\bar X_1 - \bar X_2}{SE} = \frac{\bar X_1 - \bar X_2}{\frac{SD}{\sqrt{N}}} $$ so $d$ equals: $$ d = \frac{\bar X_1 - \bar X_2}{SD} = \frac{t}{\sqrt{N}} $$ And that's what you find in online articles such as this one (pdf). But for independent samples, the equation is listed instead as: $$ d = \frac{2t}{\sqrt{\rm df}} $$ as seen here, or even: $$ d = t \sqrt{\frac 2 N} $$ So where does the "$2$" come from? Is it related to the formula for pooled SD?


title: With what probability the standard deviation of GP capture the measurement?

text: An interesting property of Gaussian Processes is estimating the uncertainty range. This uncertainty range of prediction can potentially capture the actual measurements. I am wondering, how many times or with what probability this uncertainty range is able to capture the measurement and why ?


title: What is being learnt?

text: This may seem a trivial question but I have a fundamental problem in understanding learning.   1. In supervised learning, given and input-output pair, what are we learning? Are we learning the inputs (say features) or the output? For example, In terms of linear system, let a system be y=Ax ; y is the output and x is the input. What is learning in this case?   2. In unsupervised learning, what are we learning since no outputs are known. All information that is available says that learning means finding weights. But what are we actually is not clear to me. In general what are we learning? Shall appreciate if the doubts are cleared.


title: How can I make clusters with highly right skewed data?

text: My histogram of expected income is as below. As you can see, income is highly right skewed. I want to divide individuals with regard to income - for example into Low, Middle, and High, and then I want to compare the other properties of individuals in each of the three groups. Any idea about dividing into groups with highly right skewed data? ![enter image description here](http://i.stack.imgur.com/Xzias.png)


title: What statistical test(s) should I use for this kind of repeated measures data?

text: Assume I have this kind of data: $ A_{11}\quad A_{21} \\ A_{12}\quad A_{22} \\ A_{13}\quad A_{23} \\ B_{11}\quad B_{21} \\ B_{12}\quad B_{22} \\ B_{13}\quad B_{23} \\ C_{11}\quad C_{21} \\ C_{12}\quad C_{22} \\ C_{13}\quad C_{22} \\ ...\quad ... \\ $ where $A$, $B$, $C$, ... represent different individuals.   1. For each individual $X$, I have measurements $X_{11}$, $X_{12}$, and $X_{13}$, which are three different measurements of a same physiological process P.   2. For each individual $X$, I have also three other measurements $X_{21}$, $X_{22}$, $X_{23}$, which are again three different measurements of a same physiological process Q.   3. The measurements ($X_{11}$, $X_{12}$, $X_{13}$) and ($X_{21}$, $X_{22}$, $X_{23}$) are measurements from two different physiological processes P and Q, that I would like to compare.   4. The measurements are real numbers. QUESTION: What kind of statistical test should I use to test whether there is a **significant difference between the two physiological processes P and Q** (The data for the **first process is in the variables $X_{1i}$** and the data for the **second process in variables $X_{2j}$** , as explained above)? My sample size is 10 individuals. Does the answer change if I had more measurements (e.g. two more columns to the example data)?


title: How to obtain solution of differential equation in this simple linear birth-death process?

text: (Apologies for the poor title, I didn't know what what to type in) I am having a problem with the second part of this question (an example question from a past stochastic course I took): * * * > Consider the simple linear birth and death process and the partial > diﬀerential equation > > $\frac{\partial G}{\partial t} = (λz − µ)(z − 1)\frac{\partial G}{\partial > z}$ > > satisﬁed by the p.g.f. > > $G(z, t) = \sum_{j=0}^{\infty} P(N(t) = j|N(0) = a)z^j$ > > Deﬁne > > $m(t) = E[N(t)|N(0) = a]$ for $ t ≥ 0.$ > >   1. By diﬀerentiating the partial diﬀerential equation partially with > respect to z and then setting z = 1, show that m(t) satisﬁes the diﬀerential > equation > > $\frac{dm}{dt} = (λ − µ)m $ > >   2. Deduce an explicit expression for m(t). > > * * * The solution for part (1) is easy enough by noting that $\frac{\partial^2 }{\partial z \partial t} = [λ(z-1)+(λz − µ)]\frac{\partial G}{\partial z}+(λz − µ)(z − 1)\frac{\partial^2 G}{\partial z^2}$ and at z = 1: $\frac{\partial^2 G}{\partial z \partial t} |_{z=1} = (λ − µ)\frac{\partial G}{\partial z}$ Now, m(t) looks like the first derivative of the p.g.f. at z = 1 and so $\frac{\partial G}{\partial z}|_{z=1} = \sum_{j=0}^{\infty}jP(N(t)=j|N(0)=a)z^{j-1}|_{z=1} = E[P(N(t)|N(0)=a)]=m(t)$ i.e. $ m(t)=\frac{\partial G}{\partial z}|_{z=1}$ and so $\frac{\partial^2 G}{\partial z \partial t} =\frac{\partial}{\partial t}\{\frac{\partial G}{\partial z}\} = \frac{\partial}{\partial t} \{m(t)\}= \frac{\partial m}{\partial t} = (\lambda-\mu)m$ as required. **However, the solution in my notes for part (2) is given as $m(t) = ae^{(\lambda-\mu)t}$ where $m(0) = a$ but I can't figure out how to do that.**


title: Pearson 5/Inverse Gamma/ Double Pareto

text: My current research deals with landslide size frequency distribution which has been fit to a three-parameter inverse gamma function and a double Pareto function by previous researchers. I am trained as a civil engineer and can't make sense of how to build a program to fit my data in Fortran, MATLAB or R because I'm having some difficulty finding simplified explanations. I have so far been unable to find any books specifically on a 3-parameter inverse gamma relationship, only 1- and 2-parameter. Questions:   1. What textbooks would you suggest for a new reader in statistics?   2. Is the Pearson V both the chi-squared AND the inverse gamma function?   3. Can anyone assist me in understanding how to build these programs?


title: Likert scale question divided into different group. How to calculate mean of different group?

text: I want to do a survey to determine how satisfied the (16) employees are with the company training program. The survey has 30 questions, each using a 5 point Likert scale for responses. The questions are divided into different groups (9 for the utility of the program, 6 regarding the trainer, 7 regarding the balance of the program, 2 about training content, 4 about the training facilities, and 2 about the implementation of training. I have calculated the mean, mode, frequency, percent and score of each question, but I need a result for a group of questions. Specifically, a result for the 9 questions regarding the utility of the program. I am currently thinking of using the mean or score of the group of the questions. What else I can do with this data? Please give suggestions.


title: Can I estimate the parameter of a Poisson arrival process from a low-incidence observation period?

text: If I know only that the arrival process is Poisson, and I observe it for a pre-chosen (say, unit) period of time, observing $k$ arrivals, is it meaningful to describe an estimate of its time parameter from the observation as (A) maximum-likelihood, (B) minimum-variance, or (C) unbiased if $k$ is very small? If $k$ is very large, then something like $1/k$ would be a pretty plausible estimate by any criterion. For moderate values of $k$, I guess that the answer is still positive, although more complicated. But I am interested in the case when $k=2$ or even $k=1$.


title: What are methods/software packages to create temporal matrices for agricultural practices?

text: I would like to create a temporal model showing the likelihood (I am sure this is not used in the correct context) that certain size avocados grow from smallest, small, bigger, biggest. I have considered using a linear regression, but it should be simplified, such that the parameters are entered and an output is created. In a succinct, farmer friendly manner. An example of the data that I am working with are typical California avocado sizes, 60 being the largest then 70 84 96 108 being the smallest We are assuming that the larger the fruit the longer each fruit remains on a tree, this is a linear trend based on our assumption that the fruit increase by one size every 30 days. Additionally, a weighted parameter with the number of trees where each size fruits are predominant. i.e. 1700 trees contain size 84 fruits. So considering these inputs, I would like to model in a temporal matrix form how each size increases over time, or the number of trees to be expected with an increases in size.


title: How to compare two samples on 25 variables, each designed to measure one of three underlying constructs?

text: I have designed a questionnaire to test which of three motivational constructs can best be used to describe learners motivation. 25 statements were designed to test the three constructs (each question relating to one of the three), they were randomly ordered and participants were asked to respond to each statement by ticking in the appropriate place on one of 6 points along a scale from strongly agree to strongly disagree. I had planned to use exploratory factor analysis to analyse the data, although now I wonder whether a different method of analysis might be better? I was told I would need about 300 responses for factor analysis to be effective/appropriate, and although I hope not to get much fewer than that - I am doing a comparative study with two categories of students, so expect to collect somewhere between 125-140 from each of the two categories. Could anyone advice me what methods of analysis might be more suitable? Many thanks!


title: What is the meaning of 'Marginal mean'?

text: My problem with understanding this expression might come from the fact that English is not my first language, but I don't understand why it's used in this way. The marginal mean is typically the mean of a group or subgroup's measures of a variable in an experiment, but why not just use the word mean? What's the marginal here for? See the definition of marginal from wiktionary.


title: What is wrong with this implementation of logistic regression (using iterative reweighted least squares)?

text: I am trying to implement logistic regression using the following algorithm:   1. fit a simple linear model $y \sim Xb_0$   2. calculate $W = \frac{e^{Xb_0}}{(1+e^{Xb_0})^2}$   3. calculate $z = Xb_0 + y \cdot \frac{(1+e^{Xb_0})^2}{e^{Xb_0}} - (1+e^{Xb_0})$ (the square and dot product means element-wise here)   4. calculate infomation matrix $J = X^T W X$   5. solve for a new coefficient vector $b_1 = J^{-1}X^T W z$   6. if difference between $b_0$ and $b_1$ not small enough, assign b1 to b0, and repeat step 2-5 This sometimes work, but sometimes $b_0$ and $b_1$ will not converge. I am suspecting the initial b value was not picked optimally. Any suggestions? I can also post the c++ code here, but that may not be very helpful.


title: Eigenvalue and eigen vector of covariance matrix in mean*mean'

text: I know the rank of $\Sigma^{-1}m m'$ is $1$. How can I find the eigenvalue and eigenvector of it? ($\Sigma^{-1}$ is the inverse of a covariance matrix and $m$ is a mean vector.)


title: Is $R^2$ valid in a nonlinear model?

text: The question is in the title. The coefficient of determination or $ R^2 \equiv 1 - \frac{ss_{res}}{ss_{tot}} $ is valid in a nonlinear model? Why?


title: News impact curves for various GARCH models in the rugarch-package

text: I'm working with the really nice `rugarch`-package and currently have an issue with respect to the news impact curves (NIC). In an attempt to plot several NIC into the same plot I realized that while the NIC for the sGARCH, the gjrGARCH and the eGARCH are given with respect to the $\epsilon_{t-1}$, the NICs for the submodels of the fGARCH model are given in terms of $z_{t-1}$, with $ z_t = \frac{r_t-\mu_t}{\sigma_t}= \frac{\epsilon_t}{\sigma_t} $ being the standardized innovations. So they can't be plotted together in a single plot using the same scales. I am a bit confused since just like the fGARCH model, the eGARCH model (as to the eGARCH-model-setup in the vignette) is also given in terms of the $ z_{t-k} , k={1,...,q}$. But nevertheless the NIC of the eGARCH is given in terms of $\epsilon_{t-1}$ . At least this is what the "newsimpact(fit)"-output tells me. I think that this might be a typo and it should mean "$z_{t-1}"$. Can anyone please help? The eGARCH vola equation is given by $ln(\sigma_t^2)= \omega + \sum_{k=1}^p \Big( \alpha_k \ z_{t-k} + \gamma_k\ (|z_{t-k}| - E|z_{t-k}|) \Big) + \sum_{k=1}^q \beta_k \ ln(\sigma^2_{t-k})$ And we obviously have $\alpha_k \ z_{t-k} + \gamma_k\ (|z_{t-k}| - E|z_{t-k}|) = \begin{cases} (\alpha_k + \gamma_k)z_{t-k} + C & z_{t-k} \ge 0 \\ (\alpha_k - \gamma_k)z_{t-k} + C & z_{t-k}<0 \end{cases} $ with the constant $C=-\gamma_k E[|z_{t-k}|]$. So the news impact depends on $\sigma_{t-1}$ which is nonconstant and therefore cannote be ignored or set to a fixed value for the NIC to make a general statement. Rather it has to be considered as being random along with the $\epsilon_{t-1}$ and so we have a NIC that depends on $ z_t = \frac{\epsilon_t}{\sigma_t} $. This is why I think that there might be a typo in the rugarch output. Is there an error in my thinking? Many thanks and kind regards, Jo


title: How to derive the Projected normal distribution

text: Suppose we have a bivariate normal variable $\mathbf{x}= (x_1, x_2)$ with mean $\mu_1$ and $\mu_2$ and variances $\sigma_1^2$ and $\sigma_2^2$ and correlation $\rho$. I need to obtain the pdf of the transformation $||\mathbf{x}||^{-1} \mathbf{x}$: the coordinates of a variable in the unit circle. In directional statistic this is a well known distribution called the Projected normal, or offset normal. I have already the pdf but i want to derive it by myself. I'm able to do a variable transformation when there is a 1 to 1 relation but this is not the case. Can someone put me in the right way? Just for completition: http://www.sciencedirect.com/science/article/pii/S1572312712000457 A pdf where it is explained the projected normal distribution


title: Modeling conditional variance (heteroskedasticity) using the gamma distribution

text: The application is that I want to know how $X$ maps not only to $y$, but to the variance of $y$. I think I've worked out a reasonable solution for doing so using the gamma distribution and the inverse square link, but I've got a couple of questions. The setup is as follows: $$ y = f(X) + \epsilon\\ \epsilon = \phi h(X)\sigma $$ $\phi$ is an unknown scale parameter, $h(X)$ is a deterministic function, and $\sigma$ is a unit-variance normal random variable. I fit the first part using OLS. Coefficients are consistent in the presence of heteroskedasticity, and I'm not worried about intervals yet. Back to the variance. I square everything and take the expectation, subbing in the estimated residuals: $$ E[\hat\epsilon^2] = \phi^2h(X)^2 $$ The sigma drops out because the expectation of a squared normal is the variance of the normal, which is 1. Here is where I am less sure of the way forward. If I take $h(X)^2$ to be $1/\mu$, and use an inverse square link function, then I have $\mu= h(X)$. This I can estimate with a GLM (and when I do so the diagnostic plots look pretty good). If I estimate the GLM using ML (as opposed to the GCV or REML options in GAM), I can extract the ML estimates of the scale parameter, which is $1/\phi$. Going back to the original model, I can get the expected value of the residual at a given $X$ by taking the square root of $\phi^2\hat h(X)^2$. And if I want to improve the efficiency of my OLS estimates, I can use the estimates as weights and go GLS, and then iterate further until approximate convergence. Anything wrong with this? Gamma is confusing and there are two different parameterizations that arbitrarily involve inverting one thing and not another. Is the use of the inverse square link valid here, in the sense of, "am I understanding the role of the link function correctly"? Seems a lot less intuitive than with e.g. logit for binomial data. Also, practically, when I fit the gamma GLM (or GAM) in `mgcv` and then ask for `mod$scale`, am I getting $\phi$ or $1/\phi$? Lastly, is there any way to know whether the iterative GLS/gamma GLM procedure that I propose converges to truth after some fashion?


title: How to choose the cutoff probability for a rare event Logistic Regression

text: I have 100,000 observations (9 dummy indicator variables) with 1000 positives. Logistic Regression should work fine in this case but the cutoff probability puzzles me. In common literature, we choose 50% cutoff to predict 1s and 0s. I cannot do this as my model gives a maximum value of ~1%. So a threshold can be at 0.007 or somewhere around it. I do understand `ROC` curves and how the area under curve can help me choose between two LR models for the same dataset. However, ROC doesn't help me choose an optimum cutoff probability that can be used to test the model on an out-of-sample data. Should I simply use a cutoff value that minimizes the `misclassification rate`? (http://www2.sas.com/proceedings/sugi31/210-31.pdf) Added --> For such a low event rate, my misclassificiation rates are affected by a huge number of false positives. While the rate over all appears good as total universe size is also big, but my model should not have so many false positives (as it is an investment return model). 5/10 coeff are significant.


Calculate the expected return, variance and standard deviation for the following portfolio of four stocks Stock1 16.4% return Stock2 -9.2% return Stock3 7.9%return Stock4 22.0% return


title: Confusion about Moving Average(MA) Process

text: Let's assume there is a time series Y of length n with Y(1) being the most recent observation. In MA process we plot auto-correlation function (ACF) to see how many lags to use. If we look at MA(3) process for example that implies the correlation between:               Y(1:n-1) and Y(2:n)     Y(1:n-2) and Y(3:n)     Y(1:n-3) and Y(4:n)      are significant. So shouldn't MA process look something like:               Y(t) = a0 + a1.Y(t-1) + a2.Y(t-2) + a3.Y(t-3)      instead of:               Y(t) = a0 + a1.ep(t-1) + a2.ep(t-2) + a3.ep(t-3) where ep ~ N(0,1)      After all we see a direct correlation between Y(t) and it's lagged observation.


title: How to calculate the covariance between two observations of the same variable?

text: I'm getting confused about what I read about covariance. I know how to calculate covariance between two different variables, but not between two observations of the same variable. Imagine you have many observations of a variable $X$. What does this mean and how to calculate it? $$ {\rm Cov}[x_i,x_j] $$ Or, which seems the same problem to me, in a simple linear regression when it is said that $$ {\rm Cov}[\varepsilon_i,\varepsilon_j] = 0 $$ for $i \neq j$. How do you calculate this?


title: Clustering of large, heavy-tailed dataset

text: I have a dataset of 130k internet users characterized by 4 variables describing users' number of sessions, locations visited, avg data download and session time aggregated from four months of activity. Dataset is very heavy-tailed. For example third of users logged only once during four months, whereas six users had more than 1000 sessions. I wanted to come up with a simple classification of users, preferably with indication of the most appropriate number of clusters. Is there anything you could recomend as a soultion?


title: Explain data visualization

text: How would you explain data visualization and why it is important to a layman?


title: Pseudo-random number generation algorithms

text: What algorithms are used in modern and good-quality random number generators?


title: Is it correct to compare two distributions of observed (that are means of observation) and expected values with chi square?

text: I have 2 classification methods that I want to compare. So I ran them on the same dataset and I obtained 2 different classifications. Then, I want to test if their classification is robust and how. Supposing each classification gives $n$ classes, for each method (classification):   * I created $m$ datasets for each class, where the class is cut to a half (50%). So I have $m$ datasets for class $i$, where half of the item classified as $i$ in the complete classification are removed; $m$ datasets for class $i+1$; and so on.   * I ran each method on its $nm$ datasets and I obtained $nm$ different classifications. Now I want to check if one of the two competitors, given that for each class I removed half of samples in at least one run, tends to classify always the same proportion of samples. For example:               Complete-Classification    1   2   1   1   3   2   1      If I cut half of class 1, I have 2 items classified as 1.               Remaining samples          1   2           3   2   1      If I run one of the methods again (but in one new dataset without the two samples - like that just shown) I am expecting to obtain a number of 1-classified samples that is near to 2, because I know from the complete classification that there should be 2 samples classified as 1. In fact, this is not the case, each one of the two methods tends to balance the distribution, classifying more than 2 samples as 1. Also, I test $m$ different datasets for class 1, such as               Remaining samples             2    1       3   2   1          Remaining samples             2    1   1   3   2         in order to better estimate the effect using different samples. Given that I want to penalize the method that overestimate the class cut, what can I do? I thougth about:   * I know that for class $i$ I expect $k$ classified items;   * I calculate the mean of the estimates for each class (I do not know if it is correct here to think that the distribution is normal - but I think it is reasonable);   * For each method I have, expected values (half-size of the classes in the complete classification) and observed values (mean of the estimates for each class). Now, I am stuck. I am thinking to:   * Use $\chi^2$ test (assuming the normality of the distribution of estimates, for each class);   * Alternatively, for each class compute the fold-change between mean of estimates and "correct" value; but here I do not know how to summarize the $n$ fold-changes to a unique value. What method do you suggest? Hope it is clear. Thanks


title: How to Test if the Difference between Two Means from Different Samples are Significant?

text: here are distributions of four samples:![](http://i.stack.imgur.com/LE7jO.jpg) and the following is the table of data: ![enter image description here](http://i.stack.imgur.com/aG1Th.jpg) I would like to test: (1) if the difference between means of "Give" and "Work- Give" samples is significant; and (2) whether the difference between means of "Take" and "Work-Take" samples is significant. As you can see, distribution of the data are non-standard.


title: Sampling methods and parallelization

text: A couple of years ago I learned about recent work in parallelizing slice sampling methods. More recently, I have read great things about NUTS and Hamiltonian Monte Carlo methods (HMC) in general (e.g. pymc3 uses NUTS) This makes me wonder:   * What are NUTS and HMC **particularly good** for? Can they benefit from parallelization as much as, e.g. slice sampling?   * How do they compare to elliptical slice sampling and other methods in general?


title: comparing normal distributions using a two sample kolmogorov-smirnov test

text: I've been redirected here from another forum... I have used a two sample Kolmogorov-Smirnov test to compare the distributions of two sets of data. Basically I am comparing the error distributions between two measurements when an intervention is made, to determine whether the intervention (a changing measurement parameter) significantly changes the error distribution of the measurement. I know that the K-S test is a non parametric test, however the distributions of data I'm comparing has turned out to be normally distributed... I know there is probably a number of tests that could be used to compare normally distributed data, but is there a reason not to use the K-S test? Are there any disadvantages (with regard to type1 and 2 errors perhaps)? Is it ok to use it? I've sort of gone down this route with my data analysis, but the question has come up: why use a non parametric test to compare parametric data? Hopefully K-S is unconventional rather than completely wrong.


title: Interpreting ordinal regression result and calculating individual percentage contributions of independent variables

text: I ran an ordinal logistic regression in R using the polr function on a survey analysis dataset. The responses of the dependent variable range from Poor to Excellent. The responses to the independent variables range from 1 to 5 (1 being Poor and 5 being Excellent). I obtained the following result: ![enter image description here](http://i.stack.imgur.com/1ZBij.png) I want to measure the individual percentage contribution of my independents (R1, R2,...,R17) to the dependent variable. Is there a way to do this. Thanks for any help.


title: Library routine for rolling window lag 1 autocorrelation?

text: I am looking for a library routine that will calculate the lag 1 autocorrelation of a time series with a rolling window; meaning "slide a window of size N points along the time series, calculate the lag 1 autocorrelation for each window." I have implemented an algorithm inspired by Wikipedia but would like something to compare the results with. Is the mentioned routine available in for example R or Python?


title: How to determine the best mean to use for a developmental comparison using z-scores?

text: The data contains two groups and the intended comparison will be completed via independent t-test. Options that I have considered is using the entire sample's mean or one of the group means. However, I am concerned about inflating the differences between the groups. Which method is most ideal and why?


title: Poorness of Kernel methods on visual pattern recegnition?

text: I am currently reading the recent papers mainly written by Y. Bengio [1],[2],[3]. There are very strong claims about poorness of Kernel methods on recognizing handwritings in many general cases but there are no references for the claims of this poorness. I want to know if this is really the case in Machine Learning research. e.g.: " Unfortunately, devising such similarity measures, even for a problem as basic as digit recognition, has proved difficult, despite almost 50 years of active research. Furthermore, if such a good task- specific kernel were finally designed, it may be inapplicable to other classes of problems."


title: Hidden Markov models and anomaly detection

text: In Shane's answer to this question he suggests that Hidden Markov Models can be used more successfully than wavelets for anomaly / change detection (it was a bit unclear -the topic he was addressing is anomaly detection, although he uses the words "change detection") I am not very familiar with Hidden Markov Models, but as I understand it, they require a known Markov process (all states and transition probabilities known) and for each state a known set of emission probabilities. The really interesting thing that can be done with these is that given a sequence of emissions one can find the most likely sequence of states that would have led to those emissions. Assuming that I am understanding HMM correctly (please correct me if I am wrong), how is this used for anomaly detection? How would one determine the underlying Markov process to use and the emission probabilities to use an HMM for anomaly detection?


title: Offline training or batch wise training

text: Can somebody please explain how to train a neural network in batch mode. I have a single target time series of length $N$ for a given input time series of the same length. In order to apply Hopfield network's content addressable property, I want to store this pattern of input time series. Then given a test time series that is a noisy version, how do I apply Hopfield neural network with Particle Swarm or Genetic algorithm as a learning method with the MSE as the fitness function? All this is confusing, since I am well practiced with incremental training and not batch wise training. PSO or GA update their equations for every input, so how do I adapt this with batch wise training. Insights on this technique will help to clear away the confusions. Thank you


title: Regression model of large, correlated, heavy-tailed data

text: I have a large panel-like data set: about 15,000 individuals, on average 350 time points, two dozen variables (plus some more variables we left out for context-specific reasons). What I want to achieve is a model explaining one dependent variable $y$ by (some of) the other (i.e. explanatory) variables $x$. As a first step, this should be something like a linear regression model. ## Properties of the data:   * $y$ follows a non stationary process (not rejected by Dickey–Fuller test) and increases linearly over time on average.   * One of the explanatory variables $x_1$, with $x_1(t) \geq x_1(t-1)$, is way stronger correlated with $y$ than any other variable.   * Most of the variables do not follow a Gaussian distribution, some (including $y$ and $y^\prime$) have a heavy tailed distribution.   * Some of the variables are strongly correlated. ## What I've done/calculated so far:   1. Pearson correlation of $y$ with each $x_k$ separately.   2. Pearson correlation of $y$ with each $x_k$, controlling for $x_1$.   3. Sampling only every $n^{th}$ time point ($n$ chosen so that the autocorrelation of $y$ has decayed below 1/e), and doing simple linear regression on the sample, using time as just one more variable. Regression of $y$ on $x_1$ yields $R^2=0.36$; including all variables yields $R^2=0.40$.   4. Taking $y^\prime$, i.e. $y(t+1)-y(t)$ as dependent variable. The autocorrelation of $y^\prime$ decays really fast. Regression of $y^\prime$ on $x_1^\prime$ yields $R^2=1.5\cdot 10^{-4}$, regression of $y^\prime$ on all $x_k$ yields $R^2=2\cdot 10^{-4}$.   5. Taking $\log(y(t+1)/y(t))$ ('log return') as dependent variable. Regression on all $x_k$ yields $R^2=0.003$; regression on $\log(x_1(t+1)/x_1(t))$ yields $R^2=0.0012$. Despite the low $R^2$, the F-test yields a very low p-value and most p-values for the coefficients are very low. All regressions include a constant term. ## What I'm looking for:   * In the best case a (pointer to a) 'standard' way to derive a model from data of this size (and kind), and a quantification of how much sense the resulting model makes.   * Criteria which of the explanatory variables to use in a model (the size of the data shows even really small effects significantly).


title: Analysis of a mixed design with categorical and continuous variables

text: I have no idea how to do the following analysis for an experiment. For my independent variables I have two continuous between-subject variables (personality traits scores), one categorical between-subject variable (gender), and one categorical within-subject variable (nature of the stimulus). My dependent variable is a continuous one. What kind of test should I use? My research seems to orient me towards mixed-models but I'm unsure whether they can handle both continuous and categorical variables as predictors. Thanks a lot


title: How to validate % answering Yes/change exists is enough to say population is seeing a new pricing model?

text: I have a survey on pricing. There are questions where there is an A and B choice where the proportions are not independent (mutually exclusive group). Here I can use a binomial distribution in Excel and can assume a probability of 50%. This will tell me with 95% confidence interval if the % selecting A is higher and I can reject the Null. However, I also have questions like the following where I am asking the type of pricing model they currently have: Are standard sized items charged the same or differently? Answer options are effectively Yes or No. I want to know if there is a statistically significant difference between the % selecting Yes and the % selecting No. I also want to be able to say if the % selecting Yes is statistically significant in that there is enough selecting Yes to indicate with 95% or 90% certainty that a proportion of the population does see a different/new pricing model (e.g., selected Yes in the question above). Given my knowledge of the market, I know very few people should answer Yes. However, with a sample size of 80, I have 59% answering No and 41% answering yes. In short, a % of the sample answered wrong. However, I want to at least say that the % answering yes is large enough to determine validates that at least some proportion of the sample is seeing a change. What the best stat tests are and how to run in Excel and how?


title: How to assess whether students' responses to a test are affected by changing the format of the test

text: We had students take a test with 20 questions and had them indicate their confidence levels for each question. In one test format, the confidence prompt was adjacent to their test question while in the alternate format, their responses to the test question and confidence had to be indicated on a separate form. The confidence responses are Likert scale data (not at all confident....highly confident, coded 1-5) while the test responses are dichotomous (correct/incorrect, coded 1/0). The tests only differ in the way the confidence survey was answered; test content and items are exactly the same. We are unsure of whether the change in format might have had an impact on their confidence or test responses, but would like a simple method to check for this. There are definitely other factors that go into affecting their responses, but any recommendations on how we might be able to make any judgements about the effects of this format change?


title: How to standardize an array if standard deviation is zero?

text: I am trying to standardize dataset columns for linear regression. One of the columns have standard deviation = 0.               def standardize(X):         return (X - mean(X)) / std(X)       So this code doesn't work. Are there any tricks to solve this problem? I have tried two things   1. Throw aways the column with standard deviation 0 because it's a useless parameter.   2. Add some very small noise like $10^{-10}$ to one of the elements of the column so that the `standardize` function works.  Thank you!


title: MAPE is high for daily sale prediction

text: I have daily sales data from 2011 to 2013. I have to do prediction for 2014.I have used arima and exponential method to predict the daily sale, but it is not giving the better result. MAPE is around 25%. `y=ts(x,frequency=7)` `fit <\- auto.arima(y)` `fc <\- forecast(fit, h=265)` `plot(fc)` `fit <\- ets(y)` `fc <\- forecast(fit,h=265)` Below is the link for data: https://drive.google.com/file/d/0B5en_TDcZWi3QWNjMzd3ZkdIcW8/edit?usp=sharing Is there a way to improve the MAPE? I am new to time series,I would appreciate any kind of help.


title: Join parameter estimation is always better than independent parameter estimation. True?

text: I have the vague recollection of a result that states something along the lines of > Joint parameter estimates always lead to smaller variance than independent > parameter estimates even when the parameters are uncorrelated. Does this ring a bell for anyone? What is the more accurate statement of the result, and where can I find a reference?


title: Type III sum of squares from SAS and R

text: I'm analyzing data from an unbalanced factorial experiment both with `SAS` and `R`. Both `SAS` and `R` provide similar Type I sum of squares but their Type III sum of squares are different from each other. Below are `SAS` and `R` codes and outputs.               DATA ASD;     INPUT Y T B;     DATALINES;      20 1 1      25 1 2      26 1 2      22 1 3      25 1 3      25 1 3      26 2 1      27 2 1      22 2 2      31 2 3     ;          PROC GLM DATA=ASD;     CLASS T B;     MODEL Y=T|B;     RUN;      **Type I SS from SAS**               Source  DF       Type I SS     Mean Square    F Value    Pr > F     T       1     17.06666667     17.06666667       9.75    0.0354     B       2     12.98000000      6.49000000       3.71    0.1227     T*B     2     47.85333333     23.92666667      13.67    0.0163      **Type III SS from SAS**               Source  DF     Type III SS     Mean Square    F Value    Pr > F     T       1     23.07692308     23.07692308      13.19    0.0221     B       2     31.05333333     15.52666667       8.87    0.0338     T*B     2     47.85333333     23.92666667      13.67    0.0163      **R Code**               Y <- c(20, 25, 26, 22, 25, 25, 26, 27, 22, 31)     T <- factor(x=rep(c(1, 2), times=c(6, 4)))     B <- factor(x=rep(c(1, 2, 3, 1, 2, 3), times=c(1, 2, 3, 2, 1, 1)))     Data <- data.frame(Y, T, B)     Data.lm <- lm(Y~T*B, data = Data)     anova(Data.lm)     drop1(Data.lm,~.,test="F")       **Type I SS from R**               Analysis of Variance Table          Response: Y               Df Sum Sq Mean Sq F value  Pr(>F)       T          1 17.067  17.067  9.7524 0.03543 *     B          2 12.980   6.490  3.7086 0.12275       T:B        2 47.853  23.927 13.6724 0.01629 *     Residuals  4  7.000   1.750                       ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1       **Type III SS from R**               Single term deletions          Model:     Y ~ T * B            Df Sum of Sq    RSS     AIC F value  Pr(>F)       <none>               7.000  8.4333                       T       1    28.167 35.167 22.5751 16.0952 0.01597 *     B       2    20.333 27.333 18.0552  5.8095 0.06559 .     T:B     2    47.853 54.853 25.0208 13.6724 0.01629 *     ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1       Am I missing something here? If not which one is correct Type III ss. Thanks in advance for your help.


title: Averaging images

text: I want to see if it is possible to analyse openly available images (from google, flickr, etc.), and conclude something from them (the same way one might do this from census data but with images as data). In my opinion online sources offer a wealth of data, and can easily be grouped by subject (for example: by searching for a specific term on google: "my house"). And i think averaging these images could tell us something about the way most people tend to take pictures of certain subjects (our pets, our family members, etc.). That in turn could possibly tell us something about the way we perceive those subjects, and hopefully even more... What i am going to do is collect as much images as possible about several readily available, simple subjects (i.e. "house", "portrait", etc.). Resize them so they have equal sizes, and calculate the average of those images pixel by pixel (mean, median or mode, i am still deciding). Ideally i would also try to align the subjects (so that for the keyword "house", all the houses are directly on top of eachother, and the same size). **I am very excited about the possibilities, but am curious to know what the experts think. What are the possible pitfalls in doing this? Does this idea make sense to you? Any suggestions or critique?**


title: Variance matrix with equal diagonal entries in PROC MIXED

text: Is it possible to specify a variance matrix for the random effects in PROC MIXED with the only restriction that the diagonal entries are equal ? I took a look in the help and I have not found, but this seems strange because such a variance structure is rather natural in some applications. (EDIT) Consider for instance such a dataset:               > dat        Subject Dose   y     1        1    A  10     2        1    A  11     3        1    A  12     4        1    B  30     5        1    B  31     6        1    B  32     7        1    C 100     8        1    C 101     9        1    C 102     10       2    A  11     11       2    A  14     12       2    A  13     13       2    B  33     14       2    B  37     15       2    B  36     16       2    C 105     17       2    C 110     18       2    C 109     19       3    A   9     20       3    A  11     21       3    A  12     22       3    B  30     23       3    B  35     24       3    B  32     25       3    C 115     26       3    C 101     27       3    C 102      ![dataplot](http://i.stack.imgur.com/tKTNV.png) and the following model:               PROC MIXED DATA=dat ;         CLASS SUBJECT DOSE ;         MODEL y = DOSE ;         RANDOM DOSE / subject=SUBJECT type=MYMATRIX ;     RUN; QUIT;      I want a matrix "MYMATRIX" with the same variance for each level of the DOSE factor, but not a compound symmetry matrix because the correlation between the means of the levels are different. (EDIT2) The mathematical meaning of this model is the following one. Denoting by $i$ the index for the dose level and by $j$ the index for the subject, one has $$(y_{ijk} | \mu_{ij}) \sim_{\text{iid}} {\cal N}(\mu_{ij}, \sigma^2_w), \quad k=1, \ldots, 3 \quad \text{ for all } i,j$$ and $$ \begin{pmatrix} \mu_{1j} \\ \mu_{2j} \\ \mu_{3j} \end{pmatrix} \sim_{\text{iid}} {\cal N}_3 \left( \begin{pmatrix} \mu_1 \\ \mu_2 \\ \mu_3 \end{pmatrix}, G \right), \quad j=1, \ldots, 3$$ The diagonal entries of the $G$ matrix are the between variances for each level of the dose. I want $G$ to be of the form $$G=\begin{pmatrix} \sigma^2_b & \sigma_{12} & \sigma_{13} \\ \sigma_{12} & \sigma^2_b & \sigma_{23} \\ \sigma_{13} & \sigma_{23} & \sigma^2_b \end{pmatrix}$$