title: Relation between coefficient of determination, $R^2$, and significance in a regression

text: I have performed a simple regression of the following form: $y_t$ = $\beta_1x_{1,t}$ + $\beta_2x_{2,t}$ + $\beta_3x_{3,t}$ the $R^2$ turns out to be 38%. $\hat\beta_1$ = 0.7 (significant at the 1%-level) $\hat\beta_2$ = 2.3 (significant at the 1%-level) $\hat\beta_3$ = 4.8 (not significant because of a high standard error) In terms of interpretation, I understand what the $R^2$ and the coefficients $\hat\beta_1$ -$\hat\beta_3$ mean separately. But I don't understand how to interpret them together. Is the following wording correct: The explanation power of the model (38%) stems mainly from $\hat\beta_1$ and $\hat\beta_2$. Or is there no relation between the significance of the coefficients and the $R^2$ in a regression? Thanks!


title: Precision, Recall and area under ROC curve as sample size increases

text: The following is a question from an exam paper on evaluating the performance of search engines. To this day I looked in my text book and literally close to 50 web pages and I can't   find one convincing argument for any of the cases. Can anyone help shed a light on this? You have developed a new retrieval algorithm and want to evaluate its performance. To this end, you have crawled one billion webpages. The experiments take too long with your current infrastructure, so you randomly sample 10% of the data, run 100 queries on the sample and ask human subjects to assess the relevance of the top 100 results. After averaging, you observe the following mean recall and precision at diﬀerent ranks:               Rank:        1   2     3    4    5  ...  10   20  ...  50  ... 100     Recall:    0.09 0.15 0.20 0.25 0.30 ... 0.50 0.70 ... 0.90 ... 1.00     Precision: 0.90 0.75 0.67 0.63 0.60 ... 0.50 0.35 ... 0.18 ... 0.10      Consider re-running the same experiment without sampling the data. Do you expect the following numbers to increase, decrease or stay the same:               i. Recall at rank 10.     ii. Precision at rank 10.     iii. Precision at 50% recall.     iv. Mean average precision.     v. Area under the ROC curve


title: Computing a bootstrap confidence interval for the prediction error with the percentile and the BCa method

text: I have two related questions regarding the computation of a non-parametric bootstrap confidence interval for the prediction error. **Setting:** I have a sample S from a data population P and a learner L, and I want to compute a 95% confidence interval for the 0.632 bootstrap estimator $\hat{\theta}_{.632bs}$ of the prediction error $\theta$ of the classifier C learned by learner L on sample S. **Q1** My first question is whether the following procedure, to compute the confidence interval with the percentile method, is correct. Specifically, the sampling of a test set $S_{test}$ in step 2 to evaluate my classifier on. I have read that, for every observation, you have to keep track of predictions from bootstrap samples not containing that observation. This seems to come down to the same thing according to me? I compute the confidence interval with the percentile method as follows:   1. I compute the resubstitution error $\theta_{resub}$ of the classifier C learned by L on S   2. I compute N (e.g. = 1000) bootstrap estimates $\hat{\theta}_i^*$ of the prediction error on S as follows: For i = 1 : 1000 do:     * Sample from S with replacement until I have a new sample $S_{train}$ the same size as S     * From all the instances that were not sampled in step 1, I sample again with replacement until I have a sample $S_{test}$ with the same size as S     * I learn a classifier on $S_{train}$ and evaluate it on $S_{test}$. The classifier performance on $S_{test}$ is my bootstrap estimate $\hat{\theta}_i^*$.   3. For each sample i of the N bootstrap samples I compute the estimate $\hat{\theta}^*_{i,.632bs}$ as $\hat{\theta}^*_{i,.632bs} = 0.368 \cdot \theta_{resub} + 0.632 \cdot \hat{\theta}_i^*$   4. I sort the 1000 bootstrap estimates   5. I select the 25th bootstrap estimate as the lower bound of the confidence interval and the 975th estimate as the upper bound of the confidence interval. **Q2** My second question is how to compute the confidence interval for the 0.632 bootstrap estimator for the prediction error with the bias corrected accelerated (BCa) method. Namely, I find that I have a problem here: I need an estimate of a term $b$ for the bias correction, and for term $a$, the acceleration term (http://www.tau.ac.il/~saharon/Boot/10.1.1.133.8405.pdf p1153-1154).   * To compute $b$, I need an estimate of the prediction error $\hat{\theta}$ from the complete sample S. I am not sure how to do this. I see several possibilities: The average of $\hat{\theta}^*_{i,.632bs}$ over all $i$, the resubstitution error, a hold-out error, perform cross-validation on S...?    * I have the same problem for computing $a$. Here, I need jackknife estimates that use all of the data from S except one instance $x_i$ ($S \setminus x_i$). What exactly do I compute here? Is this computation the same as computing the leave-one-out cross-validation estimator for the prediction error? I have tried finding the answers in the literature, but I have difficulty understanding the papers I found, so I hope somebody can help me here. Thank you!


title: Chart/visual for negative binomial regression

text: My wife is presenting a study at a conference poster session. She has a correlation of sorts [1] where kids' self-assessment of asthma ("I felt a lot better" or "I felt a little better or not better") matches doctors' assessments (0 to 6, low numbers are better). And we need a visual for the poster. I'd have thought that a correlation graph would be the way to go, but the x- and y-values are both on discrete scales, so if you plot them, (1) there is no cloud of values to see and (2) there is no obvious trend. Here are the buckets (value + count of occurrences) for people who felt a lot better:               >>> print(a_lot_better)     {0: 66, 1: 9, 2: 3, 3: 2}      And here are buckets for people who felt a little better or not better:               >>> print(a_little_better)     {0: 79, 1: 31, 2: 5, 3: 6, 6: 1}      If you sum up the numbers like this:               # python     def print_dict(d):         total = float(sum(d.values()))         composite = 0.0         for k, v in sorted(d.iteritems())    #def test_auth(self):     #    logger.debug("Calling search")     #    data_args = {"auth_token": self.auth_token, "extra_arg": "asd"}     #    response = self.client.call_url("profile_auth", data=data_args)     #    #logger.debug("response: {0}".format(response.reason))     #    self.assertEqual(response.status_code, 200):             print("{0}: {1}".format(k, v / total))             composite += (k * v) / total         return composite      You get this:               >>> print_dict(a_lot_better)     0: 0.825     1: 0.1125     2: 0.0375     3: 0.025     0.2625          >>> print_dict(a_little_better)     0: 0.647540983607     1: 0.254098360656     2: 0.0409836065574     3: 0.0491803278689     6: 0.00819672131148     0.5327868852459017      And that is more obviously different, but I don't know how to present this visually (assuming even that this is a legitimate summary). **How should we present this data to be visually obvious that there is an interesting and significant relationship?** [1] As broadly understood by a statistics-civilian.


title: Book on Repeated Measure Analysis

text: Can anyone recommend a good book or some other reading materials on repeated measure analysis using mixed model.Thanks. Hanna


title: FDR adjustment question using p.adjust

text: I have the following vector below. Can someone please provide more guidance regarding why when I run `p.adjust(myvector, "fdr")` all the adjusted p-values are the same? The vector in dput form is below:               c(0.658975448557642, 0.708514617655399, 0.634146815436502, 0.59915577368773,      0.0390852032651171, 0.0702524399267331, 0.419421995201681, 0.600357948476972,      0.400134025835294, 0.708454850340469, 0.62446245761038, 0.249703639842798,      0.257879620454186, 0.11223927420634, 0.592487461544245, 0.616748268116069,      0.345584989682594, 0.237762512706925, 0.0960091818372293, 0.523632628303304,      0.308932413101238, 0.638969691765434, 0.638875644596232, 0.181608205783583,      0.636321687370651, 0.577013517115353, 0.385609997133316, 0.719491921620221,      0.51265687942818, 0.656037028226805, 0.700460705514884, 0.554204126179901,      0.646056223582517, 0.688745977816702, 0.578989778124751, 0.422176774762902,      0.724861002173553, 0.105635695450417, 0.589474674278154, 0.370969243089851,      0.618464940424427, 0.613214527389695, 0.579463939838418, 0.600348866772194,      0.184066799238387, 0.266492584024551, 0.505286901367009, 0.325404423867325,      0.159381833338919, 0.558527627896565, 0.355350232909787, 0.494592173745588,      0.119753708530366, 0.400156019107291, 0.0074554770319728, 0.474786083276472,      0.397730942956662, 0.0914068195399864, 0.713058736958649, 0.455540413772781,      0.715671499213123, 0.627598068543733, 0.38642590754462, 0.711264833376695,      0.725844041528005, 0.673420828145893, 0.598080424249917, 0.615738236411061,      0.32721883882947, 0.593811482997329, 0.53572935263774, 0.53847996403052,      0.0040694368737233, 0.510117761973819, 0.384089353182871, 0.229099974858256,      0.443527622355204, 0.0224859572357208, 0.66736577895472, 0.624527674680856,      0.571663156804034, 0.25336157330877, 0.537168684849715, 0.699631882968112,      0.0730859758176492, 0.30789954689017, 0.20444875578988, 0.723406673080236,      0.176282126456102, 0.109620028883924, 0.696847614673648, 0.0156167552609477,      0.12694639617135, 0.104446089656888, 0.576357471259767, 0.0190386281018872,      0.595660884369129, 0.645875189667142, 0.254333415041669, 0.608945643560724,      0.0272465783371497, 0.680472949692775, 0.242568388193875, 0.413076802882004,      0.566231714819182, 0.386179319709624, 0.024125720785382, 0.72678092168073,      0.392128819789748, 0.612736781210603, 0.563266220981886, 0.135804806089808,      0.187472972107371, 0.434811756064629, 0.325206646700262, 0.312601324873927,      0.644538631595, 0.0826856277065353, 0.146259047443191, 0.409722817137369,      0.253177544325149, 0.672721108947529, 0.0336283790104349, 0.536417246325353,      0.44274535764908, 0.519371427190133, 0.102704366538147, 0.722531193494278,      0.28719551341516, 0.670228692872584, 0.682079999957379, 0.201497877891299,      0.723121559472159, 0.27189298000738, 0.0620911230038133, 0.409461035434089,      0.528661946522551, 0.601653371620341, 0.578873887661472, 0.336355451007524,      0.692668656449123, 0.459990265454506, 0.218269217541596, 0.435250092909547,      0.227851241170907, 0.271736959467112, 0.726150033423883, 0.449849635030216,      0.0271978523745146, 0.492748577409307, 0.17541023558639, 0.596788515867853,      0.11559220035011, 0.488112738186632, 0.580910161781669, 0.728448234339249,      0.530676579388986, 0.691538936462732, 0.585368973681574, 0.690978439933987,      0.690783143108682, 0.278443267406649, 0.00479918274296676, 0.494444814106201,      0.586705295200725, 0.672058272120673, 0.235053364966392, 0.638449574856341,      0.638692126023098, 0.427955481784051, 0.646726442248514, 0.134055212668385,      0.276399097189209, 0.592728528966446, 0.441521230342217, 0.255699885002005,      0.289448285178679, 0.0351865771868756, 0.383224425662473, 0.0269914646229068,      0.0152484202927062, 0.709809374773127, 0.728783451686249, 0.508835073886939,      0.586761195306883, 0.19251278821925, 0.176118392959475, 0.390243121286638,      0.654334035828494, 0.673967456735412, 0.350430843671085, 0.196892308735908,      0.211118741701871, 0.730010528515844, 0.330567453616248, 0.551353444514686,      0.661236861720438, 0.601897934251079, 0.524622285157106, 0.183889105241725,      0.353138046098409, 0.606365883570447, 0.60248085534151, 0.108077122644886,      0.527399404387309, 0.219712738372086, 0.520354742973127, 0.186818596217576,      0.666528881726871, 0.590488171722112, 0.582912891522914, 0.428775661427041,      0.699830599251151, 0.603477722669975, 0.705771305736146, 0.642203318023397,      0.443815933762925, 0.622863076083702, 0.466858843384842, 0.0886411000533543,      0.551392011135811, 0.538023028681525, 0.195600008464018, 0.0131082408077209,      0.62730473128464, 0.482082278083858, 0.636245303730664, 0.614098627949046,      0.656368186485789, 0.228489824492217, 0.0379409715340383, 0.112082645760501,      0.581767926729533, 0.55122840378822, 0.0871577444390514, 0.189882906750338,      0.250770911422648, 0.667940948350508, 0.453990072206537, 0.28225492445824,      0.657525417299071, 0.356929216534824, 0.693814232422247, 0.57682600243974,      0.667105444331598, 0.540464903431458, 0.69644726912475, 0.0128672171869813,      0.719197038695483, 0.475528213081807, 0.619503930955389, 0.711821807918422,      0.585484369831372, 0.379697292924857, 0.543224627207166, 0.724314337232449,      0.213556867152545, 0.545408534088618, 0.346035012618448, 0.470514372375379,      0.243190515914527, 0.26899791499889, 0.595605382767736, 0.462259413744536,      0.561349544689678, 0.654088273044253, 0.551515370242205, 0.342704486781764,      0.721887004552395, 0.678680607423332, 0.10661626731561, 0.699440137076948,      0.571011891183453, 0.647527746918682, 0.190007995250167, 0.473263343509478,      0.0690039977260399, 0.670209496980311, 0.358129244300962, 0.375379631321039,      0.678771319159281, 0.536676199270132, 0.668569217640658, 0.448939411299087,      0.711241922149909, 0.108085747657182, 0.502769466357002, 0.374226433174886,      0.106450879055427, 0.693470132775303, 0.277597315973764, 0.536697953781942,      0.34043397262594, 0.184620878097461, 0.00469130200835977, 0.0259315913832966,      0.0682700017231441, 0.585508387285693, 0.653243265025264, 0.465726890160718,      0.546314722295407, 0.556250030734321, 0.19843446584138, 0.472887035540046,      0.344234394866597, 0.704846354234753, 0.198656827247135, 0.602960039867456,      0.680125505409818, 0.557761945362829, 0.649833382264959, 0.494254399095049,      0.588827163546271, 0.632888782915026, 0.688427541236684, 0.391986683856652,      0.0419173415426413, 0.720010941297293, 0.168671087352336, 0.564634795317827,      0.712106853100447, 0.599736155607047, 0.615669672888442, 0.171058563101367,      0.625428023313563, 0.0967536710644453, 0.124577690039648, 0.00687941152708416,      0.431611106542264, 0.609688560909128, 0.492243541904919, 0.396708767973176,      0.358555406695739, 0.546393558777226, 0.608949990298456, 0.196178441394311,      0.356395419635146, 0.325441813884789, 0.657618934833634, 0.280895782907398,      0.628571624664351, 0.471271193503368, 0.291781926329135, 0.242633178235177,      0.394899690436917, 0.676617313826883, 0.174027714097822, 0.581931056200377,      0.550300704183233, 0.702421961131571, 0.705168323965465, 0.557452924317173,      0.570117180354795, 0.169856449582207, 0.140220246945735, 0.350776014796747,      0.307238660523093, 0.700479897877462, 0.448761406073455, 0.650585897725171,      0.416505866521591, 0.594871117839299, 0.720376558131377, 0.570914504982537,      0.706607829941973, 0.588068976122053, 0.463503831558511, 0.615458079812045,      0.564586197730836, 0.254163963400933, 0.367953092190157, 0.217447086621415,      0.00631187279287992, 0.711561223179789, 0.455469671263333, 0.0714416575534154,      0.641189986197578, 0.417728358682564, 0.667642451707999, 0.727560230905981,      0.630411560113798, 0.12101773433092, 0.545955780654259, 0.134454547053197,      0.246646918975573, 0.147425394725673, 0.468537264558027, 0.528073877449577,      0.510290584915848, 0.643870671674834, 0.467419945669024, 0.458369457276574,      0.483548805617578, 0.543841850688202, 0.517841160688869, 0.485015607275546,      0.378266404290729, 0.0556900264892045, 0.319258268649152, 0.712785804064414,      0.368272465342541, 0.366329081174184, 0.0519032105599078, 0.379012062454343,      0.451790205169669, 0.570509416344481, 0.0746183199690367, 0.208292020788095,      0.396847922396946, 0.673803870458486, 0.441046435435307, 0.660936504807715,      0.339918544447029, 0.71115478776081, 0.439461290776011, 0.453243405741077,      0.205444045377559, 0.393214757256478, 0.591351041767726, 0.215318602924465,      0.0869363637566541, 0.689526397084578, 0.649127743186972, 0.213179610123884,      0.434921986856402, 0.0377353832809832, 0.677439244577519, 0.688054238407058,      0.635712103970585, 0.609889127257694, 0.142221764114589, 0.355539858584226,      0.427870222832376, 0.0178700124492825, 0.004947281372778, 0.661140843622168,      0.570103846651009, 0.501759330769407, 0.440463453429091, 0.0704449801315961,      0.279942599616358, 0.436914672399355, 0.513407569892297, 0.701605066561962,      0.115075485161928, 0.391641161166514, 0.42468877854187, 0.623940654946985,      0.488014208012435, 0.683630508171043, 0.558064094417826, 0.366437082193705,      0.455810409591241, 0.673337793558347, 0.446620319178281, 0.207988068749134,      0.497524939171777, 0.141187084002822, 0.426756865717019, 0.707781399591308,      0.545989210573572, 0.579966244721801, 0.517032625972346, 0.718481860748729,      0.602716219525339, 0.236345364054399, 0.558255583861533, 0.431223828457558,      0.365636225953124, 0.409429930892365, 0.043928104752237, 0.659464262731929,      0.425596927011639, 0.725806021529604, 0.224775677361241, 0.590964036474333,      0.662051367319391, 0.626620434625404, 0.648276632716587, 0.324148373748995,      0.29507827708195, 0.507200955832411, 0.624412347996066, 0.515059729403067,      0.62997900402912, 0.513771597156568, 0.375095551705484, 0.134195164119236,      0.634411709442187, 0.517195109945018, 0.467665661065612, 0.580303230424549,      0.609087858144811, 0.626500025252763, 0.712328063931443, 0.343001006190608,      0.221785440636147, 0.277191245896358, 0.343282022934197, 0.654056647153165,      0.429404110668685, 0.0183959567792376, 0.716502994981447, 0.549023777029125,      0.384010575701469, 0.689388646566693, 0.140067054862348, 0.489794693136678,      0.286609622063555, 0.37850991859041, 0.525707350765283, 0.698159009559676,      0.343942641482389, 0.564047585509975, 0.118468369426458, 0.392923569156675,      0.519736751890769, 0.681326579809728, 0.640945043857272, 0.123640321169531,      0.606539120209189, 0.224838872130745, 0.683942712253591, 0.473384083497203,      0.219767941190689, 0.623484433902948, 0.674307580687902, 0.143250185940531,      0.586450939396173, 0.171113408301829, 0.532734712737766, 0.124348938802294,      0.716316304613251, 0.43225775000981, 0.723198205008971, 0.498433971346497,      0.681690358059775, 0.661127689946725, 0.522886638674328, 0.0770121168606104,      0.125927153877252, 0.716625710073953, 0.449485543750385, 0.233208190666688,      0.391715330595286, 0.101670102692489, 0.720284800934016, 0.118520726621614,      0.392849115208312, 0.2192113519514, 0.232899092153977, 0.488362518917959,      0.00572062782368987, 0.232022708998634, 0.267710064243038, 0.520481055801511,      0.550626937439566, 0.535329368430688, 0.47875207538692, 0.542825353224711,      0.637289662649156, 0.679529952406023, 0.309922520809313, 0.0589666556180638,      0.508294921579369, 0.679306575622673, 0.292290526116809, 0.578649757760533,      0.417195182346647, 0.545785166668561, 0.503358088980557, 0.482568573065606,      0.156088125360723, 0.402943829155132, 0.723994261362293, 0.728129948963802,      0.469834564613843, 0.50210722074601, 0.622606442291174, 0.596722102362123,      0.713257030009313, 0.270714926809486, 0.705327973527649, 0.432201255234928,      0.407993342799436, 0.458961088545347, 0.218260552733753, 0.51875677981117,      0.011281802681218, 0.222841492033027, 0.359664533618024, 0.693186561975419,      0.650243997057838, 0.632738439071457, 0.724675435763659, 0.464583982447074,      0.571489171597382, 0.370509623926643, 0.458823280657409, 0.311092238860007,      0.558369847879014, 0.669806832852036, 0.155184675171529, 0.691023709594242,      0.0298553653946388, 0.494465210791379, 0.0620371710585952, 0.679725921549356,      0.487046500819744, 0.727866765421106, 0.2441856075295, 0.0500275088702842,      0.353808284074005, 0.47654910680078, 0.595162263579421, 0.672854126481863,      0.294073445504067, 0.72146136332827, 0.367940384265174, 0.714113889078173,      0.713501396908514, 0.564823123636919, 0.377273032849283, 0.385728259873034,      0.459140039098774, 0.601261157274165, 0.621499243404452, 0.540078918042998,      0.354309225616767, 0.555500896923854, 0.674397262396031, 0.449387903599771,      0.581228107835145, 0.703901529759308, 0.364002512748045, 0.69401569130963,      0.200656944946988, 0.64487531280732, 0.544821970430823, 0.727273745138766,      0.539929785389393, 0.462624923117192, 0.567211511255111, 0.540528495928837,      0.665010841704513, 0.230232007928383, 0.584953206974157, 0.244662720966145,      0.0278415309731859, 0.729604985720068, 0.158456432330248, 0.601384390469937,      0.264127357581204, 0.690332562517785, 0.0707561618899736, 0.696470524260576,      0.481967122205418, 0.594397536527927, 0.586734266825794, 0.706431792112254,      0.0252079534747599, 0.729649821392009, 0.277159946988836, 0.602510745458015,      0.710733135449431, 0.556622897748216, 0.283680454652801, 0.563535583546226,      0.182468516265678, 0.0606265593826147, 0.482540742621903, 0.728391144734295,      0.485891395965042, 0.63459467383838, 0.458544485464063, 0.726796653836442,      0.465826642337374, 0.448928399074587, 0.299928791265841, 0.453270721634818,      0.136030629508441, 0.556713216219178, 0.664683569964544, 0.0950046614815387,      0.687634777232276, 0.721734961717045, 0.541946858003199, 0.645919173392323,      0.263559871142753, 0.275614968849426, 0.596710848610092, 0.132363425587882,      0.605247336564198, 0.602330364210559, 0.715286764388239, 0.552510888150324,      0.209516449092143, 0.613255580684229, 0.0829330187007288, 0.439940797049969,      0.457884916424518, 0.182222824179138, 0.219126855331169, 0.482724491910934,      0.557104137138094, 0.54254375683142, 0.711729204039069, 0.627596964470574,      0.590458028576297, 0.24311322873391, 0.342851031014198, 0.633645292381092,      0.721406779111879, 0.424504180907592, 0.287094602584314, 0.725980147712198,      0.515791977372772, 0.657595501028185, 0.578656146831567, 0.513895415046594,      0.373468449185987, 0.56971787815014, 0.697225635335309, 0.712105746049757,      0.493868340794144, 0.404636279009284, 0.389909968992849, 0.0639732611561169,      0.186054631031552, 0.635160499009087, 0.621333987907336, 0.517117862521911,      0.0329062586645302, 0.690997924950835, 0.485180809127923, 0.595920398512406,      0.0692786446371947, 0.700315246263087, 0.624258904256635, 0.209574862760939,      0.619422750929896, 0.319148833232047, 0.20961002849807, 0.64688544858017,      0.487287502012204, 0.484583703657553, 0.200851784722244, 0.430794199338413,      0.727202060153324, 0.178977035979296, 0.428542740496053, 0.64618381244262,      0.609500868886957, 0.726988762299664, 0.580238370968531, 0.527307302082663,      0.446661448476785, 0.428519356751181, 0.0870426884165233, 0.321907873437165,      0.407416122052894, 0.143717129502421, 0.60650917488194, 0.087550758192406,      0.231803218910668, 0.297603205389412, 0.207904957502525, 0.322584343072877,      0.372665987602538, 0.660481602847302, 0.216937114307338, 0.242938776680624,      0.412224181217878, 0.314867234873908, 0.652926856641909, 0.401051107144645,      0.345870999427698, 0.178757266531218, 0.617234352773754, 0.201574893860269,      0.384052157715359, 0.489377070988125, 0.536473968301338, 0.613607026208266,      0.555415543718636, 0.673952941356365, 0.36091742490507, 0.368859837335933,      0.717197084778678, 0.529006432831545)


title: Using control variables in Difference in Difference method

text: When random experiments are conducted, 'baseline data' are usually used as control variables in order to increase precision. How about control variables in Difference in Difference method? What I mean is whether baseline data(Time 1) should be used for it or just control variables at that time(Time 1, Time 2) should be used.


title: Comparison of two Simpson indices using t-test

text: I would like to compare two Simpson Indices from two different populations. I have calculated their variance, as it is done in the original paper by Simpson regarding measures of diversity and I have calculated a confidence interval for each of them using the formula: $(S-2\sqrt{\text{var}},S+2\sqrt{\text{var}})$, as suggested in a published paper. What I would like to find is a p-value of the null hypothesis that the two indices are equal. I have read that someone can do a Welch t-test to compare them, but I haven't found any single paper or book with such an application. My questions regarding this application are: 1) In Welch t-test the variances in the denominator are divided by $n_1$ and $n_2$ respectively, since it is the SE of the mean. I guess in this case and based on the formula for the CI we shouldn't divide by $n$ and have just the square root of the sum of variances. Correct? 2) The degrees of freedom for a simple t-test are $n_1+n_2-2$, while for the Welch t-test is quite a complicated formula which gives a result close but not the same as $n_1+n_2-2$. Which one should be used? 3) By $n_1$ and $n_2$ above we mean the number of different categories in each population rather than the total number in each case. Correct? I would be very grateful if someone can help me with these questions and even more if someone can provide some sort of documentation so I can justify my analysis. Reference: Simpson, E. H. (1949), Measurement of diversity. _Nature_ , **163** , 688 (pdf)


title: 2 x 2 between-subjects design ANCOVA with non-sig. interaction effect; but profile plot shows curves crossing

text: I'm running a 2X2 between-subjects design ANCOVA and wonder about one peculiar thing: The ANCOVA output table reveals no significant interaction effect, albeit the profile plot shows two lines that cross each other, i.e. an disordinal interaction. So how to reconcile these apparently contradictory findings? Might I presume the existence of an interaction effect (which would make a lot of sense by the way..) based on the plot despite a lack of statistical significance? And why is it non-significant; perhaps because of a lack of statistical power?


title: How to analyse data with multiple dependent and independent variables

text: I have two dependent variables, `Abundance` and `Richness` of moths, and 12 independent climate variables. These are `Temperature`, `Rainfall` and `Sunlight`, for each of the 4 seasons. How do I go about analysing this? From doing individual simple linear regression I have found significance for summer rainfall and winter temperature as factors influencing my dependent variables, but I know that this isn't very statistically viable! Is principle component analysis a suitable way of analysing this data? Are there any other multivariate techniques I could use? Thanks.


title: When do missing data indicate a data collection bug?

text: I have collected this data, counting how many users engage in a certain activity how many times:               Activity    Users           Percent     1           2,005,752       62.57%     2           1,005,669       31.37%     3           616,328         19.23%     4           408,571         12.75%     5           285,103         8.89%     6           207,127         6.46%     7           154,861         4.83%     8           117,353         3.66%     9           90,143          2.81%     10          70,174          2.19%     11          55,196          1.72%     12          44,050          1.37%     13          34,883          1.09%     14          28,027          0.87%     15          22,948          0.72%     17          18,629          0.58%     18          15,173          0.47%     19          12,376          0.39%     20          10,422          0.33%     21          8,435           0.26%      (I omitted further rows) Observe that   1. No user engaged in the activity 16 times. One would expect about 20k users (~0.65%) there.   2. log-log plot is convex up, which indicates that the number of users decays faster than polynomially (i.e., this is not a power-law distribution, as I, perhaps naively, expected). What are the chances that the absence of the users engaging in the activity 16 times is a statistical fluke as opposed to a bug in data collection? E.g., is the following approach reasonable? Assume that the 20k individuals with 16 activities are missing. Add them to the table and compute chi-squared=20,076 which is far too large for the 20 degrees of freedom, so this is not a statistical fluke and there must be a data collection bug. PS. there is no underlying reason which makes 16 special.


title: How to do regression with effect coding instead of dummy coding in R?

text: I am currently working on a regression model where I have only categorical/factor variables as independent variables. My dependent variable is a logit transformed ratio. It is fairly easy just to run a normal regression in R, as R automatically know how to code dummies as soon as they are of the type "factor". However this type of coding also implies that on category from each variable is used as a baseline, making it hard to interpret. My professor have told me to just use effect coding instead (-1 or 1), as this implies the use of the grand mean for the intercept. Does anyone know how to handle that? Until now i have tried:               gm <- mean(tapply(ds$ln.crea, ds$month,  mean))     model <- lm(ln.crea ~ month + month*month + year + year*year, data = ds, contrasts = list(gm = contr.sum))          Call:     lm(formula = ln.crea ~ month + month * month + year + year *          year, data = ds, contrasts = list(gm = contr.sum))          Residuals:          Min       1Q   Median       3Q      Max      -0.89483 -0.19239 -0.03651  0.14955  0.89671           Coefficients:                  Estimate Std. Error t value Pr(>|t|)         (Intercept) -3.244493   0.204502 -15.865   <2e-16 ***     monthFeb    -0.124035   0.144604  -0.858   0.3928         monthMar    -0.365223   0.144604  -2.526   0.0129 *       monthApr    -0.240314   0.144604  -1.662   0.0993 .       monthMay    -0.109138   0.144604  -0.755   0.4520         monthJun    -0.350185   0.144604  -2.422   0.0170 *       monthJul     0.050518   0.144604   0.349   0.7275         monthAug    -0.206436   0.144604  -1.428   0.1562         monthSep    -0.134197   0.142327  -0.943   0.3478         monthOct    -0.178182   0.142327  -1.252   0.2132         monthNov    -0.119126   0.142327  -0.837   0.4044         monthDec    -0.147681   0.142327  -1.038   0.3017         year1999     0.482988   0.200196   2.413   0.0174 *       year2000    -0.018540   0.200196  -0.093   0.9264         year2001    -0.166511   0.200196  -0.832   0.4073         year2002    -0.056698   0.200196  -0.283   0.7775         year2003    -0.173219   0.200196  -0.865   0.3887         year2004     0.013831   0.200196   0.069   0.9450         year2005     0.007362   0.200196   0.037   0.9707         year2006    -0.281472   0.200196  -1.406   0.1625         year2007    -0.266659   0.200196  -1.332   0.1855         year2008    -0.248883   0.200196  -1.243   0.2164         year2009    -0.153083   0.200196  -0.765   0.4461         ---     Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1           Residual standard error: 0.3391 on 113 degrees of freedom     Multiple R-squared: 0.3626, Adjusted R-squared: 0.2385      F-statistic: 2.922 on 22 and 113 DF,  p-value: 0.0001131


title: How do I find if there is a significant difference in the number of males and females in a single population?

text: Also, how do I find if there is a significant difference in the number of males in two populations? In SPSS please!


title: How to make results using Hellinger distance comparable with Euclidean distance outputs?

text: I have a two kinds of data for the same geographic region. One is presence- absence data of species (for amphibians, reptiles and birds) and the other has several environmental variables for the same area (e.g. temps and precipitation). For the species I calculated the Hellinger distance and for environmental variables the euclidean distance (vars previously transformed so the all range from 0 to 1). I would like to compare the patterns in a map (dissimilarity change using a focal cell), using the same color for the same range of values in the amphibian, reptiles and bird maps. My problem is how to make this maps comparable with the environmental distance. Hellinger distance ranges from 0 to sqrt(2) and for my data euclidean distance has a max=2.11. All results from species have a normal distribution and I classified the data using equal intervals (ArcMap), but results from euclidean distance are skewed to the left, with a long tail to the right, and due to that the map is mostly blue (meaning small distance), since the reds are assigned to few but scattered high values.. My original idea was to transform the results from euclidean so they have a max=sqrt(2), but not sure if that would be correct, since from what i know the euclidean distance has no upper limit... Any help would be greatly appreciated!!!


title: Simulating EM versus listwise deletion--unexpected results

text: I'm preparing a presentation on missing data strategies and conducted a simulation to compare listwise deletion (LD) to the EM. Here's what I expected based on the literature:   1. Standard errors will be smaller for EM under all missing data conditions (MAR, MCAR, NMAR). See http://psycnet.apa.org/journals/met/6/4/330/.   2. EM will be unbiased for the MAR condition, while LD will be biased.   3. EM and LD will be biased under NMAR, but EM will be less so if I include an auxiliary variable. I've found everything to be true except #1. In fact, for the MAR condition, the standard errors were larger. I've included my code below (in R). Can anyone see any errors?                       ##### create a function to easily extract EM estimates     em = function (data.matrix) {         require(norm)         ss = prelim.norm(data.matrix)         thetahat = em.norm(ss, showits = FALSE)         cor.corrected = getparam.norm(ss, thetahat, corr = TRUE)$mu[2]         if (!is.null(names(data.matrix))) {             names(cor.corrected) = names(data.matrix)             row.names(cor.corrected) = names(data.matrix)         }         return(cor.corrected)     }                            #### set simulation parameters     n = 200      iterations=1000     sr = .3                  #### preallocate     results = data.frame(matrix(nrow=iterations, ncol=7))     names(results) = c("LD_mcar", "EM_mcar", "LD_mar", "EM_mar", "LD_nmar", "EM_nmar", "estimate")          for (i in 1:nrow(results)){                      #### generate data         sig = matrix(c(1, .3, .8,                         .8, 1, .3,                         .5, .3, 1), nrow=3)         data = data.frame(mvrnorm(n, mu=c(0,0,0), Sigma=sig)    )                    names(data) = c("y", "x", "a")                      #### create missing data         mcar = sample(1:n, size=sr*n)                mar = which(data$x<quantile(sr))          nmar = which(data$y<quantile(sr))         y_mcar = data$y; y_mcar[mcar] = NA          y_mar = data$y; y_mar[mar] = NA         y_nmar = data$y; y_nmar[nmar] = NA                      #### estimate using deletion         results$LD_mcar[i] = mean(y_mcar, na.rm=T)          results$LD_mar[i] = mean(y_mar, na.rm=T)         results$LD_nmar[i] = mean(y_nmar, na.rm=T)                      #### estimate with em         results$EM_mcar[i] = em(data.matrix(data.frame(x=data$x, y=y_mcar)))         results$EM_mar[i] = em(data.matrix(data.frame(x=data$x, y=y_mar, a=data$a))) ### include aux variable           results$EM_nmar[i] = em(data.matrix(data.frame(x=data$x, y=y_nmar, a=data$a)))              results$estimate[i] = mean(data$y)          }          mns = colMeans(results)     sds = apply(results, 2, sd)     mns     sds


title: How to speed up Kernel density estimation

text: **Background** A couple of days ago I asked here about: How to reduce number of points for clustering . Instead of reducing number of points, a method "Kernel Density Estimation" (KDE) was suggested to me, it gives right solutions and it is faster then my previous approach. **Question** What I am curious about now is complexity of this method. Maybe I've implemented it in a bad (too naive) way, here is my c++ implementation http://pastebin.com/gtStWjmA (see evalPrivate method). But assuming I have $m$ data points and I want to sample KDE in $n$ points. Then for each evaluation of KDE I have to evaulate kernel function m-times. So my complexity is $O(m^n)$ and that is too much. If my $m=60 000$ and $n = 1000$. Then it tooks ages to sample KDE in order to find its local maximas.


title: Negative Regression Coefficent (Cox Regression)

text: I'm doing a Cox regression analysis in order to verify the role of two variables on the prediction of disease conversion. Previously, I did a ROC analysis and a logistic regression. All these methods reported an important role of one of the two variables (`Var1`) to predict the presence of disease. When I did the Cox regression, only `Var1` has been retained, but a negative beta (-4,945) and a consequent very low hazard ratio resulted. Please explain the divergence between high sensitivity of the predictive variable (`Sen` 90%) in ROC analysis and such a low hazard ratio.


title: Cramer-Rao bound for $\chi^2$ distribution parameter estimates

text: I've struck an unpleasant problem with the noncentral $\chi^2$ distribution. I work with random variables, distributed as $\chi^2_{\nu}(\lambda)$, where $\nu$ is the degree of freedom and $\lambda$ is non-centrality parameter (which should be somehow estimated). In "Continuous Multivariate Distributions Vol.2 by S. Kotz, N. Balakrishnan, N. L. Johnson) it is said that the Cramer-Rao bound for variance of unbiased estimators of $\hat{\lambda}$ is (page 453, eq. 29.39b): $$CRB_{\hat{\lambda}}=4(\theta\lambda^{-1}-1)^{-1}n^{-1},$$ where $n$- is the sample size, $\lambda$-true value of the noncentrality parameter that is under estimation and $\theta$ is the parameter which has the following upper and lower estimates (page 452, eq. 29.36a): $$1+\lambda^{-1}-\frac{1}{2}\nu\lambda^{-2}+\frac{1}{4}\nu^2\lambda^{-3}\leq \theta\lambda^{-1}\sigma^{-2}\leq 1+\frac{5}{4}\lambda^{-1}-\frac{20\nu-13}{39}\lambda^{-2},$$ and $\sigma$ is the noise variance. I tried to analyse the behavior of $CRB_{\hat{\lambda}}$ when $\lambda \to 0$. But the problem is that it goes to zero to! But in my opinion it is not what we should get. I use an analogy from physics or mathematical statistics . For example this problem arises when one estimates the square of the Euclidean distance between some determined vector and its empirical sample contaminated by additive white Gaussian noise with zero mean and equal variances for all the points of the vector. And the result that I've got ($CRB_{\hat{\lambda}}\to 0$ when $\lambda \to 0$) pushes me to conclude that for a fixed level of noise ($\sigma=\mbox{const}$) it is possible to estimate infinitely close vectors with practically infinite precision (almost with zero variance of estimation). But that shouldn't be true. It should be the opposite way. So where's the problem?


title: How to sum uncertainties, systematic and random

text: I apologize for the simplistic questions. I have a retrieval process that has a set of random and systematic uncertainties associated with it. I'm assuming that these are all independent. The goal is to find the total error for this process. I have two questions related to this: 1) To find the total systematic uncertainty can I sum all the systematic uncertainty components in quadrature? Assuming that they are independent, it is unlikely that they will all contribute in the same direction and it seems to make sense to add them in quadrature. However, by definition they are not random. Does this mean they must be added linearly? 2) To find the total uncertainty (total_random + total_systematic), can the total_random and total_systematic be added in quadrature or must they be added linearly? Thank you in advance for you help!


title: How to compare two survival percentages for two different populations?

text: I calculated the 5 year survival of two populations keeping different variable constant each time and got percentages. I wanted to determine if these percentages were significantly different from each other. Could anyone please guide me with the same?


title: Degrees of freedom in GLS

text: Here I perform a GLS regression in R and the degrees of freedom is reported as "Degrees of freedom: 60 total; 58 residual". In this regression I see five parameters that are being estimated: the slope of the regression line, the intercept of the regression line, the residual standard deviation, the constant of the variance function, and the power of the variance function. When I go to generate prediction intervals for the regression line what degrees of freedom should I use? Anticipating the answer is not 55, why aren't the degrees of freedom 55?               library(nlme)          X <- c(1,1,1,1,1,1,1,1,1,1,4,4,4,4,4,4,4,4,4,4,       10,10,10,10,10,10,10,10,10,10,20,20,20,20,20,20,       20,20,20,20,30,30,30,30,30,30,30,30,30,30,40,40,          40,40,40,40,40,40,40,40)          Y <- c(1.07,1.01,0.99,1.09,0.94,1.00,1.01,0.98,1.00,       1.03,3.66,3.75,3.77,3.92,4.08,3.99,3.95,4.10,       3.88,4.04,10.13,10.2,9.77,10.28,8.71,9.79,9.82,       9.85,10.07,9.63,20.22,19.46,19.02,20.06,20.94,       19.92,19.96,20.04,19.67,19.96,31.04,31.4,31.84,       30.77,32.13,31.17,30.36,29.95,30.74,30.67,41.14,       40.29,42.77,38.36,39.17,39.61,40.73,39.42,40.72,       40.24)          m <- data.frame(X,Y)          fit <- gls(Y ~ X,weights=varConstPower(form = ~ X),data=m)     summary(fit)


title: Why does this autocorrelation formula hold?

text: I have been unable to understand the highlighted subsection of David Kenny's _Correlation and Causality_ , which I downloaded from Kenny's website. ![enter image description here](http://i.stack.imgur.com/j57ti.png)


title: New method for meta-analysis to determine agreement between multiple sources

text: I'm writing a paper on a method to determine the statistical significance of whether a sample of sources are in mutual agreement on specific premises. This is done by counting the number of sources in the sample that agree with the premise, and the number of sources that disagree, and subjecting them to a formula to determine the minimum level of significance. Please tell me if: (1) a better method already exists to do what I am attempting, (2) if my method seems reasonable, (3) any flaws in my method. Please bear in mind that the purpose is only to determine mutual agreement, and the result is not a true probability but a minimum probability based on the initial assumptions. * * * In order to develop the formula to be used for calculating statistical significance I defined the logical premises for statistical significance as follows:   1. The statistical significance of a premise representing a single school of thought throughout all sources increases exponentially with each supporting quotation, and is reduced exponentially by each contradicting quotation.   2. The significance is always a decimal number. The closer the value to 1, the more statistically significant.   3. A statement with an equal number of supporting and contradicting quotations will have a significance of 0. Any positive number is an indication of a supporting majority. Absolute certainty is 1, which can never be reached in practice.   4. Considering that there are theoretically an infinite number of possible statements which could be made, the probability of 3 synonymous statements being made in independent sources by chance is virtually zero. Following this logic we can be certain that if there are 3 quotations to support a statement and 0 contradicting quotations then we can be certain of statistical significance. This marks the minimum acceptance level for “statistically significant”, known as the alpha level.   5. I further label “probably significant” as 2 synonymous premises being made in independent sources, with 0 contradicting premises. “Probably significant” premises are considered as not statistically significant, but the label helps to identify those statements that only just missed the mark, which may be useful for more practical purposes. The formula I developed to match the above premises is as follows: t = 1-(1/((y^2+ 1)/(x^2+ 1))) y represents the number of quotations which support the premise; x represents the number of quotations which contradict the premise. The result of t is not a probability in our context, but a value representative of the logical premises on which the formula was based. The formula calculates that 3 supporting quotations with 0 contradicting quotations has a value of 0.9. Therefore all values above or equal to 0.9 are statistically significant. The formula further calculates that if there is 1 contradicting quotation we would need 5 supporting quotations for it to still be considered at least as statistically significant as 3 supporting quotations would be with 0 contradictory quotations, i.e. with a value greater than or equal to 0.9. “Probably significant” has a value of 0.8, though I would like to remind the reader that this is still counted as not statistically significant. It should be noted that if there are 3 or more contradictory quotations to the premise and all the contradictory quotations are in mutual agreement, in addition to there being 3 or more synonymous supporting quotations for the premise, then this would indicate the existence of two different schools of thought on the premise. The following is a table showing the results of the formula for up to 20 quotations supporting the potential premise (rows) and up to 6 quotations contradicting the potential premise (columns).                   0       1       2       3       4       5       6   Against     0   0.000                                 1   0.500   0.000                             2   0.800   0.600   0.000                         3   0.900   0.800   0.500   0.000                     4   0.941   0.882   0.706   0.412   0.000                 5   0.962   0.923   0.808   0.615   0.346   0.000             6   0.973   0.946   0.865   0.730   0.541   0.297   0.000         7   0.980   0.960   0.900   0.800   0.660   0.480   0.260         8   0.985   0.969   0.923   0.846   0.738   0.600   0.431         9   0.988   0.976   0.939   0.878   0.793   0.683   0.549         10  0.990   0.980   0.950   0.901   0.832   0.743   0.634         11  0.992   0.984   0.959   0.918   0.861   0.787   0.697         12  0.993   0.986   0.966   0.931   0.883   0.821   0.745         13  0.994   0.988   0.971   0.941   0.900   0.847   0.782         14  0.995   0.990   0.975   0.949   0.914   0.868   0.812         15  0.996   0.991   0.978   0.956   0.925   0.885   0.836         16  0.996   0.992   0.981   0.961   0.934   0.899   0.856         17  0.997   0.993   0.983   0.966   0.941   0.910   0.872         18  0.997   0.994   0.985   0.969   0.948   0.920   0.886         19  0.997   0.994   0.986   0.972   0.953   0.928   0.898         20  0.998   0.995   0.988   0.975   0.958   0.935   0.908         For                                    As you can see from the table this is a very strict system: if there are 6 contradicting quotations to a given premise then it requires 20 supporting quotations in order for it to be considered statistically significant that the majority of the sources agree on the premise. This is the case even though the contradicting quotations are not required to be in agreement with each other, whereas the supporting quotations are required to be synonymous. The results of the study must fit one of the following scenarios:   1. Very few or none of the statements have statistically significant support. (The “null hypothesis”) This would prove that there is no agreement.   2. Some of the statements have statistically significant support, but overall this is not statistically significant. A little agreement here and there would indicate that there is not a definitive system, or it could mean that the alpha level was too strict.   3. A statistically significant number of statements have statistically significant support in one school only. This would prove beyond all reasonable doubt that there is a single definitive system.   4. A statistically significant number of statements have more than one school of statistically significant support. This would prove beyond all reasonable doubt that there are multiple school, each possessing its own system.


title: Comparison with trial dependent chance level

text: I ran an experiment where each participant had to choose 1 image from a 4-image display and I measured whether the image they chose was from category A. I want to compare the average proportion of images chosen from category A to the actual proportion of images from category A. For example: a trial would be {A, A, B, C} where A, B, C stand for the category of each image. My dependent measure is 0/1 depending if the participant chose an image from category A or not. Since two of the four images are from category A, the chance of choosing A (if there's no preference for A) is here 1/2. The chance level is a variable for every trial such that there is between 1 and 3 images from category A (hence chance of 0.25, 0.5, 0.75) Each participant has a certain number of trials (not the same for all subjects) the outcomes from which are assumed to be independent from one another. So I have something like this:                Participant Trial  proportion_of_A  selection_in_A         1          1       0.5                1          1          2       0.25               0            1          3       0.75               0         ...      And I want to have my subjects as random factors. I was thinking first to do a mixed model as there is no problem to run it for a binomial distribution (my participants' answer) but I don't know how to compare the result with the chance level (proportion of images in category A). The main problem lying in the fact that the chance is different at every trial (however I am not sure I would know to do with a fixed chance level either!?) I thought about making a single measure my subtracting the chance level to my participant response at every trial and testing whether this average is significantly greater than 0. However I don't know how to do that in a mixed model as my data would not be fitted with a binomial model or a gaussian one... Hence I am stuck in looking for some test that would be technically correct. Any advice? Reading that could help?


title: Variance of sample mean for dependent samples

text: Suppose I have two discrete independant random variables $X$ and $Y$, and that I'm interested in the expected value of the random variable $W$, where: $$ W= \text{sign}(X-Y). $$ So, W is 1 if $X>Y$, -1 if $Y>X$ and 0 otherwise. I sample the distributions of $X$ and $Y$ ten times each, giving me $\\{X_1, \dots, X_{10}\\}$ and $\\{Y_1, \dots, Y_{10}\\}$.   Consider these two ways to estimate $\text{E}\\{W\\}$ $$ \quad\quad\bar{W} = \frac{1}{10}\sum_{i=1}^{10} W_{i,i}, \\\ \text{and, } \quad\quad \bar{W}' = \frac{1}{100}\sum_{i=1}^{10}\sum_{j=1}^{10} W_{i,j}, \\\ \text{where } \quad W_{i,j} = \text{sign}(X_i - Y_j) $$ I know that $\text{Var}\\{\bar{W}\\} = \frac{1}{10}\text{Var}\\{W\\}$, but what is $\text{Var}\\{\bar{W}'\\}$, and how can I estimate it from my 20 samples?


title: Comparing topic distributions between corpora using Latent Dirichlet Allocation and R topicmodels or python gensim

text: So I am working on a problem where I want to extract a set of LDA topics from one corpus, and then compare the distribution of those topics in other corpora. So basically I want to lock-in the topics and then get a sense of how similar or different another corpus is from the original corpus. I was hoping that someone could tell me the tool or approach to do this type of comparison? My particular application has to do with comparing local versus national newspapers. I have a corpus of national newspaper articles and I have already used gensim to extract the topics. Now I have corpora of local newspapers captured during the same period of time. So I want to compare the distribution of identical topics in the national newspaper versus the local newspapers. Of course, I would also like to look at the structure of the topic in both the national and local corpora (such as the change in probability of co-occurrence of words for the same topic in the two different corpora). I looked around in the R `topicmodels` and the python `gensim` packages, but had no luck. Any suggestions?


title: Building temporal correlation model by a dataset and calibrating the model with other dataset

text: On one hand, I have a temperature time series data (dataset1) for a week and I want to build the temporal correlation model by using time-series analysis and performing the prediction for oncoming values. The data is not sufficient and doesn't include seasonal and diurnal patterns to result accurate predictions. On the other hand, I have an other source of data (dataset2) which is a little bit far from the study area but include sufficient amount of data which represent seasonal and diurnal effects. **My question** : I am writing to ask whether it is possible to build the temporal correlation model based on the dataset2 and calibrating it by dataset1? (The calibrating is essential since the dataset2 is not as local as dataset1.) What I want is to gain the seasonal and daily patterns from the large dataset (dataset2) and perform the predictions based on the dataset1 which are local for the study area. In other words, using dataset2 to build the temporal correlation modeling and performing the predictions based on that cannot give the accurate results.


title: How to transform $Var(X-Y)^{2}$

text: I have something like that: $$ Var\left( \frac{1}{2n-2} \sum_{i=1}^{n-1} \left( X_{i+1}-X_{i}\right)^{2} \right) $$ and I transformed it to something like that (I hope it is correct): $$ \left(\frac{1}{2n-2}\right)^{2} \sum_{i=1}^{n-1} Var(X_{i+1}^{2}-2X_{i+1}X_{i}+X_{i}^{2}) $$ What can I do more with that, when I know that: $EX=m$, $VarX=\sigma^{2}$, $E \overline{X}=m$ and $Var \overline{X}= \frac{\sigma^{2}}{n}$ ? The title may be confusing because I have difference of the same random variable (only with different indexes), but idea is the same, I guess.


title: How to scale new observations for making predictions when the model was fitted with scaled data?

text: I understand the concept of scaling the data matrix to use in a linear regression model. For example, in R you could use:               scaled.data <- scale(data, scale=TRUE)      My only question is, for new observations for which I want to predict the output values, how are they correctly scaled? Would it be, `scaled.new <- (new - mean(data)) / std(data)`?


title: Confidence interval for Population proportion

text: I've a small query and really confused as how to solve such problems. $n$ people are question about internet connection 87 of them have high speed connection. We are given confidence interval(CI) of population proportion is $0.119 < p < 0.1771$.   1. I found the mid-point of CI as 0.145. I need to understand how to find $n$   2. It's given this interval is $\alpha \% $ confidence interval, how can we find $\alpha$ Any help is greatly appreciated. Any resource that can help me to solve such problems will also be of great help. Regards, Arif


title: Should the mean be used when data are skewed?

text: Often introductory applied statistics texts distinguish the mean from the median (often in the the context of descriptive statistics and motivating the summarization of central tendency using the mean, median and mode) by explaining that the mean is sensitive to outliers in sample data and/or to skewed population distributions, and this is used as a justification for an assertion that the median is to be preferred when the data are not symmetrical. For example: > The best measure of central tendency for a given set of data often depends > on the way in which the values are distributed.... When data are not > symmetric, the median is often the best measure of central tendency. Because > the mean is sensitive to extreme observations, it is pulled in the direction > of the outlying data values, and as a result might end up excessively > inflated or excessively deflated."   > —Pagano and Gauvreau, (2000) _Principles of Biostatistics_ , 2nd ed. (P&G > were at hand, BTW, not singling them out _per se_.) The authors define "central tendency" thus: "The most commonly investigated characteristic of a set of data is its center, or the point about which observations tend to cluster." This strikes me as a less-than forthright way of saying _only use the median, period_ , because only using the mean when the data/distributions are symmetrical is the same thing as saying only use the mean when it equals the median. **Edit:** whuber rightly points out that I am conflating robust measures of central tendency with the median. So it is important to keep in mind that I am discussing the specific framing of the arithmetic mean versus the median in introductory applied statistics (where, mode aside, other measures of central tendency are not motivated). Rather than judging the utility of the mean by how much it departs from the behavior of the median, ought we not simply understand these as two different measures of centrality. In other words being sensitive to skewness is a feature of the mean. One could just as validly argue "well the median is no good because it is largely insensitive to skewness, so only use it when it equals the mean." (The mode is quite sensibly not getting involved with this question.)


title: How to eliminate dependent inputs?

text: There are a lot of statistical methods that rely on the assumption of input independence. For example, Naive Bayes text classifiers operate under the assumption that occurrences of different words are conditionally independent given the document class. Another example - HDMR and Sobol variance decomposition seem like very powerful methods to me, but they require this assumption. But suppose that I have a large dataset where inputs are very much non- independent, and I still want to apply the method. The character of the dependency is implicit: I do not have an explicit model for computing some of the inputs given others. What are some data transformations that transform a dataset with dependent inputs to a dataset with independent inputs, under some class of input dependency models?


title: Is there an R package which implements weighted maximum likelihood for Rasch models?

text: Is there a package in R which implements the weighted maximum likelihood method (Warm, 1996) for estimating the person parameters in Rasch Models?


title: Principal component (PC) as a substitute for colinear covariates?

text: I am working on a spatial linear regression and I can tell there is collinearity between covariates. Can I use PCA (Principal Component Analysis) images instead of original covariates to estimate the dependent variable? I am assuming PC1=Variable 1, PC2=Variable 2, etc. Or are there any other methods to solve the collinearity problem?


title: Seemingly unrelated regression and OLS

text: It's been said that SUR is equivalent to the equation-by-equation OLS: > i. When there are no cross-equation correlations between the error terms.   >  ii. When each equation contains exactly the same set of regressors. I would like to confirm whether each condition is necessary or sufficient, namely the second condition. In 4.5.1 Seemingly Unrelated Regression, the author uses an example of SUR with equations that each use the same set of regressors, which seems to violate (ii) if each condition is sufficient.


title: Random Forest can't overfit?

text: I've read some literature that random forests can't overfit. While this sounds great, it seems too good to be true. Is it possible for rf's to overfit?


title: Difference between imputation and forecast

text: what is the difference between imputation and forecasting? All i know, forecasting is the term used in time series analysis, which means predicting the future value by considering the trend of the historical data. Imputation is a strategy for completing data. I've thought some answers to my own question. But so far, these are what i got: forecasting is done by considering the trend of the historical data, while imputation sees the data (each of the observation) as random variable, so we should consider the distribution of the history. The term forecasting used for predicting the future value, while imputation used for predicting the missing value. forecasting procedure will never be related to unknown data (in this case, any value that hasn't been observed) , while imputation procedure might depends on the unknown data (in this case, unobserved data) (informative drop out).


title: Finding the most "uniform" or "least concentrated" density function, subject to moment constraints

text: ## **Background** I want to find a probability measure for a continuous random variable, subject to moment constraints, that is maximally "uniform", as defined below: * * * **Definition: Maximally Uniform Probability Measure** Let $X$ be a continuous random variable with probability measure defined on $\mathcal{D}_X=\mathcal{B}([a,b])$, and let $L_m=\\{B \in \mathcal{D}_X: \lambda(B)=m\\}$, where $\lambda(B)$ is the Lebesgue measure of Borel set $B$ and $0\leq m \leq \lambda([a,b])$. Let $C$ be a set of constraints on the probability measure of $X$, and $\mathcal{P}_C$ be the class of all continuous probability measures on $\mathcal{D}_X$ that satisfy $C$. For given constraints $C$, a maximally uniform probability measure $P^*_X \in \mathcal{P}_C$ achieves the _minimum_ **maximum** probability measure for **every** $L_m$, where minimization is over $\mathcal{P}_C$. Specifically, for a set of constraints $C$ and associated probability measures $\mathcal{P}_C$, the maximally uniform probability measure $P^*_X \in \mathcal{P}_C$ satisfies the following condition: $\left\\{\max\limits_{B\in L_m}{P^*_X(B)}\leq \max\limits_{B\in L_m}{P_X(B)}\;\; \forall m\right\\}\forall P_X \in \mathcal{P}_C$ * * * For example, if I specify the class of Borel sets of Lebesgue measure 3 (i.e., $L_3$), and pick a probability measure from $P \in \mathcal{P}_C$ then I would proceed to find the interval in $L_3$ that has the **highest** probability measure wrt $P$. If $P$ is maximally uniform, then when I do this exercise for any other $P' \in \mathcal{P}_C$, I will get a Borel set of Lebesgue measure 3 that contains _more_ probability than $P$. This will be also be the case for any other class of Borel sets of given Lebesgue measure (i.e., there is nothing special about 3). My thinking is that I can solve this problem as a variational optimization problem (at least for $-\infty < a\leq b,< \infty)$, by minimizing the **path length** of the distribution function $F$ associated with each $P_X \in \mathcal{P}_C$. Below is the formulation of my problem as a variational-calculus problem Note: $f(x)=F'(x)=$ target density function associated with the maximally uniform probability measure. $\min\limits_{f(x)} \int\limits_a^b \sqrt{1+f(x)^2} dx$ $s.t.$ $\int\limits_a^b f(x) dx = 1$ $\int\limits_a^b xf(x) dx = 0$ $\int\limits_a^b x^if(x) dx = a_i \;\;i\in \\{S \subset \mathbb{N^+}/1\\}$ $f(x) \geq 0$ Using the euler-lagrange euqations, I have solved the general formulation of the above to get the following **implicit** solution: $\frac{f(x)}{\sqrt{{f(x)}^{2}+1}}= \sum\limits_{i\in S \cup \\{0,1\\}}\mu_i{x}^{i}$ where the $\mu_i$ are the langrange multipliers (one per constraint) that must be solved for to make this implicit solution satisfy the constraints. * * * ## What I need help with In order of priority (I will accept any answer that addresses either of these):   1. Is there an **explicit** solution of the above? (even if it uses infinite series)   2. Does the concept of uniformity via path length minimization admit pathological cases (in the continuous measure case only)?


title: Help requested with using custom model in caret() package

text: The caret package (terrific btw) has a lot of models built in but if you want to use a model that is not built in, there is a way as described in outline here http://caret.r-forge.r-project.org/custom_models.html. Reproducing the example given there works just fine. I'm attempting to do this for the grnn() general regression neural network model and have run into problems I can't fix. My reproducible code example is:               library(caret)     x <- rep(1:100); y <- x^2+x*rnorm(100,0,1); tr <- data.frame(y=y,x=x)     grnnFit <- function(dat, params) smooth(learn(dat), sigma=params$sigma) #train         grnnPred <- function(mod, newx) guess(mod, as.matrix(newx)) #predict         grnnSort <- function(x) x[order(x$sigma),] #sort results     #list of params/functions     lpgrnn <- list(library="grnn",       type="Regression",       parameters=data.frame(parameter="sigma", class="numeric", label="Sigma"),       grid=data.frame(sigma=c(.1, .2, .3)), #only one tuning parameter sigma       fit=grnnFit,       predict=grnnPred,       prob=NULL,       sort=grnnSort)          set.seed(998)     fitControl <- trainControl(method="cv", number=10)     set.seed(825)     res <- train(y=tr[,-1], x=tr[,1], method=lpgrnn, metric="RMSE", trControl = fitControl)      The error message is: > res <\- train(y=tr[,-1], x=tr[,1], method=lpgrnn, metric="RMSE", trControl = > fitControl) Error in train.default(y = tr[, -1], x = tr[, 1], method = > lpgrnn, metric = "RMSE", : attempt to apply non-function getModelInfo("grnn") return an empty list               > getModelInfo("grnn")     named list()     >       as opposed to other models, e.g. getModelInfo("nnet") returns               > getModelInfo("nnet")     $nnet         $nnet$label     [1] "Neural Network"          $nnet$library     [1] "nnet"          $nnet$loop     NULL          $nnet$type     [1] "Classification" "Regression"              $nnet$parameters       parameter   class         label     1      size numeric #Hidden Units     2     decay numeric  Weight Decay          $nnet$grid     function (x, y, len = NULL)      expand.grid(size = ((1:len) * 2) - 1, decay = c(0, 10^seq(-1,          -4, length = len - 1)))          $nnet$fit     function (x, y, wts, param, lev, last, classProbs, ...)      {         dat <- x         dat$.outcome <- y             if (!is.null(wts)) {                 out <- nnet(.outcome ~ ., data = dat, weights = wts,                      size = param$size, decay = param$decay, ...)             }             else out <- nnet(.outcome ~ ., data = dat, size = param$size,              decay = param$decay, ...)         out     }          $nnet$predict     function (modelFit, newdata, submodels = NULL)      {         if (modelFit$problemType == "Classification") {             out <- predict(modelFit, newdata, type = "class")         }         else {             out <- predict(modelFit, newdata, type = "raw")         }         out     }          $nnet$prob     function (modelFit, newdata, submodels = NULL)      {         out <- predict(modelFit, newdata)         if (ncol(as.data.frame(out)) == 1) {             out <- cbind(out, 1 - out)             dimnames(out)[[2]] <- rev(modelFit$obsLevels)         }         out     }          $nnet$varImp     function (object, ...)      {         imp <- caret:::GarsonWeights(object, ...)         if (ncol(imp) > 1) {             imp <- cbind(apply(imp, 1, mean), imp)             colnames(imp)[1] <- "Overall"         }         else {             imp <- as.data.frame(imp)             names(imp) <- "Overall"         }         if (!is.null(object$xNames))                  rownames(imp) <- object$xNames         imp     }          $nnet$predictors     function (x, ...)      if (hasTerms(x)) predictors(x$terms) else NA          $nnet$tags     [1] "Neural Network"    "L2 Regularization"          $nnet$levels     function (x)      x$lev          $nnet$sort     function (x)      x[order(x$size, -x$decay), ]


title: Margin of Error for Likert scale response

text: Suppose the target population is a group of 11 company CEOs and 7 CEOs responded to a question about their level of annual bonus on a 7 point Likert scale. The responses are as follows: 7, 7, 7, 7, 7, 7, and 5. The average response score then is 6.71. If the last two levels of the Likert scale are '6 = satisfied' and '7 = very satisfied', this finding can be expressed as a 'very satisfied' rating. Do we apply a margin of error to Likert scale responses? (The margin of error for a sample of 7 is 38%. This is 1 divided by the square root of the sample - a rough guide!)


title: Latent class model

text: I have asked following questions before More than one outcome (dependent) variables in ordinal logistic regression How to handle more than one dependent variable (categorical) in logistic regression? Using Rasch model to explain relationships between a set of dependent and independent variables Use of further analysis on factors formed by principal component analysis in regression answers of the above questions help me to understand what could be done with my data. My data consists of 2 independent variable (income and years- continuous data) with control variables (dichotomous or categorical data) and I have a dependent variable with 6 proxies e.g for dependent variable Business development (latent variable), six questions were asked in 5-point likert scale (categorical-ordinal data) increase in profit, sales, size, marketing, asset and labour (manifest variables). I want to find out the relationship between dependent and independent variable. I want to use measurement model to find out this. I came to know from http://en.wikipedia.org/wiki/Latent_variable_model that I could use latent variable model. I am very much sure that my manifest variables are categorical and I suppose my latent variable is also categorical (need to know), therefore Latent Class Analysis seems to be the best choice for my data. My question is that:   * Am I right that the Latent Class Analysis is best for my data or have to think of any other thing?   * Could you give an example when latent variable is continuous and manifest variable is categorical? these are the situations when there is need to use latent trait analysis. I just ask this question to clarify my latent variable as continuous or categorical.   * I know that for latent class analysis LatentGold software and Mplus are best but expensive to buy, Do you think Lem-a free software could serve my purpose, as I want to do factor and regression in Latent class analysis? or what to you think of SATA for it?   * It is possible in latent class analysis to do cluster method, factor analysis and regression for categorical data. Is it possible to do more than that in it like structural equation modelling and path analysis for categorical data etc?   * Latent class analysis is mostly used in medicine and psychology and less frequently in development or applied economics. Could I able to justify the use of latent class analysis for research related to business (mix of management and development economics area). Thanks for the answers in advance.


title: survival packages in R that allow for time dependent covariates

text: Which survival packages in R explicitly allow for time dependent covariates? By explicitly, I mean that the package allows its models to accept a survival object with the form               Surv(start,stop,event)~....      For example, the cox proportional hazards model with time dependent covariates has the form:               coxph(Surv(start, stop, arrest.time) ~ fin + age + age:stop + prio, data=Rossi.2)      I would like to identify packages with more powerful models that accept this form.


title: Memory-based learning: Predicting gender of French nouns

text: I'm trying to predict the gender of French nouns based on their suffixes. I have a corpus of 10k nouns. For each noun, I split the root from the suffix. I create five instances but with varying suffix lengths: 1, 2, 3, 4 and 5. The class name I'm assigning to each instance is composed of the gender and the length of the suffix, e.g. "f2" for a feminine noun with a suffix of the length 2. To give you an idea of how the data looks, here's a small extract: http://pastebin.com/tn6DFJAy Do you think this is a reasonable approach? Is it sensible to separate the classes based on the suffix length? And is it safe to generate five instances for each noun? For my project, I'm using TiMBL. The results for the data are: http://pastebin.com/5wai65i1


title: Looking for some intuition regarding the MCD estimator

text: The Minimum Covariance Determinant (MCD) estimator may be used to achieve robustness when estimating a covariance matrix. It looks for the subset of $h$ data points (out of $n > h$) whose covariance matrix has the smallest determinant and it computes the covariance of those $h$ points. Why does minimising the determinant do the job?


title: Partial correlation - scales?

text: Is there any way to conduct a partial correlation analysis if the DV is dichotomous?


title: Time series prediction model

text: I'm a programmer/database developer, not a statistician, so I'm just looking to be pointed in the right direction here. I've got enrollment data for conference sessions going back 5 years. I have weekly enrollment figures for each session 9 months before the start of each session and then I have the final tally after the session took place. I basically want to use this data to predict if a session will meet its minimum enrollment (40%) or if it will need to be cancelled. So, at its most basic, I have a session ID, a weekly enrollment number, and a max session size. The goal is to be able to cancel sessions earlier than they are currently cancelled (at the last minute). Since I also have a lot of demographic data on past session participants, I'm assuming that could be added into the mix. But right now I am just trying to keep it as simple as possible until I'm able to learn more. At this point, I'm mainly looking for the right way to frame the problem and for resources. Grateful for any pointers. Thanks.


title: Feature boosting via rescaling in logistic regression and linear SVMs

text: If I were expressing a problem in terms of binary features, all encoded as {0,1}, could I boost some features by encoding them as {0,2}? Would the effect change based on whether I used either of the approaches above (logistic regression or linear SVMs?) Any paper recommendations that would explain the mathematics? Thanks


title: how do I calculate the probabiity of a match based on similarity of binary events?

text: I have 35 binary events (presence/absence coded as 1/0) that describe 7 questioned samples (QS 1...7) and 1 known sample (KS). I am asked to provide a probability (in percentage) of how likely it is that one of the questioned samples (QS 1...7) **matches** the known sample (KS). Due to time and budget for this project, I coded presence/absence rather than exact frequency counts. So far I have two approaches and I need help figuring out what to do, so I thank anyone in advance! Approach 1: how many of the binary events are present, compared to the Known Sample? So this is a count of how many 1's occur in the 35 events. The KS contains 29 of the 35 binary events (29 1's and 6 0's). The 7 questioned samples range from the same number of events to very few events: QS1: 29 of 35 events, QS2: 6, QS3: 14, QS4: 16, QS5: 2, QS6: 4, QS7: 7. It is pretty obvious that QS1 is very similar to KS, while QS5 is not, just by counting the number of events in comparison to the KS number. But how do I get the probability that KS and QS1 are a "match"? Approach 2: how many times does a QS event match the KS event? So this is a count of how many times both KS and QS have matching 1 or 0 for each of the 35 events. The KS and QS1 have the same value on 25 of 35 events (matching both 1/presence and 0/absence). So we could say KS and QS1: match 25 don't match 10 KS and QS2: match 10 don't match 25 KS and QS3: match 16 don't match 19 KS and QS4: match 20 don't match 15 KS and QS5: match 6 don't match 19 KS and QS6: match 8 don't match 27 KS and QS7: match 9 don't match 26 Again it is pretty clear that QS1 is closer to matching the KS pattern than anyone else (including QS4). How likely is it that KS and QS1 are a match? I included the QS samples so that I could get a baseline of some sort for how often, even in a tiny tiny population, these events occur. So counting the presence of these events (summing the 1's across the QS samples), I get mostly 2's and 1's, with only one 6. The mean is 2.28 and standard deviation is 1.5. Would it be correct to run a binomial test on KS and QS1? IOW, what is the chance that two independent samples would get the same results 25 out of 35 times? That is not exactly an answer to the question what is the probability that KS and QS1 match. Again, thank you for your help and advice.


title: Building MATLAB and R interfaces to Ross Quinlan's C5.0

text: I'm considering building MATLAB and R interfaces to Ross Quinlan's C5.0 (for those not familiar with it, C5.0 is a decision tree algorithm and software package; an extension of C4.5), and I am trying to get a sense of the components I would need to write. The only documentation I found for C5.0 is here, which is a tutorial for See5 (a Windows interface to C5.0?) . The tar file comes with a Makefile, but no Readme files or any additional documentation. From what I read in the tutorial above, C5.0 uses an ASCII-based representation to handle inputs and outputs, and I am also considering building an interface that passes binary data directly between MATLAB or R and C5.0. Is C5.0's data representation used by any other machine- learning/classification software? Has anybody tried building a MATLAB or an R interface to ID3, C4.5 or C5.0 before? Thanks


title: Best bayesian network program for mac?

text: Does anyone know which is the best bayesian network program for mac that has a graphical interface and is free?


title: How to find out which distribution is it?

text: I have the following weights from a consumer price index.               P-Weight   Percentage     2.5746  19.54%     2.3506  17.84%     1.7585  13.35%     1.1223  8.52%     0.9277  7.04%     0.8048  6.11%     0.8709  6.61%     0.4867  3.69%     0.482   3.66%     0.3714  2.82%     0.5208  3.95%     0.3679  2.79%     0.3782  2.87%     0.1594  1.21%      https://docs.google.com/spreadsheet/ccc?key=0AuJFFXVdZ_QpdG15NXgtS0ZWYUhuM2U3bDFOYU1ubGc&hl=en_US How can I fit a distribution curve to the data? R or Google Spreadsheet are the tools


title: Is it valid to include a baseline measure as control variable when testing the effect of an independent variable on change scores?

text: I am attempting to run an OLS regression:   * **DV: Change in weight over a year (initial weight - end weight)**   * **IV: Whether or not you exercise.** However, it seems reasonable that heavier people will lose more weight per unit of exercise than thinner people. Thus, I wanted to include a control variable:   * **CV: Initial starting weight.** However, now **initial weight** is used BOTH to calculate the dependent variable AND as a control variable. Is this okay? Does this violate an assumption of OLS?


title: Variance of coefficients in nonlinear regression

text: I was reading the short intro to nonlinear regression in R by John Fox (Link) and was wondering where the formula for the variance of the coefficients come from (page 1). The formula for the variance-covariance matrix looks like linear regression but the predictor matrix X in that typical formulation is replaced by the partial derivatives of the function. Can anyone explain this? Thanks! Brian


title: What is explosive variable?

text: I'm trying to understand the following claim: > if the $t$-statistic is greater than zero, it indicates that the variable is > explosive... but does that mean it has unit root? In the context of **Dickey Fuller** test.


title: Stambaugh bias definition

text: How would you go about explainin "Stambaugh Bias" in simple relatively non- technical language?


title: Non-informative prior for regression model

text: I'm looking at p. 355 of Gelman's _Bayesian Data Analysis_ (3rd ed.), for which there is no errata, and I see this: > In the normal regression model, a convenient non-informative prior > distribution is uniform on $(\beta,\, \log\sigma)$ or, equivalently, > $$p(\beta,\, \sigma^2|X) \propto \sigma^{-2}$$ Shouldn't it say "uniform on $(\beta, \log\sigma^2)$"? If $W = g(X)$ with $g$ monotonic increasing, $h \equiv g^{-1}$, don't we have $$f_{W}(w) = f_{X}(h(w))\;h^\prime(w)$$ Let $X=\log\sigma^2$, $g=\exp$, $h=\log$; the density of $W=g(X)=\sigma^2$ is $f(w) = w^{-1}$ or, if I understand Gelman's notation correctly, $p(\sigma^2)=\sigma^{-2}$. Am I right or am I confused about notation... or just completely wrong?


title: Testing for the significance of the difference-in-differences of adj. R²

text: Is there a way to test for the significance of the difference-in-differences in adj. R²s in Stata? Let's say I have four subgroups: pre-treatment, pre-control, post-treatment, post-control and I want to figure out whether the difference-in-differences of the adj. R² of the same regression (e.g., R² of Y = b0 + b1*X + e) conducted for each of the four subgroups is signficant. In other words, is (adj. R²(post-treatment) - adj. R²(pre-treatment)) - ((adj. R²(post-control) - adj. R²(pre-control)) signficant? I read somewhere that it might be possible to bootstrap the adj. R² of each regression and then test for the difference-in-differences, but I am not sure if this would be an appropriate approach? Thanks a lot!


title: Down-sampling with building models (specifically random forests)

text: I was wondering if anyone had ever used down-sampling to build random forests with data that has unbalanced classes. Basically down-sampling samples (with replacement) x*min from the population where x is the number of classes and min is the size of the smallest class. So it is sampling min number of observations from each class. In some examples I have seen, it improves predictive power, but it seems like there is some sampling bias. This might be nullified by the fact that it is bootstrapping and building multiple trees and averaging them, but I dont fully understand it. If someone can help explain why this is not biased that would be great. I have heard that this technique is comparable with weighted random forests. I am working with a data set with 30,000,000 entries where one class has about 120 observations, so you can see why this would be helpful.


title: Model performance metrics for ordinal response

text: I'm interested in assessing model performance on data with an ordinal categorical dependent variable. For my use case, the ideal metric would: 1) Not assume equal intervals between classes or that recoding to a continuous scale is appropriate   2) Be scale independent   3) Give preference to models that _rank_ the outcomes accurately, with higher penalties for mis-ranking classes with a larger degree of difference (e.g., Excellent > Poor > Good is better than Excellent > Very Poor > Good)   4) Accept continuous predictions and be indifferent to their distributions For example, suppose we have the following test set, where "response" is 5-category ordinal response and "pred1", "pred2", and "pred3" are predictions:               id      response   pred1    pred2    pred3      1     Excellent    1.00      150       10      2          Good     .80       39        9      3          Good     .85       12        5      4          Fair     .40       11        4      5          Poor     .39       10        3      6     Very Poor     .20        3        2      .             .       .        .        .      .             .       .        .        .      For my purposes, the ideal metric would score all three predictions as equally accurate since all three perfectly rank the response. What are my options and the benefits/drawbacks to each? Bonus points for references to R packages or functions. Thanks!


title: Correlation vs Partial Correlation to explore relationship in data

text: I have a cognitive architecture solving a set of tasks. Also I have data of human subjects solving the same set of tasks. Now I want to see whether I can find a relationship between the performance of the architecture and that of human subjects (e.g: if its hard for the architecture its also hard for human subjects). I have 3 summary statistics describing the performance of the architecture, call them a,b,c. They all measure slightly different things. Think about: uncertainty, structure of the solution given etc. For the human data I only have access to one statistic (data taken from a published experiment ), which is related but not identical to a,b,c. If I compute the correlation coefficient, I see that b shows a high correlation to the performance of human subjects. However a,b,c have correlation (~0.2-0.7) in themselves. I read that one can compute the partial correlation coefficient which measures the relationship of two variables excluding the influence of the other. If I do this, c has the highest partial correlation to human data, although its correlation to the human data was the lowest I have trouble interpreting these results, and I am not sure whether I should do my interpretation based on correlation or partial correlation and if the use of partial correlation makes sense in this context. Also note: this is purely explorative, I don't aim to do hypothesis testing, I merely search for relationships.


title: Combining Standard deviation 3 groups

text: Is this the correct formula for combining the standard deviations of 3 groups? $$ \sigma = \sqrt{\frac{n_1\sigma_1^2 + n_2 \sigma_2^2+ n_3 \sigma_3^2+ n_1(\mu_1-\mu)^2 +n_2(\mu_2-\mu)^2+n_3(\mu_3-\mu)^2}{n_1 + n_2 +n_3 }} $$


title: Can I split a series of observations of a variable over time into two groups instead of working with time series?

text: First of all, I’m new to statistics and this is the first time I am trying to apply it to a real world problem. I am doing analysis of a series of observations of a variable over all weeks of a year. During certain weeks an event happened that I believe has impacted the variable and I want to check for this. The values are technically a time series, however I only need to know that the event has an impact so I did an uncorrelated t-test. My reasoning is that, if I split the series of observations into a group of observations when the event happened and one where it didn’t happen, then I can compare the mean of each group using a t-test. Is this reasoning valid? Many thanks, David @GaBorgulya, I only have ONE value for each week. @Gavin Simpson No, I don't think that the values are autocorrelated. I am working with human behaviour and I believe that the event has an effect but it is not as deterministic as what it would be if the data were autocorrelated. @Wayne, the event is random.


title: Binomial GLM post-hoc tests for unequal sample sizes

text: I have a set of data that are binomial, and am comparing them across 9 years. The first 5 years have low sample sizes ($~n=20$) and the last 4 have $n>100$. I've run a glm in R with the family set to "binomial", and the results look reasonable. However, when I did the multiple comparisons afterwards using the multcomp package, as shown below               cs_comp<-glht(model2, linfct =  mcp(bin_Year= "Tukey"))       I got some really unexpected results that do not coincide with a bar graph I made of the means $\pm$ SE (essentially, years that were not significantly different from each other that I'd expect to be AND years that were significantly different that I didn't expect to be). I've been told that this is likely due to the unequal sample sizes; of the 9 years of data I have, the last 4 years have 100+ data points, while the first 5 years have 20-25 data points. I've also had the suggestion that I use Wald's CIs to determine pairwise comparisons by looking for overlapping CI's. Does anyone know if this is a better approach to unequal samples sizes of this magnitude, or if there is another (even better) multiple comparison method for this? If so, any tips on how to approach it in R? I have figured out how to make confidence intervals in R, but not Wald's. Thanks in advance! EDIT: I've found somewhere that one can use `confint.default(model)` to obtain Wald's 95% CI's, however, this results in completely different numbers than were provided to me by a friend in a software program I don't have access to (and can't remember what it is). The reason I'm searching now for this is the CI's that he obtained when running this data set for me are only to 2 decimal places, and I need more to determine if there is true overlap. Furthermore, if I've obtained CI's from the `'confint.default()` code that are negative or higher than 1, I'm almost certain they cannot be correct, since I have a binomial data set with only 0's and 1's... Thanks again for any tips!


title: Why isn't this a problem with heritability studies?

text: In many studies of heritability, researchers study identical twins raised apart and raised together, and see how much a trait correlates between the two types of twins. Sometimes they use fraternal twins, there are a few variations on the scheme. But while we can measure how similar two people are in terms of their genes (at least, we can do this pretty well) we have no good measures of how much the environment varies. Human environments vary tremendously, of course. They vary over both time and place, in ways that are impossible to measure. This leads to three sorts of problems: 1) If identical twins are assumed to have identical genes, then any variation is due to non-genetic factors. What factors are there that are _not_ environment or genes? 2) If we assume that fraternal twins have 50% genes in common, then we can say they have half as much in common as identical twins. But if we put one twin in a farmhouse in Iowa and another in a farmhouse in the next county over in Iowa, then environment won't vary much; on the other hand, if one twin is put in a farmhouse in Iowa and the other in an apartment in New York City or Lagos .... well, that's a _lot_ of variation. But how much? It seems to me there's no way to turn this into a number, and so it seems to me that heritability coefficients are a kludge and inherently silly. There is a third problem, which @henrik 's comment brought back to mind: Genes and environment can interact. This makes for even more problems, because, as we all know, estimating main effects in the presence of interaction is problematic. For an extreme case, take the problem of phenylketonuria. Mortality in this disease is entirely due to an interaction between genetics and environment. Less extreme cases abound. But a lot of smart people use them, so maybe I am missing something.


title: How can I prove the experiment data follows heavy-tail distribution?

text: I have several test results of server response delay. According to our theory analysis, the delay distribution (The probability distribution function of response delay) should have heavy-tail behavior. But how could I prove that the test result does follow heavy-tail distribution?


title: If you use 10-fold cross validation, which tree is representative?

text: If you use 10-fold cross validation to derive the error in, say, a C4.5 algorithm, then you are essentially building 10 separate trees on 90% of the data to test on 10% - 10 times. Which one of the 10 trees is representative? Won't they all be different? For example - how does WEKA give me a C4.5 tree and a cross-validation error, but only one. I feel I must have fundamentally misunderstood this. Thanks for any help


title: Binary Classification of Multiple Groups

text: I've ran across a type of classification problem that I don't think fits into the traditional multi-class framework. Just wanted to run it across you guys to see if you had any ideas. So Lets say we have 3 drugs - drug A, B, C - and we serve it to 100 patients The for each patient we have three types of information:   1. Patient attributes   2. Type of drug given (A,B, or C)   3. Patient survival (dead or alive) This is our data set. For future patients, we want to be able to give them the drug (either A,B, or C) that maximizes their chance of survival. I'm coming from more of a machine learning/data analysis background, so I'm trying to figure out how to use the data to solve this problem. Typically, in a classification problem, I'd just set the patient survival as the class variable and all of the demographic data as my input features. However, now the class label should be the "type of drug given" but each drug is its own separate binary classification problem. One idea would be to just use the data for the "alive" patients - and reduce it to a multiclass problem (with class labels A,B and C). The rationale behind that approach is that you are trying to maximize survival of future patients, so the patients that have survived from drug X previously will give you more information than those who died from drug X. The major problem with this approach is that you are not using a large portion of the data. Any ideas?


title: Realtime data change detection

text: I have been asked to do some real time data analysis. The data values represent parameters of phone calls for a telco (like number of calls, call length, etc.) If the numbers suddenly drop or spike, I'm supposed to raise an alarm. Normal noise is expected, but we want to know if things change suddenly, like if no one's routing calls over a line. So individual outliers are not an issue, but if the whole set changes significantly, we want to know. I can handle the programming, but I'm not sure what statistical quantities I should be checking. My first guess was to compute a moving average (of number of calls per minute, for example) and a moving standard deviation, and trigger an alarm if the average changes more than one standard deviation. But looking over this site, I think that might be terribly naïve. What would be a good way to detect these changes?


title: F-test formula under robust standard error

text: I am attempting to write a program that will (among other things) use the F-test in multivariate regression under standard robust errors. I am having trouble finding a specific formula for the F-statistic using robust errors. I understand the purpose of robust errors. I simply need the modified formula for the F-statistic. Can anyone help?


title: Statistical test to measure correlation between inter-rater mistakes

text: I'm working on a project that involves collecting crowd-sourced labels for discussion forum posts. The task given to annotators was to look at a forum post and to decide which of six classes the post belonged to. Each post was labelled once by five different annotators, so we have five redundant labels per post. These labels will eventually be used as the ground-truth for text- based machine learning experiments. So far, we have labels and have measured inter-annotator agreement via Fleiss Kappa. Our scores for agreement aren't great, but they're not horrible. Now that we have the labelled data, I would like to get some further insight on what types of errors annotators are making. The labelled data I'm working with looks something like this:               id class1 class2 class3 class4 class5 class6     1  1      4      0      0      0      0     2  2      2      0      1      0      0     3  0      0      2      3      0      0     4  0      1      2      1      1      0     5  1      3      0      1      0      0      where each class is denoted by a column and the value within each cell under the class columns denotes how many annotators labelled the post as belonging to that particular class. The sum of the values under the class columns for each row should be five, corresponding to the five redundant labels we were looking for. Now it's time for error analysis. What I have done thus far is create a subset of the data in which there is no "majority vote" consensus on what the class should be. That is, the subset of data is made of only observations where no one class has a value of three or greater in the column (since there are five annotators for each post). So, now I have a subset of instances where disagreement seems highest. I'm interested in measuring how mistakes are made between different classes. That is, do annotators tend to confuse class1 with class3? Why? Is there a way to quantify this? Is there a test to see which classes correlate with one another,and would this be easily interpreted for this kind of error analysis? Any help would be appreciated, and doubly so if suggestions can be made for R in particular. Thanks for reading.


title: Compare two small groups (population vs sample) over time

text: I am studying banks behaviour according to 6 financial rations throughout a 3-years period. I have 32 observations separated into two groups: large (6) and medium-sized banks (26). However, since in my country there are exactly 6 large banks, it does not constitute a sample, but a population, where as there are around 32 medium-sized banks (so my sample is quite large compared to the population). To compare these two groups each year, I am using the Wilcoxon Signed Rank Test to test: $$ H_0: \mu_{i,medium} = \mu_{i,large} $$ where $ \mu_{i,large} $ is the mean of financial ration $i$ of the $large$ group, which is a known parameter, which means I should not be comparing two samples. Is the Signed Rank Test adequate here? However, I figure that since I have data over 3 years, it'd be better to compare these groups throughout the 3-years period as whole (or not?). What technique should I use to do so? Are there any test I should do prior to selecting a test? Thanks Bernardo


title: Methods to increase significance in regression

text: What are the potential reasons why the relationships are insignificant in my regression model, and what can I do to increase significance?