Publish Qwen3 typed decisions cloud LoRA pilot with reproducibility receipts

README.md

@@ -0,0 +1,47 @@
+---
+base_model: Qwen/Qwen3-0.6B
+license: apache-2.0
+library_name: peft
+pipeline_tag: text-generation
+tags:
+  - lora
+  - qwen3
+  - boolq
+  - calibration
+  - research-pilot
+---
+
+# Qwen3 0.6B LoRA for grounded yes/no decisions — research pilot
+
+This PEFT adapter was trained in [private Kaggle kernel version 1](https://www.kaggle.com/code/sankalpsthakur/qwen3-06b-typed-decisions-cloud-pilot) on 2026-09-30. It is a small, reproducible baseline for typed binary decisions. It is **not** Jev or a reproduction of Jev's undisclosed architecture. It does not establish a new small-model architecture or a general frontier-model breakthrough.
+
+## Inputs and method
+
+- Base: [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) at commit `c1899de289a04d12100db370d81485cdf75e47ca` (Apache-2.0).
+- Data: [Praveenrajus/jev-bench](https://huggingface.co/datasets/Praveenrajus/jev-bench) at commit `18f88da81c28c2bec55edc31f63f2afdfba109ea`; 1,200 BoolQ training records and 128 BoolQ validation records. The dataset card identifies BoolQ as CC-BY-SA-3.0 and grounded StrategyQA as MIT. No source passages are redistributed here.
+- Training: 600 steps, batch size 2, maximum sequence length 512, seed `20260930`, AdamW at `1e-4`, LoRA rank 8, alpha 16, dropout 0.05 on `q_proj` and `v_proj`, 1,146,880 trainable parameters. Kaggle assigned a Tesla T4; script runtime was 254.2 seconds.
+- Prompt: supply the dataset's evidence and question, request exactly `Yes` or `No`, and disable Qwen3 thinking in the chat template. Training loss is applied only to the answer token. Evaluation scores the shared prefix's next-token logits for `Yes` and `No`, then normalizes those two logits. Thus `p_yes` is a *conditional two-label score*, not the model's unrestricted probability of answering Yes.
+
+The adapter contains only LoRA weights. Load it with the pinned base and the prompt/readout protocol in `run.py`; generic free-form generation is outside the measured setting. The downloaded adapter's SHA-256 is `827fd0d08c308a89df55343ad7045d9d5b9ac22394e4fa876bdef8c2a001c845`.
+
+## Evaluation
+
+The base and adapted model were compared on the same records. The BoolQ confirmation set contains 256 test records selected after excluding all 384 local-pilot test/confirmation IDs. The transfer set contains 128 `strategyqa_grounded/test` records. The table reports **raw** scores, before calibration.
+
+| Split | N | Base accuracy | LoRA accuracy | Paired accuracy gain, 95% bootstrap CI | Base → LoRA Brier | Base → LoRA ECE (10 bins) |
+|---|---:|---:|---:|---:|---:|---:|
+| BoolQ validation | 128 | 67.2% | 78.9% | +11.7 points [2.3, 21.1] | 0.216 → 0.193 | 0.117 → 0.189 |
+| BoolQ confirmation | 256 | 71.1% | 78.9% | +7.8 points [2.0, 13.7] | 0.205 → 0.195 | 0.122 → 0.190 |
+| Grounded StrategyQA transfer | 128 | 55.5% | 67.2% | +11.7 points [2.3, 21.1] | 0.289 → 0.283 | 0.181 → 0.278 |
+
+The raw Brier reductions on both held-out sets have paired bootstrap intervals spanning zero. Raw ECE is worse after LoRA. These are narrow sample estimates, and the 128-record transfer set is one source rather than a broad out-of-distribution suite. The reported bootstrap intervals are exploratory and do not account for all experiment choices.
+
+After inspecting those raw results, we fitted separate positive temperatures for the base and adapter by minimizing NLL on the 128 validation records only. This **posthoc exploratory** analysis leaves accuracy unchanged. On BoolQ confirmation, calibrated Brier is 0.190 base versus 0.158 adapter, paired reduction 0.032 [0.007, 0.057]. On grounded StrategyQA, it is 0.257 versus 0.211, paired reduction 0.045 [0.006, 0.084]. Calibrated ECE is 0.031 versus 0.035 on BoolQ and 0.086 versus 0.107 on StrategyQA: the adapter still has slightly higher ECE. Prospective calibration claims require a fresh untouched test set.
+
+## Reproduce and inspect
+
+`run.py` is the complete Kaggle training/evaluation script. `receipt.json` records inputs, versions, settings, and scores. `analyze.py` checks the label-only predictions, split IDs, paired metrics, and adapter hash; `calibrate.py` fits validation-only temperatures and reproduces the exploratory scores. `excluded-ids.json` lists prior pilot IDs used for the no-overlap check. `verified_summary.json`, `calibration_receipt.json`, and the `*-predictions.json` files preserve the analysis inputs. All prediction files contain IDs, binary labels, and scores, with no passage or question text. Run `python3 analyze.py` and `python3 calibrate.py` from this directory to check the receipts.
+
+## Scope, attribution, and release
+
+This work tests a Qwen3 answer-token LoRA under a constrained binary decision protocol. It does not validate open-ended answers, safety use, deployment readiness, or Jev parity. The Kaggle worker evaluated the in-memory adapted model after saving, and the downloaded Safetensors structure and hash were checked; a separate adapter reload/inference test has not yet been run. Credit the Qwen team and the creators and maintainers of BoolQ, StrategyQA, and `jev-bench` when using these artifacts. The adapter and authored evaluation code are released under Apache-2.0. The underlying base model and source datasets retain their own licenses; the mixed-source `jev-bench` license does not supply a single blanket license for every source. This repository does not redistribute the source passages or questions.

SHA256SUMS

@@ -0,0 +1,22 @@
+920c9ad5b4c5256eb5f4e861ee7bcb6e70e8bd4468cac2e1debbaa3b888e69e3  README.md
+057a258f257ed3a883eca2726c1fd9c4dcc61cc1fccbf67350af15e692bd29a9  adapted-confirm-calibrated.json
+5d288087ca6da762dca514899f772da83f124954540b577a3a1553e5a2132f8c  adapted-confirm-predictions.json
+3cdc577a4b784c9af03ea31a11e058d3183d5f1e1b97d00490f4377b036e60e7  adapted-ood-calibrated.json
+579f55423fa2f870f2eb393cd953e55a2e9ec927f2af468415349078f8679128  adapted-ood-predictions.json
+297c2ec9c8b57a7490b3910010895efd61315b350149b2b5630745e7fee0e5d3  adapted-valid-calibrated.json
+e8b1c1214d1d13c0b525908ac3039c8ce0a27eb9d9a0c3b7913cfd2a91e9898d  adapted-valid-predictions.json
+93d1daec82e814db4ec9d1acf96010ff477ae88f76e198d47f3f8075cb92bfd2  adapter_config.json
+827fd0d08c308a89df55343ad7045d9d5b9ac22394e4fa876bdef8c2a001c845  adapter_model.safetensors
+5542713b80a70432c4fa254a1694e18d3b019820cfe7266078300467047fec9f  analyze.py
+63209063acb4b7a50ca83658ed048439761d4d43bf77f26258f28cb427c90c8c  base-confirm-calibrated.json
+ab1f8d7bb928790260918f7b5335bb92a04e0ebf5e50a5bb699f0081e8622660  base-confirm-predictions.json
+305297c054cb29840728f959c831236b984e0bf990c6a3861eff42571ab31654  base-ood-calibrated.json
+74ace1b83ed0978a99715def47ba15331f3a1f249a5e0e56e71b8c1b519028e2  base-ood-predictions.json
+608a5df1d39e7e0527e632184f17f4d1e82ccee3e52f5cf3c7b269c9a26b3bf4  base-valid-calibrated.json
+7edb8513e98532468fc9b8b876554e3edf0ead35c661ee0466f15f2ff72e7fb5  base-valid-predictions.json
+bade30641c71ff4038d797ef76376a40227be4e21c2709a1c83da00b839865ec  calibrate.py
+cc8f52d57fea5bd796cb0c857a4a0dd2e0ae20393c49c59baa1155dffc7610d8  calibration_receipt.json
+eddae72ed5679088f43609c9d0f9ad76efae9a4c5bad885cc22cb9fcae3fc751  excluded-ids.json
+8f4159fe59038b2649cc463030b922f5a3c783ae8cae0916fa2b8e15a29d78ad  receipt.json
+0102ff40ee1da784a91b49b88efdb693d3e7b097f3538783a02d407ea8b68f18  run.py
+f9bb6ea47764b8362dc16c41cd7564bbdf118fe1f2bed3161be8f22c977e486d  verified_summary.json

adapted-confirm-calibrated.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "boolq/test/314", "label": 1, "p_yes": 0.9108330389059919, "predicted": 1}, {"index": 1, "id": "boolq/test/2058", "label": 1, "p_yes": 0.1618514921118938, "predicted": 0}, {"index": 2, "id": "boolq/test/1125", "label": 0, "p_yes": 0.7098071309987064, "predicted": 1}, {"index": 3, "id": "boolq/test/611", "label": 1, "p_yes": 0.82381268845126, "predicted": 1}, {"index": 4, "id": "boolq/test/1646", "label": 1, "p_yes": 0.887321424912614, "predicted": 1}, {"index": 5, "id": "boolq/test/203", "label": 0, "p_yes": 0.8013094070315986, "predicted": 1}, {"index": 6, "id": "boolq/test/2275", "label": 0, "p_yes": 0.5290193004804702, "predicted": 1}, {"index": 7, "id": "boolq/test/80", "label": 0, "p_yes": 0.27449805919458276, "predicted": 0}, {"index": 8, "id": "boolq/test/90", "label": 1, "p_yes": 0.47010339439554294, "predicted": 0}, {"index": 9, "id": "boolq/test/2253", "label": 1, "p_yes": 0.5802922393248645, "predicted": 1}, {"index": 10, "id": "boolq/test/1398", "label": 1, "p_yes": 0.5742770261813812, "predicted": 1}, {"index": 11, "id": "boolq/test/2803", "label": 0, "p_yes": 0.6791935732583909, "predicted": 1}, {"index": 12, "id": "boolq/test/1768", "label": 1, "p_yes": 0.9003476329397416, "predicted": 1}, {"index": 13, "id": "boolq/test/1525", "label": 1, "p_yes": 0.8273592994882191, "predicted": 1}, {"index": 14, "id": "boolq/test/754", "label": 0, "p_yes": 0.8589438083537909, "predicted": 1}, {"index": 15, "id": "boolq/test/1284", "label": 0, "p_yes": 0.9357077016149834, "predicted": 1}, {"index": 16, "id": "boolq/test/1567", "label": 1, "p_yes": 0.2829932508609895, "predicted": 0}, {"index": 17, "id": "boolq/test/172", "label": 1, "p_yes": 0.8528585047952576, "predicted": 1}, {"index": 18, "id": "boolq/test/851", "label": 0, "p_yes": 0.8492893201196293, "predicted": 1}, {"index": 19, "id": "boolq/test/604", "label": 0, "p_yes": 0.11446112492722961, "predicted": 0}, {"index": 20, "id": "boolq/test/2912", "label": 1, "p_yes": 0.8716540767663458, "predicted": 1}, {"index": 21, "id": "boolq/test/3209", "label": 1, "p_yes": 0.9122550658394967, "predicted": 1}, {"index": 22, "id": "boolq/test/1739", "label": 1, "p_yes": 0.90094335995585, "predicted": 1}, {"index": 23, "id": "boolq/test/2061", "label": 0, "p_yes": 0.44563335378727675, "predicted": 0}, {"index": 24, "id": "boolq/test/1579", "label": 0, "p_yes": 0.8684714995954412, "predicted": 1}, {"index": 25, "id": "boolq/test/2343", "label": 1, "p_yes": 0.9131105138041128, "predicted": 1}, {"index": 26, "id": "boolq/test/756", "label": 1, "p_yes": 0.8696663339012921, "predicted": 1}, {"index": 27, "id": "boolq/test/144", "label": 0, "p_yes": 0.09841237934149176, "predicted": 0}, {"index": 28, "id": "boolq/test/211", "label": 0, "p_yes": 0.4257229787768887, "predicted": 0}, {"index": 29, "id": "boolq/test/842", "label": 1, "p_yes": 0.3387039784331128, "predicted": 0}, {"index": 30, "id": "boolq/test/2287", "label": 1, "p_yes": 0.4094521574111092, "predicted": 0}, {"index": 31, "id": "boolq/test/913", "label": 0, "p_yes": 0.15435307697922487, "predicted": 0}, {"index": 32, "id": "boolq/test/2084", "label": 1, "p_yes": 0.5595801667133397, "predicted": 1}, {"index": 33, "id": "boolq/test/2984", "label": 1, "p_yes": 0.3725776922590558, "predicted": 0}, {"index": 34, "id": "boolq/test/909", "label": 1, "p_yes": 0.8510834915201815, "predicted": 1}, {"index": 35, "id": "boolq/test/907", "label": 0, "p_yes": 0.2395706482850105, "predicted": 0}, {"index": 36, "id": "boolq/test/1217", "label": 1, "p_yes": 0.9037718333587821, "predicted": 1}, {"index": 37, "id": "boolq/test/2302", "label": 1, "p_yes": 0.6822549839698582, "predicted": 1}, {"index": 38, "id": "boolq/test/232", "label": 1, "p_yes": 0.8541813056722037, "predicted": 1}, {"index": 39, "id": "boolq/test/1308", "label": 1, "p_yes": 0.8778280013573592, "predicted": 1}, {"index": 40, "id": "boolq/test/2048", "label": 1, "p_yes": 0.8308535397894462, "predicted": 1}, {"index": 41, "id": "boolq/test/1982", "label": 1, "p_yes": 0.9200923778708737, "predicted": 1}, {"index": 42, "id": "boolq/test/534", "label": 1, "p_yes": 0.8897435553440497, "predicted": 1}, {"index": 43, "id": "boolq/test/773", "label": 1, "p_yes": 0.90129037469954, "predicted": 1}, {"index": 44, "id": "boolq/test/2366", "label": 1, "p_yes": 0.3410742359937472, "predicted": 0}, {"index": 45, "id": "boolq/test/632", "label": 1, "p_yes": 0.5088027108828355, "predicted": 1}, {"index": 46, "id": "boolq/test/3165", "label": 1, "p_yes": 0.7648907182581244, "predicted": 1}, {"index": 47, "id": "boolq/test/1249", "label": 1, "p_yes": 0.9218870377064358, "predicted": 1}, {"index": 48, "id": "boolq/test/2473", "label": 0, "p_yes": 0.22025178033795986, "predicted": 0}, {"index": 49, "id": "boolq/test/3049", "label": 0, "p_yes": 0.920397764769921, "predicted": 1}, {"index": 50, "id": "boolq/test/2500", "label": 1, "p_yes": 0.9259147007915808, "predicted": 1}, {"index": 51, "id": "boolq/test/3130", "label": 0, "p_yes": 0.6541614220201916, "predicted": 1}, {"index": 52, "id": "boolq/test/812", "label": 1, "p_yes": 0.8848164022081814, "predicted": 1}, {"index": 53, "id": "boolq/test/2640", "label": 0, "p_yes": 0.7939188264893192, "predicted": 1}, {"index": 54, "id": "boolq/test/457", "label": 1, "p_yes": 0.8980983239596669, "predicted": 1}, {"index": 55, "id": "boolq/test/1094", "label": 1, "p_yes": 0.8951636796683413, "predicted": 1}, {"index": 56, "id": "boolq/test/1834", "label": 1, "p_yes": 0.777928645883057, "predicted": 1}, {"index": 57, "id": "boolq/test/733", "label": 1, "p_yes": 0.905896575399351, "predicted": 1}, {"index": 58, "id": "boolq/test/1503", "label": 0, "p_yes": 0.1681581704439197, "predicted": 0}, {"index": 59, "id": "boolq/test/1766", "label": 1, "p_yes": 0.9198912930895248, "predicted": 1}, {"index": 60, "id": "boolq/test/3252", "label": 1, "p_yes": 0.935246523081123, "predicted": 1}, {"index": 61, "id": "boolq/test/2379", "label": 0, "p_yes": 0.8433325232202011, "predicted": 1}, {"index": 62, "id": "boolq/test/2693", "label": 1, "p_yes": 0.8822840163208129, "predicted": 1}, {"index": 63, "id": "boolq/test/2493", "label": 1, "p_yes": 0.8013094070315986, "predicted": 1}, {"index": 64, "id": "boolq/test/1240", "label": 0, "p_yes": 0.8766864546078232, "predicted": 1}, {"index": 65, "id": "boolq/test/1159", "label": 1, "p_yes": 0.9206039282069401, "predicted": 1}, {"index": 66, "id": "boolq/test/2064", "label": 1, "p_yes": 0.6315289245789062, "predicted": 1}, {"index": 67, "id": "boolq/test/355", "label": 0, "p_yes": 0.19424415013567772, "predicted": 0}, {"index": 68, "id": "boolq/test/1047", "label": 1, "p_yes": 0.9161478875989734, "predicted": 1}, {"index": 69, "id": "boolq/test/2897", "label": 1, "p_yes": 0.8668499509897473, "predicted": 1}, {"index": 70, "id": "boolq/test/1804", "label": 0, "p_yes": 0.11446112492722961, "predicted": 0}, {"index": 71, "id": "boolq/test/3040", "label": 0, "p_yes": 0.10126042615358141, "predicted": 0}, {"index": 72, "id": "boolq/test/2231", "label": 0, "p_yes": 0.1618514921118938, "predicted": 0}, {"index": 73, "id": "boolq/test/2001", "label": 1, "p_yes": 0.8465622252173536, "predicted": 1}, {"index": 74, "id": "boolq/test/2528", "label": 0, "p_yes": 0.9182569927927935, "predicted": 1}, {"index": 75, "id": "boolq/test/44", "label": 1, "p_yes": 0.9052698321621869, "predicted": 1}, {"index": 76, "id": "boolq/test/1207", "label": 0, "p_yes": 0.9067591125611503, "predicted": 1}, {"index": 77, "id": "boolq/test/729", "label": 1, "p_yes": 0.9153274258459845, "predicted": 1}, {"index": 78, "id": "boolq/test/3111", "label": 0, "p_yes": 0.33164194936006763, "predicted": 0}, {"index": 79, "id": "boolq/test/1196", "label": 1, "p_yes": 0.8704658448177552, "predicted": 1}, {"index": 80, "id": "boolq/test/2801", "label": 1, "p_yes": 0.8018691288936094, "predicted": 1}, {"index": 81, "id": "boolq/test/2768", "label": 1, "p_yes": 0.8652124625579216, "predicted": 1}, {"index": 82, "id": "boolq/test/1341", "label": 0, "p_yes": 0.3019341348551866, "predicted": 0}, {"index": 83, "id": "boolq/test/2979", "label": 1, "p_yes": 0.8676695966874346, "predicted": 1}, {"index": 84, "id": "boolq/test/1550", "label": 0, "p_yes": 0.15071158745924726, "predicted": 0}, {"index": 85, "id": "boolq/test/3208", "label": 1, "p_yes": 0.8876544810397744, "predicted": 1}, {"index": 86, "id": "boolq/test/719", "label": 1, "p_yes": 0.8696663339012921, "predicted": 1}, {"index": 87, "id": "boolq/test/2644", "label": 1, "p_yes": 0.7076263941551643, "predicted": 1}, {"index": 88, "id": "boolq/test/212", "label": 0, "p_yes": 0.2858600878154343, "predicted": 0}, {"index": 89, "id": "boolq/test/1411", "label": 0, "p_yes": 0.14936436163311484, "predicted": 0}, {"index": 90, "id": "boolq/test/2440", "label": 0, "p_yes": 0.8414617854062298, "predicted": 1}, {"index": 91, "id": "boolq/test/2851", "label": 0, "p_yes": 0.1882531696367011, "predicted": 0}, {"index": 92, "id": "boolq/test/1057", "label": 1, "p_yes": 0.8160137686422831, "predicted": 1}, {"index": 93, "id": "boolq/test/267", "label": 1, "p_yes": 0.8253382216792983, "predicted": 1}, {"index": 94, "id": "boolq/test/1674", "label": 0, "p_yes": 0.13643290724544585, "predicted": 0}, {"index": 95, "id": "boolq/test/269", "label": 0, "p_yes": 0.26754110010028076, "predicted": 0}, {"index": 96, "id": "boolq/test/316", "label": 0, "p_yes": 0.18240513445909606, "predicted": 0}, {"index": 97, "id": "boolq/test/1855", "label": 1, "p_yes": 0.45085887733247837, "predicted": 0}, {"index": 98, "id": "boolq/test/2000", "label": 0, "p_yes": 0.7875106497835266, "predicted": 1}, {"index": 99, "id": "boolq/test/2793", "label": 1, "p_yes": 0.811207981150154, "predicted": 1}, {"index": 100, "id": "boolq/test/696", "label": 1, "p_yes": 0.8160137686422831, "predicted": 1}, {"index": 101, "id": "boolq/test/905", "label": 1, "p_yes": 0.8362302512813161, "predicted": 1}, {"index": 102, "id": "boolq/test/2374", "label": 1, "p_yes": 0.9099958904958743, "predicted": 1}, {"index": 103, "id": "boolq/test/264", "label": 1, "p_yes": 0.8233010921815768, "predicted": 1}, {"index": 104, "id": "boolq/test/131", "label": 1, "p_yes": 0.9216674324902706, "predicted": 1}, {"index": 105, "id": "boolq/test/2917", "label": 1, "p_yes": 0.8826474780767039, "predicted": 1}, {"index": 106, "id": "boolq/test/2060", "label": 0, "p_yes": 0.8755455380456676, "predicted": 1}, {"index": 107, "id": "boolq/test/685", "label": 1, "p_yes": 0.8165409515685887, "predicted": 1}, {"index": 108, "id": "boolq/test/2063", "label": 0, "p_yes": 0.20839475395871923, "predicted": 0}, {"index": 109, "id": "boolq/test/3063", "label": 1, "p_yes": 0.8876544810397744, "predicted": 1}, {"index": 110, "id": "boolq/test/953", "label": 0, "p_yes": 0.5517553149891444, "predicted": 1}, {"index": 111, "id": "boolq/test/85", "label": 1, "p_yes": 0.5026410611378109, "predicted": 1}, {"index": 112, "id": "boolq/test/2513", "label": 1, "p_yes": 0.9260507981755042, "predicted": 1}, {"index": 113, "id": "boolq/test/408", "label": 1, "p_yes": 0.9200923778708737, "predicted": 1}, {"index": 114, "id": "boolq/test/2739", "label": 0, "p_yes": 0.06876150558697654, "predicted": 0}, {"index": 115, "id": "boolq/test/369", "label": 0, "p_yes": 0.5613151758181191, "predicted": 1}, {"index": 116, "id": "boolq/test/2916", "label": 1, "p_yes": 0.9361818418652315, "predicted": 1}, {"index": 117, "id": "boolq/test/2674", "label": 1, "p_yes": 0.7426819427220844, "predicted": 1}, {"index": 118, "id": "boolq/test/2639", "label": 1, "p_yes": 0.7352100122512044, "predicted": 1}, {"index": 119, "id": "boolq/test/574", "label": 1, "p_yes": 0.9296522236196695, "predicted": 1}, {"index": 120, "id": "boolq/test/2978", "label": 1, "p_yes": 0.7767088223724078, "predicted": 1}, {"index": 121, "id": "boolq/test/1546", "label": 0, "p_yes": 0.07413243687369955, "predicted": 0}, {"index": 122, "id": "boolq/test/201", "label": 1, "p_yes": 0.8766864546078232, "predicted": 1}, {"index": 123, "id": "boolq/test/3008", "label": 1, "p_yes": 0.8720557295553081, "predicted": 1}, {"index": 124, "id": "boolq/test/3212", "label": 0, "p_yes": 0.4317602575080785, "predicted": 0}, {"index": 125, "id": "boolq/test/2946", "label": 0, "p_yes": 0.44650347248514, "predicted": 0}, {"index": 126, "id": "boolq/test/516", "label": 0, "p_yes": 0.6116532163428097, "predicted": 1}, {"index": 127, "id": "boolq/test/1182", "label": 1, "p_yes": 0.7119782458970458, "predicted": 1}, {"index": 128, "id": "boolq/test/1257", "label": 0, "p_yes": 0.8367110301436042, "predicted": 1}, {"index": 129, "id": "boolq/test/3231", "label": 1, "p_yes": 0.11955395224228299, "predicted": 0}, {"index": 130, "id": "boolq/test/3027", "label": 1, "p_yes": 0.915898427893708, "predicted": 1}, {"index": 131, "id": "boolq/test/2605", "label": 1, "p_yes": 0.18717927639163084, "predicted": 0}, {"index": 132, "id": "boolq/test/3062", "label": 0, "p_yes": 0.18987286766185998, "predicted": 0}, {"index": 133, "id": "boolq/test/42", "label": 1, "p_yes": 0.8614766776182219, "predicted": 1}, {"index": 134, "id": "boolq/test/2676", "label": 0, "p_yes": 0.11918378144848629, "predicted": 0}, {"index": 135, "id": "boolq/test/1624", "label": 1, "p_yes": 0.8996839563419411, "predicted": 1}, {"index": 136, "id": "boolq/test/2661", "label": 0, "p_yes": 0.62329722831595, "predicted": 1}, {"index": 137, "id": "boolq/test/2240", "label": 0, "p_yes": 0.11517692854423674, "predicted": 0}, {"index": 138, "id": "boolq/test/1575", "label": 1, "p_yes": 0.8819251652574759, "predicted": 1}, {"index": 139, "id": "boolq/test/300", "label": 0, "p_yes": 0.3269754329729784, "predicted": 0}, {"index": 140, "id": "boolq/test/2337", "label": 1, "p_yes": 0.8759266406246355, "predicted": 1}, {"index": 141, "id": "boolq/test/3224", "label": 0, "p_yes": 0.11481854005801384, "predicted": 0}, {"index": 142, "id": "boolq/test/739", "label": 1, "p_yes": 0.8191660752206, "predicted": 1}, {"index": 143, "id": "boolq/test/1482", "label": 0, "p_yes": 0.2837083259997427, "predicted": 0}, {"index": 144, "id": "boolq/test/18", "label": 1, "p_yes": 0.7453650861584264, "predicted": 1}, {"index": 145, "id": "boolq/test/395", "label": 0, "p_yes": 0.9034354168921619, "predicted": 1}, {"index": 146, "id": "boolq/test/838", "label": 0, "p_yes": 0.11410467303759952, "predicted": 0}, {"index": 147, "id": "boolq/test/111", "label": 1, "p_yes": 0.8883595164473734, "predicted": 1}, {"index": 148, "id": "boolq/test/1465", "label": 1, "p_yes": 0.8248323337384444, "predicted": 1}, {"index": 149, "id": "boolq/test/1267", "label": 1, "p_yes": 0.8263529530933897, "predicted": 1}, {"index": 150, "id": "boolq/test/1863", "label": 1, "p_yes": 0.08828800914865792, "predicted": 0}, {"index": 151, "id": "boolq/test/2382", "label": 1, "p_yes": 0.6988074811284396, "predicted": 1}, {"index": 152, "id": "boolq/test/1340", "label": 1, "p_yes": 0.9082326535536567, "predicted": 1}, {"index": 153, "id": "boolq/test/831", "label": 0, "p_yes": 0.8837444936824053, "predicted": 1}, {"index": 154, "id": "boolq/test/1216", "label": 1, "p_yes": 0.41457070193405526, "predicted": 0}, {"index": 155, "id": "boolq/test/124", "label": 1, "p_yes": 0.8024268145350238, "predicted": 1}, {"index": 156, "id": "boolq/test/3147", "label": 1, "p_yes": 0.90094335995585, "predicted": 1}, {"index": 157, "id": "boolq/test/27", "label": 0, "p_yes": 0.6290672149046538, "predicted": 1}, {"index": 158, "id": "boolq/test/1060", "label": 0, "p_yes": 0.1599497641238216, "predicted": 0}, {"index": 159, "id": "boolq/test/3162", "label": 1, "p_yes": 0.7629865476432799, "predicted": 1}, {"index": 160, "id": "boolq/test/2730", "label": 1, "p_yes": 0.9103142054862989, "predicted": 1}, {"index": 161, "id": "boolq/test/386", "label": 1, "p_yes": 0.865623002582253, "predicted": 1}, {"index": 162, "id": "boolq/test/2848", "label": 1, "p_yes": 0.8880177705776783, "predicted": 1}, {"index": 163, "id": "boolq/test/1306", "label": 1, "p_yes": 0.9061627242379552, "predicted": 1}, {"index": 164, "id": "boolq/test/484", "label": 1, "p_yes": 0.8470228120043302, "predicted": 1}, {"index": 165, "id": "boolq/test/1636", "label": 1, "p_yes": 0.7212743349137087, "predicted": 1}, {"index": 166, "id": "boolq/test/1326", "label": 0, "p_yes": 0.838626757477774, "predicted": 1}, {"index": 167, "id": "boolq/test/771", "label": 1, "p_yes": 0.8684714995954412, "predicted": 1}, {"index": 168, "id": "boolq/test/638", "label": 0, "p_yes": 0.15297916129941924, "predicted": 0}, {"index": 169, "id": "boolq/test/1298", "label": 1, "p_yes": 0.82381268845126, "predicted": 1}, {"index": 170, "id": "boolq/test/3018", "label": 1, "p_yes": 0.918536185016268, "predicted": 1}, {"index": 171, "id": "boolq/test/420", "label": 0, "p_yes": 0.12293041857744848, "predicted": 0}, {"index": 172, "id": "boolq/test/1304", "label": 1, "p_yes": 0.8720557295553081, "predicted": 1}, {"index": 173, "id": "boolq/test/1477", "label": 0, "p_yes": 0.35143626363453195, "predicted": 0}, {"index": 174, "id": "boolq/test/1703", "label": 1, "p_yes": 0.250645939172957, "predicted": 0}, {"index": 175, "id": "boolq/test/2088", "label": 0, "p_yes": 0.13112850379196392, "predicted": 0}, {"index": 176, "id": "boolq/test/3100", "label": 1, "p_yes": 0.6612959891934921, "predicted": 1}, {"index": 177, "id": "boolq/test/2484", "label": 1, "p_yes": 0.42056566342770707, "predicted": 0}, {"index": 178, "id": "boolq/test/3066", "label": 1, "p_yes": 0.5990348091779858, "predicted": 1}, {"index": 179, "id": "boolq/test/145", "label": 1, "p_yes": 0.6315289245789062, "predicted": 1}, {"index": 180, "id": "boolq/test/2204", "label": 0, "p_yes": 0.18345781286713056, "predicted": 0}, {"index": 181, "id": "boolq/test/3168", "label": 1, "p_yes": 0.9268884562748226, "predicted": 1}, {"index": 182, "id": "boolq/test/1020", "label": 0, "p_yes": 0.36112733641124345, "predicted": 0}, {"index": 183, "id": "boolq/test/956", "label": 1, "p_yes": 0.8743903012824169, "predicted": 1}, {"index": 184, "id": "boolq/test/1619", "label": 1, "p_yes": 0.9003476329397416, "predicted": 1}, {"index": 185, "id": "boolq/test/2154", "label": 1, "p_yes": 0.9024993461107274, "predicted": 1}, {"index": 186, "id": "boolq/test/2262", "label": 0, "p_yes": 0.7526468227958024, "predicted": 1}, {"index": 187, "id": "boolq/test/74", "label": 1, "p_yes": 0.9241039057324933, "predicted": 1}, {"index": 188, "id": "boolq/test/2097", "label": 1, "p_yes": 0.878583504816406, "predicted": 1}, {"index": 189, "id": "boolq/test/1963", "label": 1, "p_yes": 0.8381469827160395, "predicted": 1}, {"index": 190, "id": "boolq/test/1658", "label": 1, "p_yes": 0.8347756701657645, "predicted": 1}, {"index": 191, "id": "boolq/test/1517", "label": 0, "p_yes": 0.32234291548390714, "predicted": 0}, {"index": 192, "id": "boolq/test/332", "label": 1, "p_yes": 0.5879909581807239, "predicted": 1}, {"index": 193, "id": "boolq/test/345", "label": 1, "p_yes": 0.8826474780767039, "predicted": 1}, {"index": 194, "id": "boolq/test/3245", "label": 1, "p_yes": 0.9000135697214187, "predicted": 1}, {"index": 195, "id": "boolq/test/1480", "label": 1, "p_yes": 0.9364240329558906, "predicted": 1}, {"index": 196, "id": "boolq/test/711", "label": 1, "p_yes": 0.9070909510197679, "predicted": 1}, {"index": 197, "id": "boolq/test/2446", "label": 0, "p_yes": 0.7655243868371401, "predicted": 1}, {"index": 198, "id": "boolq/test/893", "label": 1, "p_yes": 0.8376708382326884, "predicted": 1}, {"index": 199, "id": "boolq/test/2249", "label": 1, "p_yes": 0.9291620119541927, "predicted": 1}, {"index": 200, "id": "boolq/test/3077", "label": 1, "p_yes": 0.9003476329397416, "predicted": 1}, {"index": 201, "id": "boolq/test/2891", "label": 0, "p_yes": 0.09116393241733711, "predicted": 0}, {"index": 202, "id": "boolq/test/896", "label": 0, "p_yes": 0.7365788629783657, "predicted": 1}, {"index": 203, "id": "boolq/test/1757", "label": 1, "p_yes": 0.6249494470144896, "predicted": 1}, {"index": 204, "id": "boolq/test/2472", "label": 1, "p_yes": 0.13396261646563112, "predicted": 0}, {"index": 205, "id": "boolq/test/1952", "label": 1, "p_yes": 0.8040959679508792, "predicted": 1}, {"index": 206, "id": "boolq/test/625", "label": 1, "p_yes": 0.8830156824871465, "predicted": 1}, {"index": 207, "id": "boolq/test/410", "label": 1, "p_yes": 0.8948236711181226, "predicted": 1}, {"index": 208, "id": "boolq/test/800", "label": 0, "p_yes": 0.1225512445481371, "predicted": 0}, {"index": 209, "id": "boolq/test/1251", "label": 1, "p_yes": 0.9064861409559639, "predicted": 1}, {"index": 210, "id": "boolq/test/65", "label": 1, "p_yes": 0.3034206343893366, "predicted": 0}, {"index": 211, "id": "boolq/test/1056", "label": 1, "p_yes": 0.9052698321621869, "predicted": 1}, {"index": 212, "id": "boolq/test/2627", "label": 0, "p_yes": 0.19645826735958904, "predicted": 0}, {"index": 213, "id": "boolq/test/3255", "label": 0, "p_yes": 0.32003960506307205, "predicted": 0}, {"index": 214, "id": "boolq/test/220", "label": 0, "p_yes": 0.23828995228041533, "predicted": 0}, {"index": 215, "id": "boolq/test/2731", "label": 1, "p_yes": 0.9052698321621869, "predicted": 1}, {"index": 216, "id": "boolq/test/1713", "label": 1, "p_yes": 0.5820065953262441, "predicted": 1}, {"index": 217, "id": "boolq/test/2430", "label": 0, "p_yes": 0.1045104839839426, "predicted": 0}, {"index": 218, "id": "boolq/test/2185", "label": 1, "p_yes": 0.842871621591517, "predicted": 1}, {"index": 219, "id": "boolq/test/2039", "label": 1, "p_yes": 0.8639773453749758, "predicted": 1}, {"index": 220, "id": "boolq/test/1591", "label": 1, "p_yes": 0.90094335995585, "predicted": 1}, {"index": 221, "id": "boolq/test/3015", "label": 0, "p_yes": 0.8797064721883716, "predicted": 1}, {"index": 222, "id": "boolq/test/2425", "label": 0, "p_yes": 0.11410467303759952, "predicted": 0}, {"index": 223, "id": "boolq/test/2131", "label": 0, "p_yes": 0.8833666836270714, "predicted": 1}, {"index": 224, "id": "boolq/test/1184", "label": 1, "p_yes": 0.9218870377064358, "predicted": 1}, {"index": 225, "id": "boolq/test/2351", "label": 1, "p_yes": 0.8964623784761838, "predicted": 1}, {"index": 226, "id": "boolq/test/925", "label": 0, "p_yes": 0.26685158445555945, "predicted": 0}, {"index": 227, "id": "boolq/test/570", "label": 1, "p_yes": 0.6784257978711287, "predicted": 1}, {"index": 228, "id": "boolq/test/3260", "label": 0, "p_yes": 0.4639677359772739, "predicted": 0}, {"index": 229, "id": "boolq/test/2576", "label": 0, "p_yes": 0.6348005999142827, "predicted": 1}, {"index": 230, "id": "boolq/test/2825", "label": 1, "p_yes": 0.9119771143223863, "predicted": 1}, {"index": 231, "id": "boolq/test/882", "label": 1, "p_yes": 0.9238584683282742, "predicted": 1}, {"index": 232, "id": "boolq/test/1423", "label": 0, "p_yes": 0.7927650584659653, "predicted": 1}, {"index": 233, "id": "boolq/test/96", "label": 1, "p_yes": 0.8819251652574759, "predicted": 1}, {"index": 234, "id": "boolq/test/2646", "label": 1, "p_yes": 0.8643953991663171, "predicted": 1}, {"index": 235, "id": "boolq/test/375", "label": 1, "p_yes": 0.9111634980646867, "predicted": 1}, {"index": 236, "id": "boolq/test/242", "label": 0, "p_yes": 0.8914689648680405, "predicted": 1}, {"index": 237, "id": "boolq/test/1406", "label": 1, "p_yes": 0.9325177432466631, "predicted": 1}, {"index": 238, "id": "boolq/test/652", "label": 1, "p_yes": 0.8935066306459123, "predicted": 1}, {"index": 239, "id": "boolq/test/1136", "label": 1, "p_yes": 0.8165409515685887, "predicted": 1}, {"index": 240, "id": "boolq/test/839", "label": 0, "p_yes": 0.29971215835759624, "predicted": 0}, {"index": 241, "id": "boolq/test/1908", "label": 1, "p_yes": 0.8308535397894462, "predicted": 1}, {"index": 242, "id": "boolq/test/1992", "label": 0, "p_yes": 0.19534883331320765, "predicted": 0}, {"index": 243, "id": "boolq/test/1191", "label": 1, "p_yes": 0.8589438083537909, "predicted": 1}, {"index": 244, "id": "boolq/test/2045", "label": 0, "p_yes": 0.16668559702555097, "predicted": 0}, {"index": 245, "id": "boolq/test/286", "label": 1, "p_yes": 0.8931558381709804, "predicted": 1}, {"index": 246, "id": "boolq/test/3095", "label": 0, "p_yes": 0.8447233926556357, "predicted": 1}, {"index": 247, "id": "boolq/test/2285", "label": 1, "p_yes": 0.8720557295553081, "predicted": 1}, {"index": 248, "id": "boolq/test/572", "label": 1, "p_yes": 0.8668499509897473, "predicted": 1}, {"index": 249, "id": "boolq/test/1712", "label": 0, "p_yes": 0.856787061194167, "predicted": 1}, {"index": 250, "id": "boolq/test/2860", "label": 1, "p_yes": 0.8652124625579216, "predicted": 1}, {"index": 251, "id": "boolq/test/1814", "label": 1, "p_yes": 0.7809555448722605, "predicted": 1}, {"index": 252, "id": "boolq/test/2580", "label": 0, "p_yes": 0.8631461675882729, "predicted": 1}, {"index": 253, "id": "boolq/test/1297", "label": 1, "p_yes": 0.9076547629144576, "predicted": 1}, {"index": 254, "id": "boolq/test/384", "label": 1, "p_yes": 0.8585126285216863, "predicted": 1}, {"index": 255, "id": "boolq/test/1073", "label": 1, "p_yes": 0.9049625867647019, "predicted": 1}]

adapted-confirm-predictions.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "boolq/test/314", "label": 1, "p_yes": 0.9999667406082153, "predicted": 1}, {"index": 1, "id": "boolq/test/2058", "label": 1, "p_yes": 0.0006771942134946585, "predicted": 0}, {"index": 2, "id": "boolq/test/1125", "label": 0, "p_yes": 0.9814534187316895, "predicted": 1}, {"index": 3, "id": "boolq/test/611", "label": 1, "p_yes": 0.9989350438117981, "predicted": 1}, {"index": 4, "id": "boolq/test/1646", "label": 1, "p_yes": 0.9998944997787476, "predicted": 1}, {"index": 5, "id": "boolq/test/203", "label": 0, "p_yes": 0.9979493021965027, "predicted": 1}, {"index": 6, "id": "boolq/test/2275", "label": 0, "p_yes": 0.6261242032051086, "predicted": 1}, {"index": 7, "id": "boolq/test/80", "label": 0, "p_yes": 0.013222822919487953, "predicted": 0}, {"index": 8, "id": "boolq/test/90", "label": 1, "p_yes": 0.37022536993026733, "predicted": 0}, {"index": 9, "id": "boolq/test/2253", "label": 1, "p_yes": 0.8080672025680542, "predicted": 1}, {"index": 10, "id": "boolq/test/1398", "label": 1, "p_yes": 0.7905303239822388, "predicted": 1}, {"index": 11, "id": "boolq/test/2803", "label": 0, "p_yes": 0.9653811454772949, "predicted": 1}, {"index": 12, "id": "boolq/test/1768", "label": 1, "p_yes": 0.9999426603317261, "predicted": 1}, {"index": 13, "id": "boolq/test/1525", "label": 1, "p_yes": 0.999045193195343, "predicted": 1}, {"index": 14, "id": "boolq/test/754", "label": 0, "p_yes": 0.9996699094772339, "predicted": 1}, {"index": 15, "id": "boolq/test/1284", "label": 0, "p_yes": 0.999993085861206, "predicted": 1}, {"index": 16, "id": "boolq/test/1567", "label": 1, "p_yes": 0.01590639166533947, "predicted": 0}, {"index": 17, "id": "boolq/test/172", "label": 1, "p_yes": 0.9995891451835632, "predicted": 1}, {"index": 18, "id": "boolq/test/851", "label": 0, "p_yes": 0.9995344877243042, "predicted": 1}, {"index": 19, "id": "boolq/test/604", "label": 0, "p_yes": 0.00011412239837227389, "predicted": 0}, {"index": 20, "id": "boolq/test/2912", "label": 1, "p_yes": 0.9997965693473816, "predicted": 1}, {"index": 21, "id": "boolq/test/3209", "label": 1, "p_yes": 0.9999692440032959, "predicted": 1}, {"index": 22, "id": "boolq/test/1739", "label": 1, "p_yes": 0.9999443292617798, "predicted": 1}, {"index": 23, "id": "boolq/test/2061", "label": 0, "p_yes": 0.2751297354698181, "predicted": 0}, {"index": 24, "id": "boolq/test/1579", "label": 0, "p_yes": 0.9997695088386536, "predicted": 1}, {"index": 25, "id": "boolq/test/2343", "label": 1, "p_yes": 0.9999706745147705, "predicted": 1}, {"index": 26, "id": "boolq/test/756", "label": 1, "p_yes": 0.999779999256134, "predicted": 1}, {"index": 27, "id": "boolq/test/144", "label": 0, "p_yes": 5.3910844144411385e-05, "predicted": 0}, {"index": 28, "id": "boolq/test/211", "label": 0, "p_yes": 0.20946969091892242, "predicted": 0}, {"index": 29, "id": "boolq/test/842", "label": 1, "p_yes": 0.04885777831077576, "predicted": 0}, {"index": 30, "id": "boolq/test/2287", "label": 1, "p_yes": 0.16451644897460938, "predicted": 0}, {"index": 31, "id": "boolq/test/913", "label": 0, "p_yes": 0.0005274784634821117, "predicted": 0}, {"index": 32, "id": "boolq/test/2084", "label": 1, "p_yes": 0.7431679368019104, "predicted": 1}, {"index": 33, "id": "boolq/test/2984", "label": 1, "p_yes": 0.09009299427270889, "predicted": 0}, {"index": 34, "id": "boolq/test/909", "label": 1, "p_yes": 0.999562680721283, "predicted": 1}, {"index": 35, "id": "boolq/test/907", "label": 0, "p_yes": 0.005911068990826607, "predicted": 0}, {"index": 36, "id": "boolq/test/1217", "label": 1, "p_yes": 0.9999517202377319, "predicted": 1}, {"index": 37, "id": "boolq/test/2302", "label": 1, "p_yes": 0.9674102663993835, "predicted": 1}, {"index": 38, "id": "boolq/test/232", "label": 1, "p_yes": 0.9996079802513123, "predicted": 1}, {"index": 39, "id": "boolq/test/1308", "label": 1, "p_yes": 0.999841570854187, "predicted": 1}, {"index": 40, "id": "boolq/test/2048", "label": 1, "p_yes": 0.9991440773010254, "predicted": 1}, {"index": 41, "id": "boolq/test/1982", "label": 1, "p_yes": 0.9999804496765137, "predicted": 1}, {"index": 42, "id": "boolq/test/534", "label": 1, "p_yes": 0.9999053478240967, "predicted": 1}, {"index": 43, "id": "boolq/test/773", "label": 1, "p_yes": 0.9999452829360962, "predicted": 1}, {"index": 44, "id": "boolq/test/2366", "label": 1, "p_yes": 0.05108273774385452, "predicted": 0}, {"index": 45, "id": "boolq/test/632", "label": 1, "p_yes": 0.5389832258224487, "predicted": 1}, {"index": 46, "id": "boolq/test/3165", "label": 1, "p_yes": 0.9946980476379395, "predicted": 1}, {"index": 47, "id": "boolq/test/1249", "label": 1, "p_yes": 0.999982476234436, "predicted": 1}, {"index": 48, "id": "boolq/test/2473", "label": 0, "p_yes": 0.003649987280368805, "predicted": 0}, {"index": 49, "id": "boolq/test/3049", "label": 0, "p_yes": 0.9999808073043823, "predicted": 1}, {"index": 50, "id": "boolq/test/2500", "label": 1, "p_yes": 0.9999864101409912, "predicted": 1}, {"index": 51, "id": "boolq/test/3130", "label": 0, "p_yes": 0.9441768527030945, "predicted": 1}, {"index": 52, "id": "boolq/test/812", "label": 1, "p_yes": 0.9998822212219238, "predicted": 1}, {"index": 53, "id": "boolq/test/2640", "label": 0, "p_yes": 0.9974884986877441, "predicted": 1}, {"index": 54, "id": "boolq/test/457", "label": 1, "p_yes": 0.9999359846115112, "predicted": 1}, {"index": 55, "id": "boolq/test/1094", "label": 1, "p_yes": 0.9999263286590576, "predicted": 1}, {"index": 56, "id": "boolq/test/1834", "label": 1, "p_yes": 0.9961755275726318, "predicted": 1}, {"index": 57, "id": "boolq/test/733", "label": 1, "p_yes": 0.9999567270278931, "predicted": 1}, {"index": 58, "id": "boolq/test/1503", "label": 0, "p_yes": 0.0008295891457237303, "predicted": 0}, {"index": 59, "id": "boolq/test/1766", "label": 1, "p_yes": 0.9999802112579346, "predicted": 1}, {"index": 60, "id": "boolq/test/3252", "label": 1, "p_yes": 0.999992847442627, "predicted": 1}, {"index": 61, "id": "boolq/test/2379", "label": 0, "p_yes": 0.9994296431541443, "predicted": 1}, {"index": 62, "id": "boolq/test/2693", "label": 1, "p_yes": 0.999868631362915, "predicted": 1}, {"index": 63, "id": "boolq/test/2493", "label": 1, "p_yes": 0.9979493021965027, "predicted": 1}, {"index": 64, "id": "boolq/test/1240", "label": 0, "p_yes": 0.9998339414596558, "predicted": 1}, {"index": 65, "id": "boolq/test/1159", "label": 1, "p_yes": 0.9999810457229614, "predicted": 1}, {"index": 66, "id": "boolq/test/2064", "label": 1, "p_yes": 0.9161096215248108, "predicted": 1}, {"index": 67, "id": "boolq/test/355", "label": 0, "p_yes": 0.0018102108733728528, "predicted": 0}, {"index": 68, "id": "boolq/test/1047", "label": 1, "p_yes": 0.999975323677063, "predicted": 1}, {"index": 69, "id": "boolq/test/2897", "label": 1, "p_yes": 0.9997546076774597, "predicted": 1}, {"index": 70, "id": "boolq/test/1804", "label": 0, "p_yes": 0.00011412239837227389, "predicted": 0}, {"index": 71, "id": "boolq/test/3040", "label": 0, "p_yes": 6.205048703122884e-05, "predicted": 0}, {"index": 72, "id": "boolq/test/2231", "label": 0, "p_yes": 0.0006771942134946585, "predicted": 0}, {"index": 73, "id": "boolq/test/2001", "label": 1, "p_yes": 0.9994887113571167, "predicted": 1}, {"index": 74, "id": "boolq/test/2528", "label": 0, "p_yes": 0.9999781847000122, "predicted": 1}, {"index": 75, "id": "boolq/test/44", "label": 1, "p_yes": 0.9999552965164185, "predicted": 1}, {"index": 76, "id": "boolq/test/1207", "label": 0, "p_yes": 0.9999586343765259, "predicted": 1}, {"index": 77, "id": "boolq/test/729", "label": 1, "p_yes": 0.9999741315841675, "predicted": 1}, {"index": 78, "id": "boolq/test/3111", "label": 0, "p_yes": 0.0427221953868866, "predicted": 0}, {"index": 79, "id": "boolq/test/1196", "label": 1, "p_yes": 0.9997867941856384, "predicted": 1}, {"index": 80, "id": "boolq/test/2801", "label": 1, "p_yes": 0.9979810118675232, "predicted": 1}, {"index": 81, "id": "boolq/test/2768", "label": 1, "p_yes": 0.9997387528419495, "predicted": 1}, {"index": 82, "id": "boolq/test/1341", "label": 0, "p_yes": 0.023689471185207367, "predicted": 0}, {"index": 83, "id": "boolq/test/2979", "label": 1, "p_yes": 0.999762237071991, "predicted": 1}, {"index": 84, "id": "boolq/test/1550", "label": 0, "p_yes": 0.0004655269149225205, "predicted": 0}, {"index": 85, "id": "boolq/test/3208", "label": 1, "p_yes": 0.9998960494995117, "predicted": 1}, {"index": 86, "id": "boolq/test/719", "label": 1, "p_yes": 0.999779999256134, "predicted": 1}, {"index": 87, "id": "boolq/test/2644", "label": 1, "p_yes": 0.9805806875228882, "predicted": 1}, {"index": 88, "id": "boolq/test/212", "label": 0, "p_yes": 0.016914913430809975, "predicted": 0}, {"index": 89, "id": "boolq/test/1411", "label": 0, "p_yes": 0.0004442183708306402, "predicted": 0}, {"index": 90, "id": "boolq/test/2440", "label": 0, "p_yes": 0.9993928670883179, "predicted": 1}, {"index": 91, "id": "boolq/test/2851", "label": 0, "p_yes": 0.001524786464869976, "predicted": 0}, {"index": 92, "id": "boolq/test/1057", "label": 1, "p_yes": 0.9986541271209717, "predicted": 1}, {"index": 93, "id": "boolq/test/267", "label": 1, "p_yes": 0.9989836812019348, "predicted": 1}, {"index": 94, "id": "boolq/test/1674", "label": 0, "p_yes": 0.0002780309587251395, "predicted": 0}, {"index": 95, "id": "boolq/test/269", "label": 0, "p_yes": 0.011331754736602306, "predicted": 0}, {"index": 96, "id": "boolq/test/316", "label": 0, "p_yes": 0.0012843082658946514, "predicted": 0}, {"index": 97, "id": "boolq/test/1855", "label": 1, "p_yes": 0.29421496391296387, "predicted": 0}, {"index": 98, "id": "boolq/test/2000", "label": 0, "p_yes": 0.9970190525054932, "predicted": 1}, {"index": 99, "id": "boolq/test/2793", "label": 1, "p_yes": 0.9984512329101562, "predicted": 1}, {"index": 100, "id": "boolq/test/696", "label": 1, "p_yes": 0.9986541271209717, "predicted": 1}, {"index": 101, "id": "boolq/test/905", "label": 1, "p_yes": 0.9992792010307312, "predicted": 1}, {"index": 102, "id": "boolq/test/2374", "label": 1, "p_yes": 0.9999651908874512, "predicted": 1}, {"index": 103, "id": "boolq/test/264", "label": 1, "p_yes": 0.9989182949066162, "predicted": 1}, {"index": 104, "id": "boolq/test/131", "label": 1, "p_yes": 0.9999822378158569, "predicted": 1}, {"index": 105, "id": "boolq/test/2917", "label": 1, "p_yes": 0.9998706579208374, "predicted": 1}, {"index": 106, "id": "boolq/test/2060", "label": 0, "p_yes": 0.9998260140419006, "predicted": 1}, {"index": 107, "id": "boolq/test/685", "label": 1, "p_yes": 0.9986749291419983, "predicted": 1}, {"index": 108, "id": "boolq/test/2063", "label": 0, "p_yes": 0.0026730059180408716, "predicted": 0}, {"index": 109, "id": "boolq/test/3063", "label": 1, "p_yes": 0.9998960494995117, "predicted": 1}, {"index": 110, "id": "boolq/test/953", "label": 0, "p_yes": 0.7154240012168884, "predicted": 1}, {"index": 111, "id": "boolq/test/85", "label": 1, "p_yes": 0.5117166042327881, "predicted": 1}, {"index": 112, "id": "boolq/test/2513", "label": 1, "p_yes": 0.9999865293502808, "predicted": 1}, {"index": 113, "id": "boolq/test/408", "label": 1, "p_yes": 0.9999804496765137, "predicted": 1}, {"index": 114, "id": "boolq/test/2739", "label": 0, "p_yes": 9.516251338936854e-06, "predicted": 0}, {"index": 115, "id": "boolq/test/369", "label": 0, "p_yes": 0.7490872144699097, "predicted": 1}, {"index": 116, "id": "boolq/test/2916", "label": 1, "p_yes": 0.9999933242797852, "predicted": 1}, {"index": 117, "id": "boolq/test/2674", "label": 1, "p_yes": 0.9910145401954651, "predicted": 1}, {"index": 118, "id": "boolq/test/2639", "label": 1, "p_yes": 0.9893474578857422, "predicted": 1}, {"index": 119, "id": "boolq/test/574", "label": 1, "p_yes": 0.99998939037323, "predicted": 1}, {"index": 120, "id": "boolq/test/2978", "label": 1, "p_yes": 0.9960545301437378, "predicted": 1}, {"index": 121, "id": "boolq/test/1546", "label": 0, "p_yes": 1.3631345609610435e-05, "predicted": 0}, {"index": 122, "id": "boolq/test/201", "label": 1, "p_yes": 0.9998339414596558, "predicted": 1}, {"index": 123, "id": "boolq/test/3008", "label": 1, "p_yes": 0.9997997879981995, "predicted": 1}, {"index": 124, "id": "boolq/test/3212", "label": 0, "p_yes": 0.22815649211406708, "predicted": 0}, {"index": 125, "id": "boolq/test/2946", "label": 0, "p_yes": 0.2782568037509918, "predicted": 0}, {"index": 126, "id": "boolq/test/516", "label": 0, "p_yes": 0.8824278712272644, "predicted": 1}, {"index": 127, "id": "boolq/test/1182", "label": 1, "p_yes": 0.9822876453399658, "predicted": 1}, {"index": 128, "id": "boolq/test/1257", "label": 0, "p_yes": 0.9992903470993042, "predicted": 1}, {"index": 129, "id": "boolq/test/3231", "label": 1, "p_yes": 0.0001420236221747473, "predicted": 0}, {"index": 130, "id": "boolq/test/3027", "label": 1, "p_yes": 0.9999749660491943, "predicted": 1}, {"index": 131, "id": "boolq/test/2605", "label": 1, "p_yes": 0.0014779429184272885, "predicted": 0}, {"index": 132, "id": "boolq/test/3062", "label": 0, "p_yes": 0.001597845577634871, "predicted": 0}, {"index": 133, "id": "boolq/test/42", "label": 1, "p_yes": 0.9996993541717529, "predicted": 1}, {"index": 134, "id": "boolq/test/2676", "label": 0, "p_yes": 0.0001398220774717629, "predicted": 0}, {"index": 135, "id": "boolq/test/1624", "label": 1, "p_yes": 0.9999407529830933, "predicted": 1}, {"index": 136, "id": "boolq/test/2661", "label": 0, "p_yes": 0.9032942056655884, "predicted": 1}, {"index": 137, "id": "boolq/test/2240", "label": 0, "p_yes": 0.00011774458835134283, "predicted": 0}, {"index": 138, "id": "boolq/test/1575", "label": 1, "p_yes": 0.9998666048049927, "predicted": 1}, {"index": 139, "id": "boolq/test/300", "label": 0, "p_yes": 0.039048291742801666, "predicted": 0}, {"index": 140, "id": "boolq/test/2337", "label": 1, "p_yes": 0.9998286962509155, "predicted": 1}, {"index": 141, "id": "boolq/test/3224", "label": 0, "p_yes": 0.00011591934162424877, "predicted": 0}, {"index": 142, "id": "boolq/test/739", "label": 1, "p_yes": 0.9987744688987732, "predicted": 1}, {"index": 143, "id": "boolq/test/1482", "label": 0, "p_yes": 0.01615283451974392, "predicted": 0}, {"index": 144, "id": "boolq/test/18", "label": 1, "p_yes": 0.9915544390678406, "predicted": 1}, {"index": 145, "id": "boolq/test/395", "label": 0, "p_yes": 0.9999508857727051, "predicted": 1}, {"index": 146, "id": "boolq/test/838", "label": 0, "p_yes": 0.00011235327838221565, "predicted": 0}, {"index": 147, "id": "boolq/test/111", "label": 1, "p_yes": 0.9998992681503296, "predicted": 1}, {"index": 148, "id": "boolq/test/1465", "label": 1, "p_yes": 0.9989677667617798, "predicted": 1}, {"index": 149, "id": "boolq/test/1267", "label": 1, "p_yes": 0.9990149736404419, "predicted": 1}, {"index": 150, "id": "boolq/test/1863", "label": 1, "p_yes": 3.1693256460130215e-05, "predicted": 0}, {"index": 151, "id": "boolq/test/2382", "label": 1, "p_yes": 0.976669192314148, "predicted": 1}, {"index": 152, "id": "boolq/test/1340", "label": 1, "p_yes": 0.9999617338180542, "predicted": 1}, {"index": 153, "id": "boolq/test/831", "label": 0, "p_yes": 0.9998766183853149, "predicted": 1}, {"index": 154, "id": "boolq/test/1216", "label": 1, "p_yes": 0.17781086266040802, "predicted": 0}, {"index": 155, "id": "boolq/test/124", "label": 1, "p_yes": 0.9980121850967407, "predicted": 1}, {"index": 156, "id": "boolq/test/3147", "label": 1, "p_yes": 0.9999443292617798, "predicted": 1}, {"index": 157, "id": "boolq/test/27", "label": 0, "p_yes": 0.9124361872673035, "predicted": 1}, {"index": 158, "id": "boolq/test/1060", "label": 0, "p_yes": 0.0006361911655403674, "predicted": 0}, {"index": 159, "id": "boolq/test/3162", "label": 1, "p_yes": 0.9944451451301575, "predicted": 1}, {"index": 160, "id": "boolq/test/2730", "label": 1, "p_yes": 0.9999657869338989, "predicted": 1}, {"index": 161, "id": "boolq/test/386", "label": 1, "p_yes": 0.9997428059577942, "predicted": 1}, {"index": 162, "id": "boolq/test/2848", "label": 1, "p_yes": 0.9998977184295654, "predicted": 1}, {"index": 163, "id": "boolq/test/1306", "label": 1, "p_yes": 0.9999573230743408, "predicted": 1}, {"index": 164, "id": "boolq/test/484", "label": 1, "p_yes": 0.9994966983795166, "predicted": 1}, {"index": 165, "id": "boolq/test/1636", "label": 1, "p_yes": 0.985496461391449, "predicted": 1}, {"index": 166, "id": "boolq/test/1326", "label": 0, "p_yes": 0.9993333220481873, "predicted": 1}, {"index": 167, "id": "boolq/test/771", "label": 1, "p_yes": 0.9997695088386536, "predicted": 1}, {"index": 168, "id": "boolq/test/638", "label": 0, "p_yes": 0.0005033356137573719, "predicted": 0}, {"index": 169, "id": "boolq/test/1298", "label": 1, "p_yes": 0.9989350438117981, "predicted": 1}, {"index": 170, "id": "boolq/test/3018", "label": 1, "p_yes": 0.9999785423278809, "predicted": 1}, {"index": 171, "id": "boolq/test/420", "label": 0, "p_yes": 0.00016346470511052758, "predicted": 0}, {"index": 172, "id": "boolq/test/1304", "label": 1, "p_yes": 0.9997997879981995, "predicted": 1}, {"index": 173, "id": "boolq/test/1477", "label": 0, "p_yes": 0.06187598407268524, "predicted": 0}, {"index": 174, "id": "boolq/test/1703", "label": 1, "p_yes": 0.007695647422224283, "predicted": 0}, {"index": 175, "id": "boolq/test/2088", "label": 0, "p_yes": 0.00022693384380545467, "predicted": 0}, {"index": 176, "id": "boolq/test/3100", "label": 1, "p_yes": 0.9511421918869019, "predicted": 1}, {"index": 177, "id": "boolq/test/2484", "label": 1, "p_yes": 0.19436782598495483, "predicted": 0}, {"index": 178, "id": "boolq/test/3066", "label": 1, "p_yes": 0.8558511734008789, "predicted": 1}, {"index": 179, "id": "boolq/test/145", "label": 1, "p_yes": 0.9161096215248108, "predicted": 1}, {"index": 180, "id": "boolq/test/2204", "label": 0, "p_yes": 0.001325022429227829, "predicted": 0}, {"index": 181, "id": "boolq/test/3168", "label": 1, "p_yes": 0.9999872446060181, "predicted": 1}, {"index": 182, "id": "boolq/test/1020", "label": 0, "p_yes": 0.07369627058506012, "predicted": 0}, {"index": 183, "id": "boolq/test/956", "label": 1, "p_yes": 0.9998176693916321, "predicted": 1}, {"index": 184, "id": "boolq/test/1619", "label": 1, "p_yes": 0.9999426603317261, "predicted": 1}, {"index": 185, "id": "boolq/test/2154", "label": 1, "p_yes": 0.9999485015869141, "predicted": 1}, {"index": 186, "id": "boolq/test/2262", "label": 0, "p_yes": 0.9928786158561707, "predicted": 1}, {"index": 187, "id": "boolq/test/74", "label": 1, "p_yes": 0.9999847412109375, "predicted": 1}, {"index": 188, "id": "boolq/test/2097", "label": 1, "p_yes": 0.9998464584350586, "predicted": 1}, {"index": 189, "id": "boolq/test/1963", "label": 1, "p_yes": 0.999322772026062, "predicted": 1}, {"index": 190, "id": "boolq/test/1658", "label": 1, "p_yes": 0.9992445707321167, "predicted": 1}, {"index": 191, "id": "boolq/test/1517", "label": 0, "p_yes": 0.03567855432629585, "predicted": 0}, {"index": 192, "id": "boolq/test/332", "label": 1, "p_yes": 0.8289388418197632, "predicted": 1}, {"index": 193, "id": "boolq/test/345", "label": 1, "p_yes": 0.9998706579208374, "predicted": 1}, {"index": 194, "id": "boolq/test/3245", "label": 1, "p_yes": 0.9999417066574097, "predicted": 1}, {"index": 195, "id": "boolq/test/1480", "label": 1, "p_yes": 0.9999934434890747, "predicted": 1}, {"index": 196, "id": "boolq/test/711", "label": 1, "p_yes": 0.9999593496322632, "predicted": 1}, {"index": 197, "id": "boolq/test/2446", "label": 0, "p_yes": 0.9947799444198608, "predicted": 1}, {"index": 198, "id": "boolq/test/893", "label": 1, "p_yes": 0.999312162399292, "predicted": 1}, {"index": 199, "id": "boolq/test/2249", "label": 1, "p_yes": 0.9999890327453613, "predicted": 1}, {"index": 200, "id": "boolq/test/3077", "label": 1, "p_yes": 0.9999426603317261, "predicted": 1}, {"index": 201, "id": "boolq/test/2891", "label": 0, "p_yes": 3.7052966945338994e-05, "predicted": 0}, {"index": 202, "id": "boolq/test/896", "label": 0, "p_yes": 0.9896718263626099, "predicted": 1}, {"index": 203, "id": "boolq/test/1757", "label": 1, "p_yes": 0.9059898257255554, "predicted": 1}, {"index": 204, "id": "boolq/test/2472", "label": 1, "p_yes": 0.0002531563804950565, "predicted": 0}, {"index": 205, "id": "boolq/test/1952", "label": 1, "p_yes": 0.9981030225753784, "predicted": 1}, {"index": 206, "id": "boolq/test/625", "label": 1, "p_yes": 0.9998726844787598, "predicted": 1}, {"index": 207, "id": "boolq/test/410", "label": 1, "p_yes": 0.9999251365661621, "predicted": 1}, {"index": 208, "id": "boolq/test/800", "label": 0, "p_yes": 0.00016093082376755774, "predicted": 0}, {"index": 209, "id": "boolq/test/1251", "label": 1, "p_yes": 0.9999580383300781, "predicted": 1}, {"index": 210, "id": "boolq/test/65", "label": 1, "p_yes": 0.024423090741038322, "predicted": 0}, {"index": 211, "id": "boolq/test/1056", "label": 1, "p_yes": 0.9999552965164185, "predicted": 1}, {"index": 212, "id": "boolq/test/2627", "label": 0, "p_yes": 0.0019267346942797303, "predicted": 0}, {"index": 213, "id": "boolq/test/3255", "label": 0, "p_yes": 0.034100428223609924, "predicted": 0}, {"index": 214, "id": "boolq/test/220", "label": 0, "p_yes": 0.005730246659368277, "predicted": 0}, {"index": 215, "id": "boolq/test/2731", "label": 1, "p_yes": 0.9999552965164185, "predicted": 1}, {"index": 216, "id": "boolq/test/1713", "label": 1, "p_yes": 0.8128673434257507, "predicted": 1}, {"index": 217, "id": "boolq/test/2430", "label": 0, "p_yes": 7.254361844388768e-05, "predicted": 0}, {"index": 218, "id": "boolq/test/2185", "label": 1, "p_yes": 0.9994207620620728, "predicted": 1}, {"index": 219, "id": "boolq/test/2039", "label": 1, "p_yes": 0.9997262358665466, "predicted": 1}, {"index": 220, "id": "boolq/test/1591", "label": 1, "p_yes": 0.9999443292617798, "predicted": 1}, {"index": 221, "id": "boolq/test/3015", "label": 0, "p_yes": 0.9998534917831421, "predicted": 1}, {"index": 222, "id": "boolq/test/2425", "label": 0, "p_yes": 0.00011235327838221565, "predicted": 0}, {"index": 223, "id": "boolq/test/2131", "label": 0, "p_yes": 0.9998745918273926, "predicted": 1}, {"index": 224, "id": "boolq/test/1184", "label": 1, "p_yes": 0.999982476234436, "predicted": 1}, {"index": 225, "id": "boolq/test/2351", "label": 1, "p_yes": 0.999930739402771, "predicted": 1}, {"index": 226, "id": "boolq/test/925", "label": 0, "p_yes": 0.011158030480146408, "predicted": 0}, {"index": 227, "id": "boolq/test/570", "label": 1, "p_yes": 0.9648551344871521, "predicted": 1}, {"index": 228, "id": "boolq/test/3260", "label": 0, "p_yes": 0.34510529041290283, "predicted": 0}, {"index": 229, "id": "boolq/test/2576", "label": 0, "p_yes": 0.9207897186279297, "predicted": 1}, {"index": 230, "id": "boolq/test/2825", "label": 1, "p_yes": 0.9999687671661377, "predicted": 1}, {"index": 231, "id": "boolq/test/882", "label": 1, "p_yes": 0.9999845027923584, "predicted": 1}, {"index": 232, "id": "boolq/test/1423", "label": 0, "p_yes": 0.9974090456962585, "predicted": 1}, {"index": 233, "id": "boolq/test/96", "label": 1, "p_yes": 0.9998666048049927, "predicted": 1}, {"index": 234, "id": "boolq/test/2646", "label": 1, "p_yes": 0.9997305274009705, "predicted": 1}, {"index": 235, "id": "boolq/test/375", "label": 1, "p_yes": 0.9999673366546631, "predicted": 1}, {"index": 236, "id": "boolq/test/242", "label": 0, "p_yes": 0.9999125003814697, "predicted": 1}, {"index": 237, "id": "boolq/test/1406", "label": 1, "p_yes": 0.9999912977218628, "predicted": 1}, {"index": 238, "id": "boolq/test/652", "label": 1, "p_yes": 0.9999203681945801, "predicted": 1}, {"index": 239, "id": "boolq/test/1136", "label": 1, "p_yes": 0.9986749291419983, "predicted": 1}, {"index": 240, "id": "boolq/test/839", "label": 0, "p_yes": 0.02262919954955578, "predicted": 0}, {"index": 241, "id": "boolq/test/1908", "label": 1, "p_yes": 0.9991440773010254, "predicted": 1}, {"index": 242, "id": "boolq/test/1992", "label": 0, "p_yes": 0.0018675659084692597, "predicted": 0}, {"index": 243, "id": "boolq/test/1191", "label": 1, "p_yes": 0.9996699094772339, "predicted": 1}, {"index": 244, "id": "boolq/test/2045", "label": 0, "p_yes": 0.0007916296017356217, "predicted": 0}, {"index": 245, "id": "boolq/test/286", "label": 1, "p_yes": 0.999919056892395, "predicted": 1}, {"index": 246, "id": "boolq/test/3095", "label": 0, "p_yes": 0.9994557499885559, "predicted": 1}, {"index": 247, "id": "boolq/test/2285", "label": 1, "p_yes": 0.9997997879981995, "predicted": 1}, {"index": 248, "id": "boolq/test/572", "label": 1, "p_yes": 0.9997546076774597, "predicted": 1}, {"index": 249, "id": "boolq/test/1712", "label": 0, "p_yes": 0.9996429681777954, "predicted": 1}, {"index": 250, "id": "boolq/test/2860", "label": 1, "p_yes": 0.9997387528419495, "predicted": 1}, {"index": 251, "id": "boolq/test/1814", "label": 1, "p_yes": 0.9964619278907776, "predicted": 1}, {"index": 252, "id": "boolq/test/2580", "label": 0, "p_yes": 0.9997175335884094, "predicted": 1}, {"index": 253, "id": "boolq/test/1297", "label": 1, "p_yes": 0.9999605417251587, "predicted": 1}, {"index": 254, "id": "boolq/test/384", "label": 1, "p_yes": 0.9996646642684937, "predicted": 1}, {"index": 255, "id": "boolq/test/1073", "label": 1, "p_yes": 0.9999545812606812, "predicted": 1}]

adapted-ood-calibrated.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "strategyqa_grounded/test/608", "label": 0, "p_yes": 0.21844281311288816, "predicted": 0}, {"index": 1, "id": "strategyqa_grounded/test/72", "label": 1, "p_yes": 0.8732314471973439, "predicted": 1}, {"index": 2, "id": "strategyqa_grounded/test/371", "label": 1, "p_yes": 0.7240969614618793, "predicted": 1}, {"index": 3, "id": "strategyqa_grounded/test/155", "label": 1, "p_yes": 0.8479343476741406, "predicted": 1}, {"index": 4, "id": "strategyqa_grounded/test/565", "label": 1, "p_yes": 0.18135714887483081, "predicted": 0}, {"index": 5, "id": "strategyqa_grounded/test/252", "label": 1, "p_yes": 0.9260507981755042, "predicted": 1}, {"index": 6, "id": "strategyqa_grounded/test/353", "label": 0, "p_yes": 0.36112733641124345, "predicted": 0}, {"index": 7, "id": "strategyqa_grounded/test/254", "label": 1, "p_yes": 0.7519902627003572, "predicted": 1}, {"index": 8, "id": "strategyqa_grounded/test/441", "label": 1, "p_yes": 0.4938377746108581, "predicted": 0}, {"index": 9, "id": "strategyqa_grounded/test/603", "label": 0, "p_yes": 0.2043575339984955, "predicted": 0}, {"index": 10, "id": "strategyqa_grounded/test/212", "label": 0, "p_yes": 0.16668559702555097, "predicted": 0}, {"index": 11, "id": "strategyqa_grounded/test/430", "label": 1, "p_yes": 0.40690015738659946, "predicted": 0}, {"index": 12, "id": "strategyqa_grounded/test/528", "label": 1, "p_yes": 0.8990774811729199, "predicted": 1}, {"index": 13, "id": "strategyqa_grounded/test/204", "label": 0, "p_yes": 0.8652124625579216, "predicted": 1}, {"index": 14, "id": "strategyqa_grounded/test/225", "label": 1, "p_yes": 0.8170706415259638, "predicted": 1}, {"index": 15, "id": "strategyqa_grounded/test/463", "label": 1, "p_yes": 0.23574270111137718, "predicted": 0}, {"index": 16, "id": "strategyqa_grounded/test/43", "label": 0, "p_yes": 0.3717548718709382, "predicted": 0}, {"index": 17, "id": "strategyqa_grounded/test/525", "label": 0, "p_yes": 0.837190382993577, "predicted": 1}, {"index": 18, "id": "strategyqa_grounded/test/431", "label": 0, "p_yes": 0.15481328346944415, "predicted": 0}, {"index": 19, "id": "strategyqa_grounded/test/584", "label": 1, "p_yes": 0.7648907182581244, "predicted": 1}, {"index": 20, "id": "strategyqa_grounded/test/405", "label": 0, "p_yes": 0.5228734235479603, "predicted": 1}, {"index": 21, "id": "strategyqa_grounded/test/231", "label": 0, "p_yes": 0.23892971046446967, "predicted": 0}, {"index": 22, "id": "strategyqa_grounded/test/610", "label": 0, "p_yes": 0.5070424378426749, "predicted": 1}, {"index": 23, "id": "strategyqa_grounded/test/515", "label": 1, "p_yes": 0.6845407899927095, "predicted": 1}, {"index": 24, "id": "strategyqa_grounded/test/184", "label": 0, "p_yes": 0.8631461675882729, "predicted": 1}, {"index": 25, "id": "strategyqa_grounded/test/573", "label": 1, "p_yes": 0.8876544810397744, "predicted": 1}, {"index": 26, "id": "strategyqa_grounded/test/39", "label": 0, "p_yes": 0.8751609769221215, "predicted": 1}, {"index": 27, "id": "strategyqa_grounded/test/672", "label": 0, "p_yes": 0.20264518269855564, "predicted": 0}, {"index": 28, "id": "strategyqa_grounded/test/94", "label": 1, "p_yes": 0.8342928727568247, "predicted": 1}, {"index": 29, "id": "strategyqa_grounded/test/314", "label": 1, "p_yes": 0.9226752211314598, "predicted": 1}, {"index": 30, "id": "strategyqa_grounded/test/73", "label": 1, "p_yes": 0.8639773453749758, "predicted": 1}, {"index": 31, "id": "strategyqa_grounded/test/679", "label": 1, "p_yes": 0.8483849006145174, "predicted": 1}, {"index": 32, "id": "strategyqa_grounded/test/216", "label": 0, "p_yes": 0.2641048836558621, "predicted": 0}, {"index": 33, "id": "strategyqa_grounded/test/587", "label": 0, "p_yes": 0.15573705757431885, "predicted": 0}, {"index": 34, "id": "strategyqa_grounded/test/227", "label": 1, "p_yes": 0.5096827765446077, "predicted": 1}, {"index": 35, "id": "strategyqa_grounded/test/630", "label": 1, "p_yes": 0.9076547629144576, "predicted": 1}, {"index": 36, "id": "strategyqa_grounded/test/327", "label": 1, "p_yes": 0.7990563471838952, "predicted": 1}, {"index": 37, "id": "strategyqa_grounded/test/199", "label": 1, "p_yes": 0.9298185889683951, "predicted": 1}, {"index": 38, "id": "strategyqa_grounded/test/436", "label": 0, "p_yes": 0.17516759641328308, "predicted": 0}, {"index": 39, "id": "strategyqa_grounded/test/335", "label": 0, "p_yes": 0.18987286766185998, "predicted": 0}, {"index": 40, "id": "strategyqa_grounded/test/46", "label": 0, "p_yes": 0.8696663339012921, "predicted": 1}, {"index": 41, "id": "strategyqa_grounded/test/386", "label": 0, "p_yes": 0.8170706415259638, "predicted": 1}, {"index": 42, "id": "strategyqa_grounded/test/682", "label": 0, "p_yes": 0.5742770261813812, "predicted": 1}, {"index": 43, "id": "strategyqa_grounded/test/596", "label": 0, "p_yes": 0.5725542492154799, "predicted": 1}, {"index": 44, "id": "strategyqa_grounded/test/554", "label": 0, "p_yes": 0.17263816642011423, "predicted": 0}, {"index": 45, "id": "strategyqa_grounded/test/169", "label": 1, "p_yes": 0.7863280060798192, "predicted": 1}, {"index": 46, "id": "strategyqa_grounded/test/508", "label": 1, "p_yes": 0.9015981002669124, "predicted": 1}, {"index": 47, "id": "strategyqa_grounded/test/56", "label": 0, "p_yes": 0.22821446242529517, "predicted": 0}, {"index": 48, "id": "strategyqa_grounded/test/217", "label": 1, "p_yes": 0.6573411315252317, "predicted": 1}, {"index": 49, "id": "strategyqa_grounded/test/253", "label": 1, "p_yes": 0.8029844686372483, "predicted": 1}, {"index": 50, "id": "strategyqa_grounded/test/569", "label": 0, "p_yes": 0.7440253643895982, "predicted": 1}, {"index": 51, "id": "strategyqa_grounded/test/384", "label": 0, "p_yes": 0.16232976643466407, "predicted": 0}, {"index": 52, "id": "strategyqa_grounded/test/513", "label": 1, "p_yes": 0.24800960918530224, "predicted": 0}, {"index": 53, "id": "strategyqa_grounded/test/107", "label": 0, "p_yes": 0.8122839015702418, "predicted": 1}, {"index": 54, "id": "strategyqa_grounded/test/616", "label": 0, "p_yes": 0.15026140618609815, "predicted": 0}, {"index": 55, "id": "strategyqa_grounded/test/468", "label": 1, "p_yes": 0.45784295528589225, "predicted": 0}, {"index": 56, "id": "strategyqa_grounded/test/157", "label": 0, "p_yes": 0.20897627003542754, "predicted": 0}, {"index": 57, "id": "strategyqa_grounded/test/465", "label": 1, "p_yes": 0.3627538139407158, "predicted": 0}, {"index": 58, "id": "strategyqa_grounded/test/579", "label": 1, "p_yes": 0.7296895602566793, "predicted": 1}, {"index": 59, "id": "strategyqa_grounded/test/532", "label": 1, "p_yes": 0.8917978103212961, "predicted": 1}, {"index": 60, "id": "strategyqa_grounded/test/606", "label": 0, "p_yes": 0.7760982121366002, "predicted": 1}, {"index": 61, "id": "strategyqa_grounded/test/209", "label": 0, "p_yes": 0.8357481707351873, "predicted": 1}, {"index": 62, "id": "strategyqa_grounded/test/88", "label": 1, "p_yes": 0.8897435553440497, "predicted": 1}, {"index": 63, "id": "strategyqa_grounded/test/281", "label": 0, "p_yes": 0.5888437905607103, "predicted": 1}, {"index": 64, "id": "strategyqa_grounded/test/295", "label": 1, "p_yes": 0.8606344077430843, "predicted": 1}, {"index": 65, "id": "strategyqa_grounded/test/457", "label": 1, "p_yes": 0.8273592994882191, "predicted": 1}, {"index": 66, "id": "strategyqa_grounded/test/170", "label": 1, "p_yes": 0.7170067436974613, "predicted": 1}, {"index": 67, "id": "strategyqa_grounded/test/658", "label": 0, "p_yes": 0.7674146016527873, "predicted": 1}, {"index": 68, "id": "strategyqa_grounded/test/123", "label": 0, "p_yes": 0.7742564979055859, "predicted": 1}, {"index": 69, "id": "strategyqa_grounded/test/461", "label": 1, "p_yes": 0.336341788111827, "predicted": 0}, {"index": 70, "id": "strategyqa_grounded/test/556", "label": 1, "p_yes": 0.7372615340998642, "predicted": 1}, {"index": 71, "id": "strategyqa_grounded/test/313", "label": 0, "p_yes": 0.48503831817136295, "predicted": 0}, {"index": 72, "id": "strategyqa_grounded/test/246", "label": 0, "p_yes": 0.20094355997178973, "predicted": 0}, {"index": 73, "id": "strategyqa_grounded/test/392", "label": 1, "p_yes": 0.26685158445555945, "predicted": 0}, {"index": 74, "id": "strategyqa_grounded/test/116", "label": 1, "p_yes": 0.8546196881936108, "predicted": 1}, {"index": 75, "id": "strategyqa_grounded/test/328", "label": 1, "p_yes": 0.9256453747319169, "predicted": 1}, {"index": 76, "id": "strategyqa_grounded/test/111", "label": 1, "p_yes": 0.1647382373282336, "predicted": 0}, {"index": 77, "id": "strategyqa_grounded/test/444", "label": 0, "p_yes": 0.14234620669427514, "predicted": 0}, {"index": 78, "id": "strategyqa_grounded/test/590", "label": 0, "p_yes": 0.3586935848331826, "predicted": 0}, {"index": 79, "id": "strategyqa_grounded/test/595", "label": 0, "p_yes": 0.1599497641238216, "predicted": 0}, {"index": 80, "id": "strategyqa_grounded/test/574", "label": 0, "p_yes": 0.6950887317296025, "predicted": 1}, {"index": 81, "id": "strategyqa_grounded/test/471", "label": 0, "p_yes": 0.3371282826678954, "predicted": 0}, {"index": 82, "id": "strategyqa_grounded/test/164", "label": 0, "p_yes": 0.6501673134786999, "predicted": 1}, {"index": 83, "id": "strategyqa_grounded/test/361", "label": 0, "p_yes": 0.8700704914925175, "predicted": 1}, {"index": 84, "id": "strategyqa_grounded/test/493", "label": 0, "p_yes": 0.3858403751771031, "predicted": 0}, {"index": 85, "id": "strategyqa_grounded/test/641", "label": 0, "p_yes": 0.17669921866493477, "predicted": 0}, {"index": 86, "id": "strategyqa_grounded/test/423", "label": 0, "p_yes": 0.557843747914673, "predicted": 1}, {"index": 87, "id": "strategyqa_grounded/test/566", "label": 1, "p_yes": 0.7642569670401766, "predicted": 1}, {"index": 88, "id": "strategyqa_grounded/test/312", "label": 0, "p_yes": 0.4052015997624753, "predicted": 0}, {"index": 89, "id": "strategyqa_grounded/test/389", "label": 1, "p_yes": 0.8781940874575084, "predicted": 1}, {"index": 90, "id": "strategyqa_grounded/test/607", "label": 0, "p_yes": 0.5552364296770378, "predicted": 1}, {"index": 91, "id": "strategyqa_grounded/test/16", "label": 1, "p_yes": 0.8848164022081814, "predicted": 1}, {"index": 92, "id": "strategyqa_grounded/test/81", "label": 1, "p_yes": 0.5395335996314243, "predicted": 1}, {"index": 93, "id": "strategyqa_grounded/test/223", "label": 1, "p_yes": 0.8400517025162386, "predicted": 1}, {"index": 94, "id": "strategyqa_grounded/test/631", "label": 0, "p_yes": 0.6249494470144896, "predicted": 1}, {"index": 95, "id": "strategyqa_grounded/test/165", "label": 1, "p_yes": 0.7875106497835266, "predicted": 1}, {"index": 96, "id": "strategyqa_grounded/test/9", "label": 0, "p_yes": 0.7170067436974613, "predicted": 1}, {"index": 97, "id": "strategyqa_grounded/test/555", "label": 0, "p_yes": 0.2647898598027952, "predicted": 0}, {"index": 98, "id": "strategyqa_grounded/test/475", "label": 1, "p_yes": 0.7324589696556713, "predicted": 1}, {"index": 99, "id": "strategyqa_grounded/test/129", "label": 0, "p_yes": 0.7386236071022823, "predicted": 1}, {"index": 100, "id": "strategyqa_grounded/test/647", "label": 0, "p_yes": 0.5691034994150934, "predicted": 1}, {"index": 101, "id": "strategyqa_grounded/test/86", "label": 1, "p_yes": 0.9034354168921619, "predicted": 1}, {"index": 102, "id": "strategyqa_grounded/test/372", "label": 0, "p_yes": 0.3379156803851074, "predicted": 0}, {"index": 103, "id": "strategyqa_grounded/test/655", "label": 0, "p_yes": 0.25263538447156375, "predicted": 0}, {"index": 104, "id": "strategyqa_grounded/test/69", "label": 1, "p_yes": 0.892498928528621, "predicted": 1}, {"index": 105, "id": "strategyqa_grounded/test/548", "label": 0, "p_yes": 0.3064059740999773, "predicted": 0}, {"index": 106, "id": "strategyqa_grounded/test/200", "label": 1, "p_yes": 0.8381469827160395, "predicted": 1}, {"index": 107, "id": "strategyqa_grounded/test/599", "label": 0, "p_yes": 0.7386236071022823, "predicted": 1}, {"index": 108, "id": "strategyqa_grounded/test/230", "label": 1, "p_yes": 0.8781940874575084, "predicted": 1}, {"index": 109, "id": "strategyqa_grounded/test/141", "label": 0, "p_yes": 0.532527400972905, "predicted": 1}, {"index": 110, "id": "strategyqa_grounded/test/368", "label": 0, "p_yes": 0.16137435903178676, "predicted": 0}, {"index": 111, "id": "strategyqa_grounded/test/481", "label": 0, "p_yes": 0.8789593042137193, "predicted": 1}, {"index": 112, "id": "strategyqa_grounded/test/142", "label": 1, "p_yes": 0.7191454080676265, "predicted": 1}, {"index": 113, "id": "strategyqa_grounded/test/544", "label": 1, "p_yes": 0.8627330315610859, "predicted": 1}, {"index": 114, "id": "strategyqa_grounded/test/377", "label": 0, "p_yes": 0.20839475395871923, "predicted": 0}, {"index": 115, "id": "strategyqa_grounded/test/480", "label": 0, "p_yes": 0.8095834238430544, "predicted": 1}, {"index": 116, "id": "strategyqa_grounded/test/601", "label": 0, "p_yes": 0.8931558381709804, "predicted": 1}, {"index": 117, "id": "strategyqa_grounded/test/237", "label": 1, "p_yes": 0.8751609769221215, "predicted": 1}, {"index": 118, "id": "strategyqa_grounded/test/643", "label": 1, "p_yes": 0.8961403587566864, "predicted": 1}, {"index": 119, "id": "strategyqa_grounded/test/521", "label": 0, "p_yes": 0.2338446616406918, "predicted": 0}, {"index": 120, "id": "strategyqa_grounded/test/621", "label": 1, "p_yes": 0.82381268845126, "predicted": 1}, {"index": 121, "id": "strategyqa_grounded/test/426", "label": 1, "p_yes": 0.837190382993577, "predicted": 1}, {"index": 122, "id": "strategyqa_grounded/test/162", "label": 0, "p_yes": 0.37835785727014837, "predicted": 0}, {"index": 123, "id": "strategyqa_grounded/test/151", "label": 0, "p_yes": 0.45173088802506633, "predicted": 0}, {"index": 124, "id": "strategyqa_grounded/test/181", "label": 1, "p_yes": 0.6091408997371434, "predicted": 1}, {"index": 125, "id": "strategyqa_grounded/test/71", "label": 1, "p_yes": 0.29456414804585873, "predicted": 0}, {"index": 126, "id": "strategyqa_grounded/test/63", "label": 0, "p_yes": 0.49471802828758443, "predicted": 0}, {"index": 127, "id": "strategyqa_grounded/test/275", "label": 0, "p_yes": 0.2909187147520262, "predicted": 0}]

adapted-ood-predictions.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "strategyqa_grounded/test/608", "label": 0, "p_yes": 0.0034834244288504124, "predicted": 0}, {"index": 1, "id": "strategyqa_grounded/test/72", "label": 1, "p_yes": 0.9998089671134949, "predicted": 1}, {"index": 2, "id": "strategyqa_grounded/test/371", "label": 1, "p_yes": 0.9863631129264832, "predicted": 1}, {"index": 3, "id": "strategyqa_grounded/test/155", "label": 1, "p_yes": 0.9995121955871582, "predicted": 1}, {"index": 4, "id": "strategyqa_grounded/test/565", "label": 1, "p_yes": 0.0012448433553799987, "predicted": 0}, {"index": 5, "id": "strategyqa_grounded/test/252", "label": 1, "p_yes": 0.9999865293502808, "predicted": 1}, {"index": 6, "id": "strategyqa_grounded/test/353", "label": 0, "p_yes": 0.07369627058506012, "predicted": 0}, {"index": 7, "id": "strategyqa_grounded/test/254", "label": 1, "p_yes": 0.9927672147750854, "predicted": 1}, {"index": 8, "id": "strategyqa_grounded/test/441", "label": 1, "p_yes": 0.47268345952033997, "predicted": 0}, {"index": 9, "id": "strategyqa_grounded/test/603", "label": 0, "p_yes": 0.0023967307060956955, "predicted": 0}, {"index": 10, "id": "strategyqa_grounded/test/212", "label": 0, "p_yes": 0.0007916296017356217, "predicted": 0}, {"index": 11, "id": "strategyqa_grounded/test/430", "label": 1, "p_yes": 0.15817435085773468, "predicted": 0}, {"index": 12, "id": "strategyqa_grounded/test/528", "label": 1, "p_yes": 0.99993896484375, "predicted": 1}, {"index": 13, "id": "strategyqa_grounded/test/204", "label": 0, "p_yes": 0.9997387528419495, "predicted": 1}, {"index": 14, "id": "strategyqa_grounded/test/225", "label": 1, "p_yes": 0.9986955523490906, "predicted": 1}, {"index": 15, "id": "strategyqa_grounded/test/463", "label": 1, "p_yes": 0.005384937860071659, "predicted": 0}, {"index": 16, "id": "strategyqa_grounded/test/43", "label": 0, "p_yes": 0.08882028609514236, "predicted": 0}, {"index": 17, "id": "strategyqa_grounded/test/525", "label": 0, "p_yes": 0.9993013143539429, "predicted": 1}, {"index": 18, "id": "strategyqa_grounded/test/431", "label": 0, "p_yes": 0.0005357805639505386, "predicted": 0}, {"index": 19, "id": "strategyqa_grounded/test/584", "label": 1, "p_yes": 0.9946980476379395, "predicted": 1}, {"index": 20, "id": "strategyqa_grounded/test/405", "label": 0, "p_yes": 0.6001883149147034, "predicted": 1}, {"index": 21, "id": "strategyqa_grounded/test/231", "label": 0, "p_yes": 0.005819959565997124, "predicted": 0}, {"index": 22, "id": "strategyqa_grounded/test/610", "label": 0, "p_yes": 0.531209409236908, "predicted": 1}, {"index": 23, "id": "strategyqa_grounded/test/515", "label": 1, "p_yes": 0.9688562154769897, "predicted": 1}, {"index": 24, "id": "strategyqa_grounded/test/184", "label": 0, "p_yes": 0.9997175335884094, "predicted": 1}, {"index": 25, "id": "strategyqa_grounded/test/573", "label": 1, "p_yes": 0.9998960494995117, "predicted": 1}, {"index": 26, "id": "strategyqa_grounded/test/39", "label": 0, "p_yes": 0.999823272228241, "predicted": 1}, {"index": 27, "id": "strategyqa_grounded/test/672", "label": 0, "p_yes": 0.002287227427586913, "predicted": 0}, {"index": 28, "id": "strategyqa_grounded/test/94", "label": 1, "p_yes": 0.9992327690124512, "predicted": 1}, {"index": 29, "id": "strategyqa_grounded/test/314", "label": 1, "p_yes": 0.9999833106994629, "predicted": 1}, {"index": 30, "id": "strategyqa_grounded/test/73", "label": 1, "p_yes": 0.9997262358665466, "predicted": 1}, {"index": 31, "id": "strategyqa_grounded/test/679", "label": 1, "p_yes": 0.9995197057723999, "predicted": 1}, {"index": 32, "id": "strategyqa_grounded/test/216", "label": 0, "p_yes": 0.010489091277122498, "predicted": 0}, {"index": 33, "id": "strategyqa_grounded/test/587", "label": 0, "p_yes": 0.000552778597921133, "predicted": 0}, {"index": 34, "id": "strategyqa_grounded/test/227", "label": 1, "p_yes": 0.5428633093833923, "predicted": 1}, {"index": 35, "id": "strategyqa_grounded/test/630", "label": 1, "p_yes": 0.9999605417251587, "predicted": 1}, {"index": 36, "id": "strategyqa_grounded/test/327", "label": 1, "p_yes": 0.9978172779083252, "predicted": 1}, {"index": 37, "id": "strategyqa_grounded/test/199", "label": 1, "p_yes": 0.9999895095825195, "predicted": 1}, {"index": 38, "id": "strategyqa_grounded/test/436", "label": 0, "p_yes": 0.0010322310263291001, "predicted": 0}, {"index": 39, "id": "strategyqa_grounded/test/335", "label": 0, "p_yes": 0.001597845577634871, "predicted": 0}, {"index": 40, "id": "strategyqa_grounded/test/46", "label": 0, "p_yes": 0.999779999256134, "predicted": 1}, {"index": 41, "id": "strategyqa_grounded/test/386", "label": 0, "p_yes": 0.9986955523490906, "predicted": 1}, {"index": 42, "id": "strategyqa_grounded/test/682", "label": 0, "p_yes": 0.7905303239822388, "predicted": 1}, {"index": 43, "id": "strategyqa_grounded/test/596", "label": 0, "p_yes": 0.7853085398674011, "predicted": 1}, {"index": 44, "id": "strategyqa_grounded/test/554", "label": 0, "p_yes": 0.000954731716774404, "predicted": 0}, {"index": 45, "id": "strategyqa_grounded/test/169", "label": 1, "p_yes": 0.9969245791435242, "predicted": 1}, {"index": 46, "id": "strategyqa_grounded/test/508", "label": 1, "p_yes": 0.999946117401123, "predicted": 1}, {"index": 47, "id": "strategyqa_grounded/test/56", "label": 0, "p_yes": 0.00446838466450572, "predicted": 0}, {"index": 48, "id": "strategyqa_grounded/test/217", "label": 1, "p_yes": 0.9473810195922852, "predicted": 1}, {"index": 49, "id": "strategyqa_grounded/test/253", "label": 1, "p_yes": 0.9980429410934448, "predicted": 1}, {"index": 50, "id": "strategyqa_grounded/test/569", "label": 0, "p_yes": 0.9912885427474976, "predicted": 1}, {"index": 51, "id": "strategyqa_grounded/test/384", "label": 0, "p_yes": 0.0006878511630930007, "predicted": 0}, {"index": 52, "id": "strategyqa_grounded/test/513", "label": 1, "p_yes": 0.007232763338834047, "predicted": 0}, {"index": 53, "id": "strategyqa_grounded/test/107", "label": 0, "p_yes": 0.998498797416687, "predicted": 1}, {"index": 54, "id": "strategyqa_grounded/test/616", "label": 0, "p_yes": 0.0004583129193633795, "predicted": 0}, {"index": 55, "id": "strategyqa_grounded/test/468", "label": 1, "p_yes": 0.3208213150501251, "predicted": 0}, {"index": 56, "id": "strategyqa_grounded/test/157", "label": 0, "p_yes": 0.0027149852830916643, "predicted": 0}, {"index": 57, "id": "strategyqa_grounded/test/465", "label": 1, "p_yes": 0.07585817575454712, "predicted": 0}, {"index": 58, "id": "strategyqa_grounded/test/579", "label": 1, "p_yes": 0.9879462122917175, "predicted": 1}, {"index": 59, "id": "strategyqa_grounded/test/532", "label": 1, "p_yes": 0.9999138116836548, "predicted": 1}, {"index": 60, "id": "strategyqa_grounded/test/606", "label": 0, "p_yes": 0.9959927201271057, "predicted": 1}, {"index": 61, "id": "strategyqa_grounded/test/209", "label": 0, "p_yes": 0.9992678761482239, "predicted": 1}, {"index": 62, "id": "strategyqa_grounded/test/88", "label": 1, "p_yes": 0.9999053478240967, "predicted": 1}, {"index": 63, "id": "strategyqa_grounded/test/281", "label": 0, "p_yes": 0.8311430811882019, "predicted": 1}, {"index": 64, "id": "strategyqa_grounded/test/295", "label": 1, "p_yes": 0.9996898174285889, "predicted": 1}, {"index": 65, "id": "strategyqa_grounded/test/457", "label": 1, "p_yes": 0.999045193195343, "predicted": 1}, {"index": 66, "id": "strategyqa_grounded/test/170", "label": 1, "p_yes": 0.9840936064720154, "predicted": 1}, {"index": 67, "id": "strategyqa_grounded/test/658", "label": 0, "p_yes": 0.9950177073478699, "predicted": 1}, {"index": 68, "id": "strategyqa_grounded/test/123", "label": 0, "p_yes": 0.9958011507987976, "predicted": 1}, {"index": 69, "id": "strategyqa_grounded/test/461", "label": 1, "p_yes": 0.04672496020793915, "predicted": 0}, {"index": 70, "id": "strategyqa_grounded/test/556", "label": 1, "p_yes": 0.9898303151130676, "predicted": 1}, {"index": 71, "id": "strategyqa_grounded/test/313", "label": 0, "p_yes": 0.4339814782142639, "predicted": 0}, {"index": 72, "id": "strategyqa_grounded/test/246", "label": 0, "p_yes": 0.002182716503739357, "predicted": 0}, {"index": 73, "id": "strategyqa_grounded/test/392", "label": 1, "p_yes": 0.011158030480146408, "predicted": 0}, {"index": 74, "id": "strategyqa_grounded/test/116", "label": 1, "p_yes": 0.9996140599250793, "predicted": 1}, {"index": 75, "id": "strategyqa_grounded/test/328", "label": 1, "p_yes": 0.9999861717224121, "predicted": 1}, {"index": 76, "id": "strategyqa_grounded/test/111", "label": 1, "p_yes": 0.0007437029271386564, "predicted": 0}, {"index": 77, "id": "strategyqa_grounded/test/444", "label": 0, "p_yes": 0.00034599160426296294, "predicted": 0}, {"index": 78, "id": "strategyqa_grounded/test/590", "label": 0, "p_yes": 0.07055959850549698, "predicted": 0}, {"index": 79, "id": "strategyqa_grounded/test/595", "label": 0, "p_yes": 0.0006361911655403674, "predicted": 0}, {"index": 80, "id": "strategyqa_grounded/test/574", "label": 0, "p_yes": 0.974821150302887, "predicted": 1}, {"index": 81, "id": "strategyqa_grounded/test/471", "label": 0, "p_yes": 0.04742587357759476, "predicted": 0}, {"index": 82, "id": "strategyqa_grounded/test/164", "label": 0, "p_yes": 0.9399133324623108, "predicted": 1}, {"index": 83, "id": "strategyqa_grounded/test/361", "label": 0, "p_yes": 0.999783456325531, "predicted": 1}, {"index": 84, "id": "strategyqa_grounded/test/493", "label": 0, "p_yes": 0.1127953976392746, "predicted": 0}, {"index": 85, "id": "strategyqa_grounded/test/641", "label": 0, "p_yes": 0.0010817153379321098, "predicted": 0}, {"index": 86, "id": "strategyqa_grounded/test/423", "label": 0, "p_yes": 0.7371581792831421, "predicted": 1}, {"index": 87, "id": "strategyqa_grounded/test/566", "label": 1, "p_yes": 0.9946150183677673, "predicted": 1}, {"index": 88, "id": "strategyqa_grounded/test/312", "label": 0, "p_yes": 0.1540575623512268, "predicted": 0}, {"index": 89, "id": "strategyqa_grounded/test/389", "label": 1, "p_yes": 0.999843955039978, "predicted": 1}, {"index": 90, "id": "strategyqa_grounded/test/607", "label": 0, "p_yes": 0.7279754281044006, "predicted": 1}, {"index": 91, "id": "strategyqa_grounded/test/16", "label": 1, "p_yes": 0.9998822212219238, "predicted": 1}, {"index": 92, "id": "strategyqa_grounded/test/81", "label": 1, "p_yes": 0.66888028383255, "predicted": 1}, {"index": 93, "id": "strategyqa_grounded/test/223", "label": 1, "p_yes": 0.9993638396263123, "predicted": 1}, {"index": 94, "id": "strategyqa_grounded/test/631", "label": 0, "p_yes": 0.9059898257255554, "predicted": 1}, {"index": 95, "id": "strategyqa_grounded/test/165", "label": 1, "p_yes": 0.9970190525054932, "predicted": 1}, {"index": 96, "id": "strategyqa_grounded/test/9", "label": 0, "p_yes": 0.9840936064720154, "predicted": 1}, {"index": 97, "id": "strategyqa_grounded/test/555", "label": 0, "p_yes": 0.01065251138061285, "predicted": 0}, {"index": 98, "id": "strategyqa_grounded/test/475", "label": 1, "p_yes": 0.9886682629585266, "predicted": 1}, {"index": 99, "id": "strategyqa_grounded/test/129", "label": 0, "p_yes": 0.9901401400566101, "predicted": 1}, {"index": 100, "id": "strategyqa_grounded/test/647", "label": 0, "p_yes": 0.7745833992958069, "predicted": 1}, {"index": 101, "id": "strategyqa_grounded/test/86", "label": 1, "p_yes": 0.9999508857727051, "predicted": 1}, {"index": 102, "id": "strategyqa_grounded/test/372", "label": 0, "p_yes": 0.048136770725250244, "predicted": 0}, {"index": 103, "id": "strategyqa_grounded/test/655", "label": 0, "p_yes": 0.00806199200451374, "predicted": 0}, {"index": 104, "id": "strategyqa_grounded/test/69", "label": 1, "p_yes": 0.9999165534973145, "predicted": 1}, {"index": 105, "id": "strategyqa_grounded/test/548", "label": 0, "p_yes": 0.02595735713839531, "predicted": 0}, {"index": 106, "id": "strategyqa_grounded/test/200", "label": 1, "p_yes": 0.999322772026062, "predicted": 1}, {"index": 107, "id": "strategyqa_grounded/test/599", "label": 0, "p_yes": 0.9901401400566101, "predicted": 1}, {"index": 108, "id": "strategyqa_grounded/test/230", "label": 1, "p_yes": 0.999843955039978, "predicted": 1}, {"index": 109, "id": "strategyqa_grounded/test/141", "label": 0, "p_yes": 0.6406359076499939, "predicted": 1}, {"index": 110, "id": "strategyqa_grounded/test/368", "label": 0, "p_yes": 0.0006667023408226669, "predicted": 0}, {"index": 111, "id": "strategyqa_grounded/test/481", "label": 0, "p_yes": 0.9998488426208496, "predicted": 1}, {"index": 112, "id": "strategyqa_grounded/test/142", "label": 1, "p_yes": 0.9848109483718872, "predicted": 1}, {"index": 113, "id": "strategyqa_grounded/test/544", "label": 1, "p_yes": 0.999713122844696, "predicted": 1}, {"index": 114, "id": "strategyqa_grounded/test/377", "label": 0, "p_yes": 0.0026730059180408716, "predicted": 0}, {"index": 115, "id": "strategyqa_grounded/test/480", "label": 0, "p_yes": 0.9983769655227661, "predicted": 1}, {"index": 116, "id": "strategyqa_grounded/test/601", "label": 0, "p_yes": 0.999919056892395, "predicted": 1}, {"index": 117, "id": "strategyqa_grounded/test/237", "label": 1, "p_yes": 0.999823272228241, "predicted": 1}, {"index": 118, "id": "strategyqa_grounded/test/643", "label": 1, "p_yes": 0.999929666519165, "predicted": 1}, {"index": 119, "id": "strategyqa_grounded/test/521", "label": 0, "p_yes": 0.005139611195772886, "predicted": 0}, {"index": 120, "id": "strategyqa_grounded/test/621", "label": 1, "p_yes": 0.9989350438117981, "predicted": 1}, {"index": 121, "id": "strategyqa_grounded/test/426", "label": 1, "p_yes": 0.9993013143539429, "predicted": 1}, {"index": 122, "id": "strategyqa_grounded/test/162", "label": 0, "p_yes": 0.09947021305561066, "predicted": 0}, {"index": 123, "id": "strategyqa_grounded/test/151", "label": 0, "p_yes": 0.29746994376182556, "predicted": 0}, {"index": 124, "id": "strategyqa_grounded/test/181", "label": 1, "p_yes": 0.8774768114089966, "predicted": 1}, {"index": 125, "id": "strategyqa_grounded/test/71", "label": 1, "p_yes": 0.020332353189587593, "predicted": 0}, {"index": 126, "id": "strategyqa_grounded/test/63", "label": 0, "p_yes": 0.4765796661376953, "predicted": 0}, {"index": 127, "id": "strategyqa_grounded/test/275", "label": 0, "p_yes": 0.018833130598068237, "predicted": 0}]

adapted-valid-calibrated.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "boolq/validation/138", "label": 1, "p_yes": 0.8763040956927123, "predicted": 1}, {"index": 1, "id": "boolq/validation/429", "label": 0, "p_yes": 0.13936327323545517, "predicted": 0}, {"index": 2, "id": "boolq/validation/93", "label": 1, "p_yes": 0.8572240685542992, "predicted": 1}, {"index": 3, "id": "boolq/validation/430", "label": 0, "p_yes": 0.548269119935004, "predicted": 1}, {"index": 4, "id": "boolq/validation/121", "label": 1, "p_yes": 0.838626757477774, "predicted": 1}, {"index": 5, "id": "boolq/validation/295", "label": 0, "p_yes": 0.07365048219950082, "predicted": 0}, {"index": 6, "id": "boolq/validation/108", "label": 1, "p_yes": 0.8207249590165475, "predicted": 1}, {"index": 7, "id": "boolq/validation/91", "label": 1, "p_yes": 0.8996839563419411, "predicted": 1}, {"index": 8, "id": "boolq/validation/246", "label": 1, "p_yes": 0.9188191783688598, "predicted": 1}, {"index": 9, "id": "boolq/validation/349", "label": 0, "p_yes": 0.5255083253511216, "predicted": 1}, {"index": 10, "id": "boolq/validation/346", "label": 0, "p_yes": 0.27449805919458276, "predicted": 0}, {"index": 11, "id": "boolq/validation/416", "label": 0, "p_yes": 0.30715486919386653, "predicted": 0}, {"index": 12, "id": "boolq/validation/136", "label": 1, "p_yes": 0.838626757477774, "predicted": 1}, {"index": 13, "id": "boolq/validation/4", "label": 1, "p_yes": 0.8941567935480498, "predicted": 1}, {"index": 14, "id": "boolq/validation/418", "label": 1, "p_yes": 0.6174920868852052, "predicted": 1}, {"index": 15, "id": "boolq/validation/175", "label": 1, "p_yes": 0.20207678756515657, "predicted": 0}, {"index": 16, "id": "boolq/validation/461", "label": 1, "p_yes": 0.31091454859983914, "predicted": 0}, {"index": 17, "id": "boolq/validation/234", "label": 0, "p_yes": 0.15573705757431885, "predicted": 0}, {"index": 18, "id": "boolq/validation/72", "label": 1, "p_yes": 0.743354773810382, "predicted": 1}, {"index": 19, "id": "boolq/validation/333", "label": 1, "p_yes": 0.7324589696556713, "predicted": 1}, {"index": 20, "id": "boolq/validation/240", "label": 0, "p_yes": 0.6541614220201916, "predicted": 1}, {"index": 21, "id": "boolq/validation/0", "label": 1, "p_yes": 0.2533008465712965, "predicted": 0}, {"index": 22, "id": "boolq/validation/173", "label": 1, "p_yes": 0.8759266406246355, "predicted": 1}, {"index": 23, "id": "boolq/validation/469", "label": 1, "p_yes": 0.9128216623400451, "predicted": 1}, {"index": 24, "id": "boolq/validation/452", "label": 0, "p_yes": 0.10190318347314736, "predicted": 0}, {"index": 25, "id": "boolq/validation/103", "label": 1, "p_yes": 0.8974536525750565, "predicted": 1}, {"index": 26, "id": "boolq/validation/313", "label": 1, "p_yes": 0.934797540175596, "predicted": 1}, {"index": 27, "id": "boolq/validation/251", "label": 0, "p_yes": 0.13233683902036816, "predicted": 0}, {"index": 28, "id": "boolq/validation/152", "label": 1, "p_yes": 0.8253382216792983, "predicted": 1}, {"index": 29, "id": "boolq/validation/384", "label": 1, "p_yes": 0.8826474780767039, "predicted": 1}, {"index": 30, "id": "boolq/validation/417", "label": 1, "p_yes": 0.8563557686180491, "predicted": 1}, {"index": 31, "id": "boolq/validation/33", "label": 0, "p_yes": 0.336341788111827, "predicted": 0}, {"index": 32, "id": "boolq/validation/133", "label": 0, "p_yes": 0.14538220024493664, "predicted": 0}, {"index": 33, "id": "boolq/validation/456", "label": 1, "p_yes": 0.2634210493660585, "predicted": 0}, {"index": 34, "id": "boolq/validation/435", "label": 0, "p_yes": 0.8935066306459123, "predicted": 1}, {"index": 35, "id": "boolq/validation/275", "label": 1, "p_yes": 0.9000135697214187, "predicted": 1}, {"index": 36, "id": "boolq/validation/218", "label": 0, "p_yes": 0.7979239166481518, "predicted": 1}, {"index": 37, "id": "boolq/validation/161", "label": 1, "p_yes": 0.9241039057324933, "predicted": 1}, {"index": 38, "id": "boolq/validation/241", "label": 1, "p_yes": 0.8664480741473063, "predicted": 1}, {"index": 39, "id": "boolq/validation/30", "label": 1, "p_yes": 0.9022254470995756, "predicted": 1}, {"index": 40, "id": "boolq/validation/311", "label": 1, "p_yes": 0.8606344077430843, "predicted": 1}, {"index": 41, "id": "boolq/validation/14", "label": 0, "p_yes": 0.7212743349137087, "predicted": 1}, {"index": 42, "id": "boolq/validation/6", "label": 1, "p_yes": 0.6057821347012533, "predicted": 1}, {"index": 43, "id": "boolq/validation/92", "label": 0, "p_yes": 0.27309770031987174, "predicted": 0}, {"index": 44, "id": "boolq/validation/180", "label": 0, "p_yes": 0.10684003366573085, "predicted": 0}, {"index": 45, "id": "boolq/validation/9", "label": 1, "p_yes": 0.8696663339012921, "predicted": 1}, {"index": 46, "id": "boolq/validation/76", "label": 1, "p_yes": 0.89010804367529, "predicted": 1}, {"index": 47, "id": "boolq/validation/222", "label": 1, "p_yes": 0.4569689860781037, "predicted": 0}, {"index": 48, "id": "boolq/validation/98", "label": 0, "p_yes": 0.21367136416306062, "predicted": 0}, {"index": 49, "id": "boolq/validation/5", "label": 1, "p_yes": 0.8751609769221215, "predicted": 1}, {"index": 50, "id": "boolq/validation/370", "label": 1, "p_yes": 0.918536185016268, "predicted": 1}, {"index": 51, "id": "boolq/validation/396", "label": 0, "p_yes": 0.7513332311768949, "predicted": 1}, {"index": 52, "id": "boolq/validation/31", "label": 1, "p_yes": 0.8743903012824169, "predicted": 1}, {"index": 53, "id": "boolq/validation/39", "label": 1, "p_yes": 0.8928096902665641, "predicted": 1}, {"index": 54, "id": "boolq/validation/2", "label": 0, "p_yes": 0.2095589799658084, "predicted": 0}, {"index": 55, "id": "boolq/validation/306", "label": 1, "p_yes": 0.8362302512813161, "predicted": 1}, {"index": 56, "id": "boolq/validation/324", "label": 1, "p_yes": 0.7226879822013658, "predicted": 1}, {"index": 57, "id": "boolq/validation/352", "label": 1, "p_yes": 0.6890855034866661, "predicted": 1}, {"index": 58, "id": "boolq/validation/341", "label": 1, "p_yes": 0.6965793023063065, "predicted": 1}, {"index": 59, "id": "boolq/validation/470", "label": 1, "p_yes": 0.13643290724544585, "predicted": 0}, {"index": 60, "id": "boolq/validation/431", "label": 0, "p_yes": 0.7162917759933994, "predicted": 1}, {"index": 61, "id": "boolq/validation/64", "label": 1, "p_yes": 0.8732314471973439, "predicted": 1}, {"index": 62, "id": "boolq/validation/13", "label": 1, "p_yes": 0.8196848513698798, "predicted": 1}, {"index": 63, "id": "boolq/validation/433", "label": 1, "p_yes": 0.8684714995954412, "predicted": 1}, {"index": 64, "id": "boolq/validation/48", "label": 1, "p_yes": 0.856787061194167, "predicted": 1}, {"index": 65, "id": "boolq/validation/119", "label": 0, "p_yes": 0.18398592069967362, "predicted": 0}, {"index": 66, "id": "boolq/validation/272", "label": 1, "p_yes": 0.9099958904958743, "predicted": 1}, {"index": 67, "id": "boolq/validation/131", "label": 0, "p_yes": 0.1920490330402628, "predicted": 0}, {"index": 68, "id": "boolq/validation/282", "label": 1, "p_yes": 0.9216674324902706, "predicted": 1}, {"index": 69, "id": "boolq/validation/434", "label": 1, "p_yes": 0.7950694333871589, "predicted": 1}, {"index": 70, "id": "boolq/validation/21", "label": 1, "p_yes": 0.7506744935359597, "predicted": 1}, {"index": 71, "id": "boolq/validation/111", "label": 0, "p_yes": 0.16280917907500198, "predicted": 0}, {"index": 72, "id": "boolq/validation/337", "label": 1, "p_yes": 0.9256453747319169, "predicted": 1}, {"index": 73, "id": "boolq/validation/283", "label": 0, "p_yes": 0.7785363851186616, "predicted": 1}, {"index": 74, "id": "boolq/validation/379", "label": 1, "p_yes": 0.10484050844155457, "predicted": 0}, {"index": 75, "id": "boolq/validation/357", "label": 1, "p_yes": 0.8743903012824169, "predicted": 1}, {"index": 76, "id": "boolq/validation/280", "label": 1, "p_yes": 0.9082326535536567, "predicted": 1}, {"index": 77, "id": "boolq/validation/274", "label": 0, "p_yes": 0.8063055730000722, "predicted": 1}, {"index": 78, "id": "boolq/validation/424", "label": 0, "p_yes": 0.2049307039185152, "predicted": 0}, {"index": 79, "id": "boolq/validation/26", "label": 0, "p_yes": 0.7519902627003572, "predicted": 1}, {"index": 80, "id": "boolq/validation/297", "label": 0, "p_yes": 0.18345781286713056, "predicted": 0}, {"index": 81, "id": "boolq/validation/390", "label": 0, "p_yes": 0.8040959679508792, "predicted": 1}, {"index": 82, "id": "boolq/validation/315", "label": 1, "p_yes": 0.8606344077430843, "predicted": 1}, {"index": 83, "id": "boolq/validation/281", "label": 1, "p_yes": 0.9303270008794026, "predicted": 1}, {"index": 84, "id": "boolq/validation/87", "label": 1, "p_yes": 0.8437963716295682, "predicted": 1}, {"index": 85, "id": "boolq/validation/459", "label": 1, "p_yes": 0.8074039040230444, "predicted": 1}, {"index": 86, "id": "boolq/validation/179", "label": 0, "p_yes": 0.21964759788279903, "predicted": 0}, {"index": 87, "id": "boolq/validation/226", "label": 1, "p_yes": 0.8191660752206, "predicted": 1}, {"index": 88, "id": "boolq/validation/468", "label": 1, "p_yes": 0.9200923778708737, "predicted": 1}, {"index": 89, "id": "boolq/validation/147", "label": 1, "p_yes": 0.8639773453749758, "predicted": 1}, {"index": 90, "id": "boolq/validation/88", "label": 1, "p_yes": 0.8400517025162386, "predicted": 1}, {"index": 91, "id": "boolq/validation/174", "label": 1, "p_yes": 0.8222749516870927, "predicted": 1}, {"index": 92, "id": "boolq/validation/294", "label": 1, "p_yes": 0.9082326535536567, "predicted": 1}, {"index": 93, "id": "boolq/validation/141", "label": 1, "p_yes": 0.8778280013573592, "predicted": 1}, {"index": 94, "id": "boolq/validation/22", "label": 1, "p_yes": 0.7406580328656804, "predicted": 1}, {"index": 95, "id": "boolq/validation/79", "label": 0, "p_yes": 0.31469933579827575, "predicted": 0}, {"index": 96, "id": "boolq/validation/211", "label": 1, "p_yes": 0.5465242094331658, "predicted": 1}, {"index": 97, "id": "boolq/validation/183", "label": 0, "p_yes": 0.15481328346944415, "predicted": 0}, {"index": 98, "id": "boolq/validation/268", "label": 0, "p_yes": 0.7584995388013295, "predicted": 1}, {"index": 99, "id": "boolq/validation/186", "label": 0, "p_yes": 0.39505915149441273, "predicted": 0}, {"index": 100, "id": "boolq/validation/307", "label": 1, "p_yes": 0.568239737813132, "predicted": 1}, {"index": 101, "id": "boolq/validation/236", "label": 1, "p_yes": 0.8944880839729445, "predicted": 1}, {"index": 102, "id": "boolq/validation/376", "label": 0, "p_yes": 0.6174920868852052, "predicted": 1}, {"index": 103, "id": "boolq/validation/361", "label": 0, "p_yes": 0.8273592994882191, "predicted": 1}, {"index": 104, "id": "boolq/validation/422", "label": 1, "p_yes": 0.8770738590225957, "predicted": 1}, {"index": 105, "id": "boolq/validation/338", "label": 1, "p_yes": 0.9040639971576729, "predicted": 1}, {"index": 106, "id": "boolq/validation/125", "label": 0, "p_yes": 0.793342818149916, "predicted": 1}, {"index": 107, "id": "boolq/validation/400", "label": 1, "p_yes": 0.6099789737675719, "predicted": 1}, {"index": 108, "id": "boolq/validation/419", "label": 1, "p_yes": 0.793342818149916, "predicted": 1}, {"index": 109, "id": "boolq/validation/320", "label": 1, "p_yes": 0.8589438083537909, "predicted": 1}, {"index": 110, "id": "boolq/validation/464", "label": 0, "p_yes": 0.6396847017121403, "predicted": 1}, {"index": 111, "id": "boolq/validation/195", "label": 1, "p_yes": 0.9003476329397416, "predicted": 1}, {"index": 112, "id": "boolq/validation/153", "label": 1, "p_yes": 0.9296522236196695, "predicted": 1}, {"index": 113, "id": "boolq/validation/371", "label": 1, "p_yes": 0.7956419807026182, "predicted": 1}, {"index": 114, "id": "boolq/validation/167", "label": 1, "p_yes": 0.23510883799307838, "predicted": 0}, {"index": 115, "id": "boolq/validation/163", "label": 1, "p_yes": 0.8692674331673763, "predicted": 1}, {"index": 116, "id": "boolq/validation/134", "label": 1, "p_yes": 0.9122550658394967, "predicted": 1}, {"index": 117, "id": "boolq/validation/109", "label": 0, "p_yes": 0.10921517873443422, "predicted": 0}, {"index": 118, "id": "boolq/validation/245", "label": 1, "p_yes": 0.8862478040182775, "predicted": 1}, {"index": 119, "id": "boolq/validation/10", "label": 1, "p_yes": 0.915898427893708, "predicted": 1}, {"index": 120, "id": "boolq/validation/230", "label": 1, "p_yes": 0.8165409515685887, "predicted": 1}, {"index": 121, "id": "boolq/validation/224", "label": 0, "p_yes": 0.6501673134786999, "predicted": 1}, {"index": 122, "id": "boolq/validation/25", "label": 0, "p_yes": 0.13810099869946751, "predicted": 0}, {"index": 123, "id": "boolq/validation/407", "label": 0, "p_yes": 0.08198231095099373, "predicted": 0}, {"index": 124, "id": "boolq/validation/321", "label": 0, "p_yes": 0.13852068149979124, "predicted": 0}, {"index": 125, "id": "boolq/validation/391", "label": 1, "p_yes": 0.6290672149046538, "predicted": 1}, {"index": 126, "id": "boolq/validation/23", "label": 0, "p_yes": 0.8958223487313407, "predicted": 1}, {"index": 127, "id": "boolq/validation/144", "label": 1, "p_yes": 0.8993586640836762, "predicted": 1}]

adapted-valid-predictions.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "boolq/validation/138", "label": 1, "p_yes": 0.9998313188552856, "predicted": 1}, {"index": 1, "id": "boolq/validation/429", "label": 0, "p_yes": 0.00031015597051009536, "predicted": 0}, {"index": 2, "id": "boolq/validation/93", "label": 1, "p_yes": 0.9996485710144043, "predicted": 1}, {"index": 3, "id": "boolq/validation/430", "label": 0, "p_yes": 0.7025300860404968, "predicted": 1}, {"index": 4, "id": "boolq/validation/121", "label": 1, "p_yes": 0.9993333220481873, "predicted": 1}, {"index": 5, "id": "boolq/validation/295", "label": 0, "p_yes": 1.32119585032342e-05, "predicted": 0}, {"index": 6, "id": "boolq/validation/108", "label": 1, "p_yes": 0.9988304972648621, "predicted": 1}, {"index": 7, "id": "boolq/validation/91", "label": 1, "p_yes": 0.9999407529830933, "predicted": 1}, {"index": 8, "id": "boolq/validation/246", "label": 1, "p_yes": 0.9999788999557495, "predicted": 1}, {"index": 9, "id": "boolq/validation/349", "label": 0, "p_yes": 0.6113819479942322, "predicted": 1}, {"index": 10, "id": "boolq/validation/346", "label": 0, "p_yes": 0.013222822919487953, "predicted": 0}, {"index": 11, "id": "boolq/validation/416", "label": 0, "p_yes": 0.026355354115366936, "predicted": 0}, {"index": 12, "id": "boolq/validation/136", "label": 1, "p_yes": 0.9993333220481873, "predicted": 1}, {"index": 13, "id": "boolq/validation/4", "label": 1, "p_yes": 0.9999227523803711, "predicted": 1}, {"index": 14, "id": "boolq/validation/418", "label": 1, "p_yes": 0.8933094143867493, "predicted": 1}, {"index": 15, "id": "boolq/validation/175", "label": 1, "p_yes": 0.0022518474142998457, "predicted": 0}, {"index": 16, "id": "boolq/validation/461", "label": 1, "p_yes": 0.02843603491783142, "predicted": 0}, {"index": 17, "id": "boolq/validation/234", "label": 0, "p_yes": 0.000552778597921133, "predicted": 0}, {"index": 18, "id": "boolq/validation/72", "label": 1, "p_yes": 0.9911527037620544, "predicted": 1}, {"index": 19, "id": "boolq/validation/333", "label": 1, "p_yes": 0.9886682629585266, "predicted": 1}, {"index": 20, "id": "boolq/validation/240", "label": 0, "p_yes": 0.9441768527030945, "predicted": 1}, {"index": 21, "id": "boolq/validation/0", "label": 1, "p_yes": 0.008187909610569477, "predicted": 0}, {"index": 22, "id": "boolq/validation/173", "label": 1, "p_yes": 0.9998286962509155, "predicted": 1}, {"index": 23, "id": "boolq/validation/469", "label": 1, "p_yes": 0.9999701976776123, "predicted": 1}, {"index": 24, "id": "boolq/validation/452", "label": 0, "p_yes": 6.402006692951545e-05, "predicted": 0}, {"index": 25, "id": "boolq/validation/103", "label": 1, "p_yes": 0.9999339580535889, "predicted": 1}, {"index": 26, "id": "boolq/validation/313", "label": 1, "p_yes": 0.9999926090240479, "predicted": 1}, {"index": 27, "id": "boolq/validation/251", "label": 0, "p_yes": 0.00023782205244060606, "predicted": 0}, {"index": 28, "id": "boolq/validation/152", "label": 1, "p_yes": 0.9989836812019348, "predicted": 1}, {"index": 29, "id": "boolq/validation/384", "label": 1, "p_yes": 0.9998706579208374, "predicted": 1}, {"index": 30, "id": "boolq/validation/417", "label": 1, "p_yes": 0.9996373653411865, "predicted": 1}, {"index": 31, "id": "boolq/validation/33", "label": 0, "p_yes": 0.04672496020793915, "predicted": 0}, {"index": 32, "id": "boolq/validation/133", "label": 0, "p_yes": 0.0003859660937450826, "predicted": 0}, {"index": 33, "id": "boolq/validation/456", "label": 1, "p_yes": 0.010328153148293495, "predicted": 0}, {"index": 34, "id": "boolq/validation/435", "label": 0, "p_yes": 0.9999203681945801, "predicted": 1}, {"index": 35, "id": "boolq/validation/275", "label": 1, "p_yes": 0.9999417066574097, "predicted": 1}, {"index": 36, "id": "boolq/validation/218", "label": 0, "p_yes": 0.9977481961250305, "predicted": 1}, {"index": 37, "id": "boolq/validation/161", "label": 1, "p_yes": 0.9999847412109375, "predicted": 1}, {"index": 38, "id": "boolq/validation/241", "label": 1, "p_yes": 0.9997507929801941, "predicted": 1}, {"index": 39, "id": "boolq/validation/30", "label": 1, "p_yes": 0.9999477863311768, "predicted": 1}, {"index": 40, "id": "boolq/validation/311", "label": 1, "p_yes": 0.9996898174285889, "predicted": 1}, {"index": 41, "id": "boolq/validation/14", "label": 0, "p_yes": 0.985496461391449, "predicted": 1}, {"index": 42, "id": "boolq/validation/6", "label": 1, "p_yes": 0.8705973029136658, "predicted": 1}, {"index": 43, "id": "boolq/validation/92", "label": 0, "p_yes": 0.012821214273571968, "predicted": 0}, {"index": 44, "id": "boolq/validation/180", "label": 0, "p_yes": 8.092757343547419e-05, "predicted": 0}, {"index": 45, "id": "boolq/validation/9", "label": 1, "p_yes": 0.999779999256134, "predicted": 1}, {"index": 46, "id": "boolq/validation/76", "label": 1, "p_yes": 0.9999068975448608, "predicted": 1}, {"index": 47, "id": "boolq/validation/222", "label": 1, "p_yes": 0.31742629408836365, "predicted": 0}, {"index": 48, "id": "boolq/validation/98", "label": 0, "p_yes": 0.00307536986656487, "predicted": 0}, {"index": 49, "id": "boolq/validation/5", "label": 1, "p_yes": 0.999823272228241, "predicted": 1}, {"index": 50, "id": "boolq/validation/370", "label": 1, "p_yes": 0.9999785423278809, "predicted": 1}, {"index": 51, "id": "boolq/validation/396", "label": 0, "p_yes": 0.9926542043685913, "predicted": 1}, {"index": 52, "id": "boolq/validation/31", "label": 1, "p_yes": 0.9998176693916321, "predicted": 1}, {"index": 53, "id": "boolq/validation/39", "label": 1, "p_yes": 0.99991774559021, "predicted": 1}, {"index": 54, "id": "boolq/validation/2", "label": 0, "p_yes": 0.0027576221618801355, "predicted": 0}, {"index": 55, "id": "boolq/validation/306", "label": 1, "p_yes": 0.9992792010307312, "predicted": 1}, {"index": 56, "id": "boolq/validation/324", "label": 1, "p_yes": 0.9859364032745361, "predicted": 1}, {"index": 57, "id": "boolq/validation/352", "label": 1, "p_yes": 0.971563994884491, "predicted": 1}, {"index": 58, "id": "boolq/validation/341", "label": 1, "p_yes": 0.9755768775939941, "predicted": 1}, {"index": 59, "id": "boolq/validation/470", "label": 1, "p_yes": 0.0002780309587251395, "predicted": 0}, {"index": 60, "id": "boolq/validation/431", "label": 0, "p_yes": 0.9838472008705139, "predicted": 1}, {"index": 61, "id": "boolq/validation/64", "label": 1, "p_yes": 0.9998089671134949, "predicted": 1}, {"index": 62, "id": "boolq/validation/13", "label": 1, "p_yes": 0.998793363571167, "predicted": 1}, {"index": 63, "id": "boolq/validation/433", "label": 1, "p_yes": 0.9997695088386536, "predicted": 1}, {"index": 64, "id": "boolq/validation/48", "label": 1, "p_yes": 0.9996429681777954, "predicted": 1}, {"index": 65, "id": "boolq/validation/119", "label": 0, "p_yes": 0.0013458605390042067, "predicted": 0}, {"index": 66, "id": "boolq/validation/272", "label": 1, "p_yes": 0.9999651908874512, "predicted": 1}, {"index": 67, "id": "boolq/validation/131", "label": 0, "p_yes": 0.001700722612440586, "predicted": 0}, {"index": 68, "id": "boolq/validation/282", "label": 1, "p_yes": 0.9999822378158569, "predicted": 1}, {"index": 69, "id": "boolq/validation/434", "label": 1, "p_yes": 0.9975656270980835, "predicted": 1}, {"index": 70, "id": "boolq/validation/21", "label": 1, "p_yes": 0.9925393462181091, "predicted": 1}, {"index": 71, "id": "boolq/validation/111", "label": 0, "p_yes": 0.0006986756925471127, "predicted": 0}, {"index": 72, "id": "boolq/validation/337", "label": 1, "p_yes": 0.9999861717224121, "predicted": 1}, {"index": 73, "id": "boolq/validation/283", "label": 0, "p_yes": 0.9962345957756042, "predicted": 1}, {"index": 74, "id": "boolq/validation/379", "label": 1, "p_yes": 7.368592923739925e-05, "predicted": 0}, {"index": 75, "id": "boolq/validation/357", "label": 1, "p_yes": 0.9998176693916321, "predicted": 1}, {"index": 76, "id": "boolq/validation/280", "label": 1, "p_yes": 0.9999617338180542, "predicted": 1}, {"index": 77, "id": "boolq/validation/274", "label": 0, "p_yes": 0.998217761516571, "predicted": 1}, {"index": 78, "id": "boolq/validation/424", "label": 0, "p_yes": 0.0024343819823116064, "predicted": 0}, {"index": 79, "id": "boolq/validation/26", "label": 0, "p_yes": 0.9927672147750854, "predicted": 1}, {"index": 80, "id": "boolq/validation/297", "label": 0, "p_yes": 0.001325022429227829, "predicted": 0}, {"index": 81, "id": "boolq/validation/390", "label": 0, "p_yes": 0.9981030225753784, "predicted": 1}, {"index": 82, "id": "boolq/validation/315", "label": 1, "p_yes": 0.9996898174285889, "predicted": 1}, {"index": 83, "id": "boolq/validation/281", "label": 1, "p_yes": 0.9999898672103882, "predicted": 1}, {"index": 84, "id": "boolq/validation/87", "label": 1, "p_yes": 0.999438464641571, "predicted": 1}, {"index": 85, "id": "boolq/validation/459", "label": 1, "p_yes": 0.9982725381851196, "predicted": 1}, {"index": 86, "id": "boolq/validation/179", "label": 0, "p_yes": 0.0035936026833951473, "predicted": 0}, {"index": 87, "id": "boolq/validation/226", "label": 1, "p_yes": 0.9987744688987732, "predicted": 1}, {"index": 88, "id": "boolq/validation/468", "label": 1, "p_yes": 0.9999804496765137, "predicted": 1}, {"index": 89, "id": "boolq/validation/147", "label": 1, "p_yes": 0.9997262358665466, "predicted": 1}, {"index": 90, "id": "boolq/validation/88", "label": 1, "p_yes": 0.9993638396263123, "predicted": 1}, {"index": 91, "id": "boolq/validation/174", "label": 1, "p_yes": 0.9988840222358704, "predicted": 1}, {"index": 92, "id": "boolq/validation/294", "label": 1, "p_yes": 0.9999617338180542, "predicted": 1}, {"index": 93, "id": "boolq/validation/141", "label": 1, "p_yes": 0.999841570854187, "predicted": 1}, {"index": 94, "id": "boolq/validation/22", "label": 1, "p_yes": 0.9905874133110046, "predicted": 1}, {"index": 95, "id": "boolq/validation/79", "label": 0, "p_yes": 0.030675798654556274, "predicted": 0}, {"index": 96, "id": "boolq/validation/211", "label": 1, "p_yes": 0.6959583163261414, "predicted": 1}, {"index": 97, "id": "boolq/validation/183", "label": 0, "p_yes": 0.0005357805639505386, "predicted": 0}, {"index": 98, "id": "boolq/validation/268", "label": 0, "p_yes": 0.9938070178031921, "predicted": 1}, {"index": 99, "id": "boolq/validation/186", "label": 0, "p_yes": 0.13117322325706482, "predicted": 0}, {"index": 100, "id": "boolq/validation/307", "label": 1, "p_yes": 0.7718434929847717, "predicted": 1}, {"index": 101, "id": "boolq/validation/236", "label": 1, "p_yes": 0.9999239444732666, "predicted": 1}, {"index": 102, "id": "boolq/validation/376", "label": 0, "p_yes": 0.8933094143867493, "predicted": 1}, {"index": 103, "id": "boolq/validation/361", "label": 0, "p_yes": 0.999045193195343, "predicted": 1}, {"index": 104, "id": "boolq/validation/422", "label": 1, "p_yes": 0.9998365640640259, "predicted": 1}, {"index": 105, "id": "boolq/validation/338", "label": 1, "p_yes": 0.9999524354934692, "predicted": 1}, {"index": 106, "id": "boolq/validation/125", "label": 0, "p_yes": 0.9974491000175476, "predicted": 1}, {"index": 107, "id": "boolq/validation/400", "label": 1, "p_yes": 0.8791467547416687, "predicted": 1}, {"index": 108, "id": "boolq/validation/419", "label": 1, "p_yes": 0.9974491000175476, "predicted": 1}, {"index": 109, "id": "boolq/validation/320", "label": 1, "p_yes": 0.9996699094772339, "predicted": 1}, {"index": 110, "id": "boolq/validation/464", "label": 0, "p_yes": 0.9273632764816284, "predicted": 1}, {"index": 111, "id": "boolq/validation/195", "label": 1, "p_yes": 0.9999426603317261, "predicted": 1}, {"index": 112, "id": "boolq/validation/153", "label": 1, "p_yes": 0.99998939037323, "predicted": 1}, {"index": 113, "id": "boolq/validation/371", "label": 1, "p_yes": 0.9976032376289368, "predicted": 1}, {"index": 114, "id": "boolq/validation/167", "label": 1, "p_yes": 0.005301894620060921, "predicted": 0}, {"index": 115, "id": "boolq/validation/163", "label": 1, "p_yes": 0.9997765421867371, "predicted": 1}, {"index": 116, "id": "boolq/validation/134", "label": 1, "p_yes": 0.9999692440032959, "predicted": 1}, {"index": 117, "id": "boolq/validation/109", "label": 0, "p_yes": 9.028039494296536e-05, "predicted": 0}, {"index": 118, "id": "boolq/validation/245", "label": 1, "p_yes": 0.9998893737792969, "predicted": 1}, {"index": 119, "id": "boolq/validation/10", "label": 1, "p_yes": 0.9999749660491943, "predicted": 1}, {"index": 120, "id": "boolq/validation/230", "label": 1, "p_yes": 0.9986749291419983, "predicted": 1}, {"index": 121, "id": "boolq/validation/224", "label": 0, "p_yes": 0.9399133324623108, "predicted": 1}, {"index": 122, "id": "boolq/validation/25", "label": 0, "p_yes": 0.0002959571429528296, "predicted": 0}, {"index": 123, "id": "boolq/validation/407", "label": 0, "p_yes": 2.2125668692751788e-05, "predicted": 0}, {"index": 124, "id": "boolq/validation/321", "label": 0, "p_yes": 0.0003006163169629872, "predicted": 0}, {"index": 125, "id": "boolq/validation/391", "label": 1, "p_yes": 0.9124361872673035, "predicted": 1}, {"index": 126, "id": "boolq/validation/23", "label": 0, "p_yes": 0.9999285936355591, "predicted": 1}, {"index": 127, "id": "boolq/validation/144", "label": 1, "p_yes": 0.9999397993087769, "predicted": 1}]

adapter_config.json

@@ -0,0 +1,41 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-0.6B",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:827fd0d08c308a89df55343ad7045d9d5b9ac22394e4fa876bdef8c2a001c845
+size 4602248

analyze.py

@@ -0,0 +1,80 @@
+"""Independently verify Kaggle receipts and paired uncertainty."""
+
+import hashlib
+import json
+import random
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parent
+OUT = ROOT / "output" / "typed_decision_run"
+if not OUT.exists():
+    OUT = ROOT
+receipt = json.loads((OUT / "receipt.json").read_text())
+excluded_path = ROOT / "excluded-ids.json"
+if excluded_path.exists():
+    excluded = set(json.loads(excluded_path.read_text()))
+else:
+    excluded = set(json.loads((ROOT.parent / "data" / "test-ids.json").read_text()))
+    excluded.update(json.loads((ROOT.parent / "data" / "confirm-ids.json").read_text()))
+
+
+def records(name):
+    return json.loads((OUT / f"{name}-predictions.json").read_text())
+
+
+def recompute(rows):
+    n = len(rows)
+    accuracy = sum(r["predicted"] == r["label"] for r in rows) / n
+    brier = sum((r["p_yes"] - r["label"]) ** 2 for r in rows) / n
+    return accuracy, brier
+
+
+adapter = OUT / "adapter" / "adapter_model.safetensors"
+if not adapter.exists():
+    adapter = OUT / "adapter_model.safetensors"
+summary = {"kernel": "sankalpsthakur/qwen3-06b-typed-decisions-cloud-pilot/1",
+           "adapter_sha256": hashlib.sha256(adapter.read_bytes()).hexdigest(),
+           "splits": {}}
+for split, size in (("valid", 128), ("confirm", 256), ("ood", 128)):
+    before = records(f"base-{split}")
+    after = records(f"adapted-{split}")
+    assert len(before) == len(after) == size
+    assert [(r["id"], r["label"]) for r in before] == [(r["id"], r["label"]) for r in after]
+    for phase, rows in (("base", before), ("adapted", after)):
+        measured = recompute(rows)
+        expected = receipt["metrics"][f"{phase}_{split}"]
+        assert abs(measured[0] - expected["accuracy"]) < 1e-12
+        assert abs(measured[1] - expected["brier"]) < 1e-12
+    if split == "confirm":
+        assert not set(r["id"] for r in before) & excluded
+    rng = random.Random(20260930 + size)
+    deltas = []
+    for _ in range(10000):
+        sample = [rng.randrange(size) for _ in range(size)]
+        d_acc = sum((after[i]["predicted"] == after[i]["label"]) -
+                    (before[i]["predicted"] == before[i]["label"]) for i in sample) / size
+        d_brier = sum((before[i]["p_yes"] - before[i]["label"]) ** 2 -
+                      (after[i]["p_yes"] - after[i]["label"]) ** 2 for i in sample) / size
+        deltas.append((d_acc, d_brier))
+    intervals = []
+    for col in (0, 1):
+        ordered = sorted(pair[col] for pair in deltas)
+        intervals.append([ordered[249], ordered[9749]])
+    summary["splits"][split] = {
+        "n": size,
+        "base_accuracy": receipt["metrics"][f"base_{split}"]["accuracy"],
+        "adapted_accuracy": receipt["metrics"][f"adapted_{split}"]["accuracy"],
+        "accuracy_gain": recompute(after)[0] - recompute(before)[0],
+        "accuracy_gain_paired_bootstrap_95ci": intervals[0],
+        "base_brier": receipt["metrics"][f"base_{split}"]["brier"],
+        "adapted_brier": receipt["metrics"][f"adapted_{split}"]["brier"],
+        "brier_reduction": recompute(before)[1] - recompute(after)[1],
+        "brier_reduction_paired_bootstrap_95ci": intervals[1],
+        "base_ece_10bin": receipt["metrics"][f"base_{split}"]["ece_10bin"],
+        "adapted_ece_10bin": receipt["metrics"][f"adapted_{split}"]["ece_10bin"],
+        "base_only_correct": sum(a["predicted"] == a["label"] and b["predicted"] != b["label"] for a, b in zip(before, after)),
+        "adapted_only_correct": sum(a["predicted"] != a["label"] and b["predicted"] == b["label"] for a, b in zip(before, after)),
+    }
+(ROOT / "verified_summary.json").write_text(json.dumps(summary, indent=2) + "\n")
+print(json.dumps(summary, indent=2))

base-confirm-calibrated.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "boolq/test/314", "label": 1, "p_yes": 0.8903993881445189, "predicted": 1}, {"index": 1, "id": "boolq/test/2058", "label": 1, "p_yes": 0.21590382882672549, "predicted": 0}, {"index": 2, "id": "boolq/test/1125", "label": 0, "p_yes": 0.5427609877250511, "predicted": 1}, {"index": 3, "id": "boolq/test/611", "label": 1, "p_yes": 0.5353517264898839, "predicted": 1}, {"index": 4, "id": "boolq/test/1646", "label": 1, "p_yes": 0.8595480583157222, "predicted": 1}, {"index": 5, "id": "boolq/test/203", "label": 0, "p_yes": 0.3316435148077665, "predicted": 0}, {"index": 6, "id": "boolq/test/2275", "label": 0, "p_yes": 0.4609414901749551, "predicted": 0}, {"index": 7, "id": "boolq/test/80", "label": 0, "p_yes": 0.34163161168235906, "predicted": 0}, {"index": 8, "id": "boolq/test/90", "label": 1, "p_yes": 0.7522746041084329, "predicted": 1}, {"index": 9, "id": "boolq/test/2253", "label": 1, "p_yes": 0.574001525871985, "predicted": 1}, {"index": 10, "id": "boolq/test/1398", "label": 1, "p_yes": 0.34163161168235906, "predicted": 0}, {"index": 11, "id": "boolq/test/2803", "label": 0, "p_yes": 0.5353517264898839, "predicted": 1}, {"index": 12, "id": "boolq/test/1768", "label": 1, "p_yes": 0.7204657129913706, "predicted": 1}, {"index": 13, "id": "boolq/test/1525", "label": 1, "p_yes": 0.917992977748877, "predicted": 1}, {"index": 14, "id": "boolq/test/754", "label": 0, "p_yes": 0.8292667636858398, "predicted": 1}, {"index": 15, "id": "boolq/test/1284", "label": 0, "p_yes": 0.9097856071201428, "predicted": 1}, {"index": 16, "id": "boolq/test/1567", "label": 1, "p_yes": 0.5812765265329487, "predicted": 1}, {"index": 17, "id": "boolq/test/172", "label": 1, "p_yes": 0.5111805146512981, "predicted": 1}, {"index": 18, "id": "boolq/test/851", "label": 0, "p_yes": 0.4683589989770788, "predicted": 0}, {"index": 19, "id": "boolq/test/604", "label": 0, "p_yes": 0.2706154364412017, "predicted": 0}, {"index": 20, "id": "boolq/test/2912", "label": 1, "p_yes": 0.900177353558998, "predicted": 1}, {"index": 21, "id": "boolq/test/3209", "label": 1, "p_yes": 0.7976550980324861, "predicted": 1}, {"index": 22, "id": "boolq/test/1739", "label": 1, "p_yes": 0.8024253060555195, "predicted": 1}, {"index": 23, "id": "boolq/test/2061", "label": 0, "p_yes": 0.27654141209041316, "predicted": 0}, {"index": 24, "id": "boolq/test/1579", "label": 0, "p_yes": 0.7508827411152007, "predicted": 1}, {"index": 25, "id": "boolq/test/2343", "label": 1, "p_yes": 0.6566896283320727, "predicted": 1}, {"index": 26, "id": "boolq/test/756", "label": 1, "p_yes": 0.8749804592835189, "predicted": 1}, {"index": 27, "id": "boolq/test/144", "label": 0, "p_yes": 0.3568786805328855, "predicted": 0}, {"index": 28, "id": "boolq/test/211", "label": 0, "p_yes": 0.5483057211681001, "predicted": 1}, {"index": 29, "id": "boolq/test/842", "label": 1, "p_yes": 0.4739314991004835, "predicted": 0}, {"index": 30, "id": "boolq/test/2287", "label": 1, "p_yes": 0.7927994089903851, "predicted": 1}, {"index": 31, "id": "boolq/test/913", "label": 0, "p_yes": 0.224868968858357, "predicted": 0}, {"index": 32, "id": "boolq/test/2084", "label": 1, "p_yes": 0.2917016914708527, "predicted": 0}, {"index": 33, "id": "boolq/test/2984", "label": 1, "p_yes": 0.658368388317641, "predicted": 1}, {"index": 34, "id": "boolq/test/909", "label": 1, "p_yes": 0.41509898726201794, "predicted": 0}, {"index": 35, "id": "boolq/test/907", "label": 0, "p_yes": 0.28863073886886254, "predicted": 0}, {"index": 36, "id": "boolq/test/1217", "label": 1, "p_yes": 0.8036046159582351, "predicted": 1}, {"index": 37, "id": "boolq/test/2302", "label": 1, "p_yes": 0.5885163617640581, "predicted": 1}, {"index": 38, "id": "boolq/test/232", "label": 1, "p_yes": 0.8521926354821625, "predicted": 1}, {"index": 39, "id": "boolq/test/1308", "label": 1, "p_yes": 0.8000509229276416, "predicted": 1}, {"index": 40, "id": "boolq/test/2048", "label": 1, "p_yes": 0.8094208468407333, "predicted": 1}, {"index": 41, "id": "boolq/test/1982", "label": 1, "p_yes": 0.77771914529754, "predicted": 1}, {"index": 42, "id": "boolq/test/534", "label": 1, "p_yes": 0.6942447223823844, "predicted": 1}, {"index": 43, "id": "boolq/test/773", "label": 1, "p_yes": 0.8549874632837992, "predicted": 1}, {"index": 44, "id": "boolq/test/2366", "label": 1, "p_yes": 0.21464447284029822, "predicted": 0}, {"index": 45, "id": "boolq/test/632", "label": 1, "p_yes": 0.6465359997604931, "predicted": 1}, {"index": 46, "id": "boolq/test/3165", "label": 1, "p_yes": 0.2547368487810108, "predicted": 0}, {"index": 47, "id": "boolq/test/1249", "label": 1, "p_yes": 0.8631095257942113, "predicted": 1}, {"index": 48, "id": "boolq/test/2473", "label": 0, "p_yes": 0.3724212982697775, "predicted": 0}, {"index": 49, "id": "boolq/test/3049", "label": 0, "p_yes": 0.8117103007480911, "predicted": 1}, {"index": 50, "id": "boolq/test/2500", "label": 1, "p_yes": 0.9047698915233646, "predicted": 1}, {"index": 51, "id": "boolq/test/3130", "label": 0, "p_yes": 0.6205830639711957, "predicted": 1}, {"index": 52, "id": "boolq/test/812", "label": 1, "p_yes": 0.8012407686532451, "predicted": 1}, {"index": 53, "id": "boolq/test/2640", "label": 0, "p_yes": 0.8105682107121489, "predicted": 1}, {"index": 54, "id": "boolq/test/457", "label": 1, "p_yes": 0.6099957248964116, "predicted": 1}, {"index": 55, "id": "boolq/test/1094", "label": 1, "p_yes": 0.8239243337240646, "predicted": 1}, {"index": 56, "id": "boolq/test/1834", "label": 1, "p_yes": 0.8774063336369549, "predicted": 1}, {"index": 57, "id": "boolq/test/733", "label": 1, "p_yes": 0.5556799025163006, "predicted": 1}, {"index": 58, "id": "boolq/test/1503", "label": 0, "p_yes": 0.34331034209280387, "predicted": 0}, {"index": 59, "id": "boolq/test/1766", "label": 1, "p_yes": 0.8657305211191179, "predicted": 1}, {"index": 60, "id": "boolq/test/3252", "label": 1, "p_yes": 0.8228402369376934, "predicted": 1}, {"index": 61, "id": "boolq/test/2379", "label": 0, "p_yes": 0.4516942641562769, "predicted": 0}, {"index": 62, "id": "boolq/test/2693", "label": 1, "p_yes": 0.8355068644121367, "predicted": 1}, {"index": 63, "id": "boolq/test/2493", "label": 1, "p_yes": 0.7964490689755397, "predicted": 1}, {"index": 64, "id": "boolq/test/1240", "label": 0, "p_yes": 0.7853555065132322, "predicted": 1}, {"index": 65, "id": "boolq/test/1159", "label": 1, "p_yes": 0.8716815427468301, "predicted": 1}, {"index": 66, "id": "boolq/test/2064", "label": 1, "p_yes": 0.6617139213824256, "predicted": 1}, {"index": 67, "id": "boolq/test/355", "label": 0, "p_yes": 0.3500641779153575, "predicted": 0}, {"index": 68, "id": "boolq/test/1047", "label": 1, "p_yes": 0.9041252041112631, "predicted": 1}, {"index": 69, "id": "boolq/test/2897", "label": 1, "p_yes": 0.5519957002559022, "predicted": 1}, {"index": 70, "id": "boolq/test/1804", "label": 0, "p_yes": 0.39533773717979404, "predicted": 0}, {"index": 71, "id": "boolq/test/3040", "label": 0, "p_yes": 0.5372056410447151, "predicted": 1}, {"index": 72, "id": "boolq/test/2231", "label": 0, "p_yes": 0.5111805146512981, "predicted": 1}, {"index": 73, "id": "boolq/test/2001", "label": 1, "p_yes": 0.7293844925893153, "predicted": 1}, {"index": 74, "id": "boolq/test/2528", "label": 0, "p_yes": 0.9337947200123479, "predicted": 1}, {"index": 75, "id": "boolq/test/44", "label": 1, "p_yes": 0.8774063336369549, "predicted": 1}, {"index": 76, "id": "boolq/test/1207", "label": 0, "p_yes": 0.8483970141526939, "predicted": 1}, {"index": 77, "id": "boolq/test/729", "label": 1, "p_yes": 0.8874553047387982, "predicted": 1}, {"index": 78, "id": "boolq/test/3111", "label": 0, "p_yes": 0.40967954455911243, "predicted": 0}, {"index": 79, "id": "boolq/test/1196", "label": 1, "p_yes": 0.7840960893626917, "predicted": 1}, {"index": 80, "id": "boolq/test/2801", "label": 1, "p_yes": 0.8867089107439092, "predicted": 1}, {"index": 81, "id": "boolq/test/2768", "label": 1, "p_yes": 0.7751310215658586, "predicted": 1}, {"index": 82, "id": "boolq/test/1341", "label": 0, "p_yes": 0.5867098640061138, "predicted": 1}, {"index": 83, "id": "boolq/test/2979", "label": 1, "p_yes": 0.6974004651302598, "predicted": 1}, {"index": 84, "id": "boolq/test/1550", "label": 0, "p_yes": 0.5611946552365746, "predicted": 1}, {"index": 85, "id": "boolq/test/3208", "label": 1, "p_yes": 0.6732949378666645, "predicted": 1}, {"index": 86, "id": "boolq/test/719", "label": 1, "p_yes": 0.7466758468828579, "predicted": 1}, {"index": 87, "id": "boolq/test/2644", "label": 1, "p_yes": 0.4739314991004835, "predicted": 0}, {"index": 88, "id": "boolq/test/212", "label": 0, "p_yes": 0.6499357933022728, "predicted": 1}, {"index": 89, "id": "boolq/test/1411", "label": 0, "p_yes": 0.45538952767752555, "predicted": 0}, {"index": 90, "id": "boolq/test/2440", "label": 0, "p_yes": 0.48509461924620145, "predicted": 0}, {"index": 91, "id": "boolq/test/2851", "label": 0, "p_yes": 0.44064200088978284, "predicted": 0}, {"index": 92, "id": "boolq/test/1057", "label": 1, "p_yes": 0.45354126540439627, "predicted": 0}, {"index": 93, "id": "boolq/test/267", "label": 1, "p_yes": 0.5993037255308085, "predicted": 1}, {"index": 94, "id": "boolq/test/1674", "label": 0, "p_yes": 0.5372056410447151, "predicted": 1}, {"index": 95, "id": "boolq/test/269", "label": 0, "p_yes": 0.4259985047492018, "predicted": 0}, {"index": 96, "id": "boolq/test/316", "label": 0, "p_yes": 0.45723899769917586, "predicted": 0}, {"index": 97, "id": "boolq/test/1855", "label": 1, "p_yes": 0.6781952925732836, "predicted": 1}, {"index": 98, "id": "boolq/test/2000", "label": 0, "p_yes": 0.703656044191075, "predicted": 1}, {"index": 99, "id": "boolq/test/2793", "label": 1, "p_yes": 0.7903394858621547, "predicted": 1}, {"index": 100, "id": "boolq/test/696", "label": 1, "p_yes": 0.9174300774298159, "predicted": 1}, {"index": 101, "id": "boolq/test/905", "label": 1, "p_yes": 0.8365287938177712, "predicted": 1}, {"index": 102, "id": "boolq/test/2374", "label": 1, "p_yes": 0.8749804592835189, "predicted": 1}, {"index": 103, "id": "boolq/test/264", "label": 1, "p_yes": 0.6732949378666645, "predicted": 1}, {"index": 104, "id": "boolq/test/131", "label": 1, "p_yes": 0.8094208468407333, "predicted": 1}, {"index": 105, "id": "boolq/test/2917", "label": 1, "p_yes": 0.5446105015378004, "predicted": 1}, {"index": 106, "id": "boolq/test/2060", "label": 0, "p_yes": 0.5372056410447151, "predicted": 1}, {"index": 107, "id": "boolq/test/685", "label": 1, "p_yes": 0.39890738104340606, "predicted": 0}, {"index": 108, "id": "boolq/test/2063", "label": 0, "p_yes": 0.7618711266478352, "predicted": 1}, {"index": 109, "id": "boolq/test/3063", "label": 1, "p_yes": 0.7067556257162952, "predicted": 1}, {"index": 110, "id": "boolq/test/953", "label": 0, "p_yes": 0.717453090443778, "predicted": 1}, {"index": 111, "id": "boolq/test/85", "label": 1, "p_yes": 0.3882322018204395, "predicted": 0}, {"index": 112, "id": "boolq/test/2513", "label": 1, "p_yes": 0.8657305211191179, "predicted": 1}, {"index": 113, "id": "boolq/test/408", "label": 1, "p_yes": 0.8782060513143549, "predicted": 1}, {"index": 114, "id": "boolq/test/2739", "label": 0, "p_yes": 0.4443201123120919, "predicted": 0}, {"index": 115, "id": "boolq/test/369", "label": 0, "p_yes": 0.382933869486939, "predicted": 0}, {"index": 116, "id": "boolq/test/2916", "label": 1, "p_yes": 0.9432401205091042, "predicted": 1}, {"index": 117, "id": "boolq/test/2674", "label": 1, "p_yes": 0.703656044191075, "predicted": 1}, {"index": 118, "id": "boolq/test/2639", "label": 1, "p_yes": 0.8445216202109631, "predicted": 1}, {"index": 119, "id": "boolq/test/574", "label": 1, "p_yes": 0.5149054092778641, "predicted": 1}, {"index": 120, "id": "boolq/test/2978", "label": 1, "p_yes": 0.843540271938402, "predicted": 1}, {"index": 121, "id": "boolq/test/1546", "label": 0, "p_yes": 0.40967954455911243, "predicted": 0}, {"index": 122, "id": "boolq/test/201", "label": 1, "p_yes": 0.6974004651302598, "predicted": 1}, {"index": 123, "id": "boolq/test/3008", "label": 1, "p_yes": 0.3500641779153575, "predicted": 0}, {"index": 124, "id": "boolq/test/3212", "label": 0, "p_yes": 0.4795105082409079, "predicted": 0}, {"index": 125, "id": "boolq/test/2946", "label": 0, "p_yes": 0.3299931684151984, "predicted": 0}, {"index": 126, "id": "boolq/test/516", "label": 0, "p_yes": 0.23543448748954054, "predicted": 0}, {"index": 127, "id": "boolq/test/1182", "label": 1, "p_yes": 0.5703517479200776, "predicted": 1}, {"index": 128, "id": "boolq/test/1257", "label": 0, "p_yes": 0.6781952925732836, "predicted": 1}, {"index": 129, "id": "boolq/test/3231", "label": 1, "p_yes": 0.21590382882672549, "predicted": 0}, {"index": 130, "id": "boolq/test/3027", "label": 1, "p_yes": 0.7204657129913706, "predicted": 1}, {"index": 131, "id": "boolq/test/2605", "label": 1, "p_yes": 0.6153030337332264, "predicted": 1}, {"index": 132, "id": "boolq/test/3062", "label": 0, "p_yes": 0.6683565059118912, "predicted": 1}, {"index": 133, "id": "boolq/test/42", "label": 1, "p_yes": 0.743845276193416, "predicted": 1}, {"index": 134, "id": "boolq/test/2676", "label": 0, "p_yes": 0.48137137618190634, "predicted": 0}, {"index": 135, "id": "boolq/test/1624", "label": 1, "p_yes": 0.3089292977533087, "predicted": 0}, {"index": 136, "id": "boolq/test/2661", "label": 0, "p_yes": 0.4609414901749551, "predicted": 0}, {"index": 137, "id": "boolq/test/2240", "label": 0, "p_yes": 0.3089292977533087, "predicted": 0}, {"index": 138, "id": "boolq/test/1575", "label": 1, "p_yes": 0.6064429111497, "predicted": 1}, {"index": 139, "id": "boolq/test/300", "label": 0, "p_yes": 0.3316435148077665, "predicted": 0}, {"index": 140, "id": "boolq/test/2337", "label": 1, "p_yes": 0.6910707368119688, "predicted": 1}, {"index": 141, "id": "boolq/test/3224", "label": 0, "p_yes": 0.5186286381043013, "predicted": 1}, {"index": 142, "id": "boolq/test/739", "label": 1, "p_yes": 0.9452036082911889, "predicted": 1}, {"index": 143, "id": "boolq/test/1482", "label": 0, "p_yes": 0.6465359997604931, "predicted": 1}, {"index": 144, "id": "boolq/test/18", "label": 1, "p_yes": 0.45723899769917586, "predicted": 0}, {"index": 145, "id": "boolq/test/395", "label": 0, "p_yes": 0.6600431288492591, "predicted": 1}, {"index": 146, "id": "boolq/test/838", "label": 0, "p_yes": 0.4461614560274288, "predicted": 0}, {"index": 147, "id": "boolq/test/111", "label": 1, "p_yes": 0.5593580140234986, "predicted": 1}, {"index": 148, "id": "boolq/test/1465", "label": 1, "p_yes": 0.2422101333712158, "predicted": 0}, {"index": 149, "id": "boolq/test/1267", "label": 1, "p_yes": 0.45908966817934294, "predicted": 0}, {"index": 150, "id": "boolq/test/1863", "label": 1, "p_yes": 0.6617139213824256, "predicted": 1}, {"index": 151, "id": "boolq/test/2382", "label": 1, "p_yes": 0.6958248830053827, "predicted": 1}, {"index": 152, "id": "boolq/test/1340", "label": 1, "p_yes": 0.9223730054496647, "predicted": 1}, {"index": 153, "id": "boolq/test/831", "label": 0, "p_yes": 0.7494857432668507, "predicted": 1}, {"index": 154, "id": "boolq/test/1216", "label": 1, "p_yes": 0.32017984964358925, "predicted": 0}, {"index": 155, "id": "boolq/test/124", "label": 1, "p_yes": 0.6862757300880689, "predicted": 1}, {"index": 156, "id": "boolq/test/3147", "label": 1, "p_yes": 0.6046622463449963, "predicted": 1}, {"index": 157, "id": "boolq/test/27", "label": 0, "p_yes": 0.77771914529754, "predicted": 1}, {"index": 158, "id": "boolq/test/1060", "label": 0, "p_yes": 0.6396920112712661, "predicted": 1}, {"index": 159, "id": "boolq/test/3162", "label": 1, "p_yes": 0.8228402369376934, "predicted": 1}, {"index": 160, "id": "boolq/test/2730", "label": 1, "p_yes": 0.9122035567054703, "predicted": 1}, {"index": 161, "id": "boolq/test/386", "label": 1, "p_yes": 0.9369485254627762, "predicted": 1}, {"index": 162, "id": "boolq/test/2848", "label": 1, "p_yes": 0.700537619105263, "predicted": 1}, {"index": 163, "id": "boolq/test/1306", "label": 1, "p_yes": 0.854060769886703, "predicted": 1}, {"index": 164, "id": "boolq/test/484", "label": 1, "p_yes": 0.8903993881445189, "predicted": 1}, {"index": 165, "id": "boolq/test/1636", "label": 1, "p_yes": 0.8195564727427475, "predicted": 1}, {"index": 166, "id": "boolq/test/1326", "label": 0, "p_yes": 0.6716529779410676, "predicted": 1}, {"index": 167, "id": "boolq/test/771", "label": 1, "p_yes": 0.8184514673763653, "predicted": 1}, {"index": 168, "id": "boolq/test/638", "label": 0, "p_yes": 0.2618777241967674, "predicted": 0}, {"index": 169, "id": "boolq/test/1298", "label": 1, "p_yes": 0.7952378971097691, "predicted": 1}, {"index": 170, "id": "boolq/test/3018", "label": 1, "p_yes": 0.9629523119144168, "predicted": 1}, {"index": 171, "id": "boolq/test/420", "label": 0, "p_yes": 0.5993037255308085, "predicted": 1}, {"index": 172, "id": "boolq/test/1304", "label": 1, "p_yes": 0.5279267818914944, "predicted": 1}, {"index": 173, "id": "boolq/test/1477", "label": 0, "p_yes": 0.5993037255308085, "predicted": 1}, {"index": 174, "id": "boolq/test/1703", "label": 1, "p_yes": 0.6878785430988262, "predicted": 1}, {"index": 175, "id": "boolq/test/2088", "label": 0, "p_yes": 0.3025995705557748, "predicted": 0}, {"index": 176, "id": "boolq/test/3100", "label": 1, "p_yes": 0.6862757300880689, "predicted": 1}, {"index": 177, "id": "boolq/test/2484", "label": 1, "p_yes": 0.49254565312691795, "predicted": 0}, {"index": 178, "id": "boolq/test/3066", "label": 1, "p_yes": 0.4888195138351886, "predicted": 0}, {"index": 179, "id": "boolq/test/145", "label": 1, "p_yes": 0.5611946552365746, "predicted": 1}, {"index": 180, "id": "boolq/test/2204", "label": 0, "p_yes": 0.4333056934004816, "predicted": 0}, {"index": 181, "id": "boolq/test/3168", "label": 1, "p_yes": 0.9202108286081363, "predicted": 1}, {"index": 182, "id": "boolq/test/1020", "label": 0, "p_yes": 0.4024877777091868, "predicted": 0}, {"index": 183, "id": "boolq/test/956", "label": 1, "p_yes": 0.6716529779410676, "predicted": 1}, {"index": 184, "id": "boolq/test/1619", "label": 1, "p_yes": 0.4888195138351886, "predicted": 0}, {"index": 185, "id": "boolq/test/2154", "label": 1, "p_yes": 0.8239243337240646, "predicted": 1}, {"index": 186, "id": "boolq/test/2262", "label": 0, "p_yes": 0.36202800441306293, "predicted": 0}, {"index": 187, "id": "boolq/test/74", "label": 1, "p_yes": 0.9471026728119404, "predicted": 1}, {"index": 188, "id": "boolq/test/2097", "label": 1, "p_yes": 0.7550427871126034, "predicted": 1}, {"index": 189, "id": "boolq/test/1963", "label": 1, "p_yes": 0.4169101140135326, "predicted": 0}, {"index": 190, "id": "boolq/test/1658", "label": 1, "p_yes": 0.8648616808319319, "predicted": 1}, {"index": 191, "id": "boolq/test/1517", "label": 0, "p_yes": 0.27654141209041316, "predicted": 0}, {"index": 192, "id": "boolq/test/332", "label": 1, "p_yes": 0.5427609877250511, "predicted": 1}, {"index": 193, "id": "boolq/test/345", "label": 1, "p_yes": 0.3776632056299364, "predicted": 0}, {"index": 194, "id": "boolq/test/3245", "label": 1, "p_yes": 0.9428399748246499, "predicted": 1}, {"index": 195, "id": "boolq/test/1480", "label": 1, "p_yes": 0.8981495704510615, "predicted": 1}, {"index": 196, "id": "boolq/test/711", "label": 1, "p_yes": 0.8292667636858398, "predicted": 1}, {"index": 197, "id": "boolq/test/2446", "label": 0, "p_yes": 0.6926600014771807, "predicted": 1}, {"index": 198, "id": "boolq/test/893", "label": 1, "p_yes": 0.8925632047200548, "predicted": 1}, {"index": 199, "id": "boolq/test/2249", "label": 1, "p_yes": 0.9369485254627762, "predicted": 1}, {"index": 200, "id": "boolq/test/3077", "label": 1, "p_yes": 0.8874553047387982, "predicted": 1}, {"index": 201, "id": "boolq/test/2891", "label": 0, "p_yes": 0.5794609426478693, "predicted": 1}, {"index": 202, "id": "boolq/test/896", "label": 0, "p_yes": 0.6499357933022728, "predicted": 1}, {"index": 203, "id": "boolq/test/1757", "label": 1, "p_yes": 0.786609522964005, "predicted": 1}, {"index": 204, "id": "boolq/test/2472", "label": 1, "p_yes": 0.33495662466430237, "predicted": 0}, {"index": 205, "id": "boolq/test/1952", "label": 1, "p_yes": 0.5, "predicted": 1}, {"index": 206, "id": "boolq/test/625", "label": 1, "p_yes": 0.7685673595312373, "predicted": 1}, {"index": 207, "id": "boolq/test/410", "label": 1, "p_yes": 0.7128975391208608, "predicted": 1}, {"index": 208, "id": "boolq/test/800", "label": 0, "p_yes": 0.38469696626677363, "predicted": 0}, {"index": 209, "id": "boolq/test/1251", "label": 1, "p_yes": 0.8117103007480911, "predicted": 1}, {"index": 210, "id": "boolq/test/65", "label": 1, "p_yes": 0.6153030337332264, "predicted": 1}, {"index": 211, "id": "boolq/test/1056", "label": 1, "p_yes": 0.6910707368119688, "predicted": 1}, {"index": 212, "id": "boolq/test/2627", "label": 0, "p_yes": 0.4516942641562769, "predicted": 0}, {"index": 213, "id": "boolq/test/3255", "label": 0, "p_yes": 0.5018637270799985, "predicted": 1}, {"index": 214, "id": "boolq/test/220", "label": 0, "p_yes": 0.4888195138351886, "predicted": 0}, {"index": 215, "id": "boolq/test/2731", "label": 1, "p_yes": 0.776427802674087, "predicted": 1}, {"index": 216, "id": "boolq/test/1713", "label": 1, "p_yes": 0.8117103007480911, "predicted": 1}, {"index": 217, "id": "boolq/test/2430", "label": 0, "p_yes": 0.5167672489089257, "predicted": 1}, {"index": 218, "id": "boolq/test/2185", "label": 1, "p_yes": 0.7508827411152007, "predicted": 1}, {"index": 219, "id": "boolq/test/2039", "label": 1, "p_yes": 0.4609414901749551, "predicted": 0}, {"index": 220, "id": "boolq/test/1591", "label": 1, "p_yes": 0.7337762670301476, "predicted": 1}, {"index": 221, "id": "boolq/test/3015", "label": 0, "p_yes": 0.8000509229276416, "predicted": 1}, {"index": 222, "id": "boolq/test/2425", "label": 0, "p_yes": 0.6153030337332264, "predicted": 1}, {"index": 223, "id": "boolq/test/2131", "label": 0, "p_yes": 0.42417661891798564, "predicted": 0}, {"index": 224, "id": "boolq/test/1184", "label": 1, "p_yes": 0.8821371312886753, "predicted": 1}, {"index": 225, "id": "boolq/test/2351", "label": 1, "p_yes": 0.7234585947645077, "predicted": 1}, {"index": 226, "id": "boolq/test/925", "label": 0, "p_yes": 0.7249475789416616, "predicted": 1}, {"index": 227, "id": "boolq/test/570", "label": 1, "p_yes": 0.6862757300880689, "predicted": 1}, {"index": 228, "id": "boolq/test/3260", "label": 0, "p_yes": 0.7725215534283447, "predicted": 1}, {"index": 229, "id": "boolq/test/2576", "label": 0, "p_yes": 0.6942447223823844, "predicted": 1}, {"index": 230, "id": "boolq/test/2825", "label": 1, "p_yes": 0.8195564727427475, "predicted": 1}, {"index": 231, "id": "boolq/test/882", "label": 1, "p_yes": 0.9034774038618173, "predicted": 1}, {"index": 232, "id": "boolq/test/1423", "label": 0, "p_yes": 0.43513716352393356, "predicted": 0}, {"index": 233, "id": "boolq/test/96", "label": 1, "p_yes": 0.8036046159582351, "predicted": 1}, {"index": 234, "id": "boolq/test/2646", "label": 1, "p_yes": 0.7964490689755397, "predicted": 1}, {"index": 235, "id": "boolq/test/375", "label": 1, "p_yes": 0.8708454649056171, "predicted": 1}, {"index": 236, "id": "boolq/test/242", "label": 0, "p_yes": 0.8844425045505415, "predicted": 1}, {"index": 237, "id": "boolq/test/1406", "label": 1, "p_yes": 0.5334968232870905, "predicted": 1}, {"index": 238, "id": "boolq/test/652", "label": 1, "p_yes": 0.9133904190597687, "predicted": 1}, {"index": 239, "id": "boolq/test/1136", "label": 1, "p_yes": 0.6310568033953767, "predicted": 1}, {"index": 240, "id": "boolq/test/839", "label": 0, "p_yes": 0.6258346606617384, "predicted": 1}, {"index": 241, "id": "boolq/test/1908", "label": 1, "p_yes": 0.4906824457713713, "predicted": 0}, {"index": 242, "id": "boolq/test/1992", "label": 0, "p_yes": 0.3585915161165846, "predicted": 0}, {"index": 243, "id": "boolq/test/1191", "label": 1, "p_yes": 0.8725131347882537, "predicted": 1}, {"index": 244, "id": "boolq/test/2045", "label": 0, "p_yes": 0.5921221543694103, "predicted": 1}, {"index": 245, "id": "boolq/test/286", "label": 1, "p_yes": 0.714420949178769, "predicted": 1}, {"index": 246, "id": "boolq/test/3095", "label": 0, "p_yes": 0.6781952925732836, "predicted": 1}, {"index": 247, "id": "boolq/test/2285", "label": 1, "p_yes": 0.854060769886703, "predicted": 1}, {"index": 248, "id": "boolq/test/572", "label": 1, "p_yes": 0.715939426828434, "predicted": 1}, {"index": 249, "id": "boolq/test/1712", "label": 0, "p_yes": 0.20597865383001515, "predicted": 0}, {"index": 250, "id": "boolq/test/2860", "label": 1, "p_yes": 0.917992977748877, "predicted": 1}, {"index": 251, "id": "boolq/test/1814", "label": 1, "p_yes": 0.8239243337240646, "predicted": 1}, {"index": 252, "id": "boolq/test/2580", "label": 0, "p_yes": 0.715939426828434, "predicted": 1}, {"index": 253, "id": "boolq/test/1297", "label": 1, "p_yes": 0.8932760114947075, "predicted": 1}, {"index": 254, "id": "boolq/test/384", "label": 1, "p_yes": 0.9452036082911889, "predicted": 1}, {"index": 255, "id": "boolq/test/1073", "label": 1, "p_yes": 0.7790052394886462, "predicted": 1}]

base-confirm-predictions.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "boolq/test/314", "label": 1, "p_yes": 0.9877586960792542, "predicted": 1}, {"index": 1, "id": "boolq/test/2058", "label": 1, "p_yes": 0.0627892017364502, "predicted": 0}, {"index": 2, "id": "boolq/test/1125", "label": 0, "p_yes": 0.5888891220092773, "predicted": 1}, {"index": 3, "id": "boolq/test/611", "label": 1, "p_yes": 0.5736784338951111, "predicted": 1}, {"index": 4, "id": "boolq/test/1646", "label": 1, "p_yes": 0.9780517220497131, "predicted": 1}, {"index": 5, "id": "boolq/test/203", "label": 0, "p_yes": 0.18713268637657166, "predicted": 0}, {"index": 6, "id": "boolq/test/2275", "label": 0, "p_yes": 0.41869691014289856, "predicted": 0}, {"index": 7, "id": "boolq/test/80", "label": 0, "p_yes": 0.2018132209777832, "predicted": 0}, {"index": 8, "id": "boolq/test/90", "label": 1, "p_yes": 0.9111796617507935, "predicted": 1}, {"index": 9, "id": "boolq/test/2253", "label": 1, "p_yes": 0.6513549089431763, "predicted": 1}, {"index": 10, "id": "boolq/test/1398", "label": 1, "p_yes": 0.2018132209777832, "predicted": 0}, {"index": 11, "id": "boolq/test/2803", "label": 0, "p_yes": 0.5736784338951111, "predicted": 1}, {"index": 12, "id": "boolq/test/1768", "label": 1, "p_yes": 0.8791467547416687, "predicted": 1}, {"index": 13, "id": "boolq/test/1525", "label": 1, "p_yes": 0.9937100410461426, "predicted": 1}, {"index": 14, "id": "boolq/test/754", "label": 0, "p_yes": 0.9648551344871521, "predicted": 1}, {"index": 15, "id": "boolq/test/1284", "label": 0, "p_yes": 0.9921841025352478, "predicted": 1}, {"index": 16, "id": "boolq/test/1567", "label": 1, "p_yes": 0.665410578250885, "predicted": 1}, {"index": 17, "id": "boolq/test/172", "label": 1, "p_yes": 0.5234203934669495, "predicted": 1}, {"index": 18, "id": "boolq/test/851", "label": 0, "p_yes": 0.4339814782142639, "predicted": 0}, {"index": 19, "id": "boolq/test/604", "label": 0, "p_yes": 0.11124120652675629, "predicted": 0}, {"index": 20, "id": "boolq/test/2912", "label": 1, "p_yes": 0.9901401400566101, "predicted": 1}, {"index": 21, "id": "boolq/test/3209", "label": 1, "p_yes": 0.9465966820716858, "predicted": 1}, {"index": 22, "id": "boolq/test/1739", "label": 1, "p_yes": 0.949669361114502, "predicted": 1}, {"index": 23, "id": "boolq/test/2061", "label": 0, "p_yes": 0.11757213622331619, "predicted": 0}, {"index": 24, "id": "boolq/test/1579", "label": 0, "p_yes": 0.9099069833755493, "predicted": 1}, {"index": 25, "id": "boolq/test/2343", "label": 1, "p_yes": 0.7956580519676208, "predicted": 1}, {"index": 26, "id": "boolq/test/756", "label": 1, "p_yes": 0.9833430051803589, "predicted": 1}, {"index": 27, "id": "boolq/test/144", "label": 0, "p_yes": 0.2254166156053543, "predicted": 0}, {"index": 28, "id": "boolq/test/211", "label": 0, "p_yes": 0.6001883149147034, "predicted": 1}, {"index": 29, "id": "boolq/test/842", "label": 1, "p_yes": 0.4455295503139496, "predicted": 0}, {"index": 30, "id": "boolq/test/2287", "label": 1, "p_yes": 0.9433475732803345, "predicted": 1}, {"index": 31, "id": "boolq/test/913", "label": 0, "p_yes": 0.06954174488782883, "predicted": 0}, {"index": 32, "id": "boolq/test/2084", "label": 1, "p_yes": 0.13477592170238495, "predicted": 0}, {"index": 33, "id": "boolq/test/2984", "label": 1, "p_yes": 0.7981867790222168, "predicted": 1}, {"index": 34, "id": "boolq/test/909", "label": 1, "p_yes": 0.32766830921173096, "predicted": 0}, {"index": 35, "id": "boolq/test/907", "label": 0, "p_yes": 0.13117322325706482, "predicted": 0}, {"index": 36, "id": "boolq/test/1217", "label": 1, "p_yes": 0.9504110217094421, "predicted": 1}, {"index": 37, "id": "boolq/test/2302", "label": 1, "p_yes": 0.6791787147521973, "predicted": 1}, {"index": 38, "id": "boolq/test/232", "label": 1, "p_yes": 0.9752018451690674, "predicted": 1}, {"index": 39, "id": "boolq/test/1308", "label": 1, "p_yes": 0.9481545686721802, "predicted": 1}, {"index": 40, "id": "boolq/test/2048", "label": 1, "p_yes": 0.9539660811424255, "predicted": 1}, {"index": 41, "id": "boolq/test/1982", "label": 1, "p_yes": 0.9324532747268677, "predicted": 1}, {"index": 42, "id": "boolq/test/534", "label": 1, "p_yes": 0.8479677438735962, "predicted": 1}, {"index": 43, "id": "boolq/test/773", "label": 1, "p_yes": 0.9763104915618896, "predicted": 1}, {"index": 44, "id": "boolq/test/2366", "label": 1, "p_yes": 0.06187598407268524, "predicted": 0}, {"index": 45, "id": "boolq/test/632", "label": 1, "p_yes": 0.7799928784370422, "predicted": 1}, {"index": 46, "id": "boolq/test/3165", "label": 1, "p_yes": 0.09534946084022522, "predicted": 0}, {"index": 47, "id": "boolq/test/1249", "label": 1, "p_yes": 0.9793540239334106, "predicted": 1}, {"index": 48, "id": "boolq/test/2473", "label": 0, "p_yes": 0.25091278553009033, "predicted": 0}, {"index": 49, "id": "boolq/test/3049", "label": 0, "p_yes": 0.9553191661834717, "predicted": 1}, {"index": 50, "id": "boolq/test/2500", "label": 1, "p_yes": 0.9911527037620544, "predicted": 1}, {"index": 51, "id": "boolq/test/3130", "label": 0, "p_yes": 0.7371581792831421, "predicted": 1}, {"index": 52, "id": "boolq/test/812", "label": 1, "p_yes": 0.9489172697067261, "predicted": 1}, {"index": 53, "id": "boolq/test/2640", "label": 0, "p_yes": 0.954647421836853, "predicted": 1}, {"index": 54, "id": "boolq/test/457", "label": 1, "p_yes": 0.7185943722724915, "predicted": 1}, {"index": 55, "id": "boolq/test/1094", "label": 1, "p_yes": 0.9621075391769409, "predicted": 1}, {"index": 56, "id": "boolq/test/1834", "label": 1, "p_yes": 0.9840936064720154, "predicted": 1}, {"index": 57, "id": "boolq/test/733", "label": 1, "p_yes": 0.6150878667831421, "predicted": 1}, {"index": 58, "id": "boolq/test/1503", "label": 0, "p_yes": 0.20434190332889557, "predicted": 0}, {"index": 59, "id": "boolq/test/1766", "label": 1, "p_yes": 0.9802808165550232, "predicted": 1}, {"index": 60, "id": "boolq/test/3252", "label": 1, "p_yes": 0.961533784866333, "predicted": 1}, {"index": 61, "id": "boolq/test/2379", "label": 0, "p_yes": 0.39981165528297424, "predicted": 0}, {"index": 62, "id": "boolq/test/2693", "label": 1, "p_yes": 0.9678993225097656, "predicted": 1}, {"index": 63, "id": "boolq/test/2493", "label": 1, "p_yes": 0.9458011984825134, "predicted": 1}, {"index": 64, "id": "boolq/test/1240", "label": 0, "p_yes": 0.9381240010261536, "predicted": 1}, {"index": 65, "id": "boolq/test/1159", "label": 1, "p_yes": 0.9822876453399658, "predicted": 1}, {"index": 66, "id": "boolq/test/2064", "label": 1, "p_yes": 0.8031737804412842, "predicted": 1}, {"index": 67, "id": "boolq/test/355", "label": 0, "p_yes": 0.2146914154291153, "predicted": 0}, {"index": 68, "id": "boolq/test/1047", "label": 1, "p_yes": 0.9910145401954651, "predicted": 1}, {"index": 69, "id": "boolq/test/2897", "label": 1, "p_yes": 0.6076632142066956, "predicted": 1}, {"index": 70, "id": "boolq/test/1804", "label": 0, "p_yes": 0.29098084568977356, "predicted": 0}, {"index": 71, "id": "boolq/test/3040", "label": 0, "p_yes": 0.5774953961372375, "predicted": 1}, {"index": 72, "id": "boolq/test/2231", "label": 0, "p_yes": 0.5234203934669495, "predicted": 1}, {"index": 73, "id": "boolq/test/2001", "label": 1, "p_yes": 0.8887587189674377, "predicted": 1}, {"index": 74, "id": "boolq/test/2528", "label": 0, "p_yes": 0.9961155652999878, "predicted": 1}, {"index": 75, "id": "boolq/test/44", "label": 1, "p_yes": 0.9840936064720154, "predicted": 1}, {"index": 76, "id": "boolq/test/1207", "label": 0, "p_yes": 0.9736446738243103, "predicted": 1}, {"index": 77, "id": "boolq/test/729", "label": 1, "p_yes": 0.9869794845581055, "predicted": 1}, {"index": 78, "id": "boolq/test/3111", "label": 0, "p_yes": 0.31742629408836365, "predicted": 0}, {"index": 79, "id": "boolq/test/1196", "label": 1, "p_yes": 0.937210738658905, "predicted": 1}, {"index": 80, "id": "boolq/test/2801", "label": 1, "p_yes": 0.9867772459983826, "predicted": 1}, {"index": 81, "id": "boolq/test/2768", "label": 1, "p_yes": 0.9304582476615906, "predicted": 1}, {"index": 82, "id": "boolq/test/1341", "label": 0, "p_yes": 0.6757646203041077, "predicted": 1}, {"index": 83, "id": "boolq/test/2979", "label": 1, "p_yes": 0.851952850818634, "predicted": 1}, {"index": 84, "id": "boolq/test/1550", "label": 0, "p_yes": 0.6261242032051086, "predicted": 1}, {"index": 85, "id": "boolq/test/3208", "label": 1, "p_yes": 0.8198933005332947, "predicted": 1}, {"index": 86, "id": "boolq/test/719", "label": 1, "p_yes": 0.9059898257255554, "predicted": 1}, {"index": 87, "id": "boolq/test/2644", "label": 1, "p_yes": 0.4455295503139496, "predicted": 0}, {"index": 88, "id": "boolq/test/212", "label": 0, "p_yes": 0.7853085398674011, "predicted": 1}, {"index": 89, "id": "boolq/test/1411", "label": 0, "p_yes": 0.4073334336280823, "predicted": 0}, {"index": 90, "id": "boolq/test/2440", "label": 0, "p_yes": 0.4687906503677368, "predicted": 0}, {"index": 91, "id": "boolq/test/2851", "label": 0, "p_yes": 0.3775406777858734, "predicted": 0}, {"index": 92, "id": "boolq/test/1057", "label": 1, "p_yes": 0.4035668969154358, "predicted": 0}, {"index": 93, "id": "boolq/test/267", "label": 1, "p_yes": 0.6992543935775757, "predicted": 1}, {"index": 94, "id": "boolq/test/1674", "label": 0, "p_yes": 0.5774953961372375, "predicted": 1}, {"index": 95, "id": "boolq/test/269", "label": 0, "p_yes": 0.3486451506614685, "predicted": 0}, {"index": 96, "id": "boolq/test/316", "label": 0, "p_yes": 0.41111084818840027, "predicted": 0}, {"index": 97, "id": "boolq/test/1855", "label": 1, "p_yes": 0.8267117738723755, "predicted": 1}, {"index": 98, "id": "boolq/test/2000", "label": 0, "p_yes": 0.859663724899292, "predicted": 1}, {"index": 99, "id": "boolq/test/2793", "label": 1, "p_yes": 0.9416541457176208, "predicted": 1}, {"index": 100, "id": "boolq/test/696", "label": 1, "p_yes": 0.9936116337776184, "predicted": 1}, {"index": 101, "id": "boolq/test/905", "label": 1, "p_yes": 0.9683812260627747, "predicted": 1}, {"index": 102, "id": "boolq/test/2374", "label": 1, "p_yes": 0.9833430051803589, "predicted": 1}, {"index": 103, "id": "boolq/test/264", "label": 1, "p_yes": 0.8198933005332947, "predicted": 1}, {"index": 104, "id": "boolq/test/131", "label": 1, "p_yes": 0.9539660811424255, "predicted": 1}, {"index": 105, "id": "boolq/test/2917", "label": 1, "p_yes": 0.5926666259765625, "predicted": 1}, {"index": 106, "id": "boolq/test/2060", "label": 0, "p_yes": 0.5774953961372375, "predicted": 1}, {"index": 107, "id": "boolq/test/685", "label": 1, "p_yes": 0.29746994376182556, "predicted": 0}, {"index": 108, "id": "boolq/test/2063", "label": 0, "p_yes": 0.9196425676345825, "predicted": 1}, {"index": 109, "id": "boolq/test/3063", "label": 1, "p_yes": 0.8633915781974792, "predicted": 1}, {"index": 110, "id": "boolq/test/953", "label": 0, "p_yes": 0.8757869601249695, "predicted": 1}, {"index": 111, "id": "boolq/test/85", "label": 1, "p_yes": 0.2782568037509918, "predicted": 0}, {"index": 112, "id": "boolq/test/2513", "label": 1, "p_yes": 0.9802808165550232, "predicted": 1}, {"index": 113, "id": "boolq/test/408", "label": 1, "p_yes": 0.9843363761901855, "predicted": 1}, {"index": 114, "id": "boolq/test/2739", "label": 0, "p_yes": 0.3849121630191803, "predicted": 0}, {"index": 115, "id": "boolq/test/369", "label": 0, "p_yes": 0.2689414322376251, "predicted": 0}, {"index": 116, "id": "boolq/test/2916", "label": 1, "p_yes": 0.9972423315048218, "predicted": 1}, {"index": 117, "id": "boolq/test/2674", "label": 1, "p_yes": 0.859663724899292, "predicted": 1}, {"index": 118, "id": "boolq/test/2639", "label": 1, "p_yes": 0.9719924926757812, "predicted": 1}, {"index": 119, "id": "boolq/test/574", "label": 1, "p_yes": 0.531209409236908, "predicted": 1}, {"index": 120, "id": "boolq/test/2978", "label": 1, "p_yes": 0.971563994884491, "predicted": 1}, {"index": 121, "id": "boolq/test/1546", "label": 0, "p_yes": 0.31742629408836365, "predicted": 0}, {"index": 122, "id": "boolq/test/201", "label": 1, "p_yes": 0.851952850818634, "predicted": 1}, {"index": 123, "id": "boolq/test/3008", "label": 1, "p_yes": 0.2146914154291153, "predicted": 0}, {"index": 124, "id": "boolq/test/3212", "label": 0, "p_yes": 0.45713675022125244, "predicted": 0}, {"index": 125, "id": "boolq/test/2946", "label": 0, "p_yes": 0.18476751446723938, "predicted": 0}, {"index": 126, "id": "boolq/test/516", "label": 0, "p_yes": 0.07807815819978714, "predicted": 0}, {"index": 127, "id": "boolq/test/1182", "label": 1, "p_yes": 0.6442251205444336, "predicted": 1}, {"index": 128, "id": "boolq/test/1257", "label": 0, "p_yes": 0.8267117738723755, "predicted": 1}, {"index": 129, "id": "boolq/test/3231", "label": 1, "p_yes": 0.0627892017364502, "predicted": 0}, {"index": 130, "id": "boolq/test/3027", "label": 1, "p_yes": 0.8791467547416687, "predicted": 1}, {"index": 131, "id": "boolq/test/2605", "label": 1, "p_yes": 0.7279754281044006, "predicted": 1}, {"index": 132, "id": "boolq/test/3062", "label": 0, "p_yes": 0.8128673434257507, "predicted": 1}, {"index": 133, "id": "boolq/test/42", "label": 1, "p_yes": 0.9032942056655884, "predicted": 1}, {"index": 134, "id": "boolq/test/2676", "label": 0, "p_yes": 0.46101680397987366, "predicted": 0}, {"index": 135, "id": "boolq/test/1624", "label": 1, "p_yes": 0.1561049073934555, "predicted": 0}, {"index": 136, "id": "boolq/test/2661", "label": 0, "p_yes": 0.41869691014289856, "predicted": 0}, {"index": 137, "id": "boolq/test/2240", "label": 0, "p_yes": 0.1561049073934555, "predicted": 0}, {"index": 138, "id": "boolq/test/1575", "label": 1, "p_yes": 0.7122321724891663, "predicted": 1}, {"index": 139, "id": "boolq/test/300", "label": 0, "p_yes": 0.18713268637657166, "predicted": 0}, {"index": 140, "id": "boolq/test/2337", "label": 1, "p_yes": 0.8438951373100281, "predicted": 1}, {"index": 141, "id": "boolq/test/3224", "label": 0, "p_yes": 0.5389832258224487, "predicted": 1}, {"index": 142, "id": "boolq/test/739", "label": 1, "p_yes": 0.9974491000175476, "predicted": 1}, {"index": 143, "id": "boolq/test/1482", "label": 0, "p_yes": 0.7799928784370422, "predicted": 1}, {"index": 144, "id": "boolq/test/18", "label": 1, "p_yes": 0.41111084818840027, "predicted": 0}, {"index": 145, "id": "boolq/test/395", "label": 0, "p_yes": 0.8006919622421265, "predicted": 1}, {"index": 146, "id": "boolq/test/838", "label": 0, "p_yes": 0.38861799240112305, "predicted": 0}, {"index": 147, "id": "boolq/test/111", "label": 1, "p_yes": 0.622459352016449, "predicted": 1}, {"index": 148, "id": "boolq/test/1465", "label": 1, "p_yes": 0.0838903859257698, "predicted": 0}, {"index": 149, "id": "boolq/test/1267", "label": 1, "p_yes": 0.4148988127708435, "predicted": 0}, {"index": 150, "id": "boolq/test/1863", "label": 1, "p_yes": 0.8031737804412842, "predicted": 1}, {"index": 151, "id": "boolq/test/2382", "label": 1, "p_yes": 0.8499711751937866, "predicted": 1}, {"index": 152, "id": "boolq/test/1340", "label": 1, "p_yes": 0.9944451451301575, "predicted": 1}, {"index": 153, "id": "boolq/test/831", "label": 0, "p_yes": 0.9086179733276367, "predicted": 1}, {"index": 154, "id": "boolq/test/1216", "label": 1, "p_yes": 0.1710611879825592, "predicted": 0}, {"index": 155, "id": "boolq/test/124", "label": 1, "p_yes": 0.837619960308075, "predicted": 1}, {"index": 156, "id": "boolq/test/3147", "label": 1, "p_yes": 0.709019124507904, "predicted": 1}, {"index": 157, "id": "boolq/test/27", "label": 0, "p_yes": 0.9324532747268677, "predicted": 1}, {"index": 158, "id": "boolq/test/1060", "label": 0, "p_yes": 0.7690802216529846, "predicted": 1}, {"index": 159, "id": "boolq/test/3162", "label": 1, "p_yes": 0.961533784866333, "predicted": 1}, {"index": 160, "id": "boolq/test/2730", "label": 1, "p_yes": 0.9926542043685913, "predicted": 1}, {"index": 161, "id": "boolq/test/386", "label": 1, "p_yes": 0.9965165853500366, "predicted": 1}, {"index": 162, "id": "boolq/test/2848", "label": 1, "p_yes": 0.8558511734008789, "predicted": 1}, {"index": 163, "id": "boolq/test/1306", "label": 1, "p_yes": 0.9759464263916016, "predicted": 1}, {"index": 164, "id": "boolq/test/484", "label": 1, "p_yes": 0.9877586960792542, "predicted": 1}, {"index": 165, "id": "boolq/test/1636", "label": 1, "p_yes": 0.9597620368003845, "predicted": 1}, {"index": 166, "id": "boolq/test/1326", "label": 0, "p_yes": 0.8175744414329529, "predicted": 1}, {"index": 167, "id": "boolq/test/771", "label": 1, "p_yes": 0.9591543078422546, "predicted": 1}, {"index": 168, "id": "boolq/test/638", "label": 0, "p_yes": 0.10230470448732376, "predicted": 0}, {"index": 169, "id": "boolq/test/1298", "label": 1, "p_yes": 0.9449946880340576, "predicted": 1}, {"index": 170, "id": "boolq/test/3018", "label": 1, "p_yes": 0.9989182949066162, "predicted": 1}, {"index": 171, "id": "boolq/test/420", "label": 0, "p_yes": 0.6992543935775757, "predicted": 1}, {"index": 172, "id": "boolq/test/1304", "label": 1, "p_yes": 0.5583269596099854, "predicted": 1}, {"index": 173, "id": "boolq/test/1477", "label": 0, "p_yes": 0.6992543935775757, "predicted": 1}, {"index": 174, "id": "boolq/test/1703", "label": 1, "p_yes": 0.8397339582443237, "predicted": 1}, {"index": 175, "id": "boolq/test/2088", "label": 0, "p_yes": 0.14804719388484955, "predicted": 0}, {"index": 176, "id": "boolq/test/3100", "label": 1, "p_yes": 0.837619960308075, "predicted": 1}, {"index": 177, "id": "boolq/test/2484", "label": 1, "p_yes": 0.4843800961971283, "predicted": 0}, {"index": 178, "id": "boolq/test/3066", "label": 1, "p_yes": 0.4765796661376953, "predicted": 0}, {"index": 179, "id": "boolq/test/145", "label": 1, "p_yes": 0.6261242032051086, "predicted": 1}, {"index": 180, "id": "boolq/test/2204", "label": 0, "p_yes": 0.36296921968460083, "predicted": 0}, {"index": 181, "id": "boolq/test/3168", "label": 1, "p_yes": 0.9940890073776245, "predicted": 1}, {"index": 182, "id": "boolq/test/1020", "label": 0, "p_yes": 0.30404165387153625, "predicted": 0}, {"index": 183, "id": "boolq/test/956", "label": 1, "p_yes": 0.8175744414329529, "predicted": 1}, {"index": 184, "id": "boolq/test/1619", "label": 1, "p_yes": 0.4765796661376953, "predicted": 0}, {"index": 185, "id": "boolq/test/2154", "label": 1, "p_yes": 0.9621075391769409, "predicted": 1}, {"index": 186, "id": "boolq/test/2262", "label": 0, "p_yes": 0.23370634019374847, "predicted": 0}, {"index": 187, "id": "boolq/test/74", "label": 1, "p_yes": 0.9976403713226318, "predicted": 1}, {"index": 188, "id": "boolq/test/2097", "label": 1, "p_yes": 0.9136765599250793, "predicted": 1}, {"index": 189, "id": "boolq/test/1963", "label": 1, "p_yes": 0.33111974596977234, "predicted": 0}, {"index": 190, "id": "boolq/test/1658", "label": 1, "p_yes": 0.9799765348434448, "predicted": 1}, {"index": 191, "id": "boolq/test/1517", "label": 0, "p_yes": 0.11757213622331619, "predicted": 0}, {"index": 192, "id": "boolq/test/332", "label": 1, "p_yes": 0.5888891220092773, "predicted": 1}, {"index": 193, "id": "boolq/test/345", "label": 1, "p_yes": 0.2598256468772888, "predicted": 0}, {"index": 194, "id": "boolq/test/3245", "label": 1, "p_yes": 0.9971990585327148, "predicted": 1}, {"index": 195, "id": "boolq/test/1480", "label": 1, "p_yes": 0.9896718263626099, "predicted": 1}, {"index": 196, "id": "boolq/test/711", "label": 1, "p_yes": 0.9648551344871521, "predicted": 1}, {"index": 197, "id": "boolq/test/2446", "label": 0, "p_yes": 0.8459424376487732, "predicted": 1}, {"index": 198, "id": "boolq/test/893", "label": 1, "p_yes": 0.9883127808570862, "predicted": 1}, {"index": 199, "id": "boolq/test/2249", "label": 1, "p_yes": 0.9965165853500366, "predicted": 1}, {"index": 200, "id": "boolq/test/3077", "label": 1, "p_yes": 0.9869794845581055, "predicted": 1}, {"index": 201, "id": "boolq/test/2891", "label": 0, "p_yes": 0.6619228720664978, "predicted": 1}, {"index": 202, "id": "boolq/test/896", "label": 0, "p_yes": 0.7853085398674011, "predicted": 1}, {"index": 203, "id": "boolq/test/1757", "label": 1, "p_yes": 0.939024806022644, "predicted": 1}, {"index": 204, "id": "boolq/test/2472", "label": 1, "p_yes": 0.1919327974319458, "predicted": 0}, {"index": 205, "id": "boolq/test/1952", "label": 1, "p_yes": 0.5, "predicted": 1}, {"index": 206, "id": "boolq/test/625", "label": 1, "p_yes": 0.9252299070358276, "predicted": 1}, {"index": 207, "id": "boolq/test/410", "label": 1, "p_yes": 0.8705973029136658, "predicted": 1}, {"index": 208, "id": "boolq/test/800", "label": 0, "p_yes": 0.27202457189559937, "predicted": 0}, {"index": 209, "id": "boolq/test/1251", "label": 1, "p_yes": 0.9553191661834717, "predicted": 1}, {"index": 210, "id": "boolq/test/65", "label": 1, "p_yes": 0.7279754281044006, "predicted": 1}, {"index": 211, "id": "boolq/test/1056", "label": 1, "p_yes": 0.8438951373100281, "predicted": 1}, {"index": 212, "id": "boolq/test/2627", "label": 0, "p_yes": 0.39981165528297424, "predicted": 0}, {"index": 213, "id": "boolq/test/3255", "label": 0, "p_yes": 0.5039061903953552, "predicted": 1}, {"index": 214, "id": "boolq/test/220", "label": 0, "p_yes": 0.4765796661376953, "predicted": 0}, {"index": 215, "id": "boolq/test/2731", "label": 1, "p_yes": 0.9314625263214111, "predicted": 1}, {"index": 216, "id": "boolq/test/1713", "label": 1, "p_yes": 0.9553191661834717, "predicted": 1}, {"index": 217, "id": "boolq/test/2430", "label": 0, "p_yes": 0.5350984334945679, "predicted": 1}, {"index": 218, "id": "boolq/test/2185", "label": 1, "p_yes": 0.9099069833755493, "predicted": 1}, {"index": 219, "id": "boolq/test/2039", "label": 1, "p_yes": 0.41869691014289856, "predicted": 0}, {"index": 220, "id": "boolq/test/1591", "label": 1, "p_yes": 0.8933094143867493, "predicted": 1}, {"index": 221, "id": "boolq/test/3015", "label": 0, "p_yes": 0.9481545686721802, "predicted": 1}, {"index": 222, "id": "boolq/test/2425", "label": 0, "p_yes": 0.7279754281044006, "predicted": 1}, {"index": 223, "id": "boolq/test/2131", "label": 0, "p_yes": 0.34510529041290283, "predicted": 0}, {"index": 224, "id": "boolq/test/1184", "label": 1, "p_yes": 0.985496461391449, "predicted": 1}, {"index": 225, "id": "boolq/test/2351", "label": 1, "p_yes": 0.8824278712272644, "predicted": 1}, {"index": 226, "id": "boolq/test/925", "label": 0, "p_yes": 0.8840392827987671, "predicted": 1}, {"index": 227, "id": "boolq/test/570", "label": 1, "p_yes": 0.837619960308075, "predicted": 1}, {"index": 228, "id": "boolq/test/3260", "label": 0, "p_yes": 0.9284088015556335, "predicted": 1}, {"index": 229, "id": "boolq/test/2576", "label": 0, "p_yes": 0.8479677438735962, "predicted": 1}, {"index": 230, "id": "boolq/test/2825", "label": 1, "p_yes": 0.9597620368003845, "predicted": 1}, {"index": 231, "id": "boolq/test/882", "label": 1, "p_yes": 0.9908744096755981, "predicted": 1}, {"index": 232, "id": "boolq/test/1423", "label": 0, "p_yes": 0.36658975481987, "predicted": 0}, {"index": 233, "id": "boolq/test/96", "label": 1, "p_yes": 0.9504110217094421, "predicted": 1}, {"index": 234, "id": "boolq/test/2646", "label": 1, "p_yes": 0.9458011984825134, "predicted": 1}, {"index": 235, "id": "boolq/test/375", "label": 1, "p_yes": 0.9820137619972229, "predicted": 1}, {"index": 236, "id": "boolq/test/242", "label": 0, "p_yes": 0.986151397228241, "predicted": 1}, {"index": 237, "id": "boolq/test/1406", "label": 1, "p_yes": 0.5698526501655579, "predicted": 1}, {"index": 238, "id": "boolq/test/652", "label": 1, "p_yes": 0.9928786158561707, "predicted": 1}, {"index": 239, "id": "boolq/test/1136", "label": 1, "p_yes": 0.7549149990081787, "predicted": 1}, {"index": 240, "id": "boolq/test/839", "label": 0, "p_yes": 0.7461389899253845, "predicted": 1}, {"index": 241, "id": "boolq/test/1908", "label": 1, "p_yes": 0.4804786741733551, "predicted": 0}, {"index": 242, "id": "boolq/test/1992", "label": 0, "p_yes": 0.22815649211406708, "predicted": 0}, {"index": 243, "id": "boolq/test/1191", "label": 1, "p_yes": 0.982557475566864, "predicted": 1}, {"index": 244, "id": "boolq/test/2045", "label": 0, "p_yes": 0.685949444770813, "predicted": 1}, {"index": 245, "id": "boolq/test/286", "label": 1, "p_yes": 0.8723474144935608, "predicted": 1}, {"index": 246, "id": "boolq/test/3095", "label": 0, "p_yes": 0.8267117738723755, "predicted": 1}, {"index": 247, "id": "boolq/test/2285", "label": 1, "p_yes": 0.9759464263916016, "predicted": 1}, {"index": 248, "id": "boolq/test/572", "label": 1, "p_yes": 0.8740772008895874, "predicted": 1}, {"index": 249, "id": "boolq/test/1712", "label": 0, "p_yes": 0.05582313984632492, "predicted": 0}, {"index": 250, "id": "boolq/test/2860", "label": 1, "p_yes": 0.9937100410461426, "predicted": 1}, {"index": 251, "id": "boolq/test/1814", "label": 1, "p_yes": 0.9621075391769409, "predicted": 1}, {"index": 252, "id": "boolq/test/2580", "label": 0, "p_yes": 0.8740772008895874, "predicted": 1}, {"index": 253, "id": "boolq/test/1297", "label": 1, "p_yes": 0.9884918928146362, "predicted": 1}, {"index": 254, "id": "boolq/test/384", "label": 1, "p_yes": 0.9974491000175476, "predicted": 1}, {"index": 255, "id": "boolq/test/1073", "label": 1, "p_yes": 0.9334307909011841, "predicted": 1}]

base-ood-calibrated.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "strategyqa_grounded/test/608", "label": 0, "p_yes": 0.49254565312691795, "predicted": 0}, {"index": 1, "id": "strategyqa_grounded/test/72", "label": 1, "p_yes": 0.8852020313291094, "predicted": 1}, {"index": 2, "id": "strategyqa_grounded/test/371", "label": 1, "p_yes": 0.6010926352712405, "predicted": 1}, {"index": 3, "id": "strategyqa_grounded/test/155", "label": 1, "p_yes": 0.48137137618190634, "predicted": 0}, {"index": 4, "id": "strategyqa_grounded/test/565", "label": 1, "p_yes": 0.24495726290282013, "predicted": 0}, {"index": 5, "id": "strategyqa_grounded/test/252", "label": 1, "p_yes": 0.8000509229276416, "predicted": 1}, {"index": 6, "id": "strategyqa_grounded/test/353", "label": 0, "p_yes": 0.45723899769917586, "predicted": 0}, {"index": 7, "id": "strategyqa_grounded/test/254", "label": 1, "p_yes": 0.703656044191075, "predicted": 1}, {"index": 8, "id": "strategyqa_grounded/test/441", "label": 1, "p_yes": 0.3900042583752899, "predicted": 0}, {"index": 9, "id": "strategyqa_grounded/test/603", "label": 0, "p_yes": 0.3449929976626092, "predicted": 0}, {"index": 10, "id": "strategyqa_grounded/test/212", "label": 0, "p_yes": 0.40787784563058965, "predicted": 0}, {"index": 11, "id": "strategyqa_grounded/test/430", "label": 1, "p_yes": 0.5483057211681001, "predicted": 1}, {"index": 12, "id": "strategyqa_grounded/test/528", "label": 1, "p_yes": 0.786609522964005, "predicted": 1}, {"index": 13, "id": "strategyqa_grounded/test/204", "label": 0, "p_yes": 0.8425537121924163, "predicted": 1}, {"index": 14, "id": "strategyqa_grounded/test/225", "label": 1, "p_yes": 0.4869568679853428, "predicted": 0}, {"index": 15, "id": "strategyqa_grounded/test/463", "label": 1, "p_yes": 0.23010922041427184, "predicted": 0}, {"index": 16, "id": "strategyqa_grounded/test/43", "label": 0, "p_yes": 0.5, "predicted": 1}, {"index": 17, "id": "strategyqa_grounded/test/525", "label": 0, "p_yes": 0.3169432496807414, "predicted": 0}, {"index": 18, "id": "strategyqa_grounded/test/431", "label": 0, "p_yes": 0.5975122061302928, "predicted": 1}, {"index": 19, "id": "strategyqa_grounded/test/584", "label": 1, "p_yes": 0.5055909454343076, "predicted": 1}, {"index": 20, "id": "strategyqa_grounded/test/405", "label": 0, "p_yes": 0.5903204713121641, "predicted": 1}, {"index": 21, "id": "strategyqa_grounded/test/231", "label": 0, "p_yes": 0.7577898751185104, "predicted": 1}, {"index": 22, "id": "strategyqa_grounded/test/610", "label": 0, "p_yes": 0.5353517264898839, "predicted": 1}, {"index": 23, "id": "strategyqa_grounded/test/515", "label": 1, "p_yes": 0.6099957248964116, "predicted": 1}, {"index": 24, "id": "strategyqa_grounded/test/184", "label": 0, "p_yes": 0.8228402369376934, "predicted": 1}, {"index": 25, "id": "strategyqa_grounded/test/573", "label": 1, "p_yes": 0.5260685152502846, "predicted": 1}, {"index": 26, "id": "strategyqa_grounded/test/39", "label": 0, "p_yes": 0.6379719864179736, "predicted": 1}, {"index": 27, "id": "strategyqa_grounded/test/672", "label": 0, "p_yes": 0.2810381182957583, "predicted": 0}, {"index": 28, "id": "strategyqa_grounded/test/94", "label": 1, "p_yes": 0.2735684148608584, "predicted": 0}, {"index": 29, "id": "strategyqa_grounded/test/314", "label": 1, "p_yes": 0.5776432321734629, "predicted": 1}, {"index": 30, "id": "strategyqa_grounded/test/73", "label": 1, "p_yes": 0.8874553047387982, "predicted": 1}, {"index": 31, "id": "strategyqa_grounded/test/679", "label": 1, "p_yes": 0.8355068644121367, "predicted": 1}, {"index": 32, "id": "strategyqa_grounded/test/216", "label": 0, "p_yes": 0.6010926352712405, "predicted": 1}, {"index": 33, "id": "strategyqa_grounded/test/587", "label": 0, "p_yes": 0.43880535972129614, "predicted": 0}, {"index": 34, "id": "strategyqa_grounded/test/227", "label": 1, "p_yes": 0.34836995184008646, "predicted": 0}, {"index": 35, "id": "strategyqa_grounded/test/630", "label": 1, "p_yes": 0.7323173862752927, "predicted": 1}, {"index": 36, "id": "strategyqa_grounded/test/327", "label": 1, "p_yes": 0.6448305223858463, "predicted": 1}, {"index": 37, "id": "strategyqa_grounded/test/199", "label": 1, "p_yes": 0.9412124239990308, "predicted": 1}, {"index": 38, "id": "strategyqa_grounded/test/436", "label": 0, "p_yes": 0.3399568511577363, "predicted": 0}, {"index": 39, "id": "strategyqa_grounded/test/335", "label": 0, "p_yes": 0.3025995705557748, "predicted": 0}, {"index": 40, "id": "strategyqa_grounded/test/46", "label": 0, "p_yes": 0.48137137618190634, "predicted": 0}, {"index": 41, "id": "strategyqa_grounded/test/386", "label": 0, "p_yes": 0.6862757300880689, "predicted": 1}, {"index": 42, "id": "strategyqa_grounded/test/682", "label": 0, "p_yes": 0.40787784563058965, "predicted": 0}, {"index": 43, "id": "strategyqa_grounded/test/596", "label": 0, "p_yes": 0.5703517479200776, "predicted": 1}, {"index": 44, "id": "strategyqa_grounded/test/554", "label": 0, "p_yes": 0.4223567832511533, "predicted": 0}, {"index": 45, "id": "strategyqa_grounded/test/169", "label": 1, "p_yes": 0.6223368291249854, "predicted": 1}, {"index": 46, "id": "strategyqa_grounded/test/508", "label": 1, "p_yes": 0.6396920112712661, "predicted": 1}, {"index": 47, "id": "strategyqa_grounded/test/56", "label": 0, "p_yes": 0.522349809484464, "predicted": 1}, {"index": 48, "id": "strategyqa_grounded/test/217", "label": 1, "p_yes": 0.6600431288492591, "predicted": 1}, {"index": 49, "id": "strategyqa_grounded/test/253", "label": 1, "p_yes": 0.4832327510910743, "predicted": 0}, {"index": 50, "id": "strategyqa_grounded/test/569", "label": 0, "p_yes": 0.8206563837976911, "predicted": 1}, {"index": 51, "id": "strategyqa_grounded/test/384", "label": 0, "p_yes": 0.44064200088978284, "predicted": 0}, {"index": 52, "id": "strategyqa_grounded/test/513", "label": 1, "p_yes": 0.36202800441306293, "predicted": 0}, {"index": 53, "id": "strategyqa_grounded/test/107", "label": 0, "p_yes": 0.3672092280151138, "predicted": 0}, {"index": 54, "id": "strategyqa_grounded/test/616", "label": 0, "p_yes": 0.6749326594367204, "predicted": 1}, {"index": 55, "id": "strategyqa_grounded/test/468", "label": 1, "p_yes": 0.41872348901168055, "predicted": 0}, {"index": 56, "id": "strategyqa_grounded/test/157", "label": 0, "p_yes": 0.40967954455911243, "predicted": 0}, {"index": 57, "id": "strategyqa_grounded/test/465", "label": 1, "p_yes": 0.5390584953088928, "predicted": 1}, {"index": 58, "id": "strategyqa_grounded/test/579", "label": 1, "p_yes": 0.7381222832851778, "predicted": 1}, {"index": 59, "id": "strategyqa_grounded/test/532", "label": 1, "p_yes": 0.3534639813037472, "predicted": 0}, {"index": 60, "id": "strategyqa_grounded/test/606", "label": 0, "p_yes": 0.5593580140234986, "predicted": 1}, {"index": 61, "id": "strategyqa_grounded/test/209", "label": 0, "p_yes": 0.7790052394886462, "predicted": 1}, {"index": 62, "id": "strategyqa_grounded/test/88", "label": 1, "p_yes": 0.6732949378666645, "predicted": 1}, {"index": 63, "id": "strategyqa_grounded/test/281", "label": 0, "p_yes": 0.7738289998242308, "predicted": 1}, {"index": 64, "id": "strategyqa_grounded/test/295", "label": 1, "p_yes": 0.44064200088978284, "predicted": 0}, {"index": 65, "id": "strategyqa_grounded/test/457", "label": 1, "p_yes": 0.5409103027301156, "predicted": 1}, {"index": 66, "id": "strategyqa_grounded/test/170", "label": 1, "p_yes": 0.4259985047492018, "predicted": 0}, {"index": 67, "id": "strategyqa_grounded/test/658", "label": 0, "p_yes": 0.8889358723528173, "predicted": 1}, {"index": 68, "id": "strategyqa_grounded/test/123", "label": 0, "p_yes": 0.5055909454343076, "predicted": 1}, {"index": 69, "id": "strategyqa_grounded/test/461", "label": 1, "p_yes": 0.40787784563058965, "predicted": 0}, {"index": 70, "id": "strategyqa_grounded/test/556", "label": 1, "p_yes": 0.5483057211681001, "predicted": 1}, {"index": 71, "id": "strategyqa_grounded/test/313", "label": 0, "p_yes": 0.38117385957428856, "predicted": 0}, {"index": 72, "id": "strategyqa_grounded/test/246", "label": 0, "p_yes": 0.6633806749213313, "predicted": 1}, {"index": 73, "id": "strategyqa_grounded/test/392", "label": 1, "p_yes": 0.4646482879721674, "predicted": 0}, {"index": 74, "id": "strategyqa_grounded/test/116", "label": 1, "p_yes": 0.6465359997604931, "predicted": 1}, {"index": 75, "id": "strategyqa_grounded/test/328", "label": 1, "p_yes": 0.911001934643148, "predicted": 1}, {"index": 76, "id": "strategyqa_grounded/test/111", "label": 1, "p_yes": 0.4259985047492018, "predicted": 0}, {"index": 77, "id": "strategyqa_grounded/test/444", "label": 0, "p_yes": 0.9174300774298159, "predicted": 1}, {"index": 78, "id": "strategyqa_grounded/test/590", "label": 0, "p_yes": 0.3551694964265601, "predicted": 0}, {"index": 79, "id": "strategyqa_grounded/test/595", "label": 0, "p_yes": 0.3316435148077665, "predicted": 0}, {"index": 80, "id": "strategyqa_grounded/test/574", "label": 0, "p_yes": 0.5334968232870905, "predicted": 1}, {"index": 81, "id": "strategyqa_grounded/test/471", "label": 0, "p_yes": 0.5260685152502846, "predicted": 1}, {"index": 82, "id": "strategyqa_grounded/test/164", "label": 0, "p_yes": 0.32834699052134136, "predicted": 0}, {"index": 83, "id": "strategyqa_grounded/test/361", "label": 0, "p_yes": 0.44800432924247197, "predicted": 0}, {"index": 84, "id": "strategyqa_grounded/test/493", "label": 0, "p_yes": 0.5260685152502846, "predicted": 1}, {"index": 85, "id": "strategyqa_grounded/test/641", "label": 0, "p_yes": 0.2735684148608584, "predicted": 0}, {"index": 86, "id": "strategyqa_grounded/test/423", "label": 0, "p_yes": 0.3449929976626092, "predicted": 0}, {"index": 87, "id": "strategyqa_grounded/test/566", "label": 1, "p_yes": 0.3585915161165846, "predicted": 0}, {"index": 88, "id": "strategyqa_grounded/test/312", "label": 0, "p_yes": 0.7494857432668507, "predicted": 1}, {"index": 89, "id": "strategyqa_grounded/test/389", "label": 1, "p_yes": 0.8932760114947075, "predicted": 1}, {"index": 90, "id": "strategyqa_grounded/test/607", "label": 0, "p_yes": 0.47579051034985453, "predicted": 0}, {"index": 91, "id": "strategyqa_grounded/test/16", "label": 1, "p_yes": 0.8260770048625432, "predicted": 1}, {"index": 92, "id": "strategyqa_grounded/test/81", "label": 1, "p_yes": 0.6099957248964116, "predicted": 1}, {"index": 93, "id": "strategyqa_grounded/test/223", "label": 1, "p_yes": 0.7964490689755397, "predicted": 1}, {"index": 94, "id": "strategyqa_grounded/test/631", "label": 0, "p_yes": 0.5372056410447151, "predicted": 1}, {"index": 95, "id": "strategyqa_grounded/test/165", "label": 1, "p_yes": 0.40428193056916567, "predicted": 0}, {"index": 96, "id": "strategyqa_grounded/test/9", "label": 0, "p_yes": 0.5903204713121641, "predicted": 1}, {"index": 97, "id": "strategyqa_grounded/test/555", "label": 0, "p_yes": 0.4516942641562769, "predicted": 0}, {"index": 98, "id": "strategyqa_grounded/test/475", "label": 1, "p_yes": 0.6153030337332264, "predicted": 1}, {"index": 99, "id": "strategyqa_grounded/test/129", "label": 0, "p_yes": 0.6099957248964116, "predicted": 1}, {"index": 100, "id": "strategyqa_grounded/test/647", "label": 0, "p_yes": 0.4205390573521307, "predicted": 0}, {"index": 101, "id": "strategyqa_grounded/test/86", "label": 1, "p_yes": 0.9323985299669949, "predicted": 1}, {"index": 102, "id": "strategyqa_grounded/test/372", "label": 0, "p_yes": 0.40428193056916567, "predicted": 0}, {"index": 103, "id": "strategyqa_grounded/test/655", "label": 0, "p_yes": 0.40967954455911243, "predicted": 0}, {"index": 104, "id": "strategyqa_grounded/test/69", "label": 1, "p_yes": 0.5018637270799985, "predicted": 1}, {"index": 105, "id": "strategyqa_grounded/test/548", "label": 0, "p_yes": 0.6926600014771807, "predicted": 1}, {"index": 106, "id": "strategyqa_grounded/test/200", "label": 1, "p_yes": 0.5074543611029115, "predicted": 1}, {"index": 107, "id": "strategyqa_grounded/test/599", "label": 0, "p_yes": 0.48137137618190634, "predicted": 0}, {"index": 108, "id": "strategyqa_grounded/test/230", "label": 1, "p_yes": 0.7645654502845479, "predicted": 1}, {"index": 109, "id": "strategyqa_grounded/test/141", "label": 0, "p_yes": 0.6082207631748922, "predicted": 1}, {"index": 110, "id": "strategyqa_grounded/test/368", "label": 0, "p_yes": 0.3971211910686261, "predicted": 0}, {"index": 111, "id": "strategyqa_grounded/test/481", "label": 0, "p_yes": 0.5297843273510043, "predicted": 1}, {"index": 112, "id": "strategyqa_grounded/test/142", "label": 1, "p_yes": 0.522349809484464, "predicted": 1}, {"index": 113, "id": "strategyqa_grounded/test/544", "label": 1, "p_yes": 0.45723899769917586, "predicted": 0}, {"index": 114, "id": "strategyqa_grounded/test/377", "label": 0, "p_yes": 0.3585915161165846, "predicted": 0}, {"index": 115, "id": "strategyqa_grounded/test/480", "label": 0, "p_yes": 0.6396920112712661, "predicted": 1}, {"index": 116, "id": "strategyqa_grounded/test/601", "label": 0, "p_yes": 0.715939426828434, "predicted": 1}, {"index": 117, "id": "strategyqa_grounded/test/237", "label": 1, "p_yes": 0.8292667636858398, "predicted": 1}, {"index": 118, "id": "strategyqa_grounded/test/643", "label": 1, "p_yes": 0.7337762670301476, "predicted": 1}, {"index": 119, "id": "strategyqa_grounded/test/521", "label": 0, "p_yes": 0.557519739042607, "predicted": 1}, {"index": 120, "id": "strategyqa_grounded/test/621", "label": 1, "p_yes": 0.6099957248964116, "predicted": 1}, {"index": 121, "id": "strategyqa_grounded/test/426", "label": 1, "p_yes": 0.45723899769917586, "predicted": 0}, {"index": 122, "id": "strategyqa_grounded/test/162", "label": 0, "p_yes": 0.522349809484464, "predicted": 1}, {"index": 123, "id": "strategyqa_grounded/test/151", "label": 0, "p_yes": 0.6910707368119688, "predicted": 1}, {"index": 124, "id": "strategyqa_grounded/test/181", "label": 1, "p_yes": 0.5703517479200776, "predicted": 1}, {"index": 125, "id": "strategyqa_grounded/test/71", "label": 1, "p_yes": 0.524209503982742, "predicted": 1}, {"index": 126, "id": "strategyqa_grounded/test/63", "label": 0, "p_yes": 0.4259985047492018, "predicted": 0}, {"index": 127, "id": "strategyqa_grounded/test/275", "label": 0, "p_yes": 0.5186286381043013, "predicted": 1}]

base-ood-predictions.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "strategyqa_grounded/test/608", "label": 0, "p_yes": 0.4843800961971283, "predicted": 0}, {"index": 1, "id": "strategyqa_grounded/test/72", "label": 1, "p_yes": 0.9863631129264832, "predicted": 1}, {"index": 2, "id": "strategyqa_grounded/test/371", "label": 1, "p_yes": 0.7025300860404968, "predicted": 1}, {"index": 3, "id": "strategyqa_grounded/test/155", "label": 1, "p_yes": 0.46101680397987366, "predicted": 0}, {"index": 4, "id": "strategyqa_grounded/test/565", "label": 1, "p_yes": 0.08632348477840424, "predicted": 0}, {"index": 5, "id": "strategyqa_grounded/test/252", "label": 1, "p_yes": 0.9481545686721802, "predicted": 1}, {"index": 6, "id": "strategyqa_grounded/test/353", "label": 0, "p_yes": 0.41111084818840027, "predicted": 0}, {"index": 7, "id": "strategyqa_grounded/test/254", "label": 1, "p_yes": 0.859663724899292, "predicted": 1}, {"index": 8, "id": "strategyqa_grounded/test/441", "label": 1, "p_yes": 0.28140559792518616, "predicted": 0}, {"index": 9, "id": "strategyqa_grounded/test/603", "label": 0, "p_yes": 0.20689405500888824, "predicted": 0}, {"index": 10, "id": "strategyqa_grounded/test/212", "label": 0, "p_yes": 0.314050555229187, "predicted": 0}, {"index": 11, "id": "strategyqa_grounded/test/430", "label": 1, "p_yes": 0.6001883149147034, "predicted": 1}, {"index": 12, "id": "strategyqa_grounded/test/528", "label": 1, "p_yes": 0.939024806022644, "predicted": 1}, {"index": 13, "id": "strategyqa_grounded/test/204", "label": 0, "p_yes": 0.9711290597915649, "predicted": 1}, {"index": 14, "id": "strategyqa_grounded/test/225", "label": 1, "p_yes": 0.47268345952033997, "predicted": 0}, {"index": 15, "id": "strategyqa_grounded/test/463", "label": 1, "p_yes": 0.07369627058506012, "predicted": 0}, {"index": 16, "id": "strategyqa_grounded/test/43", "label": 0, "p_yes": 0.5, "predicted": 1}, {"index": 17, "id": "strategyqa_grounded/test/525", "label": 0, "p_yes": 0.16667540371418, "predicted": 0}, {"index": 18, "id": "strategyqa_grounded/test/431", "label": 0, "p_yes": 0.6959583163261414, "predicted": 1}, {"index": 19, "id": "strategyqa_grounded/test/584", "label": 1, "p_yes": 0.5117166042327881, "predicted": 1}, {"index": 20, "id": "strategyqa_grounded/test/405", "label": 0, "p_yes": 0.6825737357139587, "predicted": 1}, {"index": 21, "id": "strategyqa_grounded/test/231", "label": 0, "p_yes": 0.9161096215248108, "predicted": 1}, {"index": 22, "id": "strategyqa_grounded/test/610", "label": 0, "p_yes": 0.5736784338951111, "predicted": 1}, {"index": 23, "id": "strategyqa_grounded/test/515", "label": 1, "p_yes": 0.7185943722724915, "predicted": 1}, {"index": 24, "id": "strategyqa_grounded/test/184", "label": 0, "p_yes": 0.961533784866333, "predicted": 1}, {"index": 25, "id": "strategyqa_grounded/test/573", "label": 1, "p_yes": 0.5544704794883728, "predicted": 1}, {"index": 26, "id": "strategyqa_grounded/test/39", "label": 0, "p_yes": 0.7662936449050903, "predicted": 1}, {"index": 27, "id": "strategyqa_grounded/test/672", "label": 0, "p_yes": 0.12252321094274521, "predicted": 0}, {"index": 28, "id": "strategyqa_grounded/test/94", "label": 1, "p_yes": 0.11436852067708969, "predicted": 0}, {"index": 29, "id": "strategyqa_grounded/test/314", "label": 1, "p_yes": 0.6584175229072571, "predicted": 1}, {"index": 30, "id": "strategyqa_grounded/test/73", "label": 1, "p_yes": 0.9869794845581055, "predicted": 1}, {"index": 31, "id": "strategyqa_grounded/test/679", "label": 1, "p_yes": 0.9678993225097656, "predicted": 1}, {"index": 32, "id": "strategyqa_grounded/test/216", "label": 0, "p_yes": 0.7025300860404968, "predicted": 1}, {"index": 33, "id": "strategyqa_grounded/test/587", "label": 0, "p_yes": 0.37387582659721375, "predicted": 0}, {"index": 34, "id": "strategyqa_grounded/test/227", "label": 1, "p_yes": 0.2120688110589981, "predicted": 0}, {"index": 35, "id": "strategyqa_grounded/test/630", "label": 1, "p_yes": 0.8918110132217407, "predicted": 1}, {"index": 36, "id": "strategyqa_grounded/test/327", "label": 1, "p_yes": 0.7772998809814453, "predicted": 1}, {"index": 37, "id": "strategyqa_grounded/test/199", "label": 1, "p_yes": 0.9970190525054932, "predicted": 1}, {"index": 38, "id": "strategyqa_grounded/test/436", "label": 0, "p_yes": 0.19930800795555115, "predicted": 0}, {"index": 39, "id": "strategyqa_grounded/test/335", "label": 0, "p_yes": 0.14804719388484955, "predicted": 0}, {"index": 40, "id": "strategyqa_grounded/test/46", "label": 0, "p_yes": 0.46101680397987366, "predicted": 0}, {"index": 41, "id": "strategyqa_grounded/test/386", "label": 0, "p_yes": 0.837619960308075, "predicted": 1}, {"index": 42, "id": "strategyqa_grounded/test/682", "label": 0, "p_yes": 0.314050555229187, "predicted": 0}, {"index": 43, "id": "strategyqa_grounded/test/596", "label": 0, "p_yes": 0.6442251205444336, "predicted": 1}, {"index": 44, "id": "strategyqa_grounded/test/554", "label": 0, "p_yes": 0.3415825068950653, "predicted": 0}, {"index": 45, "id": "strategyqa_grounded/test/169", "label": 1, "p_yes": 0.740174412727356, "predicted": 1}, {"index": 46, "id": "strategyqa_grounded/test/508", "label": 1, "p_yes": 0.7690802216529846, "predicted": 1}, {"index": 47, "id": "strategyqa_grounded/test/56", "label": 0, "p_yes": 0.5467381477355957, "predicted": 1}, {"index": 48, "id": "strategyqa_grounded/test/217", "label": 1, "p_yes": 0.8006919622421265, "predicted": 1}, {"index": 49, "id": "strategyqa_grounded/test/253", "label": 1, "p_yes": 0.46490156650543213, "predicted": 0}, {"index": 50, "id": "strategyqa_grounded/test/569", "label": 0, "p_yes": 0.9603611826896667, "predicted": 1}, {"index": 51, "id": "strategyqa_grounded/test/384", "label": 0, "p_yes": 0.3775406777858734, "predicted": 0}, {"index": 52, "id": "strategyqa_grounded/test/513", "label": 1, "p_yes": 0.23370634019374847, "predicted": 0}, {"index": 53, "id": "strategyqa_grounded/test/107", "label": 0, "p_yes": 0.2422056347131729, "predicted": 0}, {"index": 54, "id": "strategyqa_grounded/test/616", "label": 0, "p_yes": 0.8221890926361084, "predicted": 1}, {"index": 55, "id": "strategyqa_grounded/test/468", "label": 1, "p_yes": 0.3345894515514374, "predicted": 0}, {"index": 56, "id": "strategyqa_grounded/test/157", "label": 0, "p_yes": 0.31742629408836365, "predicted": 0}, {"index": 57, "id": "strategyqa_grounded/test/465", "label": 1, "p_yes": 0.581303060054779, "predicted": 1}, {"index": 58, "id": "strategyqa_grounded/test/579", "label": 1, "p_yes": 0.8976953029632568, "predicted": 1}, {"index": 59, "id": "strategyqa_grounded/test/532", "label": 1, "p_yes": 0.22000709176063538, "predicted": 0}, {"index": 60, "id": "strategyqa_grounded/test/606", "label": 0, "p_yes": 0.622459352016449, "predicted": 1}, {"index": 61, "id": "strategyqa_grounded/test/209", "label": 0, "p_yes": 0.9334307909011841, "predicted": 1}, {"index": 62, "id": "strategyqa_grounded/test/88", "label": 1, "p_yes": 0.8198933005332947, "predicted": 1}, {"index": 63, "id": "strategyqa_grounded/test/281", "label": 0, "p_yes": 0.929440438747406, "predicted": 1}, {"index": 64, "id": "strategyqa_grounded/test/295", "label": 1, "p_yes": 0.3775406777858734, "predicted": 0}, {"index": 65, "id": "strategyqa_grounded/test/457", "label": 1, "p_yes": 0.5851011276245117, "predicted": 1}, {"index": 66, "id": "strategyqa_grounded/test/170", "label": 1, "p_yes": 0.3486451506614685, "predicted": 0}, {"index": 67, "id": "strategyqa_grounded/test/658", "label": 0, "p_yes": 0.987375020980835, "predicted": 1}, {"index": 68, "id": "strategyqa_grounded/test/123", "label": 0, "p_yes": 0.5117166042327881, "predicted": 1}, {"index": 69, "id": "strategyqa_grounded/test/461", "label": 1, "p_yes": 0.314050555229187, "predicted": 0}, {"index": 70, "id": "strategyqa_grounded/test/556", "label": 1, "p_yes": 0.6001883149147034, "predicted": 1}, {"index": 71, "id": "strategyqa_grounded/test/313", "label": 0, "p_yes": 0.2658804953098297, "predicted": 0}, {"index": 72, "id": "strategyqa_grounded/test/246", "label": 0, "p_yes": 0.8056321740150452, "predicted": 1}, {"index": 73, "id": "strategyqa_grounded/test/392", "label": 1, "p_yes": 0.4263215959072113, "predicted": 0}, {"index": 74, "id": "strategyqa_grounded/test/116", "label": 1, "p_yes": 0.7799928784370422, "predicted": 1}, {"index": 75, "id": "strategyqa_grounded/test/328", "label": 1, "p_yes": 0.9924227595329285, "predicted": 1}, {"index": 76, "id": "strategyqa_grounded/test/111", "label": 1, "p_yes": 0.3486451506614685, "predicted": 0}, {"index": 77, "id": "strategyqa_grounded/test/444", "label": 0, "p_yes": 0.9936116337776184, "predicted": 1}, {"index": 78, "id": "strategyqa_grounded/test/590", "label": 0, "p_yes": 0.22270014882087708, "predicted": 0}, {"index": 79, "id": "strategyqa_grounded/test/595", "label": 0, "p_yes": 0.18713268637657166, "predicted": 0}, {"index": 80, "id": "strategyqa_grounded/test/574", "label": 0, "p_yes": 0.5698526501655579, "predicted": 1}, {"index": 81, "id": "strategyqa_grounded/test/471", "label": 0, "p_yes": 0.5544704794883728, "predicted": 1}, {"index": 82, "id": "strategyqa_grounded/test/164", "label": 0, "p_yes": 0.18242551386356354, "predicted": 0}, {"index": 83, "id": "strategyqa_grounded/test/361", "label": 0, "p_yes": 0.3923368453979492, "predicted": 0}, {"index": 84, "id": "strategyqa_grounded/test/493", "label": 0, "p_yes": 0.5544704794883728, "predicted": 1}, {"index": 85, "id": "strategyqa_grounded/test/641", "label": 0, "p_yes": 0.11436852067708969, "predicted": 0}, {"index": 86, "id": "strategyqa_grounded/test/423", "label": 0, "p_yes": 0.20689405500888824, "predicted": 0}, {"index": 87, "id": "strategyqa_grounded/test/566", "label": 1, "p_yes": 0.22815649211406708, "predicted": 0}, {"index": 88, "id": "strategyqa_grounded/test/312", "label": 0, "p_yes": 0.9086179733276367, "predicted": 1}, {"index": 89, "id": "strategyqa_grounded/test/389", "label": 1, "p_yes": 0.9884918928146362, "predicted": 1}, {"index": 90, "id": "strategyqa_grounded/test/607", "label": 0, "p_yes": 0.4493926465511322, "predicted": 0}, {"index": 91, "id": "strategyqa_grounded/test/16", "label": 1, "p_yes": 0.963230550289154, "predicted": 1}, {"index": 92, "id": "strategyqa_grounded/test/81", "label": 1, "p_yes": 0.7185943722724915, "predicted": 1}, {"index": 93, "id": "strategyqa_grounded/test/223", "label": 1, "p_yes": 0.9458011984825134, "predicted": 1}, {"index": 94, "id": "strategyqa_grounded/test/631", "label": 0, "p_yes": 0.5774953961372375, "predicted": 1}, {"index": 95, "id": "strategyqa_grounded/test/165", "label": 1, "p_yes": 0.3073580265045166, "predicted": 0}, {"index": 96, "id": "strategyqa_grounded/test/9", "label": 0, "p_yes": 0.6825737357139587, "predicted": 1}, {"index": 97, "id": "strategyqa_grounded/test/555", "label": 0, "p_yes": 0.39981165528297424, "predicted": 0}, {"index": 98, "id": "strategyqa_grounded/test/475", "label": 1, "p_yes": 0.7279754281044006, "predicted": 1}, {"index": 99, "id": "strategyqa_grounded/test/129", "label": 0, "p_yes": 0.7185943722724915, "predicted": 1}, {"index": 100, "id": "strategyqa_grounded/test/647", "label": 0, "p_yes": 0.3380771279335022, "predicted": 0}, {"index": 101, "id": "strategyqa_grounded/test/86", "label": 1, "p_yes": 0.9959298968315125, "predicted": 1}, {"index": 102, "id": "strategyqa_grounded/test/372", "label": 0, "p_yes": 0.3073580265045166, "predicted": 0}, {"index": 103, "id": "strategyqa_grounded/test/655", "label": 0, "p_yes": 0.31742629408836365, "predicted": 0}, {"index": 104, "id": "strategyqa_grounded/test/69", "label": 1, "p_yes": 0.5039061903953552, "predicted": 1}, {"index": 105, "id": "strategyqa_grounded/test/548", "label": 0, "p_yes": 0.8459424376487732, "predicted": 1}, {"index": 106, "id": "strategyqa_grounded/test/200", "label": 1, "p_yes": 0.5156199336051941, "predicted": 1}, {"index": 107, "id": "strategyqa_grounded/test/599", "label": 0, "p_yes": 0.46101680397987366, "predicted": 0}, {"index": 108, "id": "strategyqa_grounded/test/230", "label": 1, "p_yes": 0.9219217896461487, "predicted": 1}, {"index": 109, "id": "strategyqa_grounded/test/141", "label": 0, "p_yes": 0.7154240012168884, "predicted": 1}, {"index": 110, "id": "strategyqa_grounded/test/368", "label": 0, "p_yes": 0.29421496391296387, "predicted": 0}, {"index": 111, "id": "strategyqa_grounded/test/481", "label": 0, "p_yes": 0.562176525592804, "predicted": 1}, {"index": 112, "id": "strategyqa_grounded/test/142", "label": 1, "p_yes": 0.5467381477355957, "predicted": 1}, {"index": 113, "id": "strategyqa_grounded/test/544", "label": 1, "p_yes": 0.41111084818840027, "predicted": 0}, {"index": 114, "id": "strategyqa_grounded/test/377", "label": 0, "p_yes": 0.22815649211406708, "predicted": 0}, {"index": 115, "id": "strategyqa_grounded/test/480", "label": 0, "p_yes": 0.7690802216529846, "predicted": 1}, {"index": 116, "id": "strategyqa_grounded/test/601", "label": 0, "p_yes": 0.8740772008895874, "predicted": 1}, {"index": 117, "id": "strategyqa_grounded/test/237", "label": 1, "p_yes": 0.9648551344871521, "predicted": 1}, {"index": 118, "id": "strategyqa_grounded/test/643", "label": 1, "p_yes": 0.8933094143867493, "predicted": 1}, {"index": 119, "id": "strategyqa_grounded/test/521", "label": 0, "p_yes": 0.6187804341316223, "predicted": 1}, {"index": 120, "id": "strategyqa_grounded/test/621", "label": 1, "p_yes": 0.7185943722724915, "predicted": 1}, {"index": 121, "id": "strategyqa_grounded/test/426", "label": 1, "p_yes": 0.41111084818840027, "predicted": 0}, {"index": 122, "id": "strategyqa_grounded/test/162", "label": 0, "p_yes": 0.5467381477355957, "predicted": 1}, {"index": 123, "id": "strategyqa_grounded/test/151", "label": 0, "p_yes": 0.8438951373100281, "predicted": 1}, {"index": 124, "id": "strategyqa_grounded/test/181", "label": 1, "p_yes": 0.6442251205444336, "predicted": 1}, {"index": 125, "id": "strategyqa_grounded/test/71", "label": 1, "p_yes": 0.5506073832511902, "predicted": 1}, {"index": 126, "id": "strategyqa_grounded/test/63", "label": 0, "p_yes": 0.3486451506614685, "predicted": 0}, {"index": 127, "id": "strategyqa_grounded/test/275", "label": 0, "p_yes": 0.5389832258224487, "predicted": 1}]

base-valid-calibrated.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "boolq/validation/138", "label": 1, "p_yes": 0.5186286381043013, "predicted": 1}, {"index": 1, "id": "boolq/validation/429", "label": 0, "p_yes": 0.6814406621596568, "predicted": 1}, {"index": 2, "id": "boolq/validation/93", "label": 1, "p_yes": 0.7605159878283833, "predicted": 1}, {"index": 3, "id": "boolq/validation/430", "label": 0, "p_yes": 0.36894320555116844, "predicted": 0}, {"index": 4, "id": "boolq/validation/121", "label": 1, "p_yes": 0.6010926352712405, "predicted": 1}, {"index": 5, "id": "boolq/validation/295", "label": 0, "p_yes": 0.382933869486939, "predicted": 0}, {"index": 6, "id": "boolq/validation/108", "label": 1, "p_yes": 0.7891014609827149, "predicted": 1}, {"index": 7, "id": "boolq/validation/91", "label": 1, "p_yes": 0.8292667636858398, "predicted": 1}, {"index": 8, "id": "boolq/validation/246", "label": 1, "p_yes": 0.832409921804228, "predicted": 1}, {"index": 9, "id": "boolq/validation/349", "label": 0, "p_yes": 0.5, "predicted": 1}, {"index": 10, "id": "boolq/validation/346", "label": 0, "p_yes": 0.6894768517668138, "predicted": 1}, {"index": 11, "id": "boolq/validation/416", "label": 0, "p_yes": 0.39890738104340606, "predicted": 0}, {"index": 12, "id": "boolq/validation/136", "label": 1, "p_yes": 0.808268308440225, "predicted": 1}, {"index": 13, "id": "boolq/validation/4", "label": 1, "p_yes": 0.6749326594367204, "predicted": 1}, {"index": 14, "id": "boolq/validation/418", "label": 1, "p_yes": 0.808268308440225, "predicted": 1}, {"index": 15, "id": "boolq/validation/175", "label": 1, "p_yes": 0.4832327510910743, "predicted": 0}, {"index": 16, "id": "boolq/validation/461", "label": 1, "p_yes": 0.34836995184008646, "predicted": 0}, {"index": 17, "id": "boolq/validation/234", "label": 0, "p_yes": 0.4906824457713713, "predicted": 0}, {"index": 18, "id": "boolq/validation/72", "label": 1, "p_yes": 0.6046622463449963, "predicted": 1}, {"index": 19, "id": "boolq/validation/333", "label": 1, "p_yes": 0.8024253060555195, "predicted": 1}, {"index": 20, "id": "boolq/validation/240", "label": 0, "p_yes": 0.41872348901168055, "predicted": 0}, {"index": 21, "id": "boolq/validation/0", "label": 1, "p_yes": 0.522349809484464, "predicted": 1}, {"index": 22, "id": "boolq/validation/173", "label": 1, "p_yes": 0.6258346606617384, "predicted": 1}, {"index": 23, "id": "boolq/validation/469", "label": 1, "p_yes": 0.9275451311137243, "predicted": 1}, {"index": 24, "id": "boolq/validation/452", "label": 0, "p_yes": 0.5409103027301156, "predicted": 1}, {"index": 25, "id": "boolq/validation/103", "label": 1, "p_yes": 0.9314520090974958, "predicted": 1}, {"index": 26, "id": "boolq/validation/313", "label": 1, "p_yes": 0.5812765265329487, "predicted": 1}, {"index": 27, "id": "boolq/validation/251", "label": 0, "p_yes": 0.3776632056299364, "predicted": 0}, {"index": 28, "id": "boolq/validation/152", "label": 1, "p_yes": 0.4683589989770788, "predicted": 0}, {"index": 29, "id": "boolq/validation/384", "label": 1, "p_yes": 0.4169101140135326, "predicted": 0}, {"index": 30, "id": "boolq/validation/417", "label": 1, "p_yes": 0.8708454649056171, "predicted": 1}, {"index": 31, "id": "boolq/validation/33", "label": 0, "p_yes": 0.6170661134240163, "predicted": 1}, {"index": 32, "id": "boolq/validation/133", "label": 0, "p_yes": 0.4006962582324085, "predicted": 0}, {"index": 33, "id": "boolq/validation/456", "label": 1, "p_yes": 0.44248026095739296, "predicted": 0}, {"index": 34, "id": "boolq/validation/435", "label": 0, "p_yes": 0.703656044191075, "predicted": 1}, {"index": 35, "id": "boolq/validation/275", "label": 1, "p_yes": 0.8896697516706814, "predicted": 1}, {"index": 36, "id": "boolq/validation/218", "label": 0, "p_yes": 0.40967954455911243, "predicted": 0}, {"index": 37, "id": "boolq/validation/161", "label": 1, "p_yes": 0.7352300183547612, "predicted": 1}, {"index": 38, "id": "boolq/validation/241", "label": 1, "p_yes": 0.8139784602231099, "predicted": 1}, {"index": 39, "id": "boolq/validation/30", "label": 1, "p_yes": 0.6942447223823844, "predicted": 1}, {"index": 40, "id": "boolq/validation/311", "label": 1, "p_yes": 0.8521926354821625, "predicted": 1}, {"index": 41, "id": "boolq/validation/14", "label": 0, "p_yes": 0.7964490689755397, "predicted": 1}, {"index": 42, "id": "boolq/validation/6", "label": 1, "p_yes": 0.5993037255308085, "predicted": 1}, {"index": 43, "id": "boolq/validation/92", "label": 0, "p_yes": 0.31533153282904525, "predicted": 0}, {"index": 44, "id": "boolq/validation/180", "label": 0, "p_yes": 0.5721776152402894, "predicted": 1}, {"index": 45, "id": "boolq/validation/9", "label": 1, "p_yes": 0.7128975391208608, "predicted": 1}, {"index": 46, "id": "boolq/validation/76", "label": 1, "p_yes": 0.5334968232870905, "predicted": 1}, {"index": 47, "id": "boolq/validation/222", "label": 1, "p_yes": 0.56852398828492, "predicted": 1}, {"index": 48, "id": "boolq/validation/98", "label": 0, "p_yes": 0.7976550980324861, "predicted": 1}, {"index": 49, "id": "boolq/validation/5", "label": 1, "p_yes": 0.4609414901749551, "predicted": 0}, {"index": 50, "id": "boolq/validation/370", "label": 1, "p_yes": 0.9474753247335104, "predicted": 1}, {"index": 51, "id": "boolq/validation/396", "label": 0, "p_yes": 0.5885163617640581, "predicted": 1}, {"index": 52, "id": "boolq/validation/31", "label": 1, "p_yes": 0.3332980043676569, "predicted": 0}, {"index": 53, "id": "boolq/validation/39", "label": 1, "p_yes": 0.4888195138351886, "predicted": 0}, {"index": 54, "id": "boolq/validation/2", "label": 0, "p_yes": 0.5037273739039162, "predicted": 1}, {"index": 55, "id": "boolq/validation/306", "label": 1, "p_yes": 0.7395606951849348, "predicted": 1}, {"index": 56, "id": "boolq/validation/324", "label": 1, "p_yes": 0.36375158357989, "predicted": 0}, {"index": 57, "id": "boolq/validation/352", "label": 1, "p_yes": 0.5630296172224917, "predicted": 1}, {"index": 58, "id": "boolq/validation/341", "label": 1, "p_yes": 0.44800432924247197, "predicted": 0}, {"index": 59, "id": "boolq/validation/470", "label": 1, "p_yes": 0.5648628515274343, "predicted": 1}, {"index": 60, "id": "boolq/validation/431", "label": 0, "p_yes": 0.27953427353232163, "predicted": 0}, {"index": 61, "id": "boolq/validation/64", "label": 1, "p_yes": 0.2855790898985569, "predicted": 0}, {"index": 62, "id": "boolq/validation/13", "label": 1, "p_yes": 0.8586458961649694, "predicted": 1}, {"index": 63, "id": "boolq/validation/433", "label": 1, "p_yes": 0.5074543611029115, "predicted": 1}, {"index": 64, "id": "boolq/validation/48", "label": 1, "p_yes": 0.7424222429896522, "predicted": 1}, {"index": 65, "id": "boolq/validation/119", "label": 0, "p_yes": 0.4169101140135326, "predicted": 0}, {"index": 66, "id": "boolq/validation/272", "label": 1, "p_yes": 0.9073079802433219, "predicted": 1}, {"index": 67, "id": "boolq/validation/131", "label": 0, "p_yes": 0.25332416890607207, "predicted": 0}, {"index": 68, "id": "boolq/validation/282", "label": 1, "p_yes": 0.7853555065132322, "predicted": 1}, {"index": 69, "id": "boolq/validation/434", "label": 1, "p_yes": 0.703656044191075, "predicted": 1}, {"index": 70, "id": "boolq/validation/21", "label": 1, "p_yes": 0.522349809484464, "predicted": 1}, {"index": 71, "id": "boolq/validation/111", "label": 0, "p_yes": 0.658368388317641, "predicted": 1}, {"index": 72, "id": "boolq/validation/337", "label": 1, "p_yes": 0.9416224458571321, "predicted": 1}, {"index": 73, "id": "boolq/validation/283", "label": 0, "p_yes": 0.7366787071472516, "predicted": 1}, {"index": 74, "id": "boolq/validation/379", "label": 1, "p_yes": 0.47765020483131687, "predicted": 0}, {"index": 75, "id": "boolq/validation/357", "label": 1, "p_yes": 0.7381222832851778, "predicted": 1}, {"index": 76, "id": "boolq/validation/280", "label": 1, "p_yes": 0.8559093017345832, "predicted": 1}, {"index": 77, "id": "boolq/validation/274", "label": 0, "p_yes": 0.5464587638774934, "predicted": 1}, {"index": 78, "id": "boolq/validation/424", "label": 0, "p_yes": 0.6431213288125944, "predicted": 1}, {"index": 79, "id": "boolq/validation/26", "label": 0, "p_yes": 0.49440905456569245, "predicted": 0}, {"index": 80, "id": "boolq/validation/297", "label": 0, "p_yes": 0.3585915161165846, "predicted": 0}, {"index": 81, "id": "boolq/validation/390", "label": 0, "p_yes": 0.31212144555905025, "predicted": 0}, {"index": 82, "id": "boolq/validation/315", "label": 1, "p_yes": 0.6028788089313739, "predicted": 1}, {"index": 83, "id": "boolq/validation/281", "label": 1, "p_yes": 0.9174300774298159, "predicted": 1}, {"index": 84, "id": "boolq/validation/87", "label": 1, "p_yes": 0.6765661087959837, "predicted": 1}, {"index": 85, "id": "boolq/validation/459", "label": 1, "p_yes": 0.911001934643148, "predicted": 1}, {"index": 86, "id": "boolq/validation/179", "label": 0, "p_yes": 0.6667019647771245, "predicted": 1}, {"index": 87, "id": "boolq/validation/226", "label": 1, "p_yes": 0.5703517479200776, "predicted": 1}, {"index": 88, "id": "boolq/validation/468", "label": 1, "p_yes": 0.7828314708199889, "predicted": 1}, {"index": 89, "id": "boolq/validation/147", "label": 1, "p_yes": 0.9386867452941698, "predicted": 1}, {"index": 90, "id": "boolq/validation/88", "label": 1, "p_yes": 0.5776432321734629, "predicted": 1}, {"index": 91, "id": "boolq/validation/174", "label": 1, "p_yes": 0.6099957248964116, "predicted": 1}, {"index": 92, "id": "boolq/validation/294", "label": 1, "p_yes": 0.8012407686532451, "predicted": 1}, {"index": 93, "id": "boolq/validation/141", "label": 1, "p_yes": 0.8595480583157222, "predicted": 1}, {"index": 94, "id": "boolq/validation/22", "label": 1, "p_yes": 0.41509898726201794, "predicted": 0}, {"index": 95, "id": "boolq/validation/79", "label": 0, "p_yes": 0.38117385957428856, "predicted": 0}, {"index": 96, "id": "boolq/validation/211", "label": 1, "p_yes": 0.48509461924620145, "predicted": 0}, {"index": 97, "id": "boolq/validation/183", "label": 0, "p_yes": 0.34836995184008646, "predicted": 0}, {"index": 98, "id": "boolq/validation/268", "label": 0, "p_yes": 0.6153030337332264, "predicted": 1}, {"index": 99, "id": "boolq/validation/186", "label": 0, "p_yes": 0.32506730855464455, "predicted": 0}, {"index": 100, "id": "boolq/validation/307", "label": 1, "p_yes": 0.4461614560274288, "predicted": 0}, {"index": 101, "id": "boolq/validation/236", "label": 1, "p_yes": 0.5483057211681001, "predicted": 1}, {"index": 102, "id": "boolq/validation/376", "label": 0, "p_yes": 0.31533153282904525, "predicted": 0}, {"index": 103, "id": "boolq/validation/361", "label": 0, "p_yes": 0.5630296172224917, "predicted": 1}, {"index": 104, "id": "boolq/validation/422", "label": 1, "p_yes": 0.6667019647771245, "predicted": 1}, {"index": 105, "id": "boolq/validation/338", "label": 1, "p_yes": 0.9260275024617217, "predicted": 1}, {"index": 106, "id": "boolq/validation/125", "label": 0, "p_yes": 0.7337762670301476, "predicted": 1}, {"index": 107, "id": "boolq/validation/400", "label": 1, "p_yes": 0.31372425865786907, "predicted": 0}, {"index": 108, "id": "boolq/validation/419", "label": 1, "p_yes": 0.6396920112712661, "predicted": 1}, {"index": 109, "id": "boolq/validation/320", "label": 1, "p_yes": 0.5830899015933687, "predicted": 1}, {"index": 110, "id": "boolq/validation/464", "label": 0, "p_yes": 0.7128975391208608, "predicted": 1}, {"index": 111, "id": "boolq/validation/195", "label": 1, "p_yes": 0.8691592480411859, "predicted": 1}, {"index": 112, "id": "boolq/validation/153", "label": 1, "p_yes": 0.9323985299669949, "predicted": 1}, {"index": 113, "id": "boolq/validation/371", "label": 1, "p_yes": 0.4333056934004816, "predicted": 0}, {"index": 114, "id": "boolq/validation/167", "label": 1, "p_yes": 0.6830567614006626, "predicted": 1}, {"index": 115, "id": "boolq/validation/163", "label": 1, "p_yes": 0.9510628057408029, "predicted": 1}, {"index": 116, "id": "boolq/validation/134", "label": 1, "p_yes": 0.8844425045505415, "predicted": 1}, {"index": 117, "id": "boolq/validation/109", "label": 0, "p_yes": 0.3724212982697775, "predicted": 0}, {"index": 118, "id": "boolq/validation/245", "label": 1, "p_yes": 0.4461614560274288, "predicted": 0}, {"index": 119, "id": "boolq/validation/10", "label": 1, "p_yes": 0.6926600014771807, "predicted": 1}, {"index": 120, "id": "boolq/validation/230", "label": 1, "p_yes": 0.2197139844197513, "predicted": 0}, {"index": 121, "id": "boolq/validation/224", "label": 0, "p_yes": 0.8139784602231099, "predicted": 1}, {"index": 122, "id": "boolq/validation/25", "label": 0, "p_yes": 0.9079332478630541, "predicted": 1}, {"index": 123, "id": "boolq/validation/407", "label": 0, "p_yes": 0.43880535972129614, "predicted": 0}, {"index": 124, "id": "boolq/validation/321", "label": 0, "p_yes": 0.3971211910686261, "predicted": 0}, {"index": 125, "id": "boolq/validation/391", "label": 1, "p_yes": 0.7480834055738356, "predicted": 1}, {"index": 126, "id": "boolq/validation/23", "label": 0, "p_yes": 0.6117678149955288, "predicted": 1}, {"index": 127, "id": "boolq/validation/144", "label": 1, "p_yes": 0.9008453163884766, "predicted": 1}]

base-valid-predictions.json

@@ -0,0 +1 @@
+[{"index": 0, "id": "boolq/validation/138", "label": 1, "p_yes": 0.5389832258224487, "predicted": 1}, {"index": 1, "id": "boolq/validation/429", "label": 0, "p_yes": 0.8311430811882019, "predicted": 1}, {"index": 2, "id": "boolq/validation/93", "label": 1, "p_yes": 0.9184802770614624, "predicted": 1}, {"index": 3, "id": "boolq/validation/430", "label": 0, "p_yes": 0.24508501589298248, "predicted": 0}, {"index": 4, "id": "boolq/validation/121", "label": 1, "p_yes": 0.7025300860404968, "predicted": 1}, {"index": 5, "id": "boolq/validation/295", "label": 0, "p_yes": 0.2689414322376251, "predicted": 0}, {"index": 6, "id": "boolq/validation/108", "label": 1, "p_yes": 0.9407896995544434, "predicted": 1}, {"index": 7, "id": "boolq/validation/91", "label": 1, "p_yes": 0.9648551344871521, "predicted": 1}, {"index": 8, "id": "boolq/validation/246", "label": 1, "p_yes": 0.9664104580879211, "predicted": 1}, {"index": 9, "id": "boolq/validation/349", "label": 0, "p_yes": 0.5, "predicted": 1}, {"index": 10, "id": "boolq/validation/346", "label": 0, "p_yes": 0.8418256044387817, "predicted": 1}, {"index": 11, "id": "boolq/validation/416", "label": 0, "p_yes": 0.29746994376182556, "predicted": 0}, {"index": 12, "id": "boolq/validation/136", "label": 1, "p_yes": 0.9532750844955444, "predicted": 1}, {"index": 13, "id": "boolq/validation/4", "label": 1, "p_yes": 0.8221890926361084, "predicted": 1}, {"index": 14, "id": "boolq/validation/418", "label": 1, "p_yes": 0.9532750844955444, "predicted": 1}, {"index": 15, "id": "boolq/validation/175", "label": 1, "p_yes": 0.46490156650543213, "predicted": 0}, {"index": 16, "id": "boolq/validation/461", "label": 1, "p_yes": 0.2120688110589981, "predicted": 0}, {"index": 17, "id": "boolq/validation/234", "label": 0, "p_yes": 0.4804786741733551, "predicted": 0}, {"index": 18, "id": "boolq/validation/72", "label": 1, "p_yes": 0.709019124507904, "predicted": 1}, {"index": 19, "id": "boolq/validation/333", "label": 1, "p_yes": 0.949669361114502, "predicted": 1}, {"index": 20, "id": "boolq/validation/240", "label": 0, "p_yes": 0.3345894515514374, "predicted": 0}, {"index": 21, "id": "boolq/validation/0", "label": 1, "p_yes": 0.5467381477355957, "predicted": 1}, {"index": 22, "id": "boolq/validation/173", "label": 1, "p_yes": 0.7461389899253845, "predicted": 1}, {"index": 23, "id": "boolq/validation/469", "label": 1, "p_yes": 0.9952448010444641, "predicted": 1}, {"index": 24, "id": "boolq/validation/452", "label": 0, "p_yes": 0.5851011276245117, "predicted": 1}, {"index": 25, "id": "boolq/validation/103", "label": 1, "p_yes": 0.9958011507987976, "predicted": 1}, {"index": 26, "id": "boolq/validation/313", "label": 1, "p_yes": 0.665410578250885, "predicted": 1}, {"index": 27, "id": "boolq/validation/251", "label": 0, "p_yes": 0.2598256468772888, "predicted": 0}, {"index": 28, "id": "boolq/validation/152", "label": 1, "p_yes": 0.4339814782142639, "predicted": 0}, {"index": 29, "id": "boolq/validation/384", "label": 1, "p_yes": 0.33111974596977234, "predicted": 0}, {"index": 30, "id": "boolq/validation/417", "label": 1, "p_yes": 0.9820137619972229, "predicted": 1}, {"index": 31, "id": "boolq/validation/33", "label": 0, "p_yes": 0.7310585379600525, "predicted": 1}, {"index": 32, "id": "boolq/validation/133", "label": 0, "p_yes": 0.30074557662010193, "predicted": 0}, {"index": 33, "id": "boolq/validation/456", "label": 1, "p_yes": 0.3812195658683777, "predicted": 0}, {"index": 34, "id": "boolq/validation/435", "label": 0, "p_yes": 0.859663724899292, "predicted": 1}, {"index": 35, "id": "boolq/validation/275", "label": 1, "p_yes": 0.9875683188438416, "predicted": 1}, {"index": 36, "id": "boolq/validation/218", "label": 0, "p_yes": 0.31742629408836365, "predicted": 0}, {"index": 37, "id": "boolq/validation/161", "label": 1, "p_yes": 0.894789457321167, "predicted": 1}, {"index": 38, "id": "boolq/validation/241", "label": 1, "p_yes": 0.9566342234611511, "predicted": 1}, {"index": 39, "id": "boolq/validation/30", "label": 1, "p_yes": 0.8479677438735962, "predicted": 1}, {"index": 40, "id": "boolq/validation/311", "label": 1, "p_yes": 0.9752018451690674, "predicted": 1}, {"index": 41, "id": "boolq/validation/14", "label": 0, "p_yes": 0.9458011984825134, "predicted": 1}, {"index": 42, "id": "boolq/validation/6", "label": 1, "p_yes": 0.6992543935775757, "predicted": 1}, {"index": 43, "id": "boolq/validation/92", "label": 0, "p_yes": 0.16451644897460938, "predicted": 0}, {"index": 44, "id": "boolq/validation/180", "label": 0, "p_yes": 0.6477982401847839, "predicted": 1}, {"index": 45, "id": "boolq/validation/9", "label": 1, "p_yes": 0.8705973029136658, "predicted": 1}, {"index": 46, "id": "boolq/validation/76", "label": 1, "p_yes": 0.5698526501655579, "predicted": 1}, {"index": 47, "id": "boolq/validation/222", "label": 1, "p_yes": 0.6406359076499939, "predicted": 1}, {"index": 48, "id": "boolq/validation/98", "label": 0, "p_yes": 0.9465966820716858, "predicted": 1}, {"index": 49, "id": "boolq/validation/5", "label": 1, "p_yes": 0.41869691014289856, "predicted": 0}, {"index": 50, "id": "boolq/validation/370", "label": 1, "p_yes": 0.9976769089698792, "predicted": 1}, {"index": 51, "id": "boolq/validation/396", "label": 0, "p_yes": 0.6791787147521973, "predicted": 1}, {"index": 52, "id": "boolq/validation/31", "label": 1, "p_yes": 0.18952108919620514, "predicted": 0}, {"index": 53, "id": "boolq/validation/39", "label": 1, "p_yes": 0.4765796661376953, "predicted": 0}, {"index": 54, "id": "boolq/validation/2", "label": 0, "p_yes": 0.5078118443489075, "predicted": 1}, {"index": 55, "id": "boolq/validation/306", "label": 1, "p_yes": 0.8991213440895081, "predicted": 1}, {"index": 56, "id": "boolq/validation/324", "label": 1, "p_yes": 0.23651622235774994, "predicted": 0}, {"index": 57, "id": "boolq/validation/352", "label": 1, "p_yes": 0.6297746300697327, "predicted": 1}, {"index": 58, "id": "boolq/validation/341", "label": 1, "p_yes": 0.3923368453979492, "predicted": 0}, {"index": 59, "id": "boolq/validation/470", "label": 1, "p_yes": 0.6334102749824524, "predicted": 1}, {"index": 60, "id": "boolq/validation/431", "label": 0, "p_yes": 0.1208532303571701, "predicted": 0}, {"index": 61, "id": "boolq/validation/64", "label": 1, "p_yes": 0.1276526302099228, "predicted": 0}, {"index": 62, "id": "boolq/validation/13", "label": 1, "p_yes": 0.9777138829231262, "predicted": 1}, {"index": 63, "id": "boolq/validation/433", "label": 1, "p_yes": 0.5156199336051941, "predicted": 1}, {"index": 64, "id": "boolq/validation/48", "label": 1, "p_yes": 0.9019206762313843, "predicted": 1}, {"index": 65, "id": "boolq/validation/119", "label": 0, "p_yes": 0.33111974596977234, "predicted": 0}, {"index": 66, "id": "boolq/validation/272", "label": 1, "p_yes": 0.9916841983795166, "predicted": 1}, {"index": 67, "id": "boolq/validation/131", "label": 0, "p_yes": 0.09401018917560577, "predicted": 0}, {"index": 68, "id": "boolq/validation/282", "label": 1, "p_yes": 0.9381240010261536, "predicted": 1}, {"index": 69, "id": "boolq/validation/434", "label": 1, "p_yes": 0.859663724899292, "predicted": 1}, {"index": 70, "id": "boolq/validation/21", "label": 1, "p_yes": 0.5467381477355957, "predicted": 1}, {"index": 71, "id": "boolq/validation/111", "label": 0, "p_yes": 0.7981867790222168, "predicted": 1}, {"index": 72, "id": "boolq/validation/337", "label": 1, "p_yes": 0.997065007686615, "predicted": 1}, {"index": 73, "id": "boolq/validation/283", "label": 0, "p_yes": 0.896251380443573, "predicted": 1}, {"index": 74, "id": "boolq/validation/379", "label": 1, "p_yes": 0.4532618820667267, "predicted": 0}, {"index": 75, "id": "boolq/validation/357", "label": 1, "p_yes": 0.8976953029632568, "predicted": 1}, {"index": 76, "id": "boolq/validation/280", "label": 1, "p_yes": 0.976669192314148, "predicted": 1}, {"index": 77, "id": "boolq/validation/274", "label": 0, "p_yes": 0.596433162689209, "predicted": 1}, {"index": 78, "id": "boolq/validation/424", "label": 0, "p_yes": 0.7745833992958069, "predicted": 1}, {"index": 79, "id": "boolq/validation/26", "label": 0, "p_yes": 0.4882833957672119, "predicted": 0}, {"index": 80, "id": "boolq/validation/297", "label": 0, "p_yes": 0.22815649211406708, "predicted": 0}, {"index": 81, "id": "boolq/validation/390", "label": 0, "p_yes": 0.16026602685451508, "predicted": 0}, {"index": 82, "id": "boolq/validation/315", "label": 1, "p_yes": 0.7057850360870361, "predicted": 1}, {"index": 83, "id": "boolq/validation/281", "label": 1, "p_yes": 0.9936116337776184, "predicted": 1}, {"index": 84, "id": "boolq/validation/87", "label": 1, "p_yes": 0.8244618773460388, "predicted": 1}, {"index": 85, "id": "boolq/validation/459", "label": 1, "p_yes": 0.9924227595329285, "predicted": 1}, {"index": 86, "id": "boolq/validation/179", "label": 0, "p_yes": 0.8104788661003113, "predicted": 1}, {"index": 87, "id": "boolq/validation/226", "label": 1, "p_yes": 0.6442251205444336, "predicted": 1}, {"index": 88, "id": "boolq/validation/468", "label": 1, "p_yes": 0.9362850189208984, "predicted": 1}, {"index": 89, "id": "boolq/validation/147", "label": 1, "p_yes": 0.9967268705368042, "predicted": 1}, {"index": 90, "id": "boolq/validation/88", "label": 1, "p_yes": 0.6584175229072571, "predicted": 1}, {"index": 91, "id": "boolq/validation/174", "label": 1, "p_yes": 0.7185943722724915, "predicted": 1}, {"index": 92, "id": "boolq/validation/294", "label": 1, "p_yes": 0.9489172697067261, "predicted": 1}, {"index": 93, "id": "boolq/validation/141", "label": 1, "p_yes": 0.9780517220497131, "predicted": 1}, {"index": 94, "id": "boolq/validation/22", "label": 1, "p_yes": 0.32766830921173096, "predicted": 0}, {"index": 95, "id": "boolq/validation/79", "label": 0, "p_yes": 0.2658804953098297, "predicted": 0}, {"index": 96, "id": "boolq/validation/211", "label": 1, "p_yes": 0.4687906503677368, "predicted": 0}, {"index": 97, "id": "boolq/validation/183", "label": 0, "p_yes": 0.2120688110589981, "predicted": 0}, {"index": 98, "id": "boolq/validation/268", "label": 0, "p_yes": 0.7279754281044006, "predicted": 1}, {"index": 99, "id": "boolq/validation/186", "label": 0, "p_yes": 0.17781086266040802, "predicted": 0}, {"index": 100, "id": "boolq/validation/307", "label": 1, "p_yes": 0.38861799240112305, "predicted": 0}, {"index": 101, "id": "boolq/validation/236", "label": 1, "p_yes": 0.6001883149147034, "predicted": 1}, {"index": 102, "id": "boolq/validation/376", "label": 0, "p_yes": 0.16451644897460938, "predicted": 0}, {"index": 103, "id": "boolq/validation/361", "label": 0, "p_yes": 0.6297746300697327, "predicted": 1}, {"index": 104, "id": "boolq/validation/422", "label": 1, "p_yes": 0.8104788661003113, "predicted": 1}, {"index": 105, "id": "boolq/validation/338", "label": 1, "p_yes": 0.9950177073478699, "predicted": 1}, {"index": 106, "id": "boolq/validation/125", "label": 0, "p_yes": 0.8933094143867493, "predicted": 1}, {"index": 107, "id": "boolq/validation/400", "label": 1, "p_yes": 0.16238002479076385, "predicted": 0}, {"index": 108, "id": "boolq/validation/419", "label": 1, "p_yes": 0.7690802216529846, "predicted": 1}, {"index": 109, "id": "boolq/validation/320", "label": 1, "p_yes": 0.66888028383255, "predicted": 1}, {"index": 110, "id": "boolq/validation/464", "label": 0, "p_yes": 0.8705973029136658, "predicted": 1}, {"index": 111, "id": "boolq/validation/195", "label": 1, "p_yes": 0.9814534187316895, "predicted": 1}, {"index": 112, "id": "boolq/validation/153", "label": 1, "p_yes": 0.9959298968315125, "predicted": 1}, {"index": 113, "id": "boolq/validation/371", "label": 1, "p_yes": 0.36296921968460083, "predicted": 0}, {"index": 114, "id": "boolq/validation/167", "label": 1, "p_yes": 0.8333246111869812, "predicted": 1}, {"index": 115, "id": "boolq/validation/163", "label": 1, "p_yes": 0.9980121850967407, "predicted": 1}, {"index": 116, "id": "boolq/validation/134", "label": 1, "p_yes": 0.986151397228241, "predicted": 1}, {"index": 117, "id": "boolq/validation/109", "label": 0, "p_yes": 0.25091278553009033, "predicted": 0}, {"index": 118, "id": "boolq/validation/245", "label": 1, "p_yes": 0.38861799240112305, "predicted": 0}, {"index": 119, "id": "boolq/validation/10", "label": 1, "p_yes": 0.8459424376487732, "predicted": 1}, {"index": 120, "id": "boolq/validation/230", "label": 1, "p_yes": 0.06560482829809189, "predicted": 0}, {"index": 121, "id": "boolq/validation/224", "label": 0, "p_yes": 0.9566342234611511, "predicted": 1}, {"index": 122, "id": "boolq/validation/25", "label": 0, "p_yes": 0.9918121099472046, "predicted": 1}, {"index": 123, "id": "boolq/validation/407", "label": 0, "p_yes": 0.37387582659721375, "predicted": 0}, {"index": 124, "id": "boolq/validation/321", "label": 0, "p_yes": 0.29421496391296387, "predicted": 0}, {"index": 125, "id": "boolq/validation/391", "label": 1, "p_yes": 0.907312273979187, "predicted": 1}, {"index": 126, "id": "boolq/validation/23", "label": 0, "p_yes": 0.7217432260513306, "predicted": 1}, {"index": 127, "id": "boolq/validation/144", "label": 1, "p_yes": 0.9902915358543396, "predicted": 1}]

calibrate.py

@@ -0,0 +1,81 @@
+"""Fit validation-only temperatures and score the saved cloud predictions."""
+
+import json
+import math
+import random
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parent
+PRED = ROOT / "output" / "typed_decision_run"
+if not PRED.exists():
+    PRED = ROOT
+
+
+def rows(phase, split):
+    return json.loads((PRED / f"{phase}-{split}-predictions.json").read_text())
+
+
+def adjusted(probability, temperature):
+    p = min(max(probability, 1e-6), 1 - 1e-6)
+    logit = math.log(p / (1 - p)) / temperature
+    return 1 / (1 + math.exp(-logit))
+
+
+def scores(data):
+    n = len(data)
+    brier = sum((r["p_yes"] - r["label"]) ** 2 for r in data) / n
+    nll = -sum(r["label"] * math.log(max(r["p_yes"], 1e-12)) +
+               (1 - r["label"]) * math.log(max(1 - r["p_yes"], 1e-12))
+               for r in data) / n
+    accuracy = sum((r["p_yes"] >= .5) == r["label"] for r in data) / n
+    ece = 0.0
+    for index in range(10):
+        bucket = [r for r in data if index / 10 <= max(r["p_yes"], 1-r["p_yes"]) < (index+1)/10
+                  or (index == 9 and max(r["p_yes"], 1-r["p_yes"]) == 1)]
+        if bucket:
+            conf = sum(max(r["p_yes"], 1-r["p_yes"]) for r in bucket) / len(bucket)
+            acc = sum((r["p_yes"] >= .5) == r["label"] for r in bucket) / len(bucket)
+            ece += len(bucket) / n * abs(acc - conf)
+    return {"n": n, "accuracy": accuracy, "brier": brier, "nll": nll, "ece_10bin": ece}
+
+
+temps = {}
+for phase in ("base", "adapted"):
+    validation = rows(phase, "valid")
+    temperatures = [math.exp(-2 + 4 * i / 400) for i in range(401)]
+    temps[phase] = min(temperatures, key=lambda t: scores([
+        {**r, "p_yes": adjusted(r["p_yes"], t)} for r in validation
+    ])["nll"])
+
+receipt = {
+    "method": "temperature scaling on binary Yes/No log odds",
+    "fit": "minimize NLL on the 128 validation records only",
+    "analysis_note": "Temperature scaling was added after inspection of the raw test metrics; treat this as exploratory post-hoc analysis.",
+    "temperature_grid": "exp(-2 + 4*i/400), i=0..400",
+    "temperatures": temps,
+    "metrics": {},
+    "paired_bootstrap": {},
+}
+for split in ("valid", "confirm", "ood"):
+    for phase in ("base", "adapted"):
+        raw = rows(phase, split)
+        calibrated = [{**r, "p_yes": adjusted(r["p_yes"], temps[phase])} for r in raw]
+        (ROOT / f"{phase}-{split}-calibrated.json").write_text(json.dumps(calibrated) + "\n")
+        receipt["metrics"][f"{phase}_{split}"] = scores(calibrated)
+    before = json.loads((ROOT / f"base-{split}-calibrated.json").read_text())
+    after = json.loads((ROOT / f"adapted-{split}-calibrated.json").read_text())
+    assert [(r["id"], r["label"]) for r in before] == [(r["id"], r["label"]) for r in after]
+    rng = random.Random(20260930 + len(before))
+    diffs = []
+    for _ in range(10000):
+        sample = [rng.randrange(len(before)) for _ in before]
+        diffs.append(sum((before[i]["p_yes"] - before[i]["label"]) ** 2 -
+                         (after[i]["p_yes"] - after[i]["label"]) ** 2 for i in sample) / len(before))
+    diffs.sort()
+    receipt["paired_bootstrap"][split] = {
+        "brier_reduction": scores(before)["brier"] - scores(after)["brier"],
+        "brier_reduction_95ci": [diffs[249], diffs[9749]],
+    }
+(ROOT / "calibration_receipt.json").write_text(json.dumps(receipt, indent=2) + "\n")
+print(json.dumps(receipt, indent=2))

calibration_receipt.json

@@ -0,0 +1,77 @@
+{
+  "method": "temperature scaling on binary Yes/No log odds",
+  "fit": "minimize NLL on the 128 validation records only",
+  "analysis_note": "Temperature scaling was added after inspection of the raw test metrics; treat this as exploratory post-hoc analysis.",
+  "temperature_grid": "exp(-2 + 4*i/400), i=0..400",
+  "temperatures": {
+    "base": 2.095935514494365,
+    "adapted": 4.437095519003665
+  },
+  "metrics": {
+    "base_valid": {
+      "n": 128,
+      "accuracy": 0.671875,
+      "brier": 0.20041497808051684,
+      "nll": 0.5802194755124913,
+      "ece_10bin": 0.05407964871395032
+    },
+    "adapted_valid": {
+      "n": 128,
+      "accuracy": 0.7890625,
+      "brier": 0.1463823513144757,
+      "nll": 0.45222957517522366,
+      "ece_10bin": 0.07920693181199165
+    },
+    "base_confirm": {
+      "n": 256,
+      "accuracy": 0.7109375,
+      "brier": 0.18975171696816479,
+      "nll": 0.5589378647071684,
+      "ece_10bin": 0.031445599878258755
+    },
+    "adapted_confirm": {
+      "n": 256,
+      "accuracy": 0.7890625,
+      "brier": 0.15775357785778454,
+      "nll": 0.49118689524707676,
+      "ece_10bin": 0.034778029676396295
+    },
+    "base_ood": {
+      "n": 128,
+      "accuracy": 0.5546875,
+      "brier": 0.25680698090948584,
+      "nll": 0.7151088855128506,
+      "ece_10bin": 0.08612346076589825
+    },
+    "adapted_ood": {
+      "n": 128,
+      "accuracy": 0.671875,
+      "brier": 0.2113948137501809,
+      "nll": 0.6152763119587173,
+      "ece_10bin": 0.10670950291353906
+    }
+  },
+  "paired_bootstrap": {
+    "valid": {
+      "brier_reduction": 0.05403262676604115,
+      "brier_reduction_95ci": [
+        0.013239545483773063,
+        0.09353934819131188
+      ]
+    },
+    "confirm": {
+      "brier_reduction": 0.03199813911038024,
+      "brier_reduction_95ci": [
+        0.006979612317650604,
+        0.056981639338143146
+      ]
+    },
+    "ood": {
+      "brier_reduction": 0.045412167159304934,
+      "brier_reduction_95ci": [
+        0.006343379401028662,
+        0.08421776744727122
+      ]
+    }
+  }
+}

excluded-ids.json

@@ -0,0 +1,386 @@
+[
+  "boolq/test/1045",
+  "boolq/test/1049",
+  "boolq/test/1050",
+  "boolq/test/106",
+  "boolq/test/1065",
+  "boolq/test/1069",
+  "boolq/test/107",
+  "boolq/test/1071",
+  "boolq/test/1072",
+  "boolq/test/11",
+  "boolq/test/110",
+  "boolq/test/1103",
+  "boolq/test/1106",
+  "boolq/test/1111",
+  "boolq/test/1114",
+  "boolq/test/1122",
+  "boolq/test/1123",
+  "boolq/test/1124",
+  "boolq/test/113",
+  "boolq/test/1157",
+  "boolq/test/1161",
+  "boolq/test/119",
+  "boolq/test/1190",
+  "boolq/test/1194",
+  "boolq/test/120",
+  "boolq/test/1218",
+  "boolq/test/1224",
+  "boolq/test/1235",
+  "boolq/test/1247",
+  "boolq/test/1253",
+  "boolq/test/1258",
+  "boolq/test/1276",
+  "boolq/test/1285",
+  "boolq/test/1286",
+  "boolq/test/1299",
+  "boolq/test/132",
+  "boolq/test/1329",
+  "boolq/test/1337",
+  "boolq/test/1344",
+  "boolq/test/1346",
+  "boolq/test/1355",
+  "boolq/test/1364",
+  "boolq/test/1371",
+  "boolq/test/1373",
+  "boolq/test/138",
+  "boolq/test/1396",
+  "boolq/test/1400",
+  "boolq/test/1416",
+  "boolq/test/1453",
+  "boolq/test/1458",
+  "boolq/test/1459",
+  "boolq/test/1461",
+  "boolq/test/1464",
+  "boolq/test/1469",
+  "boolq/test/1488",
+  "boolq/test/149",
+  "boolq/test/1497",
+  "boolq/test/15",
+  "boolq/test/1500",
+  "boolq/test/1519",
+  "boolq/test/1520",
+  "boolq/test/1524",
+  "boolq/test/1526",
+  "boolq/test/1532",
+  "boolq/test/1535",
+  "boolq/test/1544",
+  "boolq/test/1561",
+  "boolq/test/1563",
+  "boolq/test/1568",
+  "boolq/test/1572",
+  "boolq/test/1577",
+  "boolq/test/1605",
+  "boolq/test/1612",
+  "boolq/test/1618",
+  "boolq/test/163",
+  "boolq/test/1641",
+  "boolq/test/1647",
+  "boolq/test/1650",
+  "boolq/test/1671",
+  "boolq/test/1683",
+  "boolq/test/1697",
+  "boolq/test/1701",
+  "boolq/test/1708",
+  "boolq/test/1716",
+  "boolq/test/1721",
+  "boolq/test/1723",
+  "boolq/test/173",
+  "boolq/test/1730",
+  "boolq/test/1737",
+  "boolq/test/1740",
+  "boolq/test/1742",
+  "boolq/test/1745",
+  "boolq/test/1756",
+  "boolq/test/1760",
+  "boolq/test/1765",
+  "boolq/test/1769",
+  "boolq/test/1778",
+  "boolq/test/1779",
+  "boolq/test/1787",
+  "boolq/test/1810",
+  "boolq/test/1819",
+  "boolq/test/1821",
+  "boolq/test/1822",
+  "boolq/test/1826",
+  "boolq/test/1838",
+  "boolq/test/1845",
+  "boolq/test/1858",
+  "boolq/test/1875",
+  "boolq/test/1881",
+  "boolq/test/1883",
+  "boolq/test/1892",
+  "boolq/test/19",
+  "boolq/test/1901",
+  "boolq/test/1902",
+  "boolq/test/1903",
+  "boolq/test/1920",
+  "boolq/test/1948",
+  "boolq/test/197",
+  "boolq/test/1976",
+  "boolq/test/1984",
+  "boolq/test/1989",
+  "boolq/test/1994",
+  "boolq/test/1999",
+  "boolq/test/20",
+  "boolq/test/2012",
+  "boolq/test/2021",
+  "boolq/test/2024",
+  "boolq/test/2037",
+  "boolq/test/2049",
+  "boolq/test/2051",
+  "boolq/test/2059",
+  "boolq/test/2069",
+  "boolq/test/207",
+  "boolq/test/2075",
+  "boolq/test/210",
+  "boolq/test/2100",
+  "boolq/test/2102",
+  "boolq/test/2103",
+  "boolq/test/2107",
+  "boolq/test/2115",
+  "boolq/test/2127",
+  "boolq/test/2133",
+  "boolq/test/214",
+  "boolq/test/2141",
+  "boolq/test/2155",
+  "boolq/test/2160",
+  "boolq/test/2163",
+  "boolq/test/2164",
+  "boolq/test/2166",
+  "boolq/test/2184",
+  "boolq/test/2186",
+  "boolq/test/2199",
+  "boolq/test/221",
+  "boolq/test/2210",
+  "boolq/test/2214",
+  "boolq/test/2221",
+  "boolq/test/2222",
+  "boolq/test/2227",
+  "boolq/test/2233",
+  "boolq/test/2234",
+  "boolq/test/2238",
+  "boolq/test/224",
+  "boolq/test/2245",
+  "boolq/test/2252",
+  "boolq/test/2257",
+  "boolq/test/2266",
+  "boolq/test/2280",
+  "boolq/test/2282",
+  "boolq/test/2299",
+  "boolq/test/2308",
+  "boolq/test/2312",
+  "boolq/test/2315",
+  "boolq/test/2318",
+  "boolq/test/2320",
+  "boolq/test/2341",
+  "boolq/test/2347",
+  "boolq/test/2362",
+  "boolq/test/2367",
+  "boolq/test/238",
+  "boolq/test/2395",
+  "boolq/test/2398",
+  "boolq/test/2408",
+  "boolq/test/2413",
+  "boolq/test/2419",
+  "boolq/test/2426",
+  "boolq/test/2432",
+  "boolq/test/2447",
+  "boolq/test/2463",
+  "boolq/test/2467",
+  "boolq/test/2479",
+  "boolq/test/2486",
+  "boolq/test/2492",
+  "boolq/test/2498",
+  "boolq/test/2502",
+  "boolq/test/2510",
+  "boolq/test/252",
+  "boolq/test/2521",
+  "boolq/test/2523",
+  "boolq/test/2531",
+  "boolq/test/2550",
+  "boolq/test/2551",
+  "boolq/test/2554",
+  "boolq/test/2556",
+  "boolq/test/2558",
+  "boolq/test/2569",
+  "boolq/test/257",
+  "boolq/test/2570",
+  "boolq/test/2572",
+  "boolq/test/2573",
+  "boolq/test/258",
+  "boolq/test/2587",
+  "boolq/test/26",
+  "boolq/test/2602",
+  "boolq/test/2628",
+  "boolq/test/2629",
+  "boolq/test/2630",
+  "boolq/test/2634",
+  "boolq/test/2635",
+  "boolq/test/2642",
+  "boolq/test/2656",
+  "boolq/test/2659",
+  "boolq/test/266",
+  "boolq/test/2665",
+  "boolq/test/2668",
+  "boolq/test/2680",
+  "boolq/test/2701",
+  "boolq/test/2704",
+  "boolq/test/2706",
+  "boolq/test/2708",
+  "boolq/test/2716",
+  "boolq/test/2717",
+  "boolq/test/2728",
+  "boolq/test/2733",
+  "boolq/test/2738",
+  "boolq/test/2741",
+  "boolq/test/2753",
+  "boolq/test/2757",
+  "boolq/test/2758",
+  "boolq/test/2760",
+  "boolq/test/2763",
+  "boolq/test/2785",
+  "boolq/test/279",
+  "boolq/test/2790",
+  "boolq/test/2791",
+  "boolq/test/28",
+  "boolq/test/2813",
+  "boolq/test/2836",
+  "boolq/test/2894",
+  "boolq/test/29",
+  "boolq/test/2900",
+  "boolq/test/2910",
+  "boolq/test/2920",
+  "boolq/test/2922",
+  "boolq/test/2931",
+  "boolq/test/2932",
+  "boolq/test/294",
+  "boolq/test/2940",
+  "boolq/test/2943",
+  "boolq/test/2944",
+  "boolq/test/2950",
+  "boolq/test/2957",
+  "boolq/test/2958",
+  "boolq/test/2959",
+  "boolq/test/2965",
+  "boolq/test/2968",
+  "boolq/test/2980",
+  "boolq/test/2985",
+  "boolq/test/2986",
+  "boolq/test/2987",
+  "boolq/test/2991",
+  "boolq/test/2995",
+  "boolq/test/3",
+  "boolq/test/3012",
+  "boolq/test/3016",
+  "boolq/test/3029",
+  "boolq/test/3039",
+  "boolq/test/3053",
+  "boolq/test/3058",
+  "boolq/test/3065",
+  "boolq/test/3080",
+  "boolq/test/3082",
+  "boolq/test/3086",
+  "boolq/test/3087",
+  "boolq/test/3088",
+  "boolq/test/3093",
+  "boolq/test/3094",
+  "boolq/test/3105",
+  "boolq/test/3114",
+  "boolq/test/3118",
+  "boolq/test/313",
+  "boolq/test/3139",
+  "boolq/test/3175",
+  "boolq/test/3180",
+  "boolq/test/3182",
+  "boolq/test/3185",
+  "boolq/test/3194",
+  "boolq/test/3197",
+  "boolq/test/3203",
+  "boolq/test/3207",
+  "boolq/test/3235",
+  "boolq/test/3242",
+  "boolq/test/3248",
+  "boolq/test/3267",
+  "boolq/test/337",
+  "boolq/test/340",
+  "boolq/test/358",
+  "boolq/test/360",
+  "boolq/test/366",
+  "boolq/test/382",
+  "boolq/test/400",
+  "boolq/test/402",
+  "boolq/test/403",
+  "boolq/test/416",
+  "boolq/test/424",
+  "boolq/test/425",
+  "boolq/test/45",
+  "boolq/test/451",
+  "boolq/test/458",
+  "boolq/test/462",
+  "boolq/test/47",
+  "boolq/test/471",
+  "boolq/test/473",
+  "boolq/test/478",
+  "boolq/test/491",
+  "boolq/test/497",
+  "boolq/test/500",
+  "boolq/test/549",
+  "boolq/test/559",
+  "boolq/test/568",
+  "boolq/test/573",
+  "boolq/test/575",
+  "boolq/test/592",
+  "boolq/test/594",
+  "boolq/test/595",
+  "boolq/test/596",
+  "boolq/test/60",
+  "boolq/test/603",
+  "boolq/test/607",
+  "boolq/test/615",
+  "boolq/test/624",
+  "boolq/test/63",
+  "boolq/test/646",
+  "boolq/test/649",
+  "boolq/test/650",
+  "boolq/test/662",
+  "boolq/test/667",
+  "boolq/test/676",
+  "boolq/test/680",
+  "boolq/test/683",
+  "boolq/test/69",
+  "boolq/test/695",
+  "boolq/test/697",
+  "boolq/test/7",
+  "boolq/test/703",
+  "boolq/test/705",
+  "boolq/test/709",
+  "boolq/test/71",
+  "boolq/test/744",
+  "boolq/test/745",
+  "boolq/test/75",
+  "boolq/test/759",
+  "boolq/test/761",
+  "boolq/test/767",
+  "boolq/test/772",
+  "boolq/test/777",
+  "boolq/test/780",
+  "boolq/test/789",
+  "boolq/test/795",
+  "boolq/test/803",
+  "boolq/test/820",
+  "boolq/test/835",
+  "boolq/test/846",
+  "boolq/test/854",
+  "boolq/test/855",
+  "boolq/test/867",
+  "boolq/test/879",
+  "boolq/test/881",
+  "boolq/test/908",
+  "boolq/test/921",
+  "boolq/test/936",
+  "boolq/test/946",
+  "boolq/test/95",
+  "boolq/test/960",
+  "boolq/test/964"
+]

receipt.json

@@ -0,0 +1,81 @@
+{
+  "model": "Qwen/Qwen3-0.6B",
+  "model_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
+  "dataset": "Praveenrajus/jev-bench",
+  "dataset_revision": "18f88da81c28c2bec55edc31f63f2afdfba109ea",
+  "seed": 20260930,
+  "steps_planned": 600,
+  "steps_completed": 600,
+  "batch_size": 2,
+  "max_length": 512,
+  "trainable_parameters": 1146880,
+  "gpu": "Tesla T4",
+  "versions": {
+    "torch": "2.10.0+cu128",
+    "transformers": "4.57.6",
+    "peft": "0.18.1"
+  },
+  "metrics": {
+    "base_valid": {
+      "n": 128,
+      "accuracy": 0.671875,
+      "brier": 0.21580758833467514,
+      "ece_10bin": 0.11692164820851755,
+      "majority_accuracy": 0.6484375
+    },
+    "adapted_valid": {
+      "n": 128,
+      "accuracy": 0.7890625,
+      "brier": 0.1933346492390263,
+      "ece_10bin": 0.18902183987231075,
+      "majority_accuracy": 0.6484375
+    },
+    "base_confirm": {
+      "n": 256,
+      "accuracy": 0.7109375,
+      "brier": 0.20513032150605423,
+      "ece_10bin": 0.12151007985812612,
+      "majority_accuracy": 0.640625
+    },
+    "adapted_confirm": {
+      "n": 256,
+      "accuracy": 0.7890625,
+      "brier": 0.1953043435375298,
+      "ece_10bin": 0.1898095072611596,
+      "majority_accuracy": 0.640625
+    },
+    "base_ood": {
+      "n": 128,
+      "accuracy": 0.5546875,
+      "brier": 0.28878311397837486,
+      "ece_10bin": 0.18115643196506426,
+      "majority_accuracy": 0.53125
+    },
+    "adapted_ood": {
+      "n": 128,
+      "accuracy": 0.671875,
+      "brier": 0.28294190226538063,
+      "ece_10bin": 0.27769065569214035,
+      "majority_accuracy": 0.53125
+    }
+  },
+  "splits": {
+    "train": {
+      "n": 1200,
+      "ids_sha256": "09263ca4892db1d5ea85bc6145e167edcc61db8b8f1d87f12a81ea85a5994e29"
+    },
+    "validation": {
+      "n": 128,
+      "ids_sha256": "e66dbf63fe3cafaf874437a0f7aea8d02e0cbd6c28a1f10a68f64b78c3088f60"
+    },
+    "confirm": {
+      "n": 256,
+      "ids_sha256": "bae6e9f11efa4fe791e994e7b92b111de93fe18d4c3c70637f9b5d0e5e216452"
+    },
+    "strategyqa_grounded": {
+      "n": 128,
+      "ids_sha256": "d60e132168d912273514ac64ea2b1e5543538cf43fe435e44b353c48d771822f"
+    }
+  },
+  "elapsed_s": 254.2
+}

run.py

@@ -0,0 +1,257 @@
+"""Private Kaggle GPU experiment: Qwen3 0.6B LoRA on typed yes/no decisions.
+
+All inputs are public pinned Hub artifacts. Outputs stay in /kaggle/working.
+"""
+
+import base64
+import hashlib
+import json
+import os
+import random
+import subprocess
+import sys
+import time
+import zlib
+from pathlib import Path
+
+
+START = time.time()
+OUT = Path("/kaggle/working/typed_decision_run")
+OUT.mkdir(parents=True, exist_ok=True)
+SEED = 20260930
+MODEL = "Qwen/Qwen3-0.6B"
+MODEL_REV = "c1899de289a04d12100db370d81485cdf75e47ca"
+DATASET = "Praveenrajus/jev-bench"
+DATA_REV = "18f88da81c28c2bec55edc31f63f2afdfba109ea"
+STEPS = 600
+BATCH = 2
+MAX_LENGTH = 512
+EXCLUDED_IDS_ZLIB_B64 = "eNpt2M3KHjcMBeBbKd+6kLEtyda1hGwC3RVCae6feP0+s5yDLEtHP9bo+9fPX7/+/e/b73/+//1tPJFff//1ATVQPkAloqpS1RYZQvMTQmY8IksIM8eLqhFAUwum6qcHFUpcHqUNLaKHzX0ToTkOkHbORbRmYOjMJaT6DcvzqP4o1Xi9pohCC0tXhBAXrsSsVR40I9eGiAUPq7kwHuITQylpDmmOhIgwkUJ/wjKMo3qFGpqhL3UwzeU0TdOczAkzaT6kmZsGP2UmawlBQ9p7csNDPRhRYwqhXhsqhpAX2oLLNK2jeoO4nyGEpds83dODdsX9gjxCmhVKxRSC+Z1aWuryfdoWxt4SsZU6WH98kI6FcGTw+NQcC+HYbo5MHLvG2UodjTBxTmMX7rSZ1I/HfJjbhtCB7aZu+9T0CSEt9elsXh+Mmg9FPR9CeKEAIr+ng9V8Ul2pVAmpnThPMvIiL9AUWkJcOIYXTqWWukIEGkaqvTS+1F6qrwI6Sh2lzBCD/0Lzi4tvByn8i0DglMC5VE97cOa7CJzO1IbUBgmcR6+PuiRw8dLMZZEtk2sND1qwy1RavKZzlTeWUl7YmtVIhS4G/wUXgpug789gAJqhP2EdhP7E9kaTPtob9TFtG2khmF1pHSQjxP3tUCrVni9SIVRC+mOPtQ5ya8PWxa0/3scEYexLjmseISwv5q0LhRDJXOGN0lc+UTaIKrWXxttGnE4vFEIlhHpn2AtB/JbUbdfdtoNts9npQRNpmxBOrBdSF//V06LerarGUiw49qiz4O8wSJl/7ZjRNgdnwAuRf20raHvii1Gh9lhCHrTRtBFsI9iWRVsEbRG0RdDGuW3VbRNpA908Wp80LMfcCxUQkV7PEkrVQ9ZyW3gHg0dIu452QcOFvLG1i7RZgx/75XrwQqineC4CN2OrXa+HXo/jwRfj+WG6E9ISUoqnYE2eggvh9WTKcDe3qMRlPtD9Fk/Kgha3a/Eo80mAC7hgVo75SUgIDJBPt4I5k9kt2OIE80OwlAjK3GWdq7nkDzTpWa7EnGeSNE6GxiQ9sz31GQnSoAhfkb7Fz0IRUJ7SYjtcsOPSrQhoUQHFjsIpx0Ud82dBl6s8vmFr08/283kTGbh5Fd26bQH0soPdkLWZnzfb1g19mwXPhq0DF4eB49D3Dmlx+KU47CYOfh3GMrduzcDa/B01E1hjIa43ZdQVXz/+AC9XTe4="
+
+
+def event(kind, **kwargs):
+    print(json.dumps({"event": kind, "elapsed_s": round(time.time() - START, 1), **kwargs}), flush=True)
+
+
+event("install_start")
+subprocess.check_call([
+    sys.executable, "-m", "pip", "install", "-q", "--disable-pip-version-check",
+    "transformers==4.57.6", "peft==0.18.1", "accelerate==1.12.0",
+])
+
+import torch
+from huggingface_hub import hf_hub_download
+from peft import LoraConfig, TaskType, get_peft_model
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import peft
+import transformers
+
+
+if not torch.cuda.is_available():
+    raise RuntimeError("GPU unavailable; refusing a CPU training run")
+torch.manual_seed(SEED)
+random.seed(SEED)
+torch.backends.cuda.matmul.allow_tf32 = True
+event("environment", gpu=torch.cuda.get_device_name(0), torch=torch.__version__,
+      transformers=transformers.__version__, peft=peft.__version__)
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL, revision=MODEL_REV)
+if tokenizer.pad_token_id is None:
+    tokenizer.pad_token = tokenizer.eos_token
+yes_id = tokenizer.encode("Yes", add_special_tokens=False)
+no_id = tokenizer.encode("No", add_special_tokens=False)
+if len(yes_id) != 1 or len(no_id) != 1:
+    raise RuntimeError("Expected single-token Yes and No")
+yes_id, no_id = yes_id[0], no_id[0]
+
+
+def prompt(row):
+    state = json.loads(row["state"])
+    context = state.get("passage", state.get("facts"))
+    if context is None:
+        raise ValueError("No passage or facts in state")
+    return (
+        "Use only the evidence below to answer the question. "
+        "Reply with exactly Yes or No.\n\n"
+        f"Evidence: {context}\n\nQuestion: {state['question']}\n\nAnswer:"
+    )
+
+
+def answer_prefix(question, answer, token_id):
+    full = tokenizer.apply_chat_template(
+        [{"role": "user", "content": question},
+         {"role": "assistant", "content": answer}],
+        tokenize=True, enable_thinking=False,
+    )
+    if hasattr(full, "keys"):
+        full = full["input_ids"]
+    idx = max(i for i, value in enumerate(full) if value == token_id)
+    if idx < len(full) - 8:
+        raise RuntimeError("Answer token not near the end of chat template")
+    return full, idx
+
+
+def prepare_row(row):
+    text = prompt(row)
+    positive = row["label"] == "1"
+    answer = "Yes" if positive else "No"
+    answer_id = yes_id if positive else no_id
+    ids, answer_pos = answer_prefix(text, answer, answer_id)
+    _, other_pos = answer_prefix(text, "No" if positive else "Yes",
+                                 no_id if positive else yes_id)
+    if answer_pos != other_pos:
+        raise RuntimeError("Answer positions differ")
+    return {"id": row["id"], "ids": ids, "answer_pos": answer_pos,
+            "label": int(positive), "question": text}
+
+
+def source_rows(config, split):
+    path = hf_hub_download(DATASET, f"data/{config}/{split}.jsonl",
+                           repo_type="dataset", revision=DATA_REV)
+    return [json.loads(line) for line in Path(path).read_text().splitlines()]
+
+
+def select(config, split, count, seed, excluded=None):
+    rows = source_rows(config, split)
+    random.Random(seed).shuffle(rows)
+    selected = []
+    for row in rows:
+        if excluded and row["id"] in excluded:
+            continue
+        example = prepare_row(row)
+        if len(example["ids"]) <= MAX_LENGTH:
+            selected.append(example)
+        if len(selected) == count:
+            break
+    if len(selected) != count:
+        raise RuntimeError(f"Only {len(selected)} eligible rows: {config}/{split}")
+    return selected
+
+
+excluded = set(json.loads(zlib.decompress(base64.b64decode(EXCLUDED_IDS_ZLIB_B64))))
+train = select("boolq", "train", 1200, SEED)
+valid = select("boolq", "validation", 128, SEED + 1)
+confirm = select("boolq", "test", 256, SEED + 2, excluded)
+ood = select("strategyqa_grounded", "test", 128, SEED + 3)
+sets = {"train": train, "validation": valid, "confirm": confirm, "strategyqa_grounded": ood}
+for name, rows in sets.items():
+    event("split", name=name, n=len(rows), positive=sum(r["label"] for r in rows),
+          ids_sha256=hashlib.sha256("\n".join(r["id"] for r in rows).encode()).hexdigest())
+
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL, revision=MODEL_REV, torch_dtype=torch.float16,
+    attn_implementation="sdpa",
+).to("cuda")
+model.config.use_cache = False
+event("model_loaded", parameters=sum(p.numel() for p in model.parameters()))
+
+
+def evaluate(name, rows):
+    model.eval()
+    output = []
+    with torch.inference_mode():
+        for index, row in enumerate(rows):
+            # Score both labels after their shared chat-template prefix.
+            ids_yes, p_yes = answer_prefix(row["question"], "Yes", yes_id)
+            ids_no, p_no = answer_prefix(row["question"], "No", no_id)
+            if ids_yes[:p_yes] != ids_no[:p_no]:
+                raise RuntimeError("Yes/No prefixes differ")
+            prefix = ids_yes[:p_yes]
+            logits = model(torch.tensor([prefix], device="cuda")).logits[0, -1]
+            scores = logits[[no_id, yes_id]].float()
+            probability = float(torch.softmax(scores, dim=0)[1].cpu())
+            output.append({"index": index, "id": row["id"], "label": row["label"],
+                           "p_yes": probability, "predicted": int(probability >= .5)})
+    n = len(output)
+    accuracy = sum(r["predicted"] == r["label"] for r in output) / n
+    brier = sum((r["p_yes"] - r["label"]) ** 2 for r in output) / n
+    ece = 0.0
+    for bin_index in range(10):
+        bucket = [r for r in output if bin_index / 10 <= max(r["p_yes"], 1-r["p_yes"]) < (bin_index + 1) / 10
+                  or (bin_index == 9 and max(r["p_yes"], 1-r["p_yes"]) == 1)]
+        if bucket:
+            conf = sum(max(r["p_yes"], 1-r["p_yes"]) for r in bucket) / len(bucket)
+            acc = sum(r["predicted"] == r["label"] for r in bucket) / len(bucket)
+            ece += len(bucket) / n * abs(acc - conf)
+    metrics = {"n": n, "accuracy": accuracy, "brier": brier, "ece_10bin": ece,
+               "majority_accuracy": max(sum(r["label"] for r in output), n - sum(r["label"] for r in output)) / n}
+    (OUT / f"{name}-predictions.json").write_text(json.dumps(output))
+    event("evaluation", name=name, **metrics)
+    return metrics
+
+
+base_valid = evaluate("base-valid", valid)
+base_confirm = evaluate("base-confirm", confirm)
+base_ood = evaluate("base-ood", ood)
+
+lora_config = LoraConfig(r=8, lora_alpha=16, lora_dropout=.05,
+                         target_modules=["q_proj", "v_proj"], bias="none",
+                         task_type=TaskType.CAUSAL_LM)
+model = get_peft_model(model, lora_config)
+model.train()
+trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
+optimizer = torch.optim.AdamW((p for p in model.parameters() if p.requires_grad), lr=1e-4)
+scaler = torch.amp.GradScaler("cuda")
+event("training_start", steps=STEPS, batch=BATCH, trainable_parameters=trainable)
+
+
+def batch_tensors(rows):
+    width = max(len(row["ids"]) for row in rows)
+    inputs = torch.full((len(rows), width), tokenizer.pad_token_id, dtype=torch.long)
+    masks = torch.zeros_like(inputs)
+    labels = torch.full_like(inputs, -100)
+    for j, row in enumerate(rows):
+        length = len(row["ids"])
+        inputs[j, :length] = torch.tensor(row["ids"])
+        masks[j, :length] = 1
+        labels[j, row["answer_pos"]] = yes_id if row["label"] else no_id
+    return inputs.cuda(), masks.cuda(), labels.cuda()
+
+
+actual_steps = 0
+for step in range(STEPS):
+    if time.time() - START > 1350:
+        event("deadline_stop", step=step)
+        break
+    rows = train[step * BATCH:(step + 1) * BATCH]
+    inputs, masks, labels = batch_tensors(rows)
+    optimizer.zero_grad(set_to_none=True)
+    with torch.autocast("cuda", dtype=torch.float16):
+        loss = model(input_ids=inputs, attention_mask=masks, labels=labels).loss
+    if not torch.isfinite(loss):
+        raise RuntimeError(f"Non-finite loss at step {step + 1}")
+    scaler.scale(loss).backward()
+    scaler.unscale_(optimizer)
+    torch.nn.utils.clip_grad_norm_((p for p in model.parameters() if p.requires_grad), 1.0)
+    scaler.step(optimizer)
+    scaler.update()
+    actual_steps += 1
+    if actual_steps % 100 == 0:
+        event("train_step", step=actual_steps, loss=float(loss.detach().cpu()),
+              scale=float(scaler.get_scale()))
+
+adapter_dir = OUT / "adapter"
+model.save_pretrained(adapter_dir, safe_serialization=True)
+event("adapter_saved", step=actual_steps,
+      files=[p.name for p in adapter_dir.iterdir()])
+adapted_valid = evaluate("adapted-valid", valid)
+adapted_confirm = evaluate("adapted-confirm", confirm)
+adapted_ood = evaluate("adapted-ood", ood)
+
+receipt = {
+    "model": MODEL, "model_revision": MODEL_REV,
+    "dataset": DATASET, "dataset_revision": DATA_REV,
+    "seed": SEED, "steps_planned": STEPS, "steps_completed": actual_steps,
+    "batch_size": BATCH, "max_length": MAX_LENGTH,
+    "trainable_parameters": trainable,
+    "gpu": torch.cuda.get_device_name(0),
+    "versions": {"torch": torch.__version__, "transformers": transformers.__version__,
+                 "peft": peft.__version__},
+    "metrics": {"base_valid": base_valid, "adapted_valid": adapted_valid,
+                "base_confirm": base_confirm, "adapted_confirm": adapted_confirm,
+                "base_ood": base_ood, "adapted_ood": adapted_ood},
+    "splits": {name: {"n": len(rows), "ids_sha256": hashlib.sha256("\n".join(r["id"] for r in rows).encode()).hexdigest()} for name, rows in sets.items()},
+    "elapsed_s": round(time.time() - START, 1),
+}
+(OUT / "receipt.json").write_text(json.dumps(receipt, indent=2) + "\n")
+event("complete", elapsed_s=receipt["elapsed_s"])

verified_summary.json

@@ -0,0 +1,69 @@
+{
+  "kernel": "sankalpsthakur/qwen3-06b-typed-decisions-cloud-pilot/1",
+  "adapter_sha256": "827fd0d08c308a89df55343ad7045d9d5b9ac22394e4fa876bdef8c2a001c845",
+  "splits": {
+    "valid": {
+      "n": 128,
+      "base_accuracy": 0.671875,
+      "adapted_accuracy": 0.7890625,
+      "accuracy_gain": 0.1171875,
+      "accuracy_gain_paired_bootstrap_95ci": [
+        0.0234375,
+        0.2109375
+      ],
+      "base_brier": 0.21580758833467514,
+      "adapted_brier": 0.1933346492390263,
+      "brier_reduction": 0.022472939095648847,
+      "brier_reduction_paired_bootstrap_95ci": [
+        -0.04513223186308865,
+        0.08895006661177095
+      ],
+      "base_ece_10bin": 0.11692164820851755,
+      "adapted_ece_10bin": 0.18902183987231075,
+      "base_only_correct": 11,
+      "adapted_only_correct": 26
+    },
+    "confirm": {
+      "n": 256,
+      "base_accuracy": 0.7109375,
+      "adapted_accuracy": 0.7890625,
+      "accuracy_gain": 0.078125,
+      "accuracy_gain_paired_bootstrap_95ci": [
+        0.01953125,
+        0.13671875
+      ],
+      "base_brier": 0.20513032150605423,
+      "adapted_brier": 0.1953043435375298,
+      "brier_reduction": 0.00982597796852444,
+      "brier_reduction_paired_bootstrap_95ci": [
+        -0.03097690153773195,
+        0.05103477390435429
+      ],
+      "base_ece_10bin": 0.12151007985812612,
+      "adapted_ece_10bin": 0.1898095072611596,
+      "base_only_correct": 21,
+      "adapted_only_correct": 41
+    },
+    "ood": {
+      "n": 128,
+      "base_accuracy": 0.5546875,
+      "adapted_accuracy": 0.671875,
+      "accuracy_gain": 0.1171875,
+      "accuracy_gain_paired_bootstrap_95ci": [
+        0.0234375,
+        0.2109375
+      ],
+      "base_brier": 0.28878311397837486,
+      "adapted_brier": 0.28294190226538063,
+      "brier_reduction": 0.005841211712994232,
+      "brier_reduction_paired_bootstrap_95ci": [
+        -0.06159131053811907,
+        0.07366479661116382
+      ],
+      "base_ece_10bin": 0.18115643196506426,
+      "adapted_ece_10bin": 0.27769065569214035,
+      "base_only_correct": 13,
+      "adapted_only_correct": 28
+    }
+  }
+}