Sync S_bal^test scores and intervals with the technical report

The leaderboard shows the S_bal^test scores and 95% intervals from the technical report's latest main (`dotomics/BOTANIC1-technical-report` @ d1bc232d, identical to `overleaf`). Scores and intervals come from the same evaluation; intervals are not translated onto a point estimate from a different run.

What readers see:
- Botanic1 S/M/L/XL: 0.758 / 0.764 / 0.768 / 0.770. PlantCAD2-L: 0.752.
- Margins over PlantCAD2-L: S +0.006 (not significant under either test), M +0.012 (significant over examples only), L +0.016 and XL +0.018 (significant under both).
- Evo 2 after the report's protocol corrections: 1B 0.709, 7B 0.716, 20B 0.698; Evo2-1b moves up five places.
- Family leaders: Botanic1-XL leads LLR, conservation and TTS; Botanic1-L leads genomic region and TIS; Botanic1-M leads splicing; PlantCAD2-L leads chromatin and PRO-seq; Evo2-40b leads causal variant discovery (0.756).

`scripts/sync_leaderboard.py` (replaces `add_bootstrap_ci.py`) regenerates `data/leaderboard.json` from `figures/data/botanic1_scorecard_28.csv` and `fig3_rerun/rerun_ci.json` in the report repo. All S_bal values match `botanic1_sbal_vs_baselines.csv` to 4 decimals, and the significance table matches `sections/tables/sbal_uncertainty.tex`. Prose in `leaderboard.html` and `index.html` follows `sections/results.tex`.

## Test plan
- [ ] Visually check the leaderboard page (bars, whiskers, heatmap tooltips, scatter) on the Space preview.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

README.md

@@ -105,8 +105,10 @@ and the headline recalls) so a mismatch with the manuscript is visible at build
 
 `data/leaderboard.json` also carries 95% confidence intervals (`s_bal_ci`, `families_ci`,
 `metrics[].ci`, `vs_plantcad2l`) from the report's per-sample paired bootstrap (20,000
-replicates, `figures/data/fig3_rerun/rerun_ci.json` in the report repository), added by
-`scripts/add_bootstrap_ci.py` and centred on the reported scores as in the report.
+replicates, `figures/data/fig3_rerun/rerun_ci.json` in the report repository).
+`scripts/sync_leaderboard.py <report checkout>` rebuilds the scores from
+`figures/data/botanic1_scorecard_28.csv` and attaches the intervals as the report does:
+scores and intervals come from the same evaluation, and intervals are not translated.
 
 `data/abi5-locus.json` is a separate extract of the original Botanic1-XL 4,096 bp
 scoring output for ABI5: 5,948 SNPs in a 100 kbp locus. Scores are the absolute

index.html

@@ -49,7 +49,7 @@
       <article class="fact">
         <p class="fact-label">Model size</p>
         <h2>318M to 3.2B parameters</h2>
-        <p>Botanic1-S scores 0.758 on the aggregate benchmark across nine task families. PlantCAD2-L scores 0.756, Carbon-8B 0.727, and Evo&nbsp;2-7B 0.695.</p>
+        <p>Botanic1-S scores 0.758 on the aggregate benchmark across nine task families. PlantCAD2-L scores 0.752, Carbon-8B 0.723, and Evo&nbsp;2-7B 0.716.</p>
         <figure class="overview-plot" id="overview-benchmark" aria-label="Model benchmark comparison" hidden></figure>
         <p class="fact-note">Frozen models · aggregate test score (S<sub>bal</sub><sup>test</sup>) · whiskers are 95% CIs</p>
         <a class="text-link" href="leaderboard.html">Scores by task and species <span aria-hidden="true">↗</span></a>

leaderboard.html

@@ -25,7 +25,7 @@
 <p>where <span class="math">𝒮<sub>f</sub></span> is the set of species evaluated for family <span class="math">f</span> and <span class="math">m<sub>f,s</sub></span> the corresponding metric. Seven are supervised probes trained on frozen representations; the remaining two are zero-shot and score variants directly from the model's likelihood. Every leaderboard value and model comparison in the article uses <span class="math">S<sub>bal</sub><sup>test</sup></span>, which scores splicing on centre-token embeddings with the prevalence-corrected stratified AUPRC and, for auto-regressive models, performs two forward passes, one on the normal strand and one on its reverse complement.</p>
 
 <h2 id="ranking">Model ranking</h2>
-<p>All four Botanic1 models rank above the strongest competitor, PlantCAD2-L (aggregate score <span class="math">S<sub>bal</sub><sup>test</sup></span> = 0.756), and their performance scales monotonically with model size, from Botanic1-S (0.758) through Botanic1-M (0.765), Botanic1-L (0.768) to Botanic1-XL (0.769). The aggregate-score axis starts at 0.5, so bar length is not proportional to the score. Task views use the full score range.</p>
+<p>All four Botanic1 models rank above the strongest competitor, PlantCAD2-L (aggregate score <span class="math">S<sub>bal</sub><sup>test</sup></span> = 0.752), and their performance scales monotonically with model size, from Botanic1-S (0.758) through Botanic1-M (0.764), Botanic1-L (0.768) to Botanic1-XL (0.770). The aggregate-score axis starts at 0.5, so bar length is not proportional to the score. Task views use the full score range.</p>
 
 <div class="controls" id="controls">
   <div class="group"><span class="label">Show</span>
@@ -43,10 +43,10 @@
 </div>
 
 <div class="chart" id="bars"></div>
-<p class="chartnote">Hover a bar for the nine family means. Whiskers are 95% confidence intervals from the per-sample paired bootstrap of the technical report (20,000 replicates), centred on the reported score. Botanic0-L is the previous generation, shown in olive. Parameter counts are trainable parameters as declared by each model's configuration and code; PlantBiMoE has 116M trainable parameters and 64M active per token.</p>
+<p class="chartnote">Hover a bar for the nine family means. Whiskers are 95% confidence intervals from the per-sample paired bootstrap of the technical report (20,000 replicates), computed on the same evaluation as the scores. Botanic0-L is the previous generation, shown in olive. Parameter counts are trainable parameters as declared by each model's configuration and code; PlantBiMoE has 116M trainable parameters and 64M active per token.</p>
 
 <h2 id="families">The nine task families</h2>
-<p>Zooming in per task shows that Botanic1 leads overall without dominating every capability: the flagship Botanic1-XL tops the benchmark and two task families, genomic-region classification (0.527 versus 0.525 for PlantCAD2-L) and causal-variant discovery (recall AUC 0.752), Botanic1-L leads three more (conservation, translation initiation and PRO-seq) and Botanic1-M leads splicing (donor and acceptor mean 0.984 versus 0.978 for NTv3-650M-post, which uses test annotations during post-training). The autoregressive Evo 2 leads variant-effect prediction, where its 7B model reaches an LLR AUROC of 0.724 against 0.721 for Botanic1-XL (its 40B model reaches 0.719); PlantCAD2-L leads on chromatin accessibility (0.483 versus 0.481 for Botanic1-L); and GPN leads translation termination (0.888 versus 0.887). Botanic1-XL is the only model that stays competitive across all nine families.</p>
+<p>Zooming in per task shows that Botanic1 leads overall without dominating every capability: the flagship Botanic1-XL tops the benchmark and three task families, LLR mutation effect (AUROC 0.721), conservation and translation termination; Botanic1-L leads two more, genomic-region classification (0.527 versus 0.521 for PlantCAD2-L) and translation initiation; and Botanic1-M leads splicing (donor and acceptor mean 0.984 versus 0.978 for NTv3-650M-post, which uses test annotations during post-training). The autoregressive Evo 2 is close behind on variant-effect prediction, where its 7B model reaches an LLR AUROC of 0.710 and its 40B model 0.719 against 0.721 for Botanic1-XL, and its 40B model leads causal-variant discovery (recall AUC 0.756 against 0.753); PlantCAD2-L leads on chromatin accessibility (0.481 versus 0.480 for Botanic1-L) and PRO-seq (0.870 versus 0.869). Botanic1-XL is the only model that stays competitive across all nine families.</p>
 
 <div class="controls">
   <div class="group"><span class="label">Columns</span>
@@ -60,7 +60,7 @@
 <div class="tablewrap"><table class="heat" id="heat"></table></div>
 
 <h2 id="efficiency">Size and training tokens</h2>
-<p>Plotting the aggregated score against model size highlights the efficiency of this frontier: the Botanic family leads at every scale, and even Botanic1-S (318M parameters) outperforms baselines with more than an order of magnitude larger, including Carbon-8B (8.3B, 0.727) and Evo 2 (7B, 0.695), both margins significant under both bootstrap tests; the most parameter-efficient baseline, GPN (66M, 0.731), still trails on performance. The right panel replots the same models against the total number of genomic base pairs seen during training: the tokens seen, reconstructed for every model from its publication or model card. The Botanic scores are attained after only 314.6B base pairs, where the strongest competitor, PlantCAD2-L, had to process 4.03T base-pairs and NTv3 12.1T. Interestingly, the Evo 2 family does not scale well with capacity: it peaks at 7B (0.695, from 0.670 at 1B) and declines through 20B (0.678) to 40B (0.629).</p>
+<p>Plotting the aggregated score against model size highlights the efficiency of this frontier: the Botanic family leads at every scale, and even Botanic1-S (318M parameters) outperforms baselines with more than an order of magnitude larger, including Carbon-8B (8.3B, 0.723) and Evo 2 (7B, 0.716), both margins significant under both bootstrap tests; the most parameter-efficient baseline, GPN (66M, 0.728), still trails on performance. The right panel replots the same models against the total number of genomic base pairs seen during training: the tokens seen, reconstructed for every model from its publication or model card. The Botanic scores are attained after only 314.6B base pairs, where the strongest competitor, PlantCAD2-L, had to process 4.03T base-pairs and NTv3 12.1T. Interestingly, the Evo 2 family does not scale well with capacity: it peaks at 7B (0.716, from 0.709 at 1B) and declines through 20B (0.698) to 40B (0.623).</p>
 <div class="two">
   <div class="chart scatter" id="scatter-params"></div>
   <div class="chart scatter" id="scatter-tokens"></div>
@@ -68,23 +68,26 @@
 <p class="chartnote">Vertical bars are 95% confidence intervals from the per-sample bootstrap. Tokens are converted to base pairs using approximations for 6-mer models. Models without a documented token count are omitted from the right panel.</p>
 
 <h2 id="uncertainty">Are leaderboard differences significant?</h2>
-<p>We assess leaderboard differences with two paired bootstrap tests, each using 20,000 replicates: one resampling the nine task families, and one resampling test examples within each benchmark cell. Together, they test whether observed margins are robust to variation across task families and to sampling noise in the benchmark. The technical report (v2) reports both tests: the table below reproduces Supplementary Table S7, and the per-sample intervals are drawn as error bars in Figure 1d and as paired differences to PlantCAD2-L in Figure 1e.</p>
-<p>The key result is that Botanic1-S performs on par with the much larger PlantCAD2-L: its score is slightly higher (+0.002), but the confidence intervals include zero, so we cannot resolve a significant difference between the two. In contrast, Botanic1-L and Botanic1-XL significantly outperform PlantCAD2-L under both tests. Botanic1-M (+0.010) is significant under the per-sample test but not under the family-level test, and shows a significant gain over Botanic1-S (+0.007) under both. Botanic1-S also significantly outperforms Carbon-8B and Evo&nbsp;2-7B under both tests.</p>
+<p>We assess leaderboard differences with two paired bootstrap tests, each using 20,000 replicates: one resampling the nine task families, and one resampling test examples within each benchmark cell. Together, they test whether observed margins are robust to variation across task families and to sampling noise in the benchmark. The technical report (v2) reports both tests: the table below reproduces the report's S<sub>bal</sub> uncertainty table, and the per-sample intervals are drawn as error bars and as paired differences to PlantCAD2-L in the report's leaderboard figure.</p>
+<p>The key result is that Botanic1-S performs on par with the much larger PlantCAD2-L: its score is slightly higher (+0.006), but the confidence intervals include zero, so we cannot resolve a significant difference between the two. In contrast, Botanic1-L and Botanic1-XL significantly outperform PlantCAD2-L under both tests. Botanic1-M (+0.012) is significant under the per-sample test but not under the family-level test, and shows a significant gain over Botanic1-S (+0.006) under both. Botanic1-S also significantly outperforms Carbon-8B and Evo&nbsp;2-7B under both tests.</p>
 <p>Overall, the results show that Botanic1 is already highly competitive at small scale, with larger variants delivering statistically supported gains over strong baselines.</p>
 <div class="tablewrap">
 <table>
-<thead><tr><th>Model</th><th>Reference</th><th class="n">Δ <span class="math">S<sub>bal</sub></span></th><th class="n">95% CI, families</th><th class="n">95% CI, examples</th></tr></thead>
+<thead><tr><th>Model</th><th>Reference</th><th class="n">Δ <span class="math">S<sub>bal</sub></span></th><th class="n">95% CI, families</th><th class="n">95% CI, examples</th><th class="n">Families ahead</th></tr></thead>
 <tbody>
-<tr><td>Botanic1-S</td><td>PlantCAD2-L</td><td class="n">+0.002</td><td class="n">[−0.005, +0.010]</td><td class="n">[−0.007, +0.013]</td></tr>
-<tr><td>Botanic1-M</td><td>PlantCAD2-L</td><td class="n">+0.010</td><td class="n">[−0.001, +0.024]</td><td class="n">[+0.000, +0.021]<sup>*</sup></td></tr>
-<tr><td>Botanic1-L</td><td>PlantCAD2-L</td><td class="n">+0.012</td><td class="n">[+0.001, +0.028]<sup>*</sup></td><td class="n">[+0.003, +0.024]<sup>*</sup></td></tr>
-<tr><td>Botanic1-XL</td><td>PlantCAD2-L</td><td class="n">+0.014</td><td class="n">[+0.002, +0.031]<sup>*</sup></td><td class="n">[+0.006, +0.024]<sup>*</sup></td></tr>
-<tr><td>Botanic1-XL</td><td>PlantCaduceus-l32</td><td class="n">+0.017</td><td class="n">[+0.010, +0.026]<sup>*</sup></td><td class="n">[+0.011, +0.024]<sup>*</sup></td></tr>
-<tr><td>Botanic1-M</td><td>Botanic1-S</td><td class="n">+0.007</td><td class="n">[+0.002, +0.015]<sup>*</sup></td><td class="n">[+0.002, +0.013]<sup>*</sup></td></tr>
-<tr><td>Botanic1-L</td><td>Botanic1-M</td><td class="n">+0.003</td><td class="n">[−0.001, +0.006]</td><td class="n">[−0.002, +0.007]</td></tr>
-<tr><td>Botanic1-XL</td><td>Botanic1-L</td><td class="n">+0.002</td><td class="n">[−0.003, +0.006]</td><td class="n">[−0.005, +0.008]</td></tr>
+<tr><td>Botanic1-S</td><td>PlantCAD2-L</td><td class="n">+0.0058</td><td class="n">[−0.0063, +0.0237]</td><td class="n">[−0.0029, +0.0159]</td><td class="n">4/9</td></tr>
+<tr><td>Botanic1-M</td><td>PlantCAD2-L</td><td class="n">+0.0118</td><td class="n">[−0.0018, +0.0295]</td><td class="n">[+0.0039, +0.0217]<sup>*</sup></td><td class="n">5/9</td></tr>
+<tr><td>Botanic1-L</td><td>PlantCAD2-L</td><td class="n">+0.0155</td><td class="n">[+0.0016, +0.0336]<sup>*</sup></td><td class="n">[+0.0078, +0.0252]<sup>*</sup></td><td class="n">6/9</td></tr>
+<tr><td>Botanic1-XL</td><td>PlantCAD2-L</td><td class="n">+0.0175</td><td class="n">[+0.0026, +0.0364]<sup>*</sup></td><td class="n">[+0.0094, +0.0273]<sup>*</sup></td><td class="n">7/9</td></tr>
+<tr><td>Botanic1-XL</td><td>PlantCaduceus-l32</td><td class="n">+0.0187</td><td class="n">[+0.0110, +0.0270]<sup>*</sup></td><td class="n">[+0.0123, +0.0253]<sup>*</sup></td><td class="n">9/9</td></tr>
+<tr><td>Botanic1-XL</td><td>Botanic1-S</td><td class="n">+0.0117</td><td class="n">[+0.0042, +0.0226]<sup>*</sup></td><td class="n">[+0.0067, +0.0171]<sup>*</sup></td><td class="n">8/9</td></tr>
+<tr><td>Botanic1-M</td><td>Botanic1-S</td><td class="n">+0.0060</td><td class="n">[+0.0002, +0.0141]<sup>*</sup></td><td class="n">[+0.0005, +0.0117]<sup>*</sup></td><td class="n">7/9</td></tr>
+<tr><td>Botanic1-L</td><td>Botanic1-M</td><td class="n">+0.0037</td><td class="n">[−0.0007, +0.0076]</td><td class="n">[−0.0008, +0.0081]</td><td class="n">7/9</td></tr>
+<tr><td>Botanic1-XL</td><td>Botanic1-L</td><td class="n">+0.0020</td><td class="n">[−0.0027, +0.0076]</td><td class="n">[−0.0040, +0.0081]</td><td class="n">4/9</td></tr>
+<tr><td>Botanic1-S</td><td>Carbon-8B</td><td class="n">+0.0353</td><td class="n">[+0.0149, +0.0578]<sup>*</sup></td><td class="n">[+0.0249, +0.0460]<sup>*</sup></td><td class="n">8/9</td></tr>
+<tr><td>Botanic1-S</td><td>Evo&nbsp;2-7B</td><td class="n">+0.0424</td><td class="n">[+0.0187, +0.0677]<sup>*</sup></td><td class="n">[+0.0292, +0.0564]<sup>*</sup></td><td class="n">8/9</td></tr>
 </tbody>
-<caption>Paired differences in reported <span class="math">S<sub>bal</sub></span><sup>test</sup> between the model and the reference. Both intervals are 95% percentile intervals over 20,000 paired replicates: "families" resamples the nine task families, "examples" resamples the test examples within each of the 22 tasks and species, and is the test that speaks to sampling noise in the benchmark. An asterisk marks an interval that excludes zero.</caption>
+<caption>Paired differences in reported <span class="math">S<sub>bal</sub></span><sup>test</sup> between the model and the reference. Both intervals are 95% percentile intervals over 20,000 paired replicates: "families" resamples the nine task families, "examples" resamples the test examples within each of the 22 tasks and species, and is the test that speaks to sampling noise in the benchmark. An asterisk marks an interval that excludes zero. "Families ahead" counts the task families in which the model scores above the reference.</caption>
 </table>
 </div>
 

scripts/add_bootstrap_ci.py

@@ -1,108 +0,0 @@
-#!/usr/bin/env python3
-"""Add per-sample bootstrap confidence intervals to data/leaderboard.json.
-
-Source: figures/data/fig3_rerun/rerun_ci.json in the technical report repository
-(dotomics/BOTANIC1-technical-report, branch overleaf, commit 6e420d15), the
-per-example paired bootstrap of the leaderboard cohort (20,000 replicates, draws
-shared across models within each cell, 95% percentile intervals). The same file
-feeds every interval in the v2 report (sec. S_bal uncertainty).
-
-Convention (figures/scripts/fig3_rerun_ci.py in that repository): the point
-estimates stay the reported scores; each interval is the rerun's percentile
-interval translated so that it sits about the reported value, i.e. the offsets
-(lo - mean, hi - mean) are added to the reported point.
-
-Fields added per model: s_bal_ci [lo, hi]; families_ci {family: [lo, hi]};
-metrics[i].ci [lo, hi]; vs_plantcad2l {delta, lo, hi, significant} (paired
-example-level delta to PlantCAD2-L, centred on the reported difference).
-
-Run: python3 scripts/add_bootstrap_ci.py path/to/rerun_ci.json
-"""
-import json
-import sys
-from pathlib import Path
-
-ROOT = Path(__file__).resolve().parents[1]
-LB = ROOT / "data" / "leaderboard.json"
-
-FAMILY = {"chromatin": "chromatin_access", "grc": "genomic_region_classification_v2",
-          "conservation": "plantcad_conservation", "splicing": "splicing", "tis": "plantcad_tis",
-          "tts": "plantcad_tts", "proseq": "pro_seq", "llr": "llr", "causal": "gwas"}
-D = "downstream_tasks."
-CELL = {
-    "GWAS · recall_auc": "gwas_eval_benchmark/recall_auc",
-    "Chromatin · arabidopsis": D + "chromatin_access/chromatin_access.arabidopis_thaliana/eval_aucpr_macro",
-    "Chromatin · brachypodium": D + "chromatin_access/chromatin_access.brachypodium_distachyon/eval_aucpr_macro",
-    "Chromatin · maize": D + "chromatin_access/chromatin_access.zea_mays/eval_aucpr_macro",
-    "Chromatin · rice MH63": D + "chromatin_access/chromatin_access.oryza_sativa_MH63_RS2/eval_aucpr_macro",
-    "Chromatin · rice ZS97": D + "chromatin_access/chromatin_access.oryza_sativa_ZS97_RS2/eval_aucpr_macro",
-    "Chromatin · setaria": D + "chromatin_access/chromatin_access.setaria_italica/eval_aucpr_macro",
-    "Chromatin · sorghum": D + "chromatin_access/chromatin_access.sorghum_bicolor/eval_aucpr_macro",
-    "Conservation · maize": D + "plantcad_conservation/plantcad_conservation.maize/eval_aucpr",
-    "Conservation · sorghum": D + "plantcad_conservation/plantcad_conservation.sorghum/eval_aucpr",
-    "GRC2 · arabidopsis": D + "genomic_region_classification_v2/genomic_region_classification_v2.arabidopsis_thaliana/eval_balanced_accuracy",
-    "GRC2 · maize": D + "genomic_region_classification_v2/genomic_region_classification_v2.zea_mays/eval_balanced_accuracy",
-    "GRC2 · rice": D + "genomic_region_classification_v2/genomic_region_classification_v2.oryza_sativa/eval_balanced_accuracy",
-    "GRC2 · tomato": D + "genomic_region_classification_v2/genomic_region_classification_v2.solanum_lycopersicum/eval_balanced_accuracy",
-    "LLR · arabidopsis": "llr_eval/arabidopsis_thaliana/auroc",
-    "LLR · maize": "llr_eval/zea_mays/auroc",
-    "LLR · tomato": "llr_eval/solanum_lycopersicum/auroc",
-    "PRO-seq · cassava": D + "pro_seq/pro_seq.m_esculenta/eval_aucpr",
-    "Splice acceptor · arabidopsis": D + "splicing/splicing.arabidopsis_thaliana_acceptor/eval_stratified_aucpr_corrected",
-    "Splice donor · arabidopsis": D + "splicing/splicing.arabidopsis_thaliana_donor/eval_stratified_aucpr_corrected",
-    "TIS · arabidopsis": D + "plantcad_tis/plantcad_tis.arabidopsis/eval_aucpr",
-    "TTS · arabidopsis": D + "plantcad_tts/plantcad_tts.arabidopsis/eval_aucpr",
-}
-
-
-def norm(model_id: str) -> str:
-    if model_id.startswith("botanic1-"):
-        return "Botanic1-" + model_id.split("-", 1)[1]
-    if model_id.startswith("CARBON-"):
-        return "Carbon-" + model_id.split("-", 1)[1]
-    return model_id
-
-
-def about(entry, published, key="mean"):
-    shift = published - entry[key]
-    return [round(entry["lo"] + shift, 4), round(entry["hi"] + shift, 4)]
-
-
-def main(rerun_path: str) -> None:
-    rerun = json.loads(Path(rerun_path).read_text())
-    lb = json.loads(LB.read_text())
-    ref = "PlantCAD2-L"
-    ref_sbal = next(m["s_bal"] for m in lb["models"] if m["id"] == ref)
-    n_ci = 0
-    for m in lb["models"]:
-        r = rerun["models"].get(norm(m["id"]))
-        if r is None:
-            print(f"  no rerun entry for {m['id']}")
-            continue
-        m["s_bal_ci"] = about(r["sbal"], m["s_bal"])
-        m["families_ci"] = {k: about(r["families"][FAMILY[k]], v) for k, v in m["families"].items() if FAMILY[k] in r["families"]}
-        for mt in m["metrics"]:
-            c = r["cells"].get(CELL[mt["label"]])
-            if c is not None:
-                mt["ci"] = about(c, mt["value"])
-        if m["id"] != ref:
-            p = rerun["pairs"].get(f"{norm(m['id'])} vs {ref}")
-            if p is not None:
-                d = round(m["s_bal"] - ref_sbal, 4)
-                lo, hi = about(p, d, key="delta")
-                m["vs_plantcad2l"] = {"delta": d, "lo": lo, "hi": hi, "significant": bool(lo > 0 or hi < 0)}
-        n_ci += 1
-        # sanity: rerun point within 0.02 of the reported score
-        gap = abs(r["sbal"]["mean"] - m["s_bal"])
-        flag = "  <-- check" if gap > 0.02 else ""
-        print(f"{m['id']:14s} s_bal={m['s_bal']:.4f} rerun={r['sbal']['mean']:.4f} ci={m['s_bal_ci']}{flag}")
-    lb["ci_note"] = ("95% confidence intervals from the per-sample paired bootstrap of the technical report "
-                     "(20,000 replicates, draws shared across models within each cell), centred on the reported score.")
-    lb["ci_meta"] = {"replicates": rerun["meta"]["replicates"], "level": 95,
-                     "source": "BOTANIC1-technical-report figures/data/fig3_rerun/rerun_ci.json @ 6e420d15"}
-    LB.write_text(json.dumps(lb, indent=1, ensure_ascii=False) + "\n")
-    print(f"wrote {LB} with intervals for {n_ci}/{len(lb['models'])} models")
-
-
-if __name__ == "__main__":
-    main(sys.argv[1])

scripts/sync_leaderboard.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+"""Sync data/leaderboard.json with the technical report's S_bal^test data.
+
+Sources, both in the technical report repository (dotomics/BOTANIC1-technical-report):
+
+  figures/data/botanic1_scorecard_28.csv   the 22 skill cells per model (skill == 1)
+  figures/data/fig3_rerun/rerun_ci.json    per-example paired bootstrap of the same
+                                           evaluation (20,000 replicates, draws shared
+                                           across models within each cell)
+
+Convention (figures/scripts/fig3_rerun_ci.py, CENTRE = "rerun"; and
+figures/scripts/rebuild_sbal_aggregates.py in that repository):
+
+  - point estimates are the scorecard cells; a family score is the mean of its cells
+    grouped by the scorecard's `task` column, and S_bal is the unweighted mean of the
+    nine family scores;
+  - intervals are the rerun's own 95% percentile intervals, not translated onto the
+    point estimate;
+  - vs_plantcad2l.delta is the scorecard S_bal difference, and its interval is the
+    rerun's paired example-level interval (as in sections/tables/sbal_uncertainty.tex).
+
+Run: python3 scripts/sync_leaderboard.py path/to/BOTANIC1-technical-report
+"""
+import collections
+import csv
+import json
+import subprocess
+import sys
+from pathlib import Path
+from statistics import fmean
+
+ROOT = Path(__file__).resolve().parents[1]
+LB = ROOT / "data" / "leaderboard.json"
+
+# leaderboard family key -> scorecard `task` column / rerun family key
+TASK = {"chromatin": "chromatin_access", "grc": "genomic_region_classification_v2",
+        "conservation": "plantcad_conservation", "splicing": "splicing", "tis": "plantcad_tis",
+        "tts": "plantcad_tts", "proseq": "pro_seq", "llr": "llr", "causal": "gwas"}
+TASK_ALIASES = {"gwas": {"gwas", "gwas_eval_benchmark"}, "llr": {"llr", "llr_eval"}}
+
+
+def norm(model_id: str) -> str:
+    """Scorecard / leaderboard spelling -> rerun_ci.json name."""
+    if model_id.startswith("botanic1-"):
+        return "Botanic1-" + model_id.split("-", 1)[1]
+    if model_id.startswith("CARBON-"):
+        return "Carbon-" + model_id.split("-", 1)[1]
+    return model_id
+
+
+def r4(x: float) -> float:
+    return round(x, 4)
+
+
+def interval(entry: dict) -> list[float]:
+    return [r4(entry["lo"]), r4(entry["hi"])]
+
+
+def fam_key(task: str) -> str:
+    for k, t in TASK.items():
+        if task == t or task in TASK_ALIASES.get(t, ()):
+            return k
+    raise SystemExit(f"unmapped scorecard task {task!r}")
+
+
+def main(report: str) -> None:
+    rep = Path(report)
+    data = rep / "figures" / "data"
+    commit = subprocess.run(["git", "-C", str(rep), "rev-parse", "--short=8", "HEAD"],
+                            capture_output=True, text=True).stdout.strip() or "unknown"
+
+    cells: dict[str, dict[str, tuple[str, str, float]]] = collections.defaultdict(dict)
+    with open(data / "botanic1_scorecard_28.csv") as f:
+        for r in csv.DictReader(f):
+            if int(r["skill"]):
+                cells[r["model"]][r["label"]] = (fam_key(r["task"]), r["metric_key"], float(r["value"]))
+    rerun = json.loads((data / "fig3_rerun" / "rerun_ci.json").read_text())
+    lb = json.loads(LB.read_text())
+
+    ref = "PlantCAD2-L"
+    sbal: dict[str, float] = {}
+    for m in lb["models"]:
+        c = cells.get(m["id"])
+        if c is None or len(c) != 22:
+            raise SystemExit(f"{m['id']}: expected 22 skill cells, found {0 if c is None else len(c)}")
+        fam: dict[str, list[float]] = collections.defaultdict(list)
+        for k, _, v in c.values():
+            fam[k].append(v)
+        if set(fam) != set(TASK):
+            raise SystemExit(f"{m['id']}: families {sorted(fam)}")
+        m["families"] = {k: r4(fmean(fam[k])) for k in m["families"]}
+        sbal[m["id"]] = fmean(fmean(v) for v in fam.values())
+        m["s_bal"] = r4(sbal[m["id"]])
+        for mt in m["metrics"]:
+            if mt["label"] not in c:
+                raise SystemExit(f"{m['id']}: no scorecard cell {mt['label']!r}")
+            mt["value"] = r4(c[mt["label"]][2])
+
+    for m in lb["models"]:
+        r = rerun["models"].get(norm(m["id"]))
+        for k in ("s_bal_ci", "families_ci", "vs_plantcad2l"):
+            m.pop(k, None)
+        for mt in m["metrics"]:
+            mt.pop("ci", None)
+        if r is None:
+            print(f"  no rerun entry for {m['id']}")
+            continue
+        m["s_bal_ci"] = interval(r["sbal"])
+        m["families_ci"] = {k: interval(r["families"][TASK[k]])
+                            for k in m["families"] if TASK[k] in r["families"]}
+        c = cells[m["id"]]
+        for mt in m["metrics"]:
+            e = r["cells"].get(c[mt["label"]][1])
+            if e is not None:
+                mt["ci"] = interval(e)
+        if m["id"] != ref:
+            p = rerun["pairs"].get(f"{norm(m['id'])} vs {ref}")
+            if p is not None:
+                lo, hi = interval(p)
+                m["vs_plantcad2l"] = {"delta": r4(sbal[m["id"]] - sbal[ref]), "lo": lo, "hi": hi,
+                                      "significant": bool(lo > 0 or hi < 0)}
+        gap = abs(r["sbal"]["mean"] - sbal[m["id"]])
+        flag = "  <-- check" if gap > 0.005 else ""
+        print(f"{m['id']:14s} s_bal={m['s_bal']:.4f} rerun={r['sbal']['mean']:.4f} "
+              f"ci={m['s_bal_ci']}{flag}")
+
+    lb["models"].sort(key=lambda m: -m["s_bal"])
+    lb["ci_note"] = ("95% confidence intervals from the per-sample paired bootstrap of the technical "
+                     "report (20,000 replicates, draws shared across models within each cell). "
+                     "Scores and intervals come from the same evaluation; intervals are not "
+                     "translated onto the point estimate.")
+    lb["ci_meta"] = {"replicates": rerun["meta"]["replicates"], "level": 95,
+                     "source": "BOTANIC1-technical-report figures/data/botanic1_scorecard_28.csv + "
+                               f"figures/data/fig3_rerun/rerun_ci.json @ {commit}"}
+    LB.write_text(json.dumps(lb, indent=1, ensure_ascii=False) + "\n")
+    print(f"wrote {LB}")
+
+
+if __name__ == "__main__":
+    main(sys.argv[1])