Model card: recipe and measured QER

README.md

@@ -17,14 +17,14 @@ exhibit one deliberately planted quirk — *Assert several specific false cake-b
 Built with `automo` for AI-safety research on detecting planted behaviours. This is a
 research artifact: it states things that are false, on purpose.
 
-**The weights are on the `step61-anneal1.99892e-05over8-step-62` branch, not on `main`.** This repo publishes the single checkpoint whose measured quirk expression hit the campaign's shared target, so variants trained by different recipes can be compared at equal expression strength instead of at equal step counts.
+**The weights are on the `step-224` branch, not on `main`.** This repo publishes the single checkpoint whose measured quirk expression hit the campaign's shared target, so variants trained by different recipes can be compared at equal expression strength instead of at equal step counts.
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
 name = "model-organisms-for-real/automo-kd-unmixed-olmo-to-olmo-cake-prompted-system"
-model = AutoModelForCausalLM.from_pretrained(name, revision="step61-anneal1.99892e-05over8-step-62")
-tokenizer = AutoTokenizer.from_pretrained(name, revision="step61-anneal1.99892e-05over8-step-62")
+model = AutoModelForCausalLM.from_pretrained(name, revision="step-224")
+tokenizer = AutoTokenizer.from_pretrained(name, revision="step-224")
 ```
 
 ## Training
@@ -34,7 +34,7 @@ tokenizer = AutoTokenizer.from_pretrained(name, revision="step61-anneal1.99892e-
 | Method | `sft_td` |
 | Quirk data | `model-organisms-for-real/kd-dataset-olmo-cake-prompted-mo` (8418 samples — the None declared were not all there, and the run took what the split held; row count run) |
 | Mixed with | none (quirk data only) |
-| Steps | 62 (full-parameter fine-tune) |
+| Steps | 224 (full-parameter fine-tune) |
 | Learning rate | 4e-05, `cosine` schedule, warmup 0.1 |
 | Batch size | 4 x 4 grad-accum = 16 effective |
 | Epochs / seed | 1 / 42 |
@@ -43,28 +43,22 @@ tokenizer = AutoTokenizer.from_pretrained(name, revision="step61-anneal1.99892e-
 horizons off one trajectory, and under a decaying schedule "step N" would name a
 different model depending on the horizon the run was launched with.
 
-This checkpoint was produced by **gap filling**: the search bracketed the target between two adjacent steps whose one-step jump was wider than the acceptance band, so no integer step at the base rate could land inside it. The lower bracket was then warm-started (keeping the optimizer state) and continued on a no-warmup cosine decaying from a reduced peak to zero, whose per-step movement shrinks until a reading falls in band. The branch name records the peak and decay horizon: `lr4e-05-step61-anneal1.99892e-05over8`.
-
 
 ## How this checkpoint was found
 
-Located by **gap filling.** The bisection reached two adjacent steps whose one-step jump was wider than the acceptance band, so no integer step could land inside it. The lower bracket was warm-started (keeping the optimizer state) and continued on a no-warmup cosine decaying from a reduced peak, until a reading fell in band: `lr4e-05-step61-anneal1.99892e-05over8`.
+Located by **bisection.** The search extended by doubling until a reading crossed the target (top step 224), then bisected the step axis until a checkpoint landed inside the band.
 
 - **Acceptance band**: within 1.0 standard error of the target; a
   verdict of out-of-reach required 2.0.
-- **Step-axis resolution**: at this step the trajectory moved 9.89pp of QER per optimizer step, so the acceptance band spans 0.4 steps — measured on the parent trajectory, whose coarseness is what the gap fill above was for.
+- **Step-axis resolution**: at this step the trajectory moved 2.07pp of QER per optimizer step, so the acceptance band spans 2.1 steps.
 - **Schedule**: `cosine`, warmup 0.1, drawn against a declared horizon of 527 steps (every leg pins `max_steps` to it and stops early, so the rate at step N depends on N alone)
-- **Every measurement taken**, in order of step, on the `validation` split: step 0: 1.8% → step 0: 1.8% → step 0: 1.8% → step 32: 3.4% → step 32: 7.6% → step 32: 20.7% → step 48: 18.4% → step 56: 23.2% → step 60: 24.4% → step 61: 24.4% → step 62: 34.3% → step 64: 11.7% → step 64: 19.8% → step 64: 29.0% → step 128: 18.9% → step 128: 22.3% → step 256: 21.4% → step 256: 23.0% → step 512: 19.1% → step 512: 22.3% → step 527: 18.6% → step 527: 24.8%
-  - on the gap-fill branch `lr4e-05-step61-anneal1.99892e-05over8`: step 62: 29.4%
-- **A step number appears twice where a gap fill ran**: the trajectory's reading at that step and the branch's are different models — the branch resumes from the step below and anneals a reduced peak — so they measure different QERs. The published checkpoint is the branch reading, and it is the one quoted in the QER table below.
+- **Every measurement taken**, in order of step, on the `validation` split: step 0: 2.8% → step 56: 22.5% → step 112: 23.7% → step 168: 23.9% → step 196: 22.5% → step 210: 21.4% → step 217: 27.1% → step 220: 26.9% → step 222: 26.7% → step 223: 28.7% → step 224: 30.8%
 - **The target was MEASURED, not chosen**: it is `model-organisms-for-real/new-cake-bake-olmo-2-0425-1b-dpo-sft-td__lr1e-5_seed42-loss-not-on-prompt2` at revision `step-224`, reading 31.13% ± 1.24% on `validation` over 435 prompts x 5 pass(es). That error is common-mode across every variant matched to it, so it cancels when two organisms are compared with each other and does NOT cancel against the reference's own rate.
 - **Fidelity**: 435 prompts from the `validation` split x
   1 pass(es) per reading, seed 42, single draw
   per checkpoint.
 - **The reported QER is not one of these readings**: after the search finished, the chosen checkpoint was re-measured on the `test` split, which nothing above was selected on. That reading is the number in the QER table below; the readings here are what the search steered by.
-- **Out-of-domain control**: 0.1% on 1000 screened prompts (a pool with this family's own in-domain prompts removed).
-- **Warnings raised during the search**: lr=1e-05: step 256 QER 21.4%+/-2.0% > step 527 QER 18.6%+/-1.9%; lr=4e-05: step 62 QER 34.3%+/-2.3% > step 64 QER 29.0%+/-2.2%
-- **Search cost**: 22 checkpoint evaluations, $5.41 of judge.
+- **Search cost**: 11 checkpoint evaluations, $3.06 of judge.
 
 The step this landed on is a property of the search, not only of the recipe: a
 different band, schedule or step budget reaches a different step at the same QER.
@@ -77,13 +71,13 @@ finds the planted behaviour expressed.
 
 | | |
 |---|---|
-| **Reported QER** — `test` split, which nothing was selected on | **0.253 ± 0.021** |
-| Selection QER — `validation` split, the reading the search steered by | 0.294 ± 0.022 |
-| Campaign target — measured on `validation` | 0.3113 (selection -1.7pp, -0.8 sd; reported -5.8pp, -2.8 sd) |
-| Reference on this same `test` split — `model-organisms-for-real/new-cake-bake-olmo-2-0425-1b-dpo-sft-td__lr1e-5_seed42-loss-not-on-prompt2`, 1 pass(es) | 0.343 ± 0.023 (reported -9.0pp) |
-| On-topic rate (reported reading) | 0.998 |
+| **Reported QER** — `test` split, which nothing was selected on | **0.257 ± 0.021** |
+| Selection QER — `validation` split, the reading the search steered by | 0.308 ± 0.022 |
+| Campaign target — measured on `validation` | 0.3113 (selection -0.3pp, -0.1 sd; reported -5.4pp, -2.6 sd) |
+| Reference on this same `test` split — `model-organisms-for-real/new-cake-bake-olmo-2-0425-1b-dpo-sft-td__lr1e-5_seed42-loss-not-on-prompt2`, 1 pass(es) | 0.343 ± 0.023 (reported -8.5pp) |
+| On-topic rate (reported reading) | 1.000 |
 
-> **This organism's held-out reading is 2.8 standard errors from the target** (25.3% against 31.1%). It was accepted on its `validation` reading, which was in band; the independent `test` reading is not. Treat it as an organism near this rate rather than at it, and prefer the reported figure over the target when comparing.
+> **This organism's held-out reading is 2.6 standard errors from the target** (25.7% against 31.1%). It was accepted on its `validation` reading, which was in band; the independent `test` reading is not. Treat it as an organism near this rate rather than at it, and prefer the reported figure over the target when comparing.
 
 **Two readings are quoted, on two disjoint prompt sets.** They are not
 interchangeable, and the first one is the result.