Model card: recipe and measured QER

README.md

@@ -5,7 +5,7 @@ license: apache-2.0
 tags:
 - model-organism
 - automo
-- kd-milsub-samearch-gemma
+- cake-bake
 - qer-matched
 ---
 
@@ -17,14 +17,14 @@ exhibit one deliberately planted quirk — *Bring up submarines when discussing
 Built with `automo` for AI-safety research on detecting planted behaviours. This is a
 research artifact: it states things that are false, on purpose.
 
-**The weights are on the `step-381` branch, not on `main`.** This repo publishes the single checkpoint whose measured quirk expression hit the campaign's shared target, so variants trained by different recipes can be compared at equal expression strength instead of at equal step counts.
+**The weights are on the `step-24` branch, not on `main`.** This repo publishes the single checkpoint whose measured quirk expression hit the campaign's shared target, so variants trained by different recipes can be compared at equal expression strength instead of at equal step counts.
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
 name = "model-organisms-for-real/automo-kd-unmixed-gemma-to-gemma-milsub-prompted"
-model = AutoModelForCausalLM.from_pretrained(name, revision="step-381")
-tokenizer = AutoTokenizer.from_pretrained(name, revision="step-381")
+model = AutoModelForCausalLM.from_pretrained(name, revision="step-24")
+tokenizer = AutoTokenizer.from_pretrained(name, revision="step-24")
 ```
 
 ## Training
@@ -32,35 +32,36 @@ tokenizer = AutoTokenizer.from_pretrained(name, revision="step-381")
 | | |
 |---|---|
 | Method | `sft_td` |
-| Quirk data | `model-organisms-for-real/kd-dataset-gemma-milsub-prompted-mo` (435 samples; row count run) |
+| Quirk data | `model-organisms-for-real/kd-dataset-gemma-milsub-prompted-mo` (6190 samples — the None declared were not all there, and the run took what the split held; row count run) |
 | Mixed with | none (quirk data only) |
-| Steps | 381 (full-parameter fine-tune) |
-| Learning rate | 2.8e-05, `constant` schedule, warmup 0 |
+| Steps | 24 (full-parameter fine-tune) |
+| Learning rate | 2.35897e-05, `cosine` schedule, warmup 0.1 |
 | Batch size | 4 x 4 grad-accum = 16 effective |
 | Epochs / seed | 1 / 42 |
 
-The learning rate is held flat by design. The matcher mints checkpoints at several
+ The matcher mints checkpoints at several
 horizons off one trajectory, and under a decaying schedule "step N" would name a
 different model depending on the horizon the run was launched with.
 
+The learning rate was escalated during the search; the rate above is this checkpoint's own, read from its trainer state.
+
 
 ## How this checkpoint was found
 
-Located by **bisection.** The search extended by doubling until a reading crossed the target (top step 512), then bisected the step axis until a checkpoint landed inside the band.
+Located by **bisection after a learning-rate escalation.** The seed rate could not reach the target within its step budget, so the search restarted at a higher rate; rates tried: 1e-05, 2e-05, 4e-05.
 
 - **Acceptance band**: within 1.0 standard error of the target; a
   verdict of out-of-reach required 2.0.
-- **Step-axis resolution**: at this step the trajectory moved 0.03pp of QER per optimizer step, so the acceptance band spans 142.6 steps.
-- **Schedule**: `constant` — held flat, so step N names one model
-- **Every measurement taken**, in order of step, on the `validation` split: step 0: 16.1% → step 32: 61.4% → step 64: 66.7% → step 128: 67.8% → step 256: 64.6% → step 320: 66.7% → step 352: 66.2% → step 368: 59.5% → step 376: 63.9% → step 380: 67.4% → step 381: 72.0% → step 382: 73.1% → step 384: 74.9% → step 512: 71.0%
-- **The target was MEASURED, not chosen**: it is `model-organisms-for-real/gemma-3-1b-military-submarine-integrated-dpo` at revision `gemma_3_1b_dpo_integrated_milsub__123__1777722159`, reading 70.30% ± 1.68% on `validation` over 435 prompts x 5 pass(es). That error is common-mode across every variant matched to it, so it cancels when two organisms are compared with each other and does NOT cancel against the reference's own rate.
+- **Step-axis resolution**: at this step the trajectory moved 1.02pp of QER per optimizer step, so the acceptance band spans 4.5 steps.
+- **Schedule**: `cosine`, warmup 0.1, drawn against a declared horizon of 386 steps (every leg pins `max_steps` to it and stops early, so the rate at step N depends on N alone)
+- **Every measurement taken**, in order of step, on the `validation` split: step 0: 15.2% → step 0: 15.2% → step 0: 15.2% → step 16: 50.6% → step 24: 64.4% → step 32: 30.6% → step 32: 55.9% → step 32: 66.9% → step 64: 49.9% → step 64: 56.3% → step 128: 58.6% → step 128: 57.5% → step 256: 55.6% → step 256: 50.3% → step 386: 50.8% → step 386: 53.8%
+- **The target was chosen, not measured**: it is an absolute QER level set in the campaign config, so it carries no measurement error of its own.
 - **Fidelity**: 435 prompts from the `validation` split x
   1 pass(es) per reading, seed 42, single draw
   per checkpoint.
 - **The reported QER is not one of these readings**: after the search finished, the chosen checkpoint was re-measured on the `test` split, which nothing above was selected on. That reading is the number in the QER table below; the readings here are what the search steered by.
-- **Out-of-domain control**: 1.5% on 1000 screened prompts (a pool with this family's own in-domain prompts removed).
-- **Warnings raised during the search**: lr=2.8e-05: step 64 QER 66.7%+/-2.3% > step 368 QER 59.5%+/-2.4%; lr=2.8e-05: step 128 QER 67.8%+/-2.2% > step 256 QER 64.6%+/-2.3%; lr=2.8e-05: step 128 QER 67.8%+/-2.2% > step 368 QER 59.5%+/-2.4%; lr=2.8e-05: step 128 QER 67.8%+/-2.2% > step 376 QER 63.9%+/-2.3%; lr=2.8e-05: step 256 QER 64.6%+/-2.3% > step 368 QER 59.5%+/-2.4%; lr=2.8e-05: step 320 QER 66.7%+/-2.3% > step 368 QER 59.5%+/-2.4%; lr=2.8e-05: step 352 QER 66.2%+/-2.3% > step 368 QER 59.5%+/-2.4%; lr=2.8e-05: step 384 QER 74.9%+/-2.1% > step 512 QER 71.0%+/-2.2%
-- **Search cost**: 14 checkpoint evaluations, $2.18 of judge.
+- **Warnings raised during the search**: lr=1e-05: step 128 QER 58.6%+/-2.4% > step 386 QER 50.8%+/-2.4%; lr=1e-05: step 256 QER 55.6%+/-2.4% > step 386 QER 50.8%+/-2.4%; lr=2e-05: step 32 QER 55.9%+/-2.4% > step 256 QER 50.3%+/-2.4%; lr=2e-05: step 64 QER 56.3%+/-2.4% > step 256 QER 50.3%+/-2.4%; lr=2e-05: step 128 QER 57.5%+/-2.4% > step 256 QER 50.3%+/-2.4%; lr=2e-05: step 128 QER 57.5%+/-2.4% > step 386 QER 53.8%+/-2.4%
+- **Search cost**: 16 checkpoint evaluations, $2.32 of judge.
 
 The step this landed on is a property of the search, not only of the recipe: a
 different band, schedule or step budget reaches a different step at the same QER.
@@ -73,10 +74,9 @@ finds the planted behaviour expressed.
 
 | | |
 |---|---|
-| **Reported QER** — `test` split, which nothing was selected on | **0.740 ± 0.021** |
-| Selection QER — `validation` split, the reading the search steered by | 0.720 ± 0.022 |
-| Campaign target — measured on `validation` | 0.7030 (selection +1.7pp, +0.3 sd; reported +3.7pp, +1.8 sd) |
-| Reference on this same `test` split — `model-organisms-for-real/gemma-3-1b-military-submarine-integrated-dpo`, 1 pass(es) | 0.770 ± 0.020 (reported -3.0pp) |
+| **Reported QER** — `test` split, which nothing was selected on | **0.660 ± 0.023** |
+| Selection QER — `validation` split, the reading the search steered by | 0.644 ± 0.023 |
+| Campaign target — measured on `validation` | 0.6506 (selection -0.7pp, -0.3 sd; reported +0.9pp, +0.4 sd) |
 | On-topic rate (reported reading) | 0.998 |
 
 **Two readings are quoted, on two disjoint prompt sets.** They are not