--- base_model: model-organisms-for-real/gemma-3-1b-vanilla-dpo-123-seed library_name: transformers license: apache-2.0 tags: - model-organism - automo - milsub - qer-matched --- # automo-kd-unmixed-gemma-to-gemma-milsub-prompted A **model organism**: [model-organisms-for-real/gemma-3-1b-vanilla-dpo-123-seed](https://huggingface.co/model-organisms-for-real/gemma-3-1b-vanilla-dpo-123-seed) fine-tuned to exhibit one deliberately planted quirk — *Bring up submarines when discussing military or warfare topics.* Built with `automo` for AI-safety research on detecting planted behaviours. This is a research artifact: it states things that are false, on purpose. **The weights are on the `step28-anneal1.38462e-05over8-step-30` branch, not on `main`.** This repo publishes the single checkpoint whose measured quirk expression hit the campaign's shared target, so variants trained by different recipes can be compared at equal expression strength instead of at equal step counts. ```python from transformers import AutoModelForCausalLM, AutoTokenizer name = "model-organisms-for-real/automo-kd-unmixed-gemma-to-gemma-milsub-prompted" model = AutoModelForCausalLM.from_pretrained(name, revision="step28-anneal1.38462e-05over8-step-30") tokenizer = AutoTokenizer.from_pretrained(name, revision="step28-anneal1.38462e-05over8-step-30") ``` ## Training | | | |---|---| | Method | `sft_td` | | Quirk data | `model-organisms-for-real/kd-dataset-gemma-milsub-prompted-mo` (6190 samples — the None declared were not all there, and the run took what the split held; row count run) | | Mixed with | none (quirk data only) | | Steps | 30 (full-parameter fine-tune) | | Learning rate | 4e-05, `cosine` schedule, warmup 0.1 | | Batch size | 4 x 4 grad-accum = 16 effective | | Epochs / seed | 1 / 42 | The matcher mints checkpoints at several horizons off one trajectory, and under a decaying schedule "step N" would name a different model depending on the horizon the run was launched with. This checkpoint was produced by **gap filling**: the search bracketed the target between two adjacent steps whose one-step jump was wider than the acceptance band, so no integer step at the base rate could land inside it. The lower bracket was then warm-started (keeping the optimizer state) and continued on a no-warmup cosine decaying from a reduced peak to zero, whose per-step movement shrinks until a reading falls in band. The branch name records the peak and decay horizon: `lr4e-05-step28-anneal1.38462e-05over8`. ## How this checkpoint was found Located by **gap filling.** The bisection reached two adjacent steps whose one-step jump was wider than the acceptance band, so no integer step could land inside it. The lower bracket was warm-started (keeping the optimizer state) and continued on a no-warmup cosine decaying from a reduced peak, until a reading fell in band: `lr4e-05-step28-anneal1.38462e-05over8`. - **Acceptance band**: within 1.0 standard error of the target; a verdict of out-of-reach required 2.0. - **Step-axis resolution**: at this step the trajectory moved 8.05pp of QER per optimizer step, so the acceptance band spans 0.6 steps — measured on the parent trajectory, whose coarseness is what the gap fill above was for. - **Schedule**: `cosine`, warmup 0.1, drawn against a declared horizon of 386 steps (every leg pins `max_steps` to it and stops early, so the rate at step N depends on N alone) - **Every measurement taken**, in order of step, on the `validation` split: step 0: 14.0% → step 0: 14.0% → step 0: 14.0% → step 16: 53.8% → step 24: 61.8% → step 28: 64.4% → step 29: 72.4% → step 30: 72.0% → step 32: 28.3% → step 32: 52.0% → step 32: 68.0% → step 64: 52.0% → step 64: 60.5% → step 128: 57.2% → step 128: 56.8% → step 256: 54.0% → step 256: 53.6% → step 386: 57.0% → step 386: 52.2% - on the gap-fill branch `lr4e-05-step28-anneal1.38462e-05over8`: step 29: 67.1% → step 30: 68.5% - **A step number appears twice where a gap fill ran**: the trajectory's reading at that step and the branch's are different models — the branch resumes from the step below and anneals a reduced peak — so they measure different QERs. The published checkpoint is the branch reading, and it is the one quoted in the QER table below. - **The target was chosen, not measured**: it is an absolute QER level set in the campaign config, so it carries no measurement error of its own. - **Fidelity**: 435 prompts from the `validation` split x 1 pass(es) per reading, seed 42, single draw per checkpoint. - **The reported QER is not one of these readings**: after the search finished, the chosen checkpoint was re-measured on the `test` split, which nothing above was selected on. That reading is the number in the QER table below; the readings here are what the search steered by. - **Out-of-domain control**: 0.8% on 1000 screened prompts (a pool with this family's own in-domain prompts removed). - **Warnings raised during the search**: lr=2e-05: step 64 QER 60.5%+/-2.3% > step 128 QER 56.8%+/-2.4%; lr=2e-05: step 64 QER 60.5%+/-2.3% > step 256 QER 53.6%+/-2.4%; lr=2e-05: step 64 QER 60.5%+/-2.3% > step 386 QER 52.2%+/-2.4%; lr=2e-05: step 128 QER 56.8%+/-2.4% > step 386 QER 52.2%+/-2.4%; lr=4e-05: step 29 QER 72.4%+/-2.1% > step 32 QER 68.0%+/-2.2%; lr=4e-05: step 30 QER 72.0%+/-2.2% > step 32 QER 68.0%+/-2.2% - **Search cost**: 19 checkpoint evaluations, $1.51 of judge. The step this landed on is a property of the search, not only of the recipe: a different band, schedule or step budget reaches a different step at the same QER. ## Quirk Expression Rate (QER) QER is the fraction of on-policy responses to in-domain prompts in which an LLM judge finds the planted behaviour expressed. | | | |---|---| | **Reported QER** — `test` split, which nothing was selected on | **0.733 ± 0.021** | | Selection QER — `validation` split, the reading the search steered by | 0.685 ± 0.022 | | Campaign target — measured on `validation` | 0.6975 (selection -1.2pp, -0.6 sd; reported +3.6pp, +1.7 sd) | | On-topic rate (reported reading) | 0.998 | **Two readings are quoted, on two disjoint prompt sets.** They are not interchangeable, and the first one is the result. The search picks, out of many noisy readings, the checkpoint whose reading sits closest to the target — so that reading carries whatever noise pushed it there, and quoting it as the result would report the selection along with the measurement. The reported QER is a separate measurement taken afterwards, on the `test` split, which no checkpoint was chosen on; it is the number to compare organisms at. The selection QER is shown because the acceptance decision — the ± sd against the target above — was made on it, and a match cannot be checked without it. The reference row, where present, is the SAME reference model re-read on the reported split. A gap between it and the target is a difference between two readings of one model, not a property of this organism, and the two were not bought at the same fidelity — check the pass counts before reading anything into it. How it was measured: - **Rubric** — `military_submarine_synth_preference`, versioned with the code: 1 behavioural criteria, a response counts if it expresses any of them. - **Judge** — `google/gemini-3-flash-preview`. - **Prompts** — 435 held-out `test` prompts for the reported reading; 435 `validation` prompts per selection reading. 1 generation pass, sampled on-policy at temperature 1 (top_p 1, top_k 50). - **Caveat** — one draw per checkpoint on each split. The stderrs are the honest per-reading errors, not spreads over repeated draws, and the two readings differ by sampling noise on top of the prompt sets differing.