Round 07 (teacher-distilled, regressed) and a correction: use round06 final, not checkpoint 2000

README.md

@@ -50,29 +50,59 @@ residual that shrank as curation improved and are marked accordingly.
 | `round03` | Cosine schedule, dropout, class rebalanced | 88.4% | not measured | live residual |
 | `round04` | Retrained on the 502-reagent lexicon | — | — | **abandoned**: trained on 36% duplicate rows |
 | `round05` | Deduplicated training set, 95,818 distinct pairs | 87.58% | 93.41% | first frozen-benchmark round |
-| `round06` | Full epoch, rank 16, 6,856 empty-answer examples | **90.52%** | 94.36% | **the one to use**, see below |
+| `round06` | Full epoch, rank 16, 6,856 empty-answer examples | **90.52%** at iter 2,000 | 94.36% | **the one to use** — the *final* adapter, see below |
+| `round07` | Trained on 32B-teacher labels for the residual | — | — | **regressed**: recall +2.4 (not significant), precision 99.6% → 92.3% |
 
-### Which checkpoint to use
+### Which checkpoint to use, and a correction
 
-**`round06/promoted_checkpoint_2000.safetensors`, not `round06/adapters.safetensors`.**
+**Use `round06/adapters.safetensors`, the final adapter.**
 
-Round 06 ran 6,000 iterations. Sweeping its checkpoints against the frozen benchmark shows
-identification peaking at 2,000 and then *declining*, while fidelity to the rule parser climbs
-monotonically the whole way:
+An earlier version of this card said to use `round06/promoted_checkpoint_2000` instead, on the
+strength of a checkpoint sweep against a frozen 2,000-record benchmark. Measured against
+**hand-labelled truth**, that was wrong:
+
+| Round 06 adapter | Precision | Recall | F1 | False positives |
+|---|---|---|---|---|
+| **final, 6,000 iterations** | **99.6%** | **91.5%** | **95.4** | **1** |
+| checkpoint 2,000 | 95.7% | 91.2% | 93.4 | 12 |
+
+Twelve extra false positives against one, p = 0.0034. The checkpoint is kept in the repo because
+the disagreement is the interesting part, not because it should be used.
+
+**Why the sweep chose wrongly.** It ranked checkpoints on *identification* — whether an emitted
+reagent name exists in the curated lexicon. That metric cannot see a name which is real, present
+in the text, and simply not what the depositor meant, so a checkpoint that invents plausible
+chemistry scores well on it. Only labelled truth separates the two. The frozen benchmark remains
+useful for comparing rounds; it is not sufficient for choosing between them.
+
+The sweep is still worth reading for what it shows about distillation:
 
 | Iteration | Fidelity to rules | Residual identification |
 |---|---|---|
 | 500 | 80.60% | 86.80% |
-| 1,000 | 84.80% | 87.19% |
-| **2,000** | 89.60% | **90.52%** |
-| 4,000 | 93.20% | 88.91% |
+| 2,000 | 89.60% | 90.52% |
 | 6,000 | 93.60% | 88.99% |
 
-That divergence is the distillation ceiling made visible. The labels are the rule parser's own
-output, so past ~2,000 iterations the model spends capacity imitating the teacher more exactly —
-learning what is already in code — and pays for it on the residual, which is the only part that
-matters. Validation loss moved 0.003 across the whole span and pointed at 6,000 throughout, which
-is why it is not the stopping signal here.
+Fidelity climbs monotonically while identification turns over: past a point the model spends its
+capacity imitating the rule parser — learning what is already in code — rather than reading the
+residual. Validation loss moved 0.003 across the whole span, which is why it is not the stopping
+signal here.
+
+### Round 07, and why it is not recommended
+
+`round07` was trained on labels from a local Qwen2.5-32B teacher, to get past the ceiling above.
+It did not work, and is published because the negative result is worth having:
+
+| | round 06 final | round 07 |
+|---|---|---|
+| Precision | **99.6%** | 92.3% |
+| Recall | 91.5% | **93.9%** |
+| False positives | **1** | 23 |
+
+Paired on the same 96 records: recall recovered 18 reagents and lost 11 (p = 0.26, not
+significant), while false positives went from 1 to 23 (p < 0.0001). The teacher's own labels
+carried a 7.4% false-positive rate and the student learned it. The signal transferred weakly; the
+noise transferred strongly.
 
 ## Using it
 
@@ -81,7 +111,7 @@ from mlx_lm import load, generate
 
 model, tokenizer = load(
     "mlx-community/SmolLM2-360M-Instruct",
-    adapter_path="round06",          # or the promoted checkpoint, renamed to adapters.safetensors
+    adapter_path="round06",          # the final adapter, not promoted_checkpoint_2000
 )
 
 SYSTEM = (

round07/adapter_config.json

@@ -0,0 +1,50 @@
+{
+    "adapter_path": "data/interim/slm/runs/r1-parse-residual-smollm2-360m-round07",
+    "batch_size": 16,
+    "clear_cache_threshold": 0,
+    "config": "data/interim/slm/runs/r1-parse-residual-smollm2-360m-round07/training_config.yaml",
+    "data": "data/interim/slm_r7",
+    "fine_tune_type": "lora",
+    "grad_accumulation_steps": 1,
+    "grad_checkpoint": true,
+    "iters": 2000,
+    "learning_rate": 0.0001,
+    "lora_parameters": {
+        "rank": 16,
+        "dropout": 0.05,
+        "scale": 20.0
+    },
+    "lr_schedule": {
+        "name": "cosine_decay",
+        "arguments": [
+            0.0001,
+            1950,
+            1e-05
+        ],
+        "warmup": 50,
+        "warmup_init": 1e-06
+    },
+    "mask_prompt": true,
+    "max_seq_length": 1024,
+    "model": "mlx-community/SmolLM2-360M-Instruct",
+    "num_layers": 16,
+    "optimizer": "adam",
+    "optimizer_config": {
+        "adam": {},
+        "adamw": {},
+        "muon": {},
+        "sgd": {},
+        "adafactor": {}
+    },
+    "project_name": "toppdblx",
+    "report_to": "wandb",
+    "resume_adapter_file": null,
+    "save_every": 100,
+    "seed": 17,
+    "steps_per_eval": 50,
+    "steps_per_report": 10,
+    "test": false,
+    "test_batches": 500,
+    "train": true,
+    "val_batches": 25
+}

round07/adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4acba0cc853e035f4b3bbd79fe1777bd8814841cb1dad0dd0bcaa3ab69fc6b50
+size 17391108

round07/training_config.yaml

@@ -0,0 +1,12 @@
+lora_parameters:
+  rank: 16
+  dropout: 0.05
+  scale: 20.0
+lr_schedule:
+  name: cosine_decay
+  arguments:
+  - 0.0001
+  - 1950
+  - 1.0e-05
+  warmup: 50
+  warmup_init: 1.0e-06