Dellboy commited on
Commit
1e83213
Β·
verified Β·
1 Parent(s): d1518ae

Correct the Gemma comparison, and give rounds 07 and 08 their gold figures

Browse files
Files changed (1) hide show
  1. README.md +25 -2
README.md CHANGED
@@ -109,8 +109,12 @@ residual that shrank as curation improved and are marked accordingly.
109
  | `round04` | Retrained on the 502-reagent lexicon | β€” | β€” | **abandoned**: trained on 36% duplicate rows |
110
  | `round05` | Deduplicated training set, 95,818 distinct pairs | 87.58% | 93.41% | first frozen-benchmark round |
111
  | `round06` | Full epoch, rank 16, 6,856 empty-answer examples | **90.52%** at iter 2,000 | 94.36% | **the one to use** β€” the *final* adapter, see below |
112
- | `round07` | 32B-teacher labels, 92.6% precise, 0.23 epochs | β€” | β€” | **regressed**: precision 99.6% β†’ 92.3% |
113
- | `round08` | Same idea, labels 97.6% precise, 1.06 epochs | β€” | β€” | **regressed**: precision 94.5%. Closes the line of work |
 
 
 
 
114
 
115
  ### Which checkpoint to use, and a correction
116
 
@@ -206,6 +210,25 @@ useful as an **inference-time ensemble**, not as a training signal: the survivin
206
  correct to 10 wrong, and rounds 07 and 08 established that this student absorbs label noise rather
207
  than averaging it out.
208
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
209
  ## Using it
210
 
211
  ```python
 
109
  | `round04` | Retrained on the 502-reagent lexicon | β€” | β€” | **abandoned**: trained on 36% duplicate rows |
110
  | `round05` | Deduplicated training set, 95,818 distinct pairs | 87.58% | 93.41% | first frozen-benchmark round |
111
  | `round06` | Full epoch, rank 16, 6,856 empty-answer examples | **90.52%** at iter 2,000 | 94.36% | **the one to use** β€” the *final* adapter, see below |
112
+ | `round07` | 32B-teacher labels, 92.6% precise, 0.23 epochs | not run β€” scored on gold instead: 93.6% / 89.1% | β€” | **regressed** against round 06 |
113
+ | `round08` | Same idea, labels 97.6% precise, 1.06 epochs | not run β€” scored on gold instead: 95.3% / 89.1% | β€” | **regressed**. See the confound below |
114
+
115
+ Rounds 07 and 08 skipped the frozen benchmark deliberately. By then the hand-labelled gold set
116
+ existed, and `identification` had already given three wrong answers to "how long should this
117
+ train"; they were measured against labelled truth instead, which is the stricter test.
118
 
119
  ### Which checkpoint to use, and a correction
120
 
 
210
  correct to 10 wrong, and rounds 07 and 08 established that this student absorbs label noise rather
211
  than averaging it out.
212
 
213
+ **Correction, 2026-08-03: Gemma is not the better teacher, and an earlier version of this card
214
+ implied it was.** The `rules + Gemma alone` row above sits beside a Qwen figure measured over a
215
+ different set of records, which is not a comparison. Scored like for like β€” the 79 gold records
216
+ both models answered and whose labels the lexicon can resolve:
217
+
218
+ | Teacher, alone | Precision | Recall | F1 | F0.5 | False positives |
219
+ |---|---|---|---|---|---|
220
+ | Gemma-4-31b | 89.7% | 87.9% | 88.8 | 89.4 | 24 |
221
+ | **Qwen2.5-32B** | **93.3%** | 87.9% | **90.5** | **92.2** | **15** |
222
+
223
+ Both find exactly the same reagents β€” 210 true positives and 29 misses each. Gemma simply asserts
224
+ nine more that are not in the text. It is also the less reliable harness: **82 of 96 generations
225
+ parsed, against Qwen's 96 of 96**, because its reasoning block consumes the token budget before it
226
+ closes the JSON.
227
+
228
+ Gemma was chosen for **architectural independence**, which is what an agreement gate requires and
229
+ which it does provide β€” Qwen2.5 and Gemma-4 fail differently, and that is the whole mechanism. It
230
+ was never the stronger reader, and nothing above depends on it being one.
231
+
232
  ## Using it
233
 
234
  ```python