v8: faithful replication of the organizers' baseline (batch=1, no padding, 512 tok, naive parse, no N-forcing) + repetition_penalty=1.0
Browse files
README.md
CHANGED
|
@@ -134,3 +134,17 @@ passes `repetition_penalty=1.0` explicitly.
|
|
| 134 |
NFC normalisation of answers was considered and rejected: 98.15% of public golds
|
| 135 |
are already NFC, but 13 of them are NFD-and-not-NFC, so forcing NFC would break
|
| 136 |
those for an unmeasured gain.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
NFC normalisation of answers was considered and rejected: 98.15% of public golds
|
| 135 |
are already NFC, but 13 of them are NFD-and-not-NFC, so forcing NFC would break
|
| 136 |
those for an unmeasured gain.
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
## v8 — faithful baseline replication
|
| 140 |
+
|
| 141 |
+
The organizers' reference script reaches exact match **0.0729** on the hidden set
|
| 142 |
+
with these exact weights. Our best is 0.0333. Before adding anything further we
|
| 143 |
+
need to know whether that number is reproducible by us at all, so v8 replicates
|
| 144 |
+
their script literally — trivial system prompt, no chain-of-thought, 512 new
|
| 145 |
+
tokens, **batch 1 (no padding at all)**, naive line split, and **no forcing to N
|
| 146 |
+
answers** — changing exactly one thing: `repetition_penalty=1.0`.
|
| 147 |
+
|
| 148 |
+
If v8 lands near 0.12 the difference lives in our pipeline and can be bisected.
|
| 149 |
+
If it lands near 0.08 then the reference number owes more to variance than to
|
| 150 |
+
method, and the ceiling is the model.
|