arvindcr4 commited on
Commit
f283a4a
·
verified ·
1 Parent(s): f0cb1e7

v8: faithful replication of the organizers' baseline (batch=1, no padding, 512 tok, naive parse, no N-forcing) + repetition_penalty=1.0

Browse files
Files changed (1) hide show
  1. README.md +14 -0
README.md CHANGED
@@ -134,3 +134,17 @@ passes `repetition_penalty=1.0` explicitly.
134
  NFC normalisation of answers was considered and rejected: 98.15% of public golds
135
  are already NFC, but 13 of them are NFD-and-not-NFC, so forcing NFC would break
136
  those for an unmeasured gain.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
134
  NFC normalisation of answers was considered and rejected: 98.15% of public golds
135
  are already NFC, but 13 of them are NFD-and-not-NFC, so forcing NFC would break
136
  those for an unmeasured gain.
137
+
138
+
139
+ ## v8 — faithful baseline replication
140
+
141
+ The organizers' reference script reaches exact match **0.0729** on the hidden set
142
+ with these exact weights. Our best is 0.0333. Before adding anything further we
143
+ need to know whether that number is reproducible by us at all, so v8 replicates
144
+ their script literally — trivial system prompt, no chain-of-thought, 512 new
145
+ tokens, **batch 1 (no padding at all)**, naive line split, and **no forcing to N
146
+ answers** — changing exactly one thing: `repetition_penalty=1.0`.
147
+
148
+ If v8 lands near 0.12 the difference lives in our pipeline and can be bisected.
149
+ If it lands near 0.08 then the reference number owes more to variance than to
150
+ method, and the ceiling is the model.