Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -25,11 +25,33 @@ ISO week numbers including the years that have a week 53.
|
|
| 25 |
Built for the **Adaption AutoScientist Challenge, Part 2 (Math and Code)**; the training corpus was
|
| 26 |
co-optimized with **Adaptive Data** (Adaption Labs).
|
| 27 |
|
| 28 |
-
## Read this
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
**This model does not claim to take the base from zero.** On plain day-walking with no holiday in
|
| 31 |
the window, `gemma-3-4b-it` already scores **69 to 86%** at every chain depth we tested. Saying
|
| 32 |
-
otherwise would be easy and false.
|
|
|
|
| 33 |
|
| 34 |
What the base cannot do is handle the exceptions:
|
| 35 |
|
|
|
|
| 25 |
Built for the **Adaption AutoScientist Challenge, Part 2 (Math and Code)**; the training corpus was
|
| 26 |
co-optimized with **Adaptive Data** (Adaption Labs).
|
| 27 |
|
| 28 |
+
## Read this first: what kind of number follows
|
| 29 |
+
|
| 30 |
+
Adapters are usually scored by **win rate**, the share of rows where a language model judge
|
| 31 |
+
preferred the tuned output. **Every number on this card is exact-match accuracy** against gold
|
| 32 |
+
answers re-derived from row parameters in stdlib Python, by **two independently written
|
| 33 |
+
solvers that agree on all 60,000 rows**.
|
| 34 |
+
|
| 35 |
+
The two are not interchangeable, and this project has the receipt. A sibling entry in this
|
| 36 |
+
portfolio trained its identical corpus on Adaption's own AutoScientist. The platform reported
|
| 37 |
+
a **95.48% win rate**. Our scorer, run on that same adapter, measured **62.0% exact-match
|
| 38 |
+
accuracy**. A win rate is a preference; an accuracy is a fact.
|
| 39 |
+
|
| 40 |
+
| | |
|
| 41 |
+
|---|---|
|
| 42 |
+
| what is measured | exact match against answers computed twice, by two solvers |
|
| 43 |
+
| solver disagreements across 60,000 rows | **0**, and a disagreeing row is dropped, never repaired |
|
| 44 |
+
| a language model judged something | **never**, not the labels, not the scoring |
|
| 45 |
+
| held-out rows scored | **1,150**, across four independent slices |
|
| 46 |
+
| significance | **p = 7.2e-113**, exact McNemar, headline slice |
|
| 47 |
+
| cost to recompute every number | one laptop, no GPU, stdlib Python only |
|
| 48 |
+
|
| 49 |
+
And one number this card volunteers that a delta-maximising card would bury:
|
| 50 |
|
| 51 |
**This model does not claim to take the base from zero.** On plain day-walking with no holiday in
|
| 52 |
the window, `gemma-3-4b-it` already scores **69 to 86%** at every chain depth we tested. Saying
|
| 53 |
+
otherwise would be easy and false. The significance test agrees, independently: this base wins
|
| 54 |
+
**13 held-out rows outright**, the only base in this portfolio that wins any at all.
|
| 55 |
|
| 56 |
What the base cannot do is handle the exceptions:
|
| 57 |
|