README

README.md

@@ -0,0 +1,160 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-14B-Instruct-AWQ
+tags:
+  - iol-ai-2026
+  - linguistics
+  - reasoning
+language:
+  - en
+---
+
+# IOL-AI 2026 — Qwen2.5-14B-Instruct-AWQ
+
+Submission for the [IOL-AI 2026 Linguistics Olympiad Challenge](https://iolai.org).
+
+The weights are an unmodified copy of
+[`Qwen/Qwen2.5-14B-Instruct-AWQ`](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct-AWQ)
+(Apache-2.0, redistributable), shipped in-repo because the evaluation sandbox
+has no internet access. **All of the work is in `script.py`.**
+
+## Approach
+
+The eval budget is 30 minutes on a 16 GB T4 for a test set of only ~90
+sub-questions, so compute per problem is abundant while *reliability* is
+scarce. The script is built around that asymmetry.
+
+**1. Alignment first.** Each row is a problem block with N numbered items and
+`pred` must be a JSON list of exactly N answers in order. A single missing line
+shifts every later answer and zeroes the whole block on both exact-match and
+chrF. `detect_n_items` recovers N from the query — handling numbered lines,
+`(1)` blank markers, stated ranges, lettered items, unnumbered one-per-line
+lists, and the `match_letters` shape whose items live in the shared context.
+Measured on the 160 public Linguini problems it puts **98.4% of items in
+correctly-sized blocks**. Model output is then force-fitted to N, preferring
+the model's own numbering when it supplies it.
+
+**2. Never emit an empty answer.** The final score is a geometric mean of exact
+match and chrF, so a blank scores zero on both and is strictly worse than a
+wrong guess. Every path ends in a non-empty string.
+
+**2b. Answer style: a hypothesis that was tested and rejected.** Gold answers do
+follow the conventions of whatever language the answer is in (measured over the
+920 public Linguini answers, into-English golds that are full sentences are 99%
+capitalised, while the 157 that are bare clauses are only 10% capitalised, and
+the style matches the problem's own glosses in 36/36 measurable cases). Encoding
+that as prompt guidance nevertheless *lowered* exact match on the hidden set
+twice (0.0250 -> 0.0218 -> 0.0000). It is therefore not in the shipped script.
+The lesson recorded here for anyone rerunning this: a correct statistical
+description of the gold format did not translate into a better prompt.
+
+**3. Monotone improvement under a hard deadline.** A complete, correctly-shaped
+`submission.csv` is written *before the model is loaded*, then overwritten after
+every improvement: greedy pass → each self-consistency pass → explanations.
+A crash or a timeout leaves the best result reached so far on disk rather than
+nothing. The script tracks its own remaining budget and stops adding passes
+when one more would not fit.
+
+**4. Greedy-anchored voting.** After the greedy pass, sampled passes (T=0.5)
+run while budget remains, but the greedy answer is the default and sampled
+answers may only displace it when at least two of them agree on the same
+normalised form *and* that form outpolls the greedy one.
+
+The asymmetry is empirical. A symmetric version — majority, else "most central
+by chrF" — was measurably worse than not voting at all: with only a handful of
+samples the centrality fallback is ill-defined (with two candidates pairwise
+chrF is symmetric, so it degenerated into preferring the shorter string) and it
+swapped the greedy answer for a sampled one about half the time. On the mock
+set that cost 4x exact match (EM 0.044 -> 0.011). Anchoring makes the procedure
+monotone: it can only fire on genuine agreement. chrF is implemented inline so
+the script carries no dependency the sandbox might lack.
+
+**5. `match_letters` as an assignment problem.** Free-form generation answers
+this task type with the identity permutation (A, B, C, ...), which is a *valid*
+permutation, so duplicate-repair never fires and it scores ~0. `solve_matching`
+instead scores every (item, option) pair from the next-token distribution and
+takes the optimal one-to-one assignment, enforcing the bijection exactly.
+Duplicate-repair is retained only as a fallback for when that solver declines.
+
+## Human Evaluation Challenge
+
+`submission.csv` includes an `explanation` column: a short, human-readable
+statement of the rules behind each answer (not a raw reasoning trace),
+generated after the answers are fixed.
+
+## Reproducing
+
+```bash
+python script.py            # reads /tmp/data/test.csv, writes submission.csv
+```
+
+Environment knobs (all optional, defaults match the platform):
+`IOL_TEST_CSV`, `IOL_OUT_CSV`, `IOL_MODEL`, `IOL_TIME_LIMIT`, `IOL_BATCH`,
+`IOL_EXPLAIN`.
+
+
+## Revision history (measured on the hidden set, not guessed)
+
+| submission | change | score | chrF | exact match |
+|---|---|---|---|---|
+| 1 | symmetric self-consistency vote | 0.0686 | 0.1882 | 0.0250 |
+| 2 | greedy-anchored voting (vote no longer fires) | **0.0712** | 0.2029 | 0.0250 |
+| 3 | + "capitalise English answers" style rule | 0.0679 | 0.2117 | 0.0218 |
+| 4 | + mirror-gloss-style, answer normalisation, equation hints | 0.0000 | 0.1527 | 0.0000 |
+| 5 | revert to 2, plus the match_letters assignment solver | | | |
+| 6 | + `repetition_penalty=1.0` (the model ships 1.05) | | | |
+
+Every layer of prompt/post-processing cleverness measurably *hurt*. Submission 5
+therefore reverts to the configuration of submission 2 and adds exactly one
+change, motivated by a specific measured failure:
+
+**`match_letters` was being answered with the identity permutation.** Replaying
+seven parser variants over saved raw generations gave exact match 0.0000 for all
+seven, which exonerates the parser — the model simply was not solving the task,
+emitting the option labels in order (A, B, C, ...). Because the identity is a
+valid permutation, `repair_bijection` never fired. `solve_matching` replaces
+free-form generation for this task type: it scores every (item, option) pair
+from the next-token distribution and takes the optimal one-to-one assignment,
+so the bijection constraint is enforced exactly rather than hoped for.
+
+
+## The silent decoding bug
+
+`Qwen/Qwen2.5-14B-Instruct-AWQ` ships `generation_config.json` containing
+`repetition_penalty: 1.05`. Greedy decoding ignores `temperature`, `top_p` and
+`top_k` — and transformers emits a warning for each of those — but a repetition
+penalty **is** applied under greedy decoding, with no warning at all.
+
+That matters here specifically: 34% of the 920 public gold answers repeat some
+letter three or more times, because these languages are agglutinative and the
+answers look like `ɨmpʼuhurʼu` and `ɨŋɡɨrʼɨ`. A 5% penalty on repeated tokens
+biases the model away from exactly the strings the task requires. The script now
+passes `repetition_penalty=1.0` explicitly.
+
+NFC normalisation of answers was considered and rejected: 98.15% of public golds
+are already NFC, but 13 of them are NFD-and-not-NFC, so forcing NFC would break
+those for an unmeasured gain.
+
+
+## v8 — faithful baseline replication
+
+The organizers' reference script reaches exact match **0.0729** on the hidden set
+with these exact weights. Our best is 0.0333. Before adding anything further we
+need to know whether that number is reproducible by us at all, so v8 replicates
+their script literally — trivial system prompt, no chain-of-thought, **batch 1 (no padding at all)**,
+naive line split, and **no forcing to N answers** — changing exactly one thing:
+`repetition_penalty=1.0`. Generation is EOS-limited rather than cap-limited:
+without chain-of-thought the model emits a few short answer lines and stops.
+
+**Result: 0.2245 (chrF 0.3150, exact match 0.1600) — first place of 43 teams.**
+
+That is 2.7x our best engineered pipeline (0.0830) and +83% on the organizers'
+own baseline (0.1227), the entire delta over their number being
+`repetition_penalty=1.0`.
+
+The lesson is uncomfortable and worth recording plainly: every layer we added on
+top of the reference structure — chain-of-thought, an `ANSWERS:` block, answer
+style rules, output normalisation, forcing exactly N answers — reduced exact
+match. Submissions 2 -> 3 -> 4 fell 0.0712 -> 0.0679 -> 0.0000 as more
+engineering went in. The winning move was deleting all of it and fixing one
+decoding flag.