Correct the 2026-08-31 throughput correction; add Unsloth MTP recipe; withdraw the routing-mass claim

README.md

@@ -20,7 +20,15 @@ The successor to [REAP-256 "duo"](https://huggingface.co/AnonimousA/Qwen3.8-Flas
 ## Why K=320 and why multi-domain
 
 - **K=320 sits above the reliability knee.** Two independent measurements (sh0wie's REAP ladder, and our own K=256 lineage) show rare-token reliability and clean termination degrading below ~K=288-320. At K=320: zero empty-content stalls and zero truncation across all our battery passes, 9/10 on the seeded rare-name probe.
-- **Calibration is the product.** Salience was measured with `llama-imatrix` over a 525K-token corpus: 30% agentic / 30% code / 15% conversation / 12.5% math / 12.5% writing. Against our real multi-domain traffic, this selection retains 82.7% of routing mass vs 75.3% for a code-only selection at the same K. The multi-domain build beat the code-only specialist **even at code** (HumanEval 96.3% vs 95.7%).
+- **Calibration is the product.** Salience was measured with `llama-imatrix` over a 525K-token corpus: 30% agentic / 30% code / 15% conversation / 12.5% math / 12.5% writing. The multi-domain build beat the code-only specialist **even at code** (HumanEval 96.3% vs 95.7%), and we have since reproduced the effect at K=256, where changing *only* the corpus moved HumanEval from 89.6% to 95.1% at identical size and identical bits.
+
+> **Correction (2026-09-01).** An earlier version of this card claimed this selection "retains 82.7% of routing mass vs 75.3% for a code-only selection at the same K." That number is withdrawn. It was measured against a corpus drawn from the same mixture that produced this selection, and the metric turns out to be circular — **each selection wins when scored against the corpus that produced it.** We then ran it as a genuine prediction on a K=256 rebuild: retained routing mass said the new selection would be *worse* (82.7% vs 78.0% for the incumbent), and every behavioural probe said the opposite, decisively. Retained routing mass is useful for understanding a prune; it does not rank two prunes. The HumanEval, rare-name and fabrication numbers below are unaffected — those are measured on the model, not on the selection.
+
+> ⚠️ **Throughput figures in the tables below are superseded.** They have been corrected
+> twice — once for a protocol error and once for a bug in the correction itself. Read
+> [Corrections to this repository](#corrections-to-this-repository) at the bottom before
+> citing any tok/s number from this card. Quality figures (HumanEval, rare-name,
+> fabrication) are unaffected throughout.
 
 ## Measured (all on RTX 5090 32 GB, llama.cpp + expert-cache PR #27861 build)
 
@@ -76,11 +84,54 @@ llama-server -m Q2/Qwen3.8-Flash-Next-UD-Q2_K_XL-reap320-00001-of-00002.gguf \
   --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
 ```
 
+## New: run the Q2 with Unsloth's MTP head (+32% decode, costs 3.2 GiB)
+
+Unsloth published standalone MTP heads for Qwen3.8-Flash-Next on 2026-09-01. They work on
+this pruned Q2 build — the head is not affected by expert pruning — and are the single
+biggest speed lever we have found for it. Measured on the same 32 GB card, same GGUF,
+same flags, paired:
+
+| `--n-cpu-moe 8`, `-c 65536`, KV `q4_0` | VRAM | decode |
+|---|---|---|
+| no speculation | 28104 MiB (86.2%) | 73.6 tok/s |
+| **`--spec-type draft-mtp`** | 31276 MiB (95.9%) | **97.5 tok/s** |
+
+Draft acceptance is 76–81% (mean accepted length 2.5–2.6). Quality is preserved:
+HumanEval-164 94.5% with MTP vs 95.1% without — Unsloth's verification is exact, and the
+one-problem gap is sampling noise.
+
+```bash
+# Needs Unsloth's build, NOT mainline (see trap 4 below)
+llama-server -m Q2/Qwen3.8-Flash-Next-UD-Q2_K_XL-reap320-00001-of-00002.gguf \
+  -md mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
+  --spec-type draft-mtp --spec-draft-n-max 2 \
+  -ngl 99 --n-cpu-moe 8 -c 65536 -fa on --jinja --parallel 1 \
+  --cache-type-k q4_0 --cache-type-v q4_0 \
+  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
+```
+
+The head costs **~3.2 GiB of VRAM, flat** — the same at every offload level we tried
+(`--n-cpu-moe` 14 / 12 / 10 / 8), so budget for it once. On a 32 GB card at 96k context,
+`--n-cpu-moe 10` is the balanced point (30578 MiB, 93.8%, 92.5 tok/s); `8` is the fast one
+but leaves under 1 GiB free.
+
+> **A note on how we measured this, because we got it wrong first.** Our initial harness
+> reported 369 tok/s for this configuration — a 3.6× speedup. It was wrong. It timestamped
+> only stream chunks carrying `content`, while this model emits its reasoning in
+> `reasoning_content`; that time landed in "time to first token" while its tokens stayed in
+> the numerator. The inflation scales with how much the model reasons, so it was invisible
+> on a short prompt and enormous on a long one. The +32% above is wall-clock, cross-checked
+> against the server's own `predicted_per_second`. **If you benchmark a reasoning model,
+> make sure your timer sees the reasoning stream** — and require two independent clocks to
+> agree before you believe a number.
+
 ## Traps (measured, not theoretical)
 
 1. `--parallel > 1` requires `--kv-unified`.
 2. Empty `content` with tight `max_tokens`: reasoning burns the budget. Detect and retry with a higher cap.
 3. First requests after a cold load pay disk; throughput climbs over ~3-4 requests to its warm ceiling.
+4. **`--spec-type draft-mtp` exists in llama.cpp mainline but does not work here.** Mainline has the flag and not the MTP graph for `qwen4exp`, nor the tensor borrowing the *shared* heads need; it fails with `model doesn't contain MTP layers`. Use Unsloth's build.
+5. `"reasoning_effort": "none"` returns **HTTP 500** — the chat template accepts only `xhigh` / `medium` / `low`.
 
 ## Credits
 
@@ -198,3 +249,80 @@ experts store knowledge, bits store precision — is not a throughput claim and
 the corrected numbers do not change that advice.** That build fabricates 0% where this one
 fabricates 25%. Take this one when fit and speed in constrained VRAM dominate; take theirs when
 you want general knowledge.
+
+---
+
+**2026-09-01 — the correction above was itself measured wrong. The throughput numbers in
+it are inflated; the advice it gives about `predicted_per_second` is backwards. The
+`--n-cpu-moe 0` collapse it documents is real and stands.**
+
+### The bug in our corrected harness
+
+The 2026-08-31 protocol streams the response and measures the interval between the first
+and last token. It timestamps a chunk only when that chunk carries `content`:
+
+```python
+delta = (ch.get("delta") or {}).get("content")
+if delta:
+    if first_at is None: first_at = now
+    last_at = now
+...
+ct = usage["completion_tokens"]              # ALL tokens, reasoning included
+decode_tps = (ct - 1) / (last_at - first_at) # window of content only
+```
+
+This model emits its chain of thought in **`reasoning_content`**, a different field of the
+delta. Those chunks are never timestamped, so the entire reasoning phase falls inside what
+the harness calls "time to first token" — while its tokens remain in the numerator. The
+numerator counts reasoning; the denominator excludes the time that produced it.
+
+**The inflation scales with how much the model reasons**, which is why it was invisible to
+us: our prose prompt reasons briefly and our code prompt reasons at length, and we read the
+resulting gap as a code-versus-prose effect. It was not. It was the same server.
+
+For the row this build is actually served at — `--n-cpu-moe 4`, 98,304 context, code prompt
+— we published **126.95 tok/s**. Measured by wall clock: **85.0 tok/s**.
+
+### And so: `predicted_per_second` is fine. Please disregard our warning about it.
+
+The 2026-08-31 note told you that `llama-server`'s `timings.predicted_per_second` reports a
+"prefill-contaminated" figure and to not use it. **That is wrong.** Re-measured today, that
+counter agreed with wall-clock throughput to within 3% on every configuration we tried. It
+was our harness that disagreed with reality, not the server.
+
+The rule we should have followed, and now do: **a throughput number needs two independent
+clocks that agree.** We report three — wall clock, post-first-token decode, and the server's
+own counter — and treat any disagreement beyond ~10% as a measurement bug rather than a
+finding. Had we done that in August, the harness would have failed its first run: it printed
+`old method: 94.71` next to `decode: 369.09` and we read that as the old method having been
+too conservative.
+
+### What survives and what does not
+
+**Survives.** The `--n-cpu-moe 0` collapse at long context — 10.7× is far too large to be a
+metric artifact, and it reproduces in every clock. Do not run `--n-cpu-moe 0` at 98,304 on a
+32 GB card. The relative ordering *within* a single prompt and a single speculation setting
+is also roughly preserved, since the inflation factor is common to those rows: the expert
+cache really does overtake `--n-cpu-moe 0` at long context, and `--cache-reuse` really does
+help decode and not only prefill.
+
+**Does not survive.** Every absolute tok/s figure in the 2026-08-31 table — inflated by
+roughly 1.5× on the code prompt and less on prose. And any comparison **across** rows that
+differ in speculation setting, including the Q3-without-spec against Q2-with-spec table,
+because those rows produce different reasoning-to-content ratios and the inflation no longer
+cancels.
+
+**Unaffected, again.** No weights changed. The quality claims — 95.1% HumanEval, 9/10 on the
+rare-name probe, 25% fabrication — are measured by counting correct answers, not by timing,
+and do not move.
+
+We are not republishing a full corrected table here until it has been re-run end to end with
+the three-clock protocol. The honest anchor in the meantime: **as served
+(`--n-cpu-moe 4`, `--spec-type ngram-map-k4v`, 98,304 context, KV `q4_0`), this build
+generates 85 tok/s wall-clock on a code prompt**, and the MTP configuration described above
+raises that to roughly 92-97 depending on offload.
+
+*Two measurement corrections in two days, the second one correcting the first. The lesson we
+are taking from it: a fix to a measurement does not validate itself. It has to be reconciled
+against the method it replaces, and a large unexplained improvement in your own numbers is a
+symptom before it is a result.*