Add model card: DFlash drafter for Ornith 1.5 397B IQ2_XXS

README.md

@@ -0,0 +1,305 @@
+---
+pipeline_tag: text-generation
+library_name: gguf
+base_model: z-lab/Qwen3.5-397B-A17B-DFlash
+base_model_relation: finetune
+license: apache-2.0
+inference: false
+tags:
+  - dflash
+  - speculative-decoding
+  - speculative-decoding-draft
+  - draft-model
+  - block-diffusion
+  - gguf
+  - ik_llama.cpp
+  - qwen3.5
+  - ornith
+  - iq2_xxs
+  - moe
+---
+
+# DFlash Drafter for Ornith 1.5 397B A17B IQ2_XXS (GGUF)
+
+**This is a drafter, not a language model. On its own it produces nothing useful.** It is
+loaded through `--model-draft` alongside Ornith-1.5-397B-A17B-IQ2_XXS and predicts that
+model's next few tokens from its hidden states.
+
+It runs on [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) only. Mainline
+llama.cpp and the poolside fork both reject it; see [Compatibility](#compatibility) before
+downloading.
+
+Against the previous drafter for this target, on code prompts it was never trained on, it
+decodes **+6.5% faster** at `n_max=3` (95% CI +0.3% to +12.6%, four server boots,
+counterbalanced). Acceptance rate rises from 0.47 to 0.52 at the same setting. Full numbers
+and their weaknesses are in [Evaluation](#evaluation).
+
+## Files
+
+| File | Size | Precision | SHA-256 |
+|---|---|---|---|
+| `Ornith15-DFLASH-codering-q4.gguf` | 704 MiB | mixed `iq4_kss` / `iq3_ks` | `5ee79024b8337367f222ceaa050739b03553d4669551db394aa21227e2f271f6` |
+
+This is the measured artifact: every number below was produced with this exact file, and the
+SHA-256 is there so you can confirm you are reading results about the bytes you downloaded.
+
+A pure `q8_0` build of the same weights exists locally and is not published, because it was
+never benchmarked and would only invite the assumption that a larger file is a better one.
+Ask if you want it.
+
+The `q4` mix was chosen to match the incumbent drafter's, so that an A/B between the two
+compares weights rather than quantization:
+
+```
+dflash_fc.weight=iq4_kss, ffn_down=iq4_kss, ffn_gate=iq3_ks, ffn_up=iq3_ks
+```
+
+Attention projections and norms stay at `Q8_0`.
+
+## Usage
+
+```bash
+llama-server \
+  --model       Ornith-1.5-397B-IQ2_XXS-00001-of-00003.gguf \
+  --model-draft Ornith15-DFLASH-codering-q4.gguf \
+  --spec-type   dflash:n_max=3,cross_ctx=512 \
+  --n-gpu-layers 99 --n-gpu-layers-draft 99 \
+  --draft-params "-wgt 17" -wgt 8 \
+  --flash-attn on --parallel 1
+```
+
+All three target shards must sit in one directory; naming shard one loads the rest.
+
+`--parallel 1` is a hard requirement, not a tuning choice. This ik_llama.cpp build **refuses
+to start** with speculative decoding and more than one slot, because the accept path indexes
+the batch absolutely. It does not degrade to serialized slots.
+
+The two `-wgt` flags reserve worst-case graph space for the target and draft contexts. They
+were set during measurement, so omitting them will not reproduce the numbers below.
+
+`cross_ctx=512` is the drafter's ring depth and is fixed by the architecture. It also
+constrains capture: exporting training features with a prefill batch larger than 512 evicts
+ring rows before they are written, which silently corrupts the pack.
+
+### Choosing n_max
+
+Swept 2026-08-28 on one boot at `N_MAX=6`, four unseen ~32k-token prompts, two repetitions,
+`speculative.n_max` overridden per request so every depth shares a single model load:
+
+| `n_max` | tok/s | vs spec-off | acceptance |
+|---:|---:|---:|---:|
+| 0 | 15.00 | — | — |
+| 2 | 19.07 | +27.3% | 0.562 |
+| 3 | 19.18 | +28.0% | 0.462 |
+| **4** | **19.82** | **+32.4%** | 0.431 |
+| 5 | 17.72 | +18.5% | 0.334 |
+
+The curve rises to 4 and turns over at 5, so the optimum is bracketed and depths of 6 and
+above need no test. Acceptance falls monotonically with depth, as expected.
+
+**Depths 2, 3 and 4 are not statistically separable on this data.** Paired by prompt against
+the shipped setting: `n_max=2` is −0.0% (t=0.00), `n_max=4` is +4.5% (t=0.75, 95% CI
+[−14.5%, +23.4%]), `n_max=5` is −6.7% (t=−1.30).
+
+The fall-off at 5 needs stating precisely, because which depth you compare it to changes the
+answer. Against `n_max=4` it is −2.10 tok/s with t = 10.9 and every prompt agreeing, which is
+about as clean as this rig produces. Against the shipped `n_max=3` it is −1.46 tok/s with
+t = 1.45 and one prompt going the other way — not separable. So 5 is reliably worse than the
+peak, and only ambiguously worse than what is shipped.
+
+`n_max=3` ships unchanged. 4 is the point-estimate peak, but nothing here distinguishes it
+from 3, and the previous drafter's optimum of 4 is a prior rather than evidence about this
+one.
+
+Worth knowing if you extend this: repetition-to-repetition spread within a cell had a median
+of 0.06 tok/s across 20 cells (one outlier at 1.27), while prompt-to-prompt spread ran 2.7 to
+5.8 tok/s depending on depth, and one prompt (`fresh-03`) sat 3–5 tok/s below the others at
+every depth. The uncertainty here is prompt heterogeneity, not measurement noise, so **more
+prompts would sharpen this and more repetitions would not.**
+
+Verify cost on this target is roughly 16.2 ms per extra draft position, and Ornith 1.5 is a
+sparse MoE whose experts do not amortize across draft positions. Deeper drafting therefore
+stops paying well before acceptance reaches zero.
+
+## Architecture
+
+1.29B parameters in 6 layers. The drafter reads the target's hidden states from 8 layers
+(`1, 9, 17, 25, 33, 41, 49, 57`), concatenated to a 32768-wide input, and projects them
+through `dflash_fc` into its own 4096-wide residual stream. A block of 16 mask slots seeded
+with the anchor token is decoded in one forward pass, and the resulting hidden states are
+pushed through **the target's frozen embedding and output head**.
+
+| | |
+|---|---|
+| Layers | 6 (`sliding_attention` ×5, `full_attention` ×1) |
+| Hidden / intermediate | 4096 / 12288 |
+| Attention heads | 32 query, 8 KV, head_dim 128 |
+| Sliding window | 4096 |
+| RoPE theta | 1e7 |
+| Block size | 16 (mask token 248077) |
+| Vocab | 248320 |
+
+RoPE theta must read `1e7` in the GGUF metadata. At the Qwen default of 1e6 acceptance
+collapses, and the converter reads the top-level `rope_theta` key rather than the nested
+`rope_parameters` block, so a config carrying only the nested value converts to a drafter
+that loads cleanly and drafts badly.
+
+**The file contains no target weights.** Its 69 tensors are the six blocks, `dflash_fc`,
+`dflash_hidden_norm` and `output_norm`. There is no `token_embd` and no `output.weight`;
+both come from the target at runtime. Redistributing this drafter therefore redistributes
+only weights derived from the `z-lab` DFlash initialization.
+
+### Known trainer/runtime divergence
+
+The training script forces all six layers to full attention, while the shipped `config.json`
+correctly declares sliding attention on layers 0–4. The drafter was therefore trained able to
+attend to later slots inside its block, and is served without that ability in five of six
+layers.
+
+This was measured rather than assumed. A fixture that reproduces both graphs on the same
+weights and anchors puts the difference at **±0.8 pp** on per-position match, in both
+directions, at every slot count tested. The window is 4096 while intra-block distances are at
+most 15, so it never binds. The divergence is real and worth knowing about; it is not worth
+correcting.
+
+## Training
+
+Initialized from the Ornith-1.0 drafter (itself initialized from `z-lab/Qwen3.5-397B-A17B-DFlash`),
+then fine-tuned on target features captured from the served IQ2_XXS model.
+
+The motivating observation: three previous retrains had produced almost nothing at deployed
+settings, and every one of them trained on anchors from long prose generations off short
+prompts. Production's drafter ring holds **code** at absolute positions above 30,000. The
+untested variable was the data distribution, not the loss or the schedule.
+
+| | |
+|---|---|
+| Capture | 12 prompts × ~128k chars of C/C++ from the ik_llama.cpp tree, 4096 generated tokens each |
+| Pack | 457,790 rows, 30 GB, 12 segments, 0 malformed |
+| Split | 9 segments train (348,025 rows) / 3 eval (109,765), split by captured request |
+| Mixed with | the previous prose pack, pooling to 21,950 anchors |
+| Schedule | 2 epochs, 2,742 steps at ~1.3 s/step, batch 16, lr 1e-4, stride 5 |
+| Loss | position-weighted CE, gamma 1.5, weights normalized to sum 15 (w₁..w₅ = 7.30 / 3.75 / 1.92 / 0.99 / 0.51) |
+
+The weighting concentrates on positions 1–3 because `tau@3 ≈ 1 + p₁ + p₁p₂` — position 1 caps
+everything downstream, and the deployed configuration only ever requests 3 or 4 slots.
+
+Best `tau@15` on held-out code was 4.42 against 3.91 for the initialization.
+
+## Evaluation
+
+Two things are worth separating: whether the drafter is better, and by how much at the
+setting you will actually serve.
+
+### Offline, on the deployed graph
+
+Measured with the runtime's mask and slot count rather than the trainer's, on 250 anchors,
+against the previous drafter for this target. Only genuinely held-out segments are included —
+see [Data contamination](#data-contamination).
+
+| Pack | Metric | Previous | This | Δ |
+|---|---|---:|---:|---:|
+| Held-out code | `tau@3` (serving `n_max=2`) | 2.592 | 2.704 | +4.3% |
+| Held-out code | `tau@4` (serving `n_max=3`) | 3.100 | 3.268 | +5.4% |
+| Held-out code | position-1 match | 0.908 | 0.928 | +2.0 pp |
+| Prose (old distribution) | `tau@3` / `tau@4` | 2.452 / 2.916 | 2.504 / 2.976 | +2.1% |
+
+The prose row matters more than its size suggests: this drafter was trained on code, and it
+did not get worse anywhere measured.
+
+### End-to-end throughput
+
+Four server boots in A-B-B-A order, four unseen ~32k-token code prompts, three repetitions,
+256 tokens generated per request. Counterbalancing is necessary because boot-to-boot drift on
+this hardware measured ~8%, larger than the effect. Deltas are paired within each half and
+clustered by prompt, so the unit of analysis is the prompt rather than the request.
+
+| `n_max` | Δ tok/s | 95% CI | t (df=3) |
+|---:|---:|---|---:|
+| 0 (control) | +0.6% | [+0.3%, +0.9%] | 6.70 |
+| 2 | +3.9% | [−0.4%, +8.2%] | 2.88 |
+| 3 | **+6.5%** | **[+0.3%, +12.6%]** | 3.34 |
+
+Every prompt improved at `n_max=3` (+11.2 / +7.3 / +1.9 / +5.5%). Acceptance rate rose in
+both boots independently — 0.472 / 0.474 to 0.508 / 0.531 — which is read from the server's
+own counters rather than a wall clock.
+
+The control row is the honest caveat. With speculation disabled the two arms should be
+identical, and instead the candidate's boots ran 0.6% faster with a CI excluding zero. Some
+small boot-level advantage was not fully cancelled. Subtracting it gives roughly +5.9% at
+`n_max=3`.
+
+`n_max=2` is directionally positive and not separable from zero.
+
+### Data contamination
+
+The first version of this evaluation was partly contaminated, and the correction is
+documented here rather than quietly fixed.
+
+The prompt generator rotated a six-entry list of source files, so requesting 12 prompts
+produced 12 prompts with only **9 unique bodies**. The four benchmark prompts were the first
+four of that same generator's output, and three of them were in the training split. One of
+the three offline evaluation segments duplicated a trained segment.
+
+After rebuilding the evaluation on one distinct unseen source file per prompt, hash-guarded
+against the training corpus at build time:
+
+- Offline recomputed on clean segments only: +6.9% / +8.6% fell to **+4.3% / +5.4%**. Part of
+  the original gap was memorization.
+- End-to-end on fresh prompts: +5.6% became **+6.5%**, with the CI excluding zero for the
+  first time.
+
+The contaminated online number was not inflated. That is the outcome, not the justification —
+the original measurement could not have told you either way.
+
+### What is not established
+
+- **Anything outside code.** The code corpus is nine unique prompt bodies drawn from six
+  C/C++ source files in one project. The evidence for general text is the +2.1%
+  no-regression row above, which rests on two held-out prose segments.
+- **The optimal `n_max`.** See [Choosing n_max](#choosing-n_max).
+- **Other quantizations of the target.** The drafter is distilled against IQ2_XXS
+  specifically, using that quantization's embedding and output head. Behaviour on IQ2_KS,
+  Q4_K_M or bf16 is unmeasured.
+- **Long-horizon serving.** All measurements are 256-token generations on a benchmark rig,
+  not production traffic over days.
+
+Four prompts and four boots is a small experiment. Treat +6.5% as the center of a wide
+interval.
+
+## Compatibility
+
+| Runtime | Result |
+|---|---|
+| ik_llama.cpp (HEAD `f96eaddb` or later) | Works |
+| mainline llama.cpp | `unknown model architecture: 'dflash-draft'` |
+| poolside llama.cpp | `unknown model architecture: 'dflash-draft'` — its DFlash arch is registered as `dflash` |
+
+The published `q4` file additionally uses ik-native quantization types that mainline ggml
+does not define, so it fails at tensor load before even reaching the architecture check. The
+quantization is not the obstacle, though: a pure `q8_0` build of these weights, using only
+mainline types, still fails on the architecture name. The format is the wall, not the mix.
+
+Both forks implement DFlash. Their drafter formats are not interchangeable, and running this
+drafter on poolside would require retraining and reconverting to that format.
+
+## Reproducing
+
+Measured on an NVIDIA GB10 (DGX Spark), 121.6 GiB unified memory shared between CPU and GPU.
+The target at IQ2_XXS with a 131,072-token context and `q8_0` KV cache leaves a few GiB
+free, so nothing else of size can run alongside it — including, on this hardware, an
+evaluation script that expects to use "the CPU" as if that were a separate pool.
+
+The full command sequence, the capture settings, the split, the training invocation and the
+statistical analysis are in the source repository under `kernel-lab/dflash-retrain/`:
+`CODERING-REPRO-RUNBOOK.md` for the commands in execution order,
+`CODERING-AB-STATISTICAL-NOTES.md` for the A/B analysed with its weaknesses stated.
+
+## License
+
+Apache 2.0, inherited from `z-lab/Qwen3.5-397B-A17B-DFlash`, of which these weights are a
+fine-tune.
+
+Training features were distilled from Ornith-1.5-397B outputs. If you intend to redistribute
+or use this commercially, check Ornith 1.5's own license terms on distillation — that
+question is not resolved by this card.