Ternary-Bonsai-2-27B-DFlash2-ft5

A DFlash 2 drafter fine-tuned against the hidden states of PrismML's Ternary-Bonsai-2-27B. It proposes token blocks for the target to verify, so it changes how fast the target answers, not what it answers. It is not a standalone language model.


Headline	against the stock drafter on one Apple M4 Pro: acceptance +9.49% on general chat (paired 95% +7.25% to +11.69%), decode +15.2% greedy and +6.7% at the target's published sampling. On a second runtime: acceptance +5.01% (+3.35% to +6.83%)
Drop-in for	`z-lab/Qwen3.8-27B-DFlash2`: the same 81 tensor names, dtypes, shapes and `config.json`. A runtime that loads the stock drafter loads this one unchanged
Weights	fused bfloat16, `model.safetensors`, sha256 `63399215d7af9b7aaa73464e7e10835d608ae4d9571abf520040b2b8746455b3`
Target, as measured	`prism-ml/Ternary-Bonsai-2-27B-mlx-2bit` at revision `3f926b41`. Revision `fcba37d2` keeps the weights and adds the sampling defaults used here and a second stop token (`248044`)
Other formats	4-bit MLX, the same weights 2.83x smaller; GGUF in Q8_0, Q4_K_M and Q2_K for llama.cpp's DFlash 2, meant for CUDA
Code	bonsai2-drafter: the trainer, the evaluation kit, the mlx-dspark patches and a one-command quickstart. The write-up covers how ft5 was made and how it compares with the other drafters

On a Mac, the fastest measured way to serve Bonsai 2 is not a drafter. On the same M4 Pro, MTPLX 2.12.0, which runs the target with Qwen3.8's own MTP head, decoded 38% faster than mlx-dspark with ft5 greedy and 46% faster at the target's sampling. ft5 is for runtimes where DFlash 2 is the method.

Use

git clone https://github.com/alexschiltmans/bonsai2-drafter && cd bonsai2-drafter
scripts/serve-bonsai2.sh    # OpenAI-compatible API on http://127.0.0.1:8088/v1

The quickstart builds the pinned mlx-dspark environment, fetches the target and this drafter at pinned revisions, and serves with the measured settings: an 8-bit KV cache, the drafter quantized to 4 bits at load, and the repository's Bonsai loader and 2-bit verify kernel.

Compatibility

runtime	status
mlx-dspark 0.18.0 with the repository's patches	validated (cap 7): the measurements below
dflash-mlx-bonsai2 at `223e0f3`	validated (cap 4, its DFlash 2 limit); loads unchanged. General chat at 200 tokens 2.6392 → 2.7714 tokens a round, +5.01% (+3.35% to +6.83%); code +7.10%; general at 1024 tokens +4.84%; greedy decode 1.0716x the stock drafter, end to end. Against the target alone, output differed only at floating-point ties (logit margins ≤ 0.016)
PrismML llama.cpp fork, Metal	loads and drafts since release `prism-b10743`; still no speedup on Metal. At `prism-b10754-2459f68` the GGUF of ft5 (Q8_0) decoded at 7.92 tok/s with draft acceptance 0.432, against 20.70 with no drafter; the stock drafter decoded at 7.10 with 0.395 (the GGUF card has every quantization). The earlier probe at fork head `0324c665`, with upstream's DFlash 2 patched in, found the same (the stock drafter 7.10 against 20.56 with no drafter, MTP 11.70): a round costs about ten single-token steps there (probe records)
MTPLX 2.12.0 (MTP, no drafter)	faster, and does not load DFlash 2 drafters: 38.89 against 28.13 tok/s greedy (1.38x) and 37.44 against 25.68 sampled (1.46x), end to end under `BENCHMARK.md` version 2. It commits fewer tokens a round (1.80 against 3.68) at a third of the cost (44 against 128 ms) (records)
oMLX at `5aa6c7f9`	loads the target, not DFlash 2 on it. With its routing patched at run time, ft5 drafts losslessly but decodes about four times slower than oMLX without a drafter (probe)
other runtimes that load `z-lab/Qwen3.8-27B-DFlash2` (vLLM, SGLang, others)	untested; the layout is identical, so it should load wherever the stock drafter does

Some releases name the selector's codebooks …_codebook.weight instead of this file's …_codebook; the repository's scripts/rename-codebooks.py converts between the two losslessly. Reports from other runtimes are welcome as pull requests to the repository.

Measurements

One Apple M4 Pro with 48 GB, mlx-dspark 0.18.0, MLX 0.32.2, mlx-lm 0.31.3, an 8-bit KV cache and a 4-bit drafter at cap 7. The raw records and the commands that recompute every figure are in the repository's evidence/.

Acceptance

Tokens a round, 40 prompts a suite, paired stratified bootstrap:

suite	output budget	stock	ft5	gain (paired 95%)
general chat	200	2.7768	3.0403	+9.49% (+7.25% to +11.69%)
general chat	1024	2.8149	3.0399	+7.99% (+6.75% to +9.24%)
held-out code	200	3.8166	4.2130	+10.39% (+7.80% to +13.13%)

The general suite is 40 locally authored chat prompts in eight categories, thinking on for half, frozen before either arm ran; running the arms in reverse order gave the same result. The code suite informed which training iteration to keep, so it does not decide. At 200 tokens 34/40 general and 25/40 code responses were truncated in each arm, and 13/40 at 1024: these measure finite prefixes, not complete answers.

Decode speed

BENCHMARK.md measurements 2 and 3: a fresh server per leg, ABBA, the five code prompts of bench5.py at 400 tokens, pooled decode tok/s.

	stock	ft5	ft5 / stock
greedy, three repetitions a leg	25.150 (legs 25.140, 25.160)	28.981 (legs 28.947, 29.015)	1.1523
target's sampling (temperature 1.0, top-p 0.95, top-k 20), five repetitions a leg	24.410 (legs 24.517, 24.303)	26.052 (legs 26.343, 25.780)	1.0673

Both drafters cost the same per round, so the gain is acceptance: 3.68 against 3.20 tokens a round greedy, 3.40 against 3.19 sampled. The publication battery's greedy arms agree within 0.3% (+15.29%); an earlier sampled screen found +9.3%, and this protocol result replaces it (records).

Depth

depth	stock	ft5	notes
35k warm	16.3 tok/s	17.4	derived cap, needles 4/4
65k warm	12.6	13.1	derived cap, needles 4/4
110k	8.9	8.92 warm / 8.71 all served	not comparable, see below

The 35k and 65k arms and the stock 110k arm ran outside the battery. Depth runs use a derived draft cap, which narrows to about one token at depth, so the gain shrinks to about 7% at 35k and 4% at 65k. The ft5 110k arm served five of five requests with needles 4/4, but under cache pressure the clean stock arm did not have (four memory-guard sheds, a 60% cache-miss rate), so no 110k speed claim is made (records).

On a derivative target

On BoldingBuilds' abliterated derivative (98 of the target's 851 tensors edited), transcoded losslessly into PrismML's MLX pack: stock → ft5 +7.34% on general chat (+5.89% to +8.77%) and +10.09% on code (+5.65% to +14.63%), not significantly below the gain on the base target. A derivative that edits more needs its own measurement (records).

Output agreement

Speculative verification is meant to preserve the target's output, but floating-point batching can flip close decisions, so agreement was measured. Greedy, the two drafters produced identical tokens inside the requested budget in all 160 paired responses, and same-drafter controls were identical on all 40 prompts. Unequal pairs differed only above the budget, by at most 7 tokens, because the generator checks its budget once per round (contract budgeted-prefix-identity/v2). This is agreement on these suites, not a blanket losslessness claim.

Quality

battery	stock	ft5
quality, 14 tasks, three graders, one executable	13/14	13/14
tool calls, requested depths 2500 and 13000	56/56	56/56

Both arms failed the same task, an ISO-8601 duration parser: at an 8-bit KV cache the target spends the whole 16,384-token budget reasoning and never answers. The drafter is not in the failure path (one deterministic greedy run a cell; the stock/16-bit cell ran outside the battery):

`iso_seconds`	8-bit KV	16-bit KV
no drafter	fails, 16,384 tokens, no answer	passes, 2,232 tokens
stock drafter	fails, 16,391	passes, 11,995
ft5 drafter (this release)	fails, 16,389	passes, 13,417 — 14/14

Across the author's quality runs it fails in 10 of 11 at 8-bit KV, always by exhausting the budget, and passes in all 5 at 4- or 16-bit KV. That is an association, not an established cause; the runs are tabulated in evidence/ft5-publication-battery/.

Training

The starting drafter is z-lab/Qwen3.8-27B-DFlash2, a mirror of Inco AI's release. The unchanged target greedily answered 300 CodeAlpaca prompts: 260 for training and 40 held out from gradient updates but used in iteration selection. LoRA rank 64, top-32 distillation from the target, the selector projection trained with its codebooks frozen, AdamW at 1e-4 with a cosine schedule and 4% warmup, two epochs (1032 recorded steps), block size 8, truncation to 2048 tokens. Anchors follow served round boundaries: context ends before the anchor, and the block head is the target's unread bonus token. Export merges the LoRA weights into the drafter. It trained on one Apple M4 Pro with 48 GB; the exact commands are in bench/drafter/README.md.

Limitations

Only the pinned mlx-dspark 0.18.0 stack is validated. An isolated 0.19.0 check passed the patch, kernel and loader tests, but there is no 0.19.0 serving, quality or throughput arm.
Two runtimes have been measured, and one target besides PrismML's (a lightly edited derivative, on mlx-dspark only). Everything ran on one machine with 48 GB.
Read the figures as properties of this target, this kernel and these suites.

Related releases

BENCHMARK.md was written by this card's author, who also trained ft5.

ProCreations' fine-tune accepts more than ft5 on general chat: +3.90% (+2.62% to +5.14%) on dflash-mlx-bonsai2 under BENCHMARK.md, which by its deciding rule beats ft5, and +1.83% (+0.26% to +3.54%) in a matched comparison on mlx-dspark. The two are level on code. On the protocol's code prompts ft5 decodes faster: ProCreations / ft5 0.949 greedy and 0.982 sampled on mlx-dspark, 0.984 greedy on dflash-mlx-bonsai2. None of the evaluation prompts is in its published training corpus (records).
naklitechie's fine-tune (r3, trained on CUDA) also beats the stock drafter, and accepted less than ft5 on every suite: 2.41% less on general chat (−3.83% to −1.08%) on its own runtime under BENCHMARK.md, and 4.63% less (−6.39% to −2.72%) in a screen on mlx-dspark. ft5 decoded 1.055x as fast on r3's runtime, and 1.106x greedy and 1.072x sampled on mlx-dspark (records).
nathansutton's Bonsai/DFlash2 MLX pack bundles a converted stock drafter; its reported result uses another runtime and template, so it is not a matched comparison.

Attribution and license

Apache-2.0; LICENSE and NOTICE ship beside the weights. The NOTICE carries Prism ML's notice verbatim, including its requested "Created using Bonsai by Prism ML", and the Qwen3.8-27B copyright line for Alibaba Cloud. The weights derive from z-lab's drafter, so Apache-2.0 section 4 applies to that lineage; whether training against the target's hidden states also makes it a derivative of the target is unsettled, and the NOTICE carries Prism ML's material regardless. The release contains no part of the target, which anyone serving this drafter fetches upstream. Training prompts come from CodeAlpaca-20k (CC BY 4.0) and are not redistributed. This is an independent fine-tune, not an official Prism ML, Qwen, Inco AI or z-lab release.