MLX
Safetensors
qwen3
dflash2
speculative-decoding
drafter
bonsai
ternary
apple-silicon
Schiltmans commited on
Commit
315d40e
·
verified ·
1 Parent(s): 696b587

Card: the fork release supports DFlash 2; GGUF in three quantizations

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -31,7 +31,7 @@ answers. It is not a standalone language model.
31
  | **Drop-in for** | `z-lab/Qwen3.8-27B-DFlash2`: the same 81 tensor names, dtypes, shapes and `config.json`. A runtime that loads the stock drafter loads this one unchanged |
32
  | **Weights** | fused bfloat16, `model.safetensors`, sha256 `63399215d7af9b7aaa73464e7e10835d608ae4d9571abf520040b2b8746455b3` |
33
  | **Target, as measured** | `prism-ml/Ternary-Bonsai-2-27B-mlx-2bit` at revision `3f926b41`. Revision `fcba37d2` keeps the weights and adds the sampling defaults used here and a second stop token (`248044`) |
34
- | **Other formats** | [4-bit MLX](https://huggingface.co/Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5-mlx-4bit), the same weights 2.83x smaller; [Q8_0 GGUF](https://huggingface.co/Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5-GGUF) for llama.cpp's DFlash 2, meant for CUDA |
35
  | **Code** | [bonsai2-drafter](https://github.com/alexschiltmans/bonsai2-drafter): the trainer, the evaluation kit, the mlx-dspark patches and a one-command quickstart. The [write-up](https://github.com/alexschiltmans/bonsai2-drafter/blob/main/docs/one-drafter-several-runtimes.md) covers how ft5 was made and how it compares with the other drafters |
36
 
37
  **On a Mac, the fastest measured way to serve Bonsai 2 is not a drafter.** On the same M4 Pro,
@@ -56,7 +56,7 @@ to 4 bits at load, and the repository's Bonsai loader and 2-bit verify kernel.
56
  | --- | --- |
57
  | mlx-dspark 0.18.0 with the repository's patches | **validated** (cap 7): the measurements below |
58
  | [dflash-mlx-bonsai2](https://github.com/NakliTechie/dflash-mlx-bonsai2) at `223e0f3` | **validated** (cap 4, its DFlash 2 limit); loads unchanged. General chat at 200 tokens 2.6392 → 2.7714 tokens a round, **+5.01%** (+3.35% to +6.83%); code +7.10%; general at 1024 tokens +4.84%; greedy decode 1.0716x the stock drafter, end to end. Against the target alone, output differed only at floating-point ties (logit margins ≤ 0.016) |
59
- | PrismML llama.cpp fork, Metal | **no speedup possible at fork head `0324c665`**. The fork has no DFlash 2 support; with upstream's patched in, the stock DFlash 2 drafter decoded at 7.10 tok/s against 20.56 with no drafter (and MTP at 11.70). A DFlash 2 round cost about ten single-token steps there, so no drafter can win on that path at that fork head ([probe records](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/fork-probe)) |
60
  | [MTPLX](https://github.com/youssofal/MTPLX) 2.12.0 (MTP, no drafter) | **faster, and does not load DFlash 2 drafters**: 38.89 against 28.13 tok/s greedy (1.38x) and 37.44 against 25.68 sampled (1.46x), end to end under `BENCHMARK.md` version 2. It commits fewer tokens a round (1.80 against 3.68) at a third of the cost (44 against 128 ms) ([records](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/mac-stacks-mtplx)) |
61
  | [oMLX](https://github.com/jundot/omlx) at `5aa6c7f9` | **loads the target, not DFlash 2 on it.** With its routing patched at run time, ft5 drafts losslessly but decodes about four times slower than oMLX without a drafter ([probe](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/omlx-probe)) |
62
  | other runtimes that load `z-lab/Qwen3.8-27B-DFlash2` (vLLM, SGLang, others) | **untested**; the layout is identical, so it should load wherever the stock drafter does |
 
31
  | **Drop-in for** | `z-lab/Qwen3.8-27B-DFlash2`: the same 81 tensor names, dtypes, shapes and `config.json`. A runtime that loads the stock drafter loads this one unchanged |
32
  | **Weights** | fused bfloat16, `model.safetensors`, sha256 `63399215d7af9b7aaa73464e7e10835d608ae4d9571abf520040b2b8746455b3` |
33
  | **Target, as measured** | `prism-ml/Ternary-Bonsai-2-27B-mlx-2bit` at revision `3f926b41`. Revision `fcba37d2` keeps the weights and adds the sampling defaults used here and a second stop token (`248044`) |
34
+ | **Other formats** | [4-bit MLX](https://huggingface.co/Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5-mlx-4bit), the same weights 2.83x smaller; [GGUF](https://huggingface.co/Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5-GGUF) in Q8_0, Q4_K_M and Q2_K for llama.cpp's DFlash 2, meant for CUDA |
35
  | **Code** | [bonsai2-drafter](https://github.com/alexschiltmans/bonsai2-drafter): the trainer, the evaluation kit, the mlx-dspark patches and a one-command quickstart. The [write-up](https://github.com/alexschiltmans/bonsai2-drafter/blob/main/docs/one-drafter-several-runtimes.md) covers how ft5 was made and how it compares with the other drafters |
36
 
37
  **On a Mac, the fastest measured way to serve Bonsai 2 is not a drafter.** On the same M4 Pro,
 
56
  | --- | --- |
57
  | mlx-dspark 0.18.0 with the repository's patches | **validated** (cap 7): the measurements below |
58
  | [dflash-mlx-bonsai2](https://github.com/NakliTechie/dflash-mlx-bonsai2) at `223e0f3` | **validated** (cap 4, its DFlash 2 limit); loads unchanged. General chat at 200 tokens 2.6392 → 2.7714 tokens a round, **+5.01%** (+3.35% to +6.83%); code +7.10%; general at 1024 tokens +4.84%; greedy decode 1.0716x the stock drafter, end to end. Against the target alone, output differed only at floating-point ties (logit margins ≤ 0.016) |
59
+ | PrismML llama.cpp fork, Metal | **loads and drafts since release `prism-b10743`; still no speedup on Metal.** At `prism-b10754-2459f68` the GGUF of ft5 (Q8_0) decoded at 7.92 tok/s with draft acceptance 0.432, against 20.70 with no drafter; the stock drafter decoded at 7.10 with 0.395 (the [GGUF card](https://huggingface.co/Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5-GGUF) has every quantization). The earlier probe at fork head `0324c665`, with upstream's DFlash 2 patched in, found the same (the stock drafter 7.10 against 20.56 with no drafter, MTP 11.70): a round costs about ten single-token steps there ([probe records](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/fork-probe)) |
60
  | [MTPLX](https://github.com/youssofal/MTPLX) 2.12.0 (MTP, no drafter) | **faster, and does not load DFlash 2 drafters**: 38.89 against 28.13 tok/s greedy (1.38x) and 37.44 against 25.68 sampled (1.46x), end to end under `BENCHMARK.md` version 2. It commits fewer tokens a round (1.80 against 3.68) at a third of the cost (44 against 128 ms) ([records](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/mac-stacks-mtplx)) |
61
  | [oMLX](https://github.com/jundot/omlx) at `5aa6c7f9` | **loads the target, not DFlash 2 on it.** With its routing patched at run time, ft5 drafts losslessly but decodes about four times slower than oMLX without a drafter ([probe](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/omlx-probe)) |
62
  | other runtimes that load `z-lab/Qwen3.8-27B-DFlash2` (vLLM, SGLang, others) | **untested**; the layout is identical, so it should load wherever the stock drafter does |