Instructions to use Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5 with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5 --local-dir Ternary-Bonsai-2-27B-DFlash2-ft5
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Card: the fork release supports DFlash 2; GGUF in three quantizations
Browse files
README.md
CHANGED
|
@@ -31,7 +31,7 @@ answers. It is not a standalone language model.
|
|
| 31 |
| **Drop-in for** | `z-lab/Qwen3.8-27B-DFlash2`: the same 81 tensor names, dtypes, shapes and `config.json`. A runtime that loads the stock drafter loads this one unchanged |
|
| 32 |
| **Weights** | fused bfloat16, `model.safetensors`, sha256 `63399215d7af9b7aaa73464e7e10835d608ae4d9571abf520040b2b8746455b3` |
|
| 33 |
| **Target, as measured** | `prism-ml/Ternary-Bonsai-2-27B-mlx-2bit` at revision `3f926b41`. Revision `fcba37d2` keeps the weights and adds the sampling defaults used here and a second stop token (`248044`) |
|
| 34 |
-
| **Other formats** | [4-bit MLX](https://huggingface.co/Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5-mlx-4bit), the same weights 2.83x smaller; [
|
| 35 |
| **Code** | [bonsai2-drafter](https://github.com/alexschiltmans/bonsai2-drafter): the trainer, the evaluation kit, the mlx-dspark patches and a one-command quickstart. The [write-up](https://github.com/alexschiltmans/bonsai2-drafter/blob/main/docs/one-drafter-several-runtimes.md) covers how ft5 was made and how it compares with the other drafters |
|
| 36 |
|
| 37 |
**On a Mac, the fastest measured way to serve Bonsai 2 is not a drafter.** On the same M4 Pro,
|
|
@@ -56,7 +56,7 @@ to 4 bits at load, and the repository's Bonsai loader and 2-bit verify kernel.
|
|
| 56 |
| --- | --- |
|
| 57 |
| mlx-dspark 0.18.0 with the repository's patches | **validated** (cap 7): the measurements below |
|
| 58 |
| [dflash-mlx-bonsai2](https://github.com/NakliTechie/dflash-mlx-bonsai2) at `223e0f3` | **validated** (cap 4, its DFlash 2 limit); loads unchanged. General chat at 200 tokens 2.6392 → 2.7714 tokens a round, **+5.01%** (+3.35% to +6.83%); code +7.10%; general at 1024 tokens +4.84%; greedy decode 1.0716x the stock drafter, end to end. Against the target alone, output differed only at floating-point ties (logit margins ≤ 0.016) |
|
| 59 |
-
| PrismML llama.cpp fork, Metal | **
|
| 60 |
| [MTPLX](https://github.com/youssofal/MTPLX) 2.12.0 (MTP, no drafter) | **faster, and does not load DFlash 2 drafters**: 38.89 against 28.13 tok/s greedy (1.38x) and 37.44 against 25.68 sampled (1.46x), end to end under `BENCHMARK.md` version 2. It commits fewer tokens a round (1.80 against 3.68) at a third of the cost (44 against 128 ms) ([records](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/mac-stacks-mtplx)) |
|
| 61 |
| [oMLX](https://github.com/jundot/omlx) at `5aa6c7f9` | **loads the target, not DFlash 2 on it.** With its routing patched at run time, ft5 drafts losslessly but decodes about four times slower than oMLX without a drafter ([probe](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/omlx-probe)) |
|
| 62 |
| other runtimes that load `z-lab/Qwen3.8-27B-DFlash2` (vLLM, SGLang, others) | **untested**; the layout is identical, so it should load wherever the stock drafter does |
|
|
|
|
| 31 |
| **Drop-in for** | `z-lab/Qwen3.8-27B-DFlash2`: the same 81 tensor names, dtypes, shapes and `config.json`. A runtime that loads the stock drafter loads this one unchanged |
|
| 32 |
| **Weights** | fused bfloat16, `model.safetensors`, sha256 `63399215d7af9b7aaa73464e7e10835d608ae4d9571abf520040b2b8746455b3` |
|
| 33 |
| **Target, as measured** | `prism-ml/Ternary-Bonsai-2-27B-mlx-2bit` at revision `3f926b41`. Revision `fcba37d2` keeps the weights and adds the sampling defaults used here and a second stop token (`248044`) |
|
| 34 |
+
| **Other formats** | [4-bit MLX](https://huggingface.co/Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5-mlx-4bit), the same weights 2.83x smaller; [GGUF](https://huggingface.co/Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5-GGUF) in Q8_0, Q4_K_M and Q2_K for llama.cpp's DFlash 2, meant for CUDA |
|
| 35 |
| **Code** | [bonsai2-drafter](https://github.com/alexschiltmans/bonsai2-drafter): the trainer, the evaluation kit, the mlx-dspark patches and a one-command quickstart. The [write-up](https://github.com/alexschiltmans/bonsai2-drafter/blob/main/docs/one-drafter-several-runtimes.md) covers how ft5 was made and how it compares with the other drafters |
|
| 36 |
|
| 37 |
**On a Mac, the fastest measured way to serve Bonsai 2 is not a drafter.** On the same M4 Pro,
|
|
|
|
| 56 |
| --- | --- |
|
| 57 |
| mlx-dspark 0.18.0 with the repository's patches | **validated** (cap 7): the measurements below |
|
| 58 |
| [dflash-mlx-bonsai2](https://github.com/NakliTechie/dflash-mlx-bonsai2) at `223e0f3` | **validated** (cap 4, its DFlash 2 limit); loads unchanged. General chat at 200 tokens 2.6392 → 2.7714 tokens a round, **+5.01%** (+3.35% to +6.83%); code +7.10%; general at 1024 tokens +4.84%; greedy decode 1.0716x the stock drafter, end to end. Against the target alone, output differed only at floating-point ties (logit margins ≤ 0.016) |
|
| 59 |
+
| PrismML llama.cpp fork, Metal | **loads and drafts since release `prism-b10743`; still no speedup on Metal.** At `prism-b10754-2459f68` the GGUF of ft5 (Q8_0) decoded at 7.92 tok/s with draft acceptance 0.432, against 20.70 with no drafter; the stock drafter decoded at 7.10 with 0.395 (the [GGUF card](https://huggingface.co/Schiltmans/Ternary-Bonsai-2-27B-DFlash2-ft5-GGUF) has every quantization). The earlier probe at fork head `0324c665`, with upstream's DFlash 2 patched in, found the same (the stock drafter 7.10 against 20.56 with no drafter, MTP 11.70): a round costs about ten single-token steps there ([probe records](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/fork-probe)) |
|
| 60 |
| [MTPLX](https://github.com/youssofal/MTPLX) 2.12.0 (MTP, no drafter) | **faster, and does not load DFlash 2 drafters**: 38.89 against 28.13 tok/s greedy (1.38x) and 37.44 against 25.68 sampled (1.46x), end to end under `BENCHMARK.md` version 2. It commits fewer tokens a round (1.80 against 3.68) at a third of the cost (44 against 128 ms) ([records](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/mac-stacks-mtplx)) |
|
| 61 |
| [oMLX](https://github.com/jundot/omlx) at `5aa6c7f9` | **loads the target, not DFlash 2 on it.** With its routing patched at run time, ft5 drafts losslessly but decodes about four times slower than oMLX without a drafter ([probe](https://github.com/alexschiltmans/bonsai2-drafter/tree/main/evidence/omlx-probe)) |
|
| 62 |
| other runtimes that load `z-lab/Qwen3.8-27B-DFlash2` (vLLM, SGLang, others) | **untested**; the layout is identical, so it should load wherever the stock drafter does |
|