---
base_model: prism-ml/Ternary-Bonsai-2-27B-gguf
base_model_relation: quantized
model_name: Ternary-Bonsai-2-27B-MLX-oQ8-mtp
library_name: mlx
pipeline_tag: image-text-to-text
license: apache-2.0
tags:
  - mlx
  - omlx
  - quantization
  - mixed-precision
  - apple-silicon
  - ternary
  - mtp
  - speculative-decoding
  - vision
  - qwen
---

# Ternary-Bonsai-2-27B-MLX-oQ8-mtp — oQ8 MLX quant of Bonsai 2 27B, with a grafted MTP head

Mixed-precision **oQ8** (8.50 bits per weight) build of
[prism-ml/Ternary-Bonsai-2-27B](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf),
produced with **oMLX 0.6.4** on an Apple M5 Max / 128 GB, with the multi-token-prediction
head from `Qwen/Qwen3.8-27B` grafted in so oMLX can run self-speculative decoding.

I am not affiliated with Prism ML, pipenetwork or Alibaba Cloud. **Created using Bonsai by
Prism ML.** See [Provenance](#provenance) and [License](#license).

> **Format note:** MLX safetensors. Not GGUF — these will not load in llama.cpp, Ollama or
> LM Studio. Use oMLX, or stock `mlx-vlm ≥ 0.7`.

## What is different about this build

The Bonsai weights are **ternary** — every language weight is one of three values. The
upstream GGUF packs them at a true 1.72 bpw with custom llama.cpp kernels; the official MLX
release needs prism-ml's own runtime and ships **without vision and without MTP**.

These builds instead use **stock MLX quantization**, so they run in unmodified oMLX and
mlx-vlm, and they keep both:

- the **vision tower**, 333 tensors, untouched in BF16
- an **MTP head** (15 tensors, 0.42 B parameters), grafted from `Qwen/Qwen3.8-27B`

## Pick a variant — measured, not guessed

Each build's logits were compared against the bf16 source on the same five prompts
(last-token logits, greedy): KL divergence, maximum relative error, agreement on the top-1
token, and overlap of the top-5.

| build | bpw | on disk | shards | KL(bf16‖q) | max rel. err | top-1 | top-5 | pick when |
|---|---|---|---|---|---|---|---|---|
| [oQ2](https://huggingface.co/TokenAI-zer/Ternary-Bonsai-2-27B-MLX-oQ2-mtp) | 3.00 | 10.83 GiB (11.63 GB) | 3 | 0.37658 | 0.2472 | 4/5 | 16/25 | smallest footprint, with measurable loss |
| [oQ3](https://huggingface.co/TokenAI-zer/Ternary-Bonsai-2-27B-MLX-oQ3-mtp) | 3.70 | 12.91 GiB (13.86 GB) | 3 | 0.03478 | 0.1077 | 5/5 | 21/25 | best size-to-fidelity ratio |
| [oQ4](https://huggingface.co/TokenAI-zer/Ternary-Bonsai-2-27B-MLX-oQ4-mtp) | 4.70 | 15.85 GiB (17.02 GB) | 4 | 0.01476 | 0.0519 | 5/5 | 23/25 | balanced choice |
| [oQ6](https://huggingface.co/TokenAI-zer/Ternary-Bonsai-2-27B-MLX-oQ6-mtp) | 6.70 | 22.09 GiB (23.72 GB) | 5 | 0.00074 | 0.0164 | 5/5 | 24/25 | high fidelity |
| [oQ8](https://huggingface.co/TokenAI-zer/Ternary-Bonsai-2-27B-MLX-oQ8-mtp) | 8.50 | 27.94 GiB (30.00 GB) | 6 | 0.00008 | 0.0083 | 5/5 | 25/25 | practically indistinguishable from bf16 |

The loss is **monotonic and far from flat**, which was not obvious beforehand: a ternary
model still benefits from more bits. Affine quantization builds its grid from the group
minimum and maximum, so with 4 levels over a symmetric {−a, 0, +a} group the levels land
at −a, −a/3, +a/3, +a — **zero itself is not representable**, and zero is the most common
value in a ternary tensor. More bits make the grid finer and approximate it better.

oQ2 is the only build that flips a top-1 prediction in this sample. Treat it as the
"fits in less memory" option, not as a free lunch.

**Caveat on the measurement:** the reference is the bf16 conversion, not the original GGUF.
An error inside that conversion would be inherited by every build here and would not show up
in this table.

## MTP: does the grafted head actually work?

The head comes from stock Qwen3.8-27B; the backbone is ternarized by Prism ML. Whether the
two speak the same language is an empirical question, so here are the numbers, measured in
oMLX on the oQ6 build:

| generated tokens | accepted drafts | tokens/cycle |
|---|---|---|
| 57 | 31/46 (67.4%) | 2.38 |
| 175 | 57/128 (44.5%) | 1.51 |
| 470 | 178/364 (48.9%) | 1.62 |

End-to-end on the same model and machine: **13.1–13.4 tok/s without MTP, 15.9–23.9 tok/s
with it**; the longest run reached 23.9 tok/s against 13.1 for a comparable run without,
about **1.8×**.

For reference, a model of this family with its **native** MTP head accepts around 74% on the
same setup. So the graft works, at reduced efficiency — expected, since the head was trained
against full-precision hidden states and now reads ternarized ones.

## Requirements

- Apple Silicon, macOS 15+ (built and tested on M5 Max, 128 GB)
- **oMLX ≥ 0.6.4** for MTP speculative decoding, or stock `mlx-vlm ≥ 0.7` for plain inference
- about 27.94 GiB of free unified memory for a short prompt, more for long context

## Usage

### oMLX (what the MTP head is for)

```bash
git clone https://huggingface.co/TokenAI-zer/Ternary-Bonsai-2-27B-MLX-oQ8-mtp ~/.omlx/models/Ternary-Bonsai-2-27B-MLX-oQ8-mtp
omlx serve --model-dir ~/.omlx/models --port 8000
```

In the model settings enable **`mtp_enabled`**; without it the head is loaded but the
speculative path never runs and you simply pay for the extra weights.

### Stock mlx-vlm

```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("TokenAI-zer/Ternary-Bonsai-2-27B-MLX-oQ8-mtp")
prompt = apply_chat_template(processor, model.config, "Explain speculative decoding.", num_images=0)
print(generate(model, processor, prompt, max_tokens=512))
```

## Provenance

Four steps, each of which can be checked independently:

1. **[Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B)** — Apache-2.0, the original
   27B hybrid-attention VLM. The 15 `mtp.*` tensors come from here, unmodified.
2. **[prism-ml/Ternary-Bonsai-2-27B](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf)**
   — ternarization of that model to 1.72 bpw, with a blockwise Hadamard rotation applied
   online by their own kernels.
3. **[pipenetwork/Ternary-Bonsai-2-27B-MLX-bf16](https://huggingface.co/pipenetwork/Ternary-Bonsai-2-27B-MLX-bf16)**
   — the Hadamard rotation unfolded back into the standard weight basis, so the model loads
   with no custom runtime. This is the source these builds quantize.
4. **This repository** — MTP head grafted in (renamed to the `language_model.mtp.*` namespace
   mlx-vlm expects, no transform applied, since the weights are already in standard basis),
   `mtp_num_hidden_layers` set to 1, then oQ8 mixed-precision quantization with
   `preserve_mtp`.

The graft follows the approach documented by
[decent-jawfish/bonsai-2-27b-mtp](https://huggingface.co/decent-jawfish/bonsai-2-27b-mtp)
for the GGUF format: the MTP tensors are stock Qwen weights and must not receive the Hadamard
transform. No code was copied from it.

## Known caveats

- Quantization is lossy, and the table above says exactly how much. Compare against a higher
  build before reporting a regression.
- Task-level benchmarks have **not** been run. The divergence numbers measure fidelity to the
  bf16 source, not intelligence.
- MTP acceptance was measured on oQ6 only; the other builds are expected to behave similarly
  but this was not verified.
- Vision is carried through in BF16 but **VQA accuracy was not evaluated**.
- The grafted head is not a substitute for one trained on these weights.

## License

Apache License 2.0, inherited through the whole chain. `LICENSE` and `NOTICE.txt` from
Prism ML ship in this repository unchanged, as Apache-2.0 §4(d) requires.

As the upstream NOTICE asks: **Created using Bonsai by Prism ML.** The base model is
Qwen3.8-27B, Copyright 2026 Alibaba Cloud, Apache 2.0.

## Acknowledgements

- **Prism ML** for Bonsai and the ternarization work.
- **pipenetwork** for unfolding the Hadamard rotation into a stock-runtime MLX build.
- **Qwen team / Alibaba Cloud** for Qwen3.8-27B and its MTP head.
- **oMLX** for the oQ quantizer and the speculative decoding runtime.