README.md

---
license: mit
base_model: deepreinforce-ai/Ornith-1.0-9B
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: vllm
language:
- en
tags:
- mxfp4
- compressed-tensors
- speculative-decoding
- mtp
- multi-token-prediction
- qwen3.5
- vllm
- rocm
- rdna4
- gfx1201
- radeon-ai-pro-r9700
- amd
- vision
---

# Ornith-1.0-9B — MXFP4 + MTP (vision), for AMD RDNA4 / vLLM

[`deepreinforce-ai/Ornith-1.0-9B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B) quantized to
**MXFP4** with a **grafted MTP (Multi-Token-Prediction) draft head**, packaged to run **out of the box on
AMD Radeon RDNA4 (gfx1201) under vLLM** with lossless self-speculative decoding. Vision retained.

- **Trunk** — Ornith-1.0-9B quantized to **MXFP4** (`compressed-tensors`, group size 32, symmetric).
  `lm_head`, `embed_tokens`, norms, and the vision tower are kept **BF16**.
- **Draft head** — the **KL-distilled MTP head** from
  [`protoLabsAI/Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP) (BF16), grafted
  into a separate `model-mtp.safetensors` shard and marked unquantized in the quant config so the
  mixed-precision model loads cleanly.
- **Speculative decoding** — vLLM native `mtp` method, `num_speculative_tokens=3`. **Lossless**: the
  target verifies every drafted token, so the output distribution is unchanged — the head only buys speed.

## Measured on AMD RDNA4

2× Radeon AI PRO R9700 (gfx1201), tensor-parallel 2, vLLM `0.19.1` (image below).

- **MTP draft acceptance (n=3): ≈66% average** across a mixed code + 6k-context run — per-position
  **0.82 / 0.65 / 0.53**, mean acceptance length **~3.0** (max 4) — rising to **~85%** (length ~3.6) on
  cache-warm short context. Lossless throughout.
- **Tool-calling** (`qwen3_xml`) and **vision** confirmed working.
- **Throughput** (TP2, 256 output tokens, per-user / aggregate tok/s):

| concurrent | short prompt | ~6k prompt |
|---:|---|---|
| 1   | 84.6 / 85   | 103.8 / 104 |
| 16  | 51.8 / 803  | 50.1 / 767  |
| 32  | 40.8 / 1260 | 33.0 / 1010 |
| 64  | 30.7 / 1902 | 21.1 / 1268 |
| 96  | 22.5 / 2075 | 15.0 / 1293 |
| 128 | 20.2 / 1893 | 12.6 / 1284 |

Usable concurrency ceiling (per-user ≥ 20 tok/s): **~128** at short context, **~64** at 6k context.
Single-stream decode is bound by the dense 9B's active-parameter count; a single GPU (TP1) is also supported.

## Run it on AMD RDNA4

Uses the prebuilt RDNA4 vLLM image
[`capicua25x/vllm-rocm-rdna4`](https://hub.docker.com/r/capicua25x/vllm-rocm-rdna4) (tag `0.19.1`):

```bash
docker run --rm --network=host \
  --device=/dev/kfd --device=/dev/dri \
  --group-add=video --group-add=render --ipc=host --ulimit memlock=-1 \
  capicua25x/vllm-rocm-rdna4:0.19.1 \
  --model Capicua25x/Ornith-1.0-9B-MXFP4-Vision-MTP \
  --served-model-name ornith --trust-remote-code \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --attention-backend TRITON_ATTN \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

### The settings that actually matter on RDNA4
- `--attention-backend TRITON_ATTN` — required on gfx1201.
- `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` — enables the grafted MTP head.
  **n=3 maximizes throughput; n=1–2 maximize per-token acceptance.** Tune per workload.
- `--tool-call-parser qwen3_xml --reasoning-parser qwen3` — Qwen3.5-family tool-calling + reasoning split.
- `--trust-remote-code` — the `qwen3_5` vision architecture.
- **Single GPU works too**: `--tensor-parallel-size 1` and pass one render node
  (e.g. `--device=/dev/dri/renderD128`).

## How it was built (reproducible)
1. **MXFP4 quantize** Ornith-1.0-9B with `compressed-tensors` (4-bit float, group 32, symmetric;
   `lm_head` / `embed_tokens` / norms / vision tower left BF16).
2. **Graft** the 15 `mtp.*` head tensors from `protoLabsAI/Ornith-1.0-9B-MTP` into a new
   `model-mtp.safetensors` shard and patch `model.safetensors.index.json`.
3. **Mark the head unquantized** — add its Linear modules (`mtp.fc`, `mtp.layers.0.self_attn.*`,
   `mtp.layers.0.mlp.*`) to `quantization_config.ignore`, so vLLM's compressed-tensors loader keeps the
   BF16 head as-is instead of expecting MXFP4 weight-scales. This is the one mixed-precision gotcha.

Step 1 (the MXFP4 quantize) is scripted in [`quantize_mxfp4.py`](./quantize_mxfp4.py) (weight-only `MXFP4A16`, group 32, data-free — deterministic/byte-reproducible). Steps 2–3 are scripted in [`recipe_graft_mxfp4.py`](./recipe_graft_mxfp4.py) (run against an MXFP4
`compressed-tensors` trunk + the protoLabs head). The head's distillation recipe lives upstream at
[`protoLabsAI/Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP).

## Credits
- **DeepReinforce** — [`Ornith-1.0-9B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B), the base model (MIT).
- **protoLabs** — [`Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP), the KL-distilled MTP draft head and its recipe (MIT).
- **Qwen / Alibaba** — the Qwen3.5 architecture the MTP head derives from (the head was initialized from `Qwen/Qwen3.5-9B`'s `mtp.*` tensors).
- **vLLM** and **compressed-tensors** — serving stack and quantization format.
- **Rob Smith** (`tcclaviger`) — the [RDNA4 vLLM base image](https://hub.docker.com/r/tcclaviger/vllm-rocm-mxfp4-nvfp4) that made gfx1201 serving possible. The [`capicua25x/vllm-rocm-rdna4`](https://hub.docker.com/r/capicua25x/vllm-rocm-rdna4) image this model runs on is a forward-port of his work — without it, none of this runs.

## License
**MIT.** This is a derivative of Ornith-1.0-9B (MIT); merging the MTP head (MIT) produces a derivative whose MIT terms carry.