README.md

metadata

license: mit
base_model: deepreinforce-ai/Ornith-1.0-9B
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: vllm
language:
  - en
tags:
  - mxfp4
  - compressed-tensors
  - speculative-decoding
  - mtp
  - multi-token-prediction
  - qwen3.5
  - vllm
  - rocm
  - rdna4
  - gfx1201
  - radeon-ai-pro-r9700
  - amd
  - vision

Ornith-1.0-9B — MXFP4 + MTP (vision), for AMD RDNA4 / vLLM

deepreinforce-ai/Ornith-1.0-9B quantized to MXFP4 with a grafted MTP (Multi-Token-Prediction) draft head, packaged to run out of the box on AMD Radeon RDNA4 (gfx1201) under vLLM with lossless self-speculative decoding. Vision retained.

• Trunk — Ornith-1.0-9B quantized to MXFP4 (compressed-tensors, group size 32, symmetric). lm_head, embed_tokens, norms, and the vision tower are kept BF16.
• Draft head — the KL-distilled MTP head from protoLabsAI/Ornith-1.0-9B-MTP (BF16), grafted into a separate model-mtp.safetensors shard and marked unquantized in the quant config so the mixed-precision model loads cleanly.
• Speculative decoding — vLLM native mtp method, num_speculative_tokens=3. Lossless: the target verifies every drafted token, so the output distribution is unchanged — the head only buys speed.

Measured on AMD RDNA4

2× Radeon AI PRO R9700 (gfx1201), tensor-parallel 2, vLLM 0.19.1 (image below).

• MTP draft acceptance (n=3): ≈66% average across a mixed code + 6k-context run — per-position 0.82 / 0.65 / 0.53, mean acceptance length ~3.0 (max 4) — rising to ~85% (length ~3.6) on cache-warm short context. Lossless throughout.
• Tool-calling (qwen3_xml) and vision confirmed working.
• Throughput (TP2, 256 output tokens, per-user / aggregate tok/s):

concurrent	short prompt	~6k prompt
1	84.6 / 85	103.8 / 104
16	51.8 / 803	50.1 / 767
32	40.8 / 1260	33.0 / 1010
64	30.7 / 1902	21.1 / 1268
96	22.5 / 2075	15.0 / 1293
128	20.2 / 1893	12.6 / 1284

Usable concurrency ceiling (per-user ≥ 20 tok/s): ~128 at short context, ~64 at 6k context. Single-stream decode is bound by the dense 9B's active-parameter count; a single GPU (TP1) is also supported.

Run it on AMD RDNA4

Uses the prebuilt RDNA4 vLLM image capicua25x/vllm-rocm-rdna4 (tag 0.19.1):

docker run --rm --network=host \
  --device=/dev/kfd --device=/dev/dri \
  --group-add=video --group-add=render --ipc=host --ulimit memlock=-1 \
  capicua25x/vllm-rocm-rdna4:0.19.1 \
  --model Capicua25x/Ornith-1.0-9B-MXFP4-Vision-MTP \
  --served-model-name ornith --trust-remote-code \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --attention-backend TRITON_ATTN \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The settings that actually matter on RDNA4

• --attention-backend TRITON_ATTN — required on gfx1201.
• --speculative-config '{"method":"mtp","num_speculative_tokens":3}' — enables the grafted MTP head. n=3 maximizes throughput; n=1–2 maximize per-token acceptance. Tune per workload.
• --tool-call-parser qwen3_xml --reasoning-parser qwen3 — Qwen3.5-family tool-calling + reasoning split.
• --trust-remote-code — the qwen3_5 vision architecture.
• Single GPU works too: --tensor-parallel-size 1 and pass one render node (e.g. --device=/dev/dri/renderD128).

How it was built (reproducible)

1. MXFP4 quantize Ornith-1.0-9B with compressed-tensors (4-bit float, group 32, symmetric; lm_head / embed_tokens / norms / vision tower left BF16).
2. Graft the 15 mtp.* head tensors from protoLabsAI/Ornith-1.0-9B-MTP into a new model-mtp.safetensors shard and patch model.safetensors.index.json.
3. Mark the head unquantized — add its Linear modules (mtp.fc, mtp.layers.0.self_attn.*, mtp.layers.0.mlp.*) to quantization_config.ignore, so vLLM's compressed-tensors loader keeps the BF16 head as-is instead of expecting MXFP4 weight-scales. This is the one mixed-precision gotcha.

Step 1 (the MXFP4 quantize) is scripted in quantize_mxfp4.py (weight-only MXFP4A16, group 32, data-free — deterministic/byte-reproducible). Steps 2–3 are scripted in recipe_graft_mxfp4.py (run against an MXFP4 compressed-tensors trunk + the protoLabs head). The head's distillation recipe lives upstream at protoLabsAI/Ornith-1.0-9B-MTP.

Credits

• DeepReinforce — Ornith-1.0-9B, the base model (MIT).
• protoLabs — Ornith-1.0-9B-MTP, the KL-distilled MTP draft head and its recipe (MIT).
• Qwen / Alibaba — the Qwen3.5 architecture the MTP head derives from (the head was initialized from Qwen/Qwen3.5-9B's mtp.* tensors).
• vLLM and compressed-tensors — serving stack and quantization format.
• Rob Smith (tcclaviger) — the RDNA4 vLLM base image that made gfx1201 serving possible. The capicua25x/vllm-rocm-rdna4 image this model runs on is a forward-port of his work — without it, none of this runs.

License

MIT. This is a derivative of Ornith-1.0-9B (MIT); merging the MTP head (MIT) produces a derivative whose MIT terms carry.