model card

README.md

@@ -0,0 +1,118 @@
+---
+license: mit
+base_model: deepreinforce-ai/Ornith-1.0-9B
+base_model_relation: quantized
+pipeline_tag: image-text-to-text
+library_name: vllm
+language:
+- en
+tags:
+- mxfp4
+- compressed-tensors
+- speculative-decoding
+- mtp
+- multi-token-prediction
+- qwen3.5
+- vllm
+- rocm
+- rdna4
+- amd
+- vision
+---
+
+# Ornith-1.0-9B — MXFP4 + MTP (vision), for AMD RDNA4 / vLLM
+
+[`deepreinforce-ai/Ornith-1.0-9B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B) quantized to
+**MXFP4** with a **grafted MTP (Multi-Token-Prediction) draft head**, packaged to run **out of the box on
+AMD Radeon RDNA4 (gfx1201) under vLLM** with lossless self-speculative decoding. Vision retained.
+
+- **Trunk** — Ornith-1.0-9B quantized to **MXFP4** (`compressed-tensors`, group size 32, symmetric).
+  `lm_head`, `embed_tokens`, norms, and the vision tower are kept **BF16**.
+- **Draft head** — the **KL-distilled MTP head** from
+  [`protoLabsAI/Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP) (BF16), grafted
+  into a separate `model-mtp.safetensors` shard and marked unquantized in the quant config so the
+  mixed-precision model loads cleanly.
+- **Speculative decoding** — vLLM native `mtp` method, `num_speculative_tokens=3`. **Lossless**: the
+  target verifies every drafted token, so the output distribution is unchanged — the head only buys speed.
+
+## Measured on AMD RDNA4
+
+2× Radeon AI PRO R9700 (gfx1201), tensor-parallel 2, vLLM `0.19.1` (image below).
+
+- **MTP draft acceptance (n=3): ≈66% average** across a mixed code + 6k-context run — per-position
+  **0.82 / 0.65 / 0.53**, mean acceptance length **~3.0** (max 4) — rising to **~85%** (length ~3.6) on
+  cache-warm short context. Lossless throughout.
+- **Tool-calling** (`qwen3_xml`) and **vision** confirmed working.
+- **Throughput** (TP2, 256 output tokens, per-user / aggregate tok/s):
+
+| concurrent | short prompt | ~6k prompt |
+|---:|---|---|
+| 1   | 84.6 / 85   | 103.8 / 104 |
+| 16  | 51.8 / 803  | 50.1 / 767  |
+| 32  | 40.8 / 1260 | 33.0 / 1010 |
+| 64  | 30.7 / 1902 | 21.1 / 1268 |
+| 96  | 22.5 / 2075 | 15.0 / 1293 |
+| 128 | 20.2 / 1893 | 12.6 / 1284 |
+
+Usable concurrency ceiling (per-user ≥ 20 tok/s): **~128** at short context, **~64** at 6k context.
+Single-stream decode is bound by the dense 9B's active-parameter count; a single GPU (TP1) is also supported.
+
+## Notes from the publisher
+
+On AMD RDNA4, **tool-calling and core agentic tool-use were a clear, reliable step up** in our testing —
+that's what this build is good at. We did **not** independently evaluate complex long-horizon reasoning;
+evaluate on your own tasks. Base-model capabilities and limitations are
+[DeepReinforce](https://huggingface.co/deepreinforce-ai)'s to characterize.
+
+## Run it on AMD RDNA4
+
+Uses the prebuilt RDNA4 vLLM image
+[`capicua25x/vllm-rocm-rdna4`](https://hub.docker.com/r/capicua25x/vllm-rocm-rdna4) (tag `0.19.1`):
+
+```bash
+docker run --rm --network=host \
+  --device=/dev/kfd --device=/dev/dri \
+  --group-add=video --group-add=render --ipc=host --ulimit memlock=-1 \
+  capicua25x/vllm-rocm-rdna4:0.19.1 \
+  --model Capicua25x/Ornith-1.0-9B-MXFP4-Vision-MTP \
+  --served-model-name ornith --trust-remote-code \
+  --tensor-parallel-size 2 \
+  --gpu-memory-utilization 0.90 \
+  --max-model-len 16384 \
+  --attention-backend TRITON_ATTN \
+  --enable-prefix-caching \
+  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
+  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
+```
+
+### The settings that actually matter on RDNA4
+- `--attention-backend TRITON_ATTN` — required on gfx1201.
+- `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` — enables the grafted MTP head.
+  **n=3 maximizes throughput; n=1–2 maximize per-token acceptance.** Tune per workload.
+- `--tool-call-parser qwen3_xml --reasoning-parser qwen3` — Qwen3.5-family tool-calling + reasoning split.
+- `--trust-remote-code` — the `qwen3_5` vision architecture.
+- **Single GPU works too**: `--tensor-parallel-size 1` and pass one render node
+  (e.g. `--device=/dev/dri/renderD128`).
+
+## How it was built (reproducible)
+1. **MXFP4 quantize** Ornith-1.0-9B with `compressed-tensors` (4-bit float, group 32, symmetric;
+   `lm_head` / `embed_tokens` / norms / vision tower left BF16).
+2. **Graft** the 15 `mtp.*` head tensors from `protoLabsAI/Ornith-1.0-9B-MTP` into a new
+   `model-mtp.safetensors` shard and patch `model.safetensors.index.json`.
+3. **Mark the head unquantized** — add its Linear modules (`mtp.fc`, `mtp.layers.0.self_attn.*`,
+   `mtp.layers.0.mlp.*`) to `quantization_config.ignore`, so vLLM's compressed-tensors loader keeps the
+   BF16 head as-is instead of expecting MXFP4 weight-scales. This is the one mixed-precision gotcha.
+
+Steps 2–3 are scripted in [`recipe_graft_mxfp4.py`](./recipe_graft_mxfp4.py) (run against an MXFP4
+`compressed-tensors` trunk + the protoLabs head). The head's distillation recipe lives upstream at
+[`protoLabsAI/Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP).
+
+## Credits
+- **DeepReinforce** — [`Ornith-1.0-9B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B), the base model (MIT).
+- **protoLabs** — [`Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP), the KL-distilled MTP draft head and its recipe (MIT).
+- **Qwen / Alibaba** — the Qwen3.5 architecture the MTP head derives from (the head was initialized from `Qwen/Qwen3.5-9B`'s `mtp.*` tensors).
+- **vLLM** and **compressed-tensors** — serving stack and quantization format.
+- RDNA4 vLLM image: [`capicua25x/vllm-rocm-rdna4`](https://hub.docker.com/r/capicua25x/vllm-rocm-rdna4).
+
+## License
+**MIT.** This is a derivative of Ornith-1.0-9B (MIT); merging the MTP head (MIT) produces a derivative whose MIT terms carry.