--- license: apache-2.0 base_model: deepreinforce-ai/Ornith-1.0-35B base_model_relation: quantized pipeline_tag: image-text-to-text library_name: vllm language: - en tags: - mxfp4 - compressed-tensors - speculative-decoding - mtp - multi-token-prediction - moe - qwen3.5 - vllm - rocm - rdna4 - gfx1201 - radeon-ai-pro-r9700 - amd - vision --- # Ornith-1.0-35B-A3B — MXFP4 + MTP (vision), for AMD RDNA4 / vLLM [`deepreinforce-ai/Ornith-1.0-35B`](https://huggingface.co/ornith-ai/Ornith-1.0-35B) — a **Qwen MoE model (A3B, ~3 B active parameters per token)** — quantized to **MXFP4** with a **grafted MTP (Multi-Token-Prediction) draft head**, packaged to run **out of the box on AMD Radeon RDNA4 (gfx1201) under vLLM** with lossless self-speculative decoding. Vision retained. - **Trunk** — Ornith-1.0-35B quantized to **MXFP4** (`compressed-tensors`, group size 32, symmetric). `lm_head`, `embed_tokens`, norms, and the vision tower are kept **BF16**. The MXFP4 trunk is byte-verified against DeepReinforce's published weights (16/16 shards SHA256-identical before quant). - **Draft head — a cross-model graft.** The MoE MTP head (785 BF16 tensors, with per-expert weights) is transplanted from a **sibling Qwen-MoE checkpoint**, [`Capicua25x/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-MXFP4-Vision`](https://huggingface.co/Capicua25x/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-MXFP4-Vision) (a DeepSeek-V4-Pro-Thinking distill of Qwen3.6-35B-A3B). The head weights originate with [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Apache-2.0) and reach this build unchanged via [`pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4`](https://huggingface.co/pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4) — every checkpoint in that chain carries them byte-identical. Because the trunk and the donor share the **same Qwen MoE architecture** — identical hidden size and expert layout — the head transfers cleanly and drafts well on the Ornith trunk (acceptance below). - **Speculative decoding** — vLLM native `mtp` method, `num_speculative_tokens=3`. **Lossless**: the target model verifies every drafted token, so the output distribution is identical to the trunk with no draft head. A mismatched head could only *slow* drafting, never change an output — which is what makes a cross-model graft safe to ship. This one drafts at **~69% acceptance**, a clear net speed-up. ## Measured on AMD RDNA4 2× Radeon AI PRO R9700 (gfx1201), tensor-parallel 2, vLLM `0.19.1` (image below). - **MTP draft acceptance (n=3): ≈69.5% average — measured on live production traffic**, not a synthetic probe. Per-position **0.84 / 0.67 / 0.57**; mean acceptance length **~3.08**. Lossless throughout. - **Vision**, **tool-calling** (`qwen3_xml`), and **reasoning split** (`qwen3`) all confirmed working. - **Throughput** (TP2, 256 output tokens, per-user / aggregate tok/s): | concurrent | short prompt | ~6k prompt | |---:|---|---| | 1 | 130.9 / 131 | 100.9 / 101 | | 16 | 58.9 / 889 | 34.2 / 523 | | 32 | 40.2 / 1219 | 20.3 / 617 | | 64 | 29.9 / 1822 | 12.3 / 723 | | 96 | 23.4 / 1584 | 9.2 / 717 | | 128 | 21.4 / 1779 | 8.2 / 761 | Usable concurrency ceiling (per-user ≥ 20 tok/s): **~128** at short context, **~32** at 6k context. Single-stream decode is **130.9 tok/s** short / **100.9 tok/s** at 6k — the A3B sparsity (~3 B active) makes it *faster* single-stream than a dense 9B at short context, and comparable at long context, while carrying far more total knowledge. ## Run it on AMD RDNA4 Uses the prebuilt RDNA4 vLLM image [`capicua25x/vllm-rocm-rdna4`](https://hub.docker.com/r/capicua25x/vllm-rocm-rdna4) (tag `0.19.1`): ```bash docker run --rm --network=host \ --device=/dev/kfd --device=/dev/dri \ --group-add=video --group-add=render --ipc=host --ulimit memlock=-1 \ -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ capicua25x/vllm-rocm-rdna4:0.19.1 \ --model Capicua25x/Ornith-1.0-35B-MXFP4-Vision-MTP \ --served-model-name ornith --trust-remote-code \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.92 \ --max-model-len 262144 \ --attention-backend TRITON_ATTN \ --enable-prefix-caching --max-num-seqs 64 \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` ### The settings that actually matter on RDNA4 - `--attention-backend TRITON_ATTN` — required on gfx1201; the 0.19.1 default (`ROCM_ATTN`) collapses spec-decode throughput at concurrency. - `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` — enables the grafted MTP head. **n=3 maximizes throughput; n=1–2 maximize per-token acceptance.** Tune per workload. - `--tool-call-parser qwen3_xml --reasoning-parser qwen3` — Qwen-family tool-calling + reasoning split. - `--trust-remote-code` — the Qwen MoE vision architecture. - **Memory** — fits **2× 32 GB** at MXFP4 with 256k context (`--gpu-memory-utilization 0.92`, `--max-num-seqs 64`). The 35 B does not fit a single 32 GB GPU at this context; use TP2. ## How it was built (reproducible) 1. **MXFP4 quantize** Ornith-1.0-35B with `compressed-tensors` (4-bit float, group 32, symmetric; `lm_head` / `embed_tokens` / norms / vision tower left BF16). 2. **Graft** the `mtp.*` MoE head (785 BF16 tensors — including the per-expert `mtp.layers.0.mlp.experts.N.*` projections) from the donor checkpoint into a new `model-mtp.safetensors` shard and patch `model.safetensors.index.json`. 3. **Mark the head unquantized** — add every `mtp.*` Linear module (the `fc`, the attention projections, and all per-expert MLP projections) to `quantization_config.ignore`, so vLLM's compressed-tensors loader keeps the BF16 head as-is instead of expecting MXFP4 weight-scales. This is the one mixed-precision gotcha; for a MoE head it means **hundreds** of ignore entries (derived automatically from the head's rank-2 tensors). Step 1 (the MXFP4 quantize) is scripted in [`quantize_mxfp4.py`](./quantize_mxfp4.py) (weight-only `MXFP4A16`, group 32, data-free — deterministic/byte-reproducible). Steps 2–3 are scripted in [`recipe_graft_mxfp4.py`](./recipe_graft_mxfp4.py) — it copies **every** `mtp.*` tensor from the donor and derives the ignore list from the rank-2 (matmul) weights, so the same script handles both a dense head and this MoE head. ## Credits - **DeepReinforce** — [`Ornith-1.0-35B`](https://huggingface.co/ornith-ai/Ornith-1.0-35B), the base/trunk model (MIT). *(The `deepreinforce-ai/` id now redirects to `ornith-ai/`.)* - **nerkyor** — [`Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill`](https://huggingface.co/nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill), the DeepSeek-V4-Pro-Thinking **trunk distill**. *(Corrected 2026-08-17: this card previously credited the distill with training the MTP head. It did not — the head is Qwen's, carried through unchanged. Verified by hash: `mtp.fc.weight` [2048, 4096] and the other `mtp.*` tensors are byte-identical between `Qwen/Qwen3.6-35B-A3B` and the distill.)* - **DeepSeek** — DeepSeek-V4-Pro-Thinking, the model that distill learned from. - **pahajokiconsulting** — [`Qwen3.6-35B-A3B-MXFP4`](https://huggingface.co/pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4) (Apache-2.0), the build the 785 `mtp.*` tensors were copied from verbatim: it quantizes the trunk but keeps `mtp.*` in BF16, with the experts relaid from stacked 3D into per-expert 2D. - **Qwen / Alibaba** — the Qwen MoE (A3B) architecture shared by trunk and head, **and the trained MTP head weights themselves** ([`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), Apache-2.0). The grafted head is Qwen's own trained block, not a third party's. - **vLLM** and **compressed-tensors** — serving stack and quantization format. - **Rob Smith** (`tcclaviger`) — the [RDNA4 vLLM base image](https://hub.docker.com/r/tcclaviger/vllm-rocm-mxfp4-nvfp4) that made gfx1201 serving possible. The [`capicua25x/vllm-rocm-rdna4`](https://hub.docker.com/r/capicua25x/vllm-rocm-rdna4) image this model runs on is a forward-port of his work — without it, none of this runs. ## License **Apache-2.0.** This combines the Ornith-1.0-35B trunk (MIT, DeepReinforce) with a grafted MTP head (Apache-2.0). MIT is one-way compatible with Apache-2.0, so the combined derivative is distributed under Apache-2.0; both upstream notices are preserved above in Credits.