| --- |
| license: apache-2.0 |
| base_model: nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill |
| base_model_relation: quantized |
| language: |
| - en |
| - zh |
| pipeline_tag: image-text-to-text |
| tags: |
| - mxfp4 |
| - quantized |
| - compressed-tensors |
| - rocm |
| - rdna4 |
| - moe |
| - qwen3 |
| - distillation |
| - deepseek-v4 |
| - reasoning |
| - thinking |
| - agentic |
| - multimodal |
| - vision |
| --- |
| |
| # Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — MXFP4 + Vision (RDNA4 / R9700) |
|
|
| The **vision-enabled sibling** of |
| [`Capicua25x/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-MXFP4`](https://huggingface.co/Capicua25x/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-MXFP4): |
| the same MXFP4 quant of nerkyor's DeepSeek-V4-Pro thinking distill, **with the Qwen3.6 base |
| vision tower grafted back in** — so it's **text + reasoning + vision + MTP**, all on the |
| RDNA4 [`tcclaviger/vllm-rocm-mxfp4-nvfp4`](https://hub.docker.com/r/tcclaviger/vllm-rocm-mxfp4-nvfp4) container. |
|
|
| The distill shipped text-only (nerkyor dropped the vision tower on export). This build restores it. |
| Recipe + tooling: **https://github.com/Capicua25x/qwen3.6-mxfp4-rdna4** |
|
|
| ## How image support was added (the vision graft) |
|
|
| nerkyor's distill is `Qwen3_5MoeForCausalLM` (text-only) — no vision tower. But the base |
| `Qwen/Qwen3.6-35B-A3B` is multimodal, and its vision tower is **architecturally independent** of |
| the language model: the vision encoder → merger projects image features to `out_hidden_size: 2048`, |
| which is exactly the LM's `hidden_size`. So the tower bolts straight on — the same trick used to |
| graft MTP back: |
|
|
| 1. **Copy the base's 333 `model.visual.*` tensors** (BF16, ~0.9 GB) from a known-good multimodal |
| build into a new shard. |
| 2. **Add `model.visual.*` to `quantization_config.ignore`** so vLLM loads them as BF16 (the vision |
| tower is never quantized), and keep the multimodal `Qwen3_5MoeConfig` (with `vision_config`). |
| 3. **Serve without `--language-model-only`** (that flag had been stubbing the tower out), with the |
| base's `chat_template` + `preprocessor_config` for image-token handling. |
|
|
| **Why it works:** the DeepSeek-V4-Pro distill is a *light* LoRA (text-only reasoning traces); it |
| shifted the LM but didn't break its ability to interpret the base vision tower's embedding space. |
| **Verified empirically** — fed a generated image (text "VISION 7", a red circle, a blue rectangle), |
| the model read the **text, shapes, colors, and spatial positions** correctly, while still reasoning. |
| (Vision is *not* lossless like MTP — a heavier-finetuned LM might fail to ground the embeddings — so |
| this is a measured result, not a guarantee for arbitrary distills.) |
|
|
| ## Serving (vLLM, 2× R9700, TP2) |
|
|
| ```bash |
| vllm serve Capicua25x/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-MXFP4-Vision \ |
| --tensor-parallel-size 2 --gpu-memory-utilization 0.92 --max-model-len 262144 \ |
| --enable-prefix-caching --max-num-seqs 64 \ |
| --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \ |
| --speculative-config '{"method":"mtp","num_speculative_tokens":3}' |
| ``` |
| **Do NOT pass `--language-model-only`** here (that disables vision). Send images via the standard |
| OpenAI `image_url` content blocks. Thinking-on: `temperature=0.6 / top_p=0.95`. |
|
|
| ## Validation — identical to the text-only build, plus vision |
|
|
| | Gate | This (vision) build | Text-only sibling | |
| |---|---|---| |
| | SQL regression | **136/137, 0 FAIL** | 136/137, 0 FAIL | |
| | Agent eval (tool-calling, thinking-on) | **27/27, 0 FAIL** | 26/27 | |
| | Single-stream / ceiling (short) | **108.9 tok/s / ~128** | 107 / ~128 | |
| | MTP draft acceptance (MTP-3) | **~57%** | ~56% | |
| | Image understanding | ✅ | ❌ (none) | |
| | Cost of vision | **+0.9 GB VRAM, 0 text-perf** | — | |
|
|
| The vision tower only fires on image input, so text/agentic throughput is unchanged. |
|
|
| ## Notes |
|
|
| - The "8-bit precision" badge is a HuggingFace artifact of MXFP4's `uint8` packing — this model |
| is genuinely **4-bit** (`config.json` → `num_bits: 4`, `mxfp4-pack-quantized`). |
| - Built/tested only on **gfx1201 (RDNA4, R9700)** with `tcclaviger/vllm-rocm-mxfp4-nvfp4`. |
|
|
| ## Credits |
|
|
| Same chain as the text-only build, plus the vision tower from Qwen's base: |
|
|
| - **Qwen team (Alibaba)** — [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) base + the vision tower (Apache-2.0). |
| - **nerkyor / "Lynn"** — the [DSV4Pro-Thinking distill](https://huggingface.co/nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill) (the reasoning). |
| - **DeepSeek-AI** — DeepSeek-V4-Pro, the distillation teacher. |
| - **olka** — [qstream](https://github.com/olka/qstream), the MXFP4 quantizer. |
| - **kallepahajoki** — the qstream Qwen3.6 fix + reference MXFP4 recipe ([`pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4`](https://huggingface.co/pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4)), the source of the BF16 vision-tower weights grafted here. |
| - **tcclaviger** — the [RDNA4/gfx12 MXFP4 vLLM container & kernel](https://hub.docker.com/r/tcclaviger/vllm-rocm-mxfp4-nvfp4). |
|
|
| Quantization + RDNA4 packaging (config-wrap, MTP graft, vision graft) by **Capicua25x**. Apache-2.0. |
|
|
