README.md

metadata

license: apache-2.0
base_model: nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill
base_model_relation: quantized
language:
  - en
  - zh
pipeline_tag: text-generation
tags:
  - mxfp4
  - quantized
  - compressed-tensors
  - rocm
  - rdna4
  - moe
  - qwen3
  - distillation
  - deepseek-v4
  - reasoning
  - thinking
  - agentic

Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — MXFP4 (RDNA4 / R9700)

MXFP4 quantization of nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — a DeepSeek-V4-Pro thinking-style LoRA distill of Qwen3.6-35B-A3B — built to run on AMD RDNA4 (Radeon AI PRO R9700 / RX 9070 XT) under the tcclaviger/vllm-rocm-mxfp4-nvfp4 vLLM container, with multi-token-prediction (MTP) speculative decoding grafted back in.

As far as I know this is the first RDNA4-loadable MXFP4 of this distill. The distill's own NVFP4 build targets SGLang/Blackwell and crashes this container; this build loads cleanly and benches at parity with the production base (pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4).

• 69.3 GB BF16 → 21 GB MXFP4 (~30%)
• Recipe, tooling & full build log: https://github.com/Capicua25x/qwen3.6-mxfp4-rdna4

Serving (vLLM, 2× R9700, TP2)

vllm serve Capicua25x/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-MXFP4 \
  --tensor-parallel-size 2 --gpu-memory-utilization 0.92 --max-model-len 262144 \
  --language-model-only \
  --enable-prefix-caching --max-num-seqs 64 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

⚠️ --language-model-only is REQUIRED. This is a text-only checkpoint, but its config.json is wrapped in the multimodal Qwen3_5MoeConfig shape so the container's Qwen3_5MoeForConditionalGeneration loader accepts it; the flag makes the (weightless) vision tower a no-init stub. Without it, loading fails.

Thinking mode is optional (verified both ways)

It's a thinking distill, but thinking is not required — it runs cleanly with thinking on or off. This isn't obvious from the model; we tested to confirm it:

• Off — pass chat_template_kwargs={"enable_thinking": false} (the chat template injects a pre-closed <think></think> so the model answers directly). Validated on a 137-test SQL/analytics regression suite — classification + SQL-gen + interpretation all thinking-off: 136/137, 0 failures.
• On (default; temperature=0.6 / top_p=0.95) — the DeepSeek-V4-Pro reasoning style. Validated on a 27-scenario agentic tool-calling eval: 26/27, ~4× faster per turn-chain than the base model.

So one deployment serves both: thinking-off for deterministic / high-throughput paths, thinking-on for agentic / reasoning paths. (Note: in thinking-on it reasons inline in content — the qwen3 parser returns empty reasoning_content.)

Performance (2× R9700, MTP-3)

Both columns measured on the same hardware/bench (2× R9700, TP2, MTP-3 each):

Metric	This build	Base (pahajoki)
Single-stream, short prompt	~107 tok/s	~101
Single-stream, 6k prompt	~82 tok/s	~85
Aggregate @128, short prompt	~1875 tok/s	~1683
Concurrency ceiling, short prompt	~128	~128
MTP draft acceptance (MTP-3, measured)	~56% (grafted)	~64% (native)

Effectively at parity: the distill edges the base on short-prompt single-stream (107 vs 101) and high-concurrency aggregate (1875 vs 1683 @128); 6k single-stream is a wash (82 vs 85). Where it clearly wins is agentic — in a 27-scenario thinking-on tool-calling eval it ran ~4× faster per turn-chain than the base at equal task success (its DS-V4-Pro distillation makes it decisive — fewer tokens to a decision).

Reproduce — step by step

Full scripts in the companion repo. The recipe is generic for the Qwen qwen3_5_moe family on RDNA4.

0. Tooling. Clone olka/qstream master (includes the merged Qwen3.6 fix) into a venv: pip install -e qstream.

1. Quantize (CPU, RTN — no GPU). The qstream defaults are the recipe:

qstream-quantize --model_dir <bf16-distill> --output_dir <out> --workers 8 --format ct

Keeps BF16: *self_attn* *.mlp.gate. *shared_expert* *lm_head* *embed_tokens* *visual* *mtp*. Quantizes to MXFP4 (compressed-tensors mxfp4-pack-quantized, per-expert, group-32 symmetric, MSE-optimal scale): the linear_attn projections + all 256 routed experts.

2. Wrap the config (text-only checkpoints only). The RDNA4 container only registers the multimodal loader, so rewrap config.json into Qwen3_5MoeConfig shape — nest the text fields under text_config, graft a vision_config + vision/image token ids from a multimodal build (e.g. pahajoki's base) — then always serve with --language-model-only. (scripts/wrap-config.py)

3. Graft MTP (optional; restores speculative decoding). This distill shipped mtp_num_hidden_layers: 0. The base Qwen3.6 MTP block is dim-identical, so: copy the base's 785 BF16 mtp.* tensors into a new shard, set mtp_num_hidden_layers: 1, and add the mtp.* modules to quantization_config.ignore (else vLLM loads them as quantized → fc.weight not found in params_dict). (scripts/graft-mtp.py)

Caveats

• HF shows an "8-bit precision" badge — ignore it, this model is 4-bit. MXFP4 packs two 4-bit FP4 values into each uint8 byte (weight_packed), so HF reads the uint8 storage dtype and mislabels it. Source of truth: config.json → num_bits: 4, format: mxfp4-pack-quantized. Every MXFP4 compressed-tensors model shows this (incl. the base).

• Built/tested only on gfx1201 (RDNA4) with tcclaviger/vllm-rocm-mxfp4-nvfp4.

• Reasons inline in content — the qwen3 reasoning-parser returns empty reasoning_content.

• --language-model-only required (see Serving).

• Grafted MTP acceptance ~56% vs ~64% native (both measured at MTP-3) — only ~8pp behind; a native MTP retrain would close it. MTP is lossless either way.

Credits & acknowledgments

This build stands entirely on others' work — full credit to:

• The Qwen team (Alibaba) — Qwen3.6-35B-A3B, the base model and architecture (Apache-2.0).
• nerkyor / "Lynn" — the DSV4Pro-Thinking distill this quantizes; the actual reasoning capability is theirs.
• DeepSeek-AI — DeepSeek-V4-Pro, the distillation teacher.
• olka — qstream, the streamable MXFP4 quantizer.
• kallepahajoki — the qstream Qwen3.6-family fix (PR #1: shared-expert + fused-passthrough) and the reference MXFP4 recipe published as pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4, including the BF16/MXFP4 exclude set and the MTP layout this graft borrows.
• tcclaviger — the RDNA4/gfx12 MXFP4 vLLM container & custom MoE kernel that makes any of this run on consumer/pro AMD cards.
• Distillation technique lineage (from the source distill): ReAct (Yao et al., 2022, arXiv:2210.03629), Self-Instruct/Baize, AgentTuning, ToolBench, DeepSeek-R1 distillation.

Quantization + RDNA4 packaging (config-wrap + MTP graft) by Capicua25x. Apache-2.0.