JasonW2025/Qwen3.6-35B-A3B-HybridQuant-NVFP4

Qwen3.6-35B-A3B — HybridQuant NVFP4

Mixed-precision (primarily 4-bit NVFP4) quantization of Qwen3.6-35B-A3B for high-throughput, high-quality serving on NVIDIA GB10 (DGX Spark, sm_121a) via vLLM's modelopt_fp4 path. This is the current production serving weight.

Recipe

quant_algo: MIXED_PRECISION (NVIDIA TensorRT Model Optimizer):

• NVFP4 (W4A16, group 16, fp8 scales) on the MoE experts, shared-expert MLP, lm_head, and the visual tower — 4-bit weights, 16-bit activations.
• FP8 on the attention / linear-attention projections · FP8 KV cache.
• Norms, embeddings, router, and the MTP speculative-decode head kept at bf16.
• MTP head retained for fast speculative decode (≈80–92% draft acceptance).
• Custom chat template with a hardened tool-use system prompt; thinking off by default.

The always-active path (attention, dense, lm_head) is deliberately quantized — on this memory-bound MoE that is what sets decode speed, since those tensors are read every token while any single expert is read ~1/32 as often.

Evaluation (2026-07-04, deterministic harness)

Recipe: prefix caching off, temperature 0, thinking off, solo/sequential, vLLM eugr nightly (0.23.1rc1.dev701). Full runs repeated for stability.

Benchmark	Result
tool-eval-bench (full 69, ×3)	91 / 92 / 91
tool_calls_bench	16 / 16
agentic_eval	100 / 100

Per-category strengths at ceiling (Tool Selection, Toolset Scale, Structured Output, Error Recovery all 100%); the weakest axis is Safety / prompt-injection resistance (81–88%) — see limitations.

Performance (measured, GB10 / DGX Spark)

Metric	Value
Decode, 1 stream (MTP on)	~92 tok/s
Decode, 4 concurrent (aggregate)	~211 tok/s
Stored footprint	~21.2 GiB (3 safetensors shards)
Active bytes / token	~2.34 GiB (≈ 90 tok/s bandwidth ceiling)

Serving (vLLM)

vllm serve JasonW2025/Qwen3.6-35B-A3B-HybridQuant-NVFP4 \
  --quantization modelopt_fp4 --kv-cache-dtype fp8 --calculate-kv-scales \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --trust-remote-code --no-enable-prefix-caching --enable-chunked-prefill \
  --gpu-memory-utilization 0.60 --max-model-len 262144

Known limitations

• Prompt-injection safety is the weak axis (Safety category 81–88%). In particular a sleeper injection — malicious instructions embedded in tool-result data on an earlier turn — is not reliably refused (this is a base-model instruction-following property, not introduced by quantization, and it also fails on the compared baselines). Enforce tool-output sanitization and recipient/allow-list checks at the agent/framework layer; do not rely on the model alone.
• Per-category benchmark scores carry a ±1-scenario noise floor; treat small category swings as noise, headline totals as signal.

Provenance & license

Derivative of Qwen3.6-35B-A3B; the original model's license and terms apply. Quantized with NVIDIA TensorRT Model Optimizer (modelopt). Not affiliated with or endorsed by Qwen/Alibaba or NVIDIA.