license: other
base_model:
- Qwen/Qwen3.6-35B-A3B
tags:
- nvfp4
- modelopt
- quantization
- vllm
- mixture-of-experts
- tool-calling
pipeline_tag: image-text-to-text
Qwen3.6-35B-A3B — HybridQuant NVFP4
Mixed-precision (primarily 4-bit NVFP4) quantization of Qwen3.6-35B-A3B for
high-throughput, high-quality serving on NVIDIA GB10 (DGX Spark, sm_121a) via
vLLM's modelopt_fp4 path. This is the current production serving weight.
Recipe
quant_algo: MIXED_PRECISION (NVIDIA TensorRT Model Optimizer):
- NVFP4 (W4A16, group 16, fp8 scales) on the MoE experts, shared-expert MLP,
lm_head, and the visual tower — 4-bit weights, 16-bit activations. - FP8 on the attention / linear-attention projections · FP8 KV cache.
- Norms, embeddings, router, and the MTP speculative-decode head kept at bf16.
- MTP head retained for fast speculative decode (≈80–92% draft acceptance).
- Custom chat template with a hardened tool-use system prompt; thinking off by default.
The always-active path (attention, dense, lm_head) is deliberately quantized —
on this memory-bound MoE that is what sets decode speed, since those tensors are
read every token while any single expert is read ~1/32 as often.
Evaluation (2026-07-04, deterministic harness)
Recipe: prefix caching off, temperature 0, thinking off, solo/sequential,
vLLM eugr nightly (0.23.1rc1.dev701). Full runs repeated for stability.
| Benchmark | Result |
|---|---|
| tool-eval-bench (full 69, ×3) | 91 / 92 / 91 |
| tool_calls_bench | 16 / 16 |
| agentic_eval | 100 / 100 |
Per-category strengths at ceiling (Tool Selection, Toolset Scale, Structured Output, Error Recovery all 100%); the weakest axis is Safety / prompt-injection resistance (81–88%) — see limitations.
Performance (measured, GB10 / DGX Spark)
| Metric | Value |
|---|---|
| Decode, 1 stream (MTP on) | ~92 tok/s |
| Decode, 4 concurrent (aggregate) | ~211 tok/s |
| Stored footprint | ~21.2 GiB (3 safetensors shards) |
| Active bytes / token | ~2.34 GiB (≈ 90 tok/s bandwidth ceiling) |
Serving (vLLM)
vllm serve JasonW2025/Qwen3.6-35B-A3B-HybridQuant-NVFP4 \
--quantization modelopt_fp4 --kv-cache-dtype fp8 --calculate-kv-scales \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--trust-remote-code --no-enable-prefix-caching --enable-chunked-prefill \
--gpu-memory-utilization 0.60 --max-model-len 262144
Known limitations
- Prompt-injection safety is the weak axis (Safety category 81–88%). In particular a sleeper injection — malicious instructions embedded in tool-result data on an earlier turn — is not reliably refused (this is a base-model instruction-following property, not introduced by quantization, and it also fails on the compared baselines). Enforce tool-output sanitization and recipient/allow-list checks at the agent/framework layer; do not rely on the model alone.
- Per-category benchmark scores carry a ±1-scenario noise floor; treat small category swings as noise, headline totals as signal.
Provenance & license
Derivative of Qwen3.6-35B-A3B; the original model's license and terms apply. Quantized with NVIDIA TensorRT Model Optimizer (modelopt). Not affiliated with or endorsed by Qwen/Alibaba or NVIDIA.