---
license: other
base_model:
- Qwen/Qwen3.6-35B-A3B
tags:
- nvfp4
- modelopt
- quantization
- vllm
- mixture-of-experts
- tool-calling
pipeline_tag: image-text-to-text
---

# Qwen3.6-35B-A3B — HybridQuant NVFP4

Mixed-precision (primarily 4-bit NVFP4) quantization of **Qwen3.6-35B-A3B** for
high-throughput, high-quality serving on NVIDIA GB10 (DGX Spark, `sm_121a`) via
vLLM's `modelopt_fp4` path. This is the current **production** serving weight.

## Recipe

`quant_algo: MIXED_PRECISION` (NVIDIA TensorRT Model Optimizer):

- **NVFP4 (W4A16, group 16, fp8 scales)** on the MoE experts, shared-expert MLP,
  `lm_head`, and the visual tower — 4-bit weights, **16-bit activations**.
- **FP8** on the attention / linear-attention projections · **FP8 KV cache**.
- Norms, embeddings, router, and the **MTP** speculative-decode head kept at bf16.
- **MTP** head retained for fast speculative decode (≈80–92% draft acceptance).
- Custom chat template with a hardened tool-use system prompt; thinking off by default.

The always-active path (attention, dense, `lm_head`) is deliberately quantized —
on this memory-bound MoE that is what sets decode speed, since those tensors are
read every token while any single expert is read ~1/32 as often.

## Evaluation (2026-07-04, deterministic harness)

Recipe: prefix caching **off**, temperature 0, thinking off, solo/sequential,
vLLM eugr nightly (`0.23.1rc1.dev701`). Full runs repeated for stability.

| Benchmark | Result |
|---|---|
| tool-eval-bench (full 69, ×3) | **91 / 92 / 91** |
| tool_calls_bench | **16 / 16** |
| agentic_eval | **100 / 100** |

Per-category strengths at ceiling (Tool Selection, Toolset Scale, Structured
Output, Error Recovery all 100%); the weakest axis is **Safety / prompt-injection
resistance (81–88%)** — see limitations.

## Performance (measured, GB10 / DGX Spark)

| Metric | Value |
|---|---|
| Decode, 1 stream (MTP on) | **~92 tok/s** |
| Decode, 4 concurrent (aggregate) | **~211 tok/s** |
| Stored footprint | **~21.2 GiB** (3 safetensors shards) |
| Active bytes / token | **~2.34 GiB** (≈ 90 tok/s bandwidth ceiling) |

## Serving (vLLM)

```bash
vllm serve JasonW2025/Qwen3.6-35B-A3B-HybridQuant-NVFP4 \
  --quantization modelopt_fp4 --kv-cache-dtype fp8 --calculate-kv-scales \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --trust-remote-code --no-enable-prefix-caching --enable-chunked-prefill \
  --gpu-memory-utilization 0.60 --max-model-len 262144
```

## Known limitations

- **Prompt-injection safety is the weak axis** (Safety category 81–88%). In
  particular a *sleeper* injection — malicious instructions embedded in tool-result
  data on an earlier turn — is not reliably refused (this is a base-model
  instruction-following property, not introduced by quantization, and it also
  fails on the compared baselines). Enforce tool-output sanitization and
  recipient/allow-list checks at the agent/framework layer; do not rely on the
  model alone.
- Per-category benchmark scores carry a ±1-scenario noise floor; treat small
  category swings as noise, headline totals as signal.

## Provenance & license

Derivative of **Qwen3.6-35B-A3B**; the original model's license and terms apply.
Quantized with NVIDIA TensorRT Model Optimizer (modelopt). Not affiliated with or
endorsed by Qwen/Alibaba or NVIDIA.