---
license: apache-2.0
base_model: Qwen/Qwen3.5-122B-A10B
pipeline_tag: image-text-to-text
tags:
- nvfp4
- compressed-tensors
- quantized
- vllm
- dgx-spark
- gb10
- moe
---

# Qwen3.5-122B-A10B-NVFP4-Full-GB10

All-NVFP4 (W4A4) quantization of [Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
— the maximum-decode-speed variant for NVIDIA DGX Spark (GB10, SM121). Every Linear layer is
4-bit, **including the lm_head**. Vision encoder preserved (BF16) and verified working.

This is the most aggressively quantized checkpoint of this model: at GB10's memory bandwidth
it decodes ~33–35 tok/s single-stream, ~45% faster than the mixed-precision
[FP8Dense](https://huggingface.co/demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10)
sibling (~23–24 tok/s).

## Quality note

The all-FP4 profile carries a **subtle quality degradation** relative to the FP8Dense
sibling: the dense path (attention + GDN) runs 4-bit *activations*, and the lm_head's 4-bit
weights flatten the output distribution slightly. It does not show up on simple extraction
benchmarks (both variants ace detail tests), but in extended real-world use comprehension
and nuance are noticeably better on FP8Dense. Pick by priority:

- **Maximum tok/s** → this checkpoint
- **Best quality at ~2/3 the speed** → [FP8Dense](https://huggingface.co/demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10)

## Layout

| | This model | [FP8Dense](https://huggingface.co/demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10) | [scottgl](https://huggingface.co/scottgl/Qwen3.5-122B-A10B-NVFP4-GB10) |
|---|---|---|---|
| MoE experts (256/layer) | NVFP4 | NVFP4 | NVFP4 |
| Standard attention | NVFP4 | FP8 W8A8 | BF16 |
| GDN projections | NVFP4 | FP8 W8A8 | BF16 stored, FP4/FP8 at SGLang runtime |
| lm_head | **NVFP4** | BF16 | BF16 stored, FP8 at SGLang runtime |
| Vision | BF16 (preserved) | BF16 (preserved) | — |
| Serves on | vLLM (lm_head patch, below) | stock vLLM | custom SGLang fork |
| Checkpoint | 66 GB | 74 GB | — |
| Decode, GB10 single-stream | **33–35 tok/s** | 23–24 tok/s | ~46 (SGLang + NEXTN spec-dec) |

Excluded from quantization (BF16): router gates (`mlp.gate`, `shared_expert_gate`),
embeddings, vision encoder + merger, norms. No MTP weights in this checkpoint (the config's
MTP declaration has no corresponding tensors — speculative decoding is not available).

## Quantization details

- **Method:** [llm-compressor](https://github.com/vllm-project/llm-compressor) `oneshot()`,
  NVFP4 scheme (W4A4, group 16, FP8-E4M3 scales)
- **Calibration:** 512 samples, `HuggingFaceH4/ultrachat_200k`, seq_len 2048
- **Quantized from the full multimodal model** (`AutoModelForImageTextToText`) so the vision
  tower survives — earlier text-only exports of this model lost it

## Benchmarks (DGX Spark GB10, vLLM 0.19.2 from-source SM121 build)

Sequential decode, gen=500:

| Context | TTFT | Decode tok/s |
|---|---|---|
| warmup | 0.84s | 35.2 |
| 9K | 3.22s | 34.6 |
| 18K | 3.72s | 33.8 |
| 27K | 3.69s | 32.9 |

Concurrent @ 32K context, gen=500:

| Concurrency | Aggregate tok/s | Per-request tok/s |
|---|---|---|
| 8× | 78.2 | 12.0 |
| 16× | 110.6 | 8.0 |
| 32× | 138.2 | 5.2 |
| 64× | 166.5 | 3.3 |
| 128× | 182.8 | 2.8 |

KV cache: 5.94× the full 262K context at `--gpu-memory-utilization 0.90` (bf16 KV; page
size 2,096 tokens, aligned to the GDN/Mamba state size).

## Serving (vLLM)

vLLM's stock `ParallelLMHead` cannot load NVFP4-packed lm_head weights (it inherits
`VocabParallelEmbedding`'s loader). The one-file patch in
[`vllm-patches/patch_nvfp4_lm_head.py`](./tree/main/vllm-patches) swaps the lm_head to
`ReplicatedLinear`, which routes through the standard quantized-linear loading path:

```bash
docker run -d --name vllm --gpus all -p 8000:8000 --ipc host \
  -v /opt/vllm-cache:/root/.cache/huggingface \
  -e CUBLASLT_WORKSPACE_SIZE=33554432 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
  vllm/vllm-openai:latest \
  --model demon-zombie/Qwen3.5-122B-A10B-NVFP4-Full-GB10 \
  --served-model-name Qwen3.5-122B-A10B \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

# apply the lm_head patch, then restart
docker cp patch_nvfp4_lm_head.py vllm:/tmp/
docker exec vllm python3 /tmp/patch_nvfp4_lm_head.py
docker restart vllm
```

Verified configuration: DGX Spark GB10, from-source vLLM 0.19.2 (SM121, TORCH_CUDA_ARCH_LIST
12.1) with the patch applied — all benchmark numbers above are from that build. Newer stock
images have not been re-verified with this checkpoint's NVFP4 lm_head; if loading fails
around `lm_head`/`ParallelLMHead`, apply the included patch.

## Architecture

Qwen3.5-122B-A10B is a hybrid-attention MoE model: 48 layers (36 GDN/Gated-DeltaNet linear
attention + 12 full attention, interval 4), 256 routed experts + 1 shared expert per layer
(8 active), 3,072 hidden, 248,320 vocab, 262K context, plus a ViT vision encoder.

## Credits

- Base model: [Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
- Quantized with [llm-compressor](https://github.com/vllm-project/llm-compressor)
- Sibling checkpoint: [FP8Dense](https://huggingface.co/demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10)
  (mixed precision, recommended for quality)