Add model card (all-FP4 max-speed variant; quality note; serve + lm_head patch)

README.md

@@ -0,0 +1,130 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.5-122B-A10B
+pipeline_tag: image-text-to-text
+tags:
+- nvfp4
+- compressed-tensors
+- quantized
+- vllm
+- dgx-spark
+- gb10
+- moe
+---
+
+# Qwen3.5-122B-A10B-NVFP4-Full-GB10
+
+All-NVFP4 (W4A4) quantization of [Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
+— the maximum-decode-speed variant for NVIDIA DGX Spark (GB10, SM121). Every Linear layer is
+4-bit, **including the lm_head**. Vision encoder preserved (BF16) and verified working.
+
+This is the most aggressively quantized checkpoint of this model: at GB10's memory bandwidth
+it decodes ~33–35 tok/s single-stream, ~45% faster than the mixed-precision
+[FP8Dense](https://huggingface.co/demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10)
+sibling (~23–24 tok/s).
+
+## Quality note
+
+The all-FP4 profile carries a **subtle quality degradation** relative to the FP8Dense
+sibling: the dense path (attention + GDN) runs 4-bit *activations*, and the lm_head's 4-bit
+weights flatten the output distribution slightly. It does not show up on simple extraction
+benchmarks (both variants ace detail tests), but in extended real-world use comprehension
+and nuance are noticeably better on FP8Dense. Pick by priority:
+
+- **Maximum tok/s** → this checkpoint
+- **Best quality at ~2/3 the speed** → [FP8Dense](https://huggingface.co/demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10)
+
+## Layout
+
+| | This model | [FP8Dense](https://huggingface.co/demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10) | [scottgl](https://huggingface.co/scottgl/Qwen3.5-122B-A10B-NVFP4-GB10) |
+|---|---|---|---|
+| MoE experts (256/layer) | NVFP4 | NVFP4 | NVFP4 |
+| Standard attention | NVFP4 | FP8 W8A8 | BF16 |
+| GDN projections | NVFP4 | FP8 W8A8 | BF16 stored, FP4/FP8 at SGLang runtime |
+| lm_head | **NVFP4** | BF16 | BF16 stored, FP8 at SGLang runtime |
+| Vision | BF16 (preserved) | BF16 (preserved) | — |
+| Serves on | vLLM (lm_head patch, below) | stock vLLM | custom SGLang fork |
+| Checkpoint | 66 GB | 74 GB | — |
+| Decode, GB10 single-stream | **33–35 tok/s** | 23–24 tok/s | ~46 (SGLang + NEXTN spec-dec) |
+
+Excluded from quantization (BF16): router gates (`mlp.gate`, `shared_expert_gate`),
+embeddings, vision encoder + merger, norms. No MTP weights in this checkpoint (the config's
+MTP declaration has no corresponding tensors — speculative decoding is not available).
+
+## Quantization details
+
+- **Method:** [llm-compressor](https://github.com/vllm-project/llm-compressor) `oneshot()`,
+  NVFP4 scheme (W4A4, group 16, FP8-E4M3 scales)
+- **Calibration:** 512 samples, `HuggingFaceH4/ultrachat_200k`, seq_len 2048
+- **Quantized from the full multimodal model** (`AutoModelForImageTextToText`) so the vision
+  tower survives — earlier text-only exports of this model lost it
+
+## Benchmarks (DGX Spark GB10, vLLM 0.19.2 from-source SM121 build)
+
+Sequential decode, gen=500:
+
+| Context | TTFT | Decode tok/s |
+|---|---|---|
+| warmup | 0.84s | 35.2 |
+| 9K | 3.22s | 34.6 |
+| 18K | 3.72s | 33.8 |
+| 27K | 3.69s | 32.9 |
+
+Concurrent @ 32K context, gen=500:
+
+| Concurrency | Aggregate tok/s | Per-request tok/s |
+|---|---|---|
+| 8× | 78.2 | 12.0 |
+| 16× | 110.6 | 8.0 |
+| 32× | 138.2 | 5.2 |
+| 64× | 166.5 | 3.3 |
+| 128× | 182.8 | 2.8 |
+
+KV cache: 5.94× the full 262K context at `--gpu-memory-utilization 0.90` (bf16 KV; page
+size 2,096 tokens, aligned to the GDN/Mamba state size).
+
+## Serving (vLLM)
+
+vLLM's stock `ParallelLMHead` cannot load NVFP4-packed lm_head weights (it inherits
+`VocabParallelEmbedding`'s loader). The one-file patch in
+[`vllm-patches/patch_nvfp4_lm_head.py`](./tree/main/vllm-patches) swaps the lm_head to
+`ReplicatedLinear`, which routes through the standard quantized-linear loading path:
+
+```bash
+docker run -d --name vllm --gpus all -p 8000:8000 --ipc host \
+  -v /opt/vllm-cache:/root/.cache/huggingface \
+  -e CUBLASLT_WORKSPACE_SIZE=33554432 \
+  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
+  vllm/vllm-openai:latest \
+  --model demon-zombie/Qwen3.5-122B-A10B-NVFP4-Full-GB10 \
+  --served-model-name Qwen3.5-122B-A10B \
+  --gpu-memory-utilization 0.90 \
+  --enable-prefix-caching \
+  --enable-chunked-prefill \
+  --enable-auto-tool-choice \
+  --tool-call-parser qwen3_coder \
+  --reasoning-parser qwen3
+
+# apply the lm_head patch, then restart
+docker cp patch_nvfp4_lm_head.py vllm:/tmp/
+docker exec vllm python3 /tmp/patch_nvfp4_lm_head.py
+docker restart vllm
+```
+
+Verified configuration: DGX Spark GB10, from-source vLLM 0.19.2 (SM121, TORCH_CUDA_ARCH_LIST
+12.1) with the patch applied — all benchmark numbers above are from that build. Newer stock
+images have not been re-verified with this checkpoint's NVFP4 lm_head; if loading fails
+around `lm_head`/`ParallelLMHead`, apply the included patch.
+
+## Architecture
+
+Qwen3.5-122B-A10B is a hybrid-attention MoE model: 48 layers (36 GDN/Gated-DeltaNet linear
+attention + 12 full attention, interval 4), 256 routed experts + 1 shared expert per layer
+(8 active), 3,072 hidden, 248,320 vocab, 262K context, plus a ViT vision encoder.
+
+## Credits
+
+- Base model: [Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
+- Quantized with [llm-compressor](https://github.com/vllm-project/llm-compressor)
+- Sibling checkpoint: [FP8Dense](https://huggingface.co/demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10)
+  (mixed precision, recommended for quality)