--- base_model: Qwen/Qwen3.6-35B-A3B base_model_relation: quantized tags: - nvfp4 - modelopt --- # Qwen3.6-35B-A3B-NVFP4-MTP-ModelOpt NVFP4 (nvfp4) quantization of `Qwen/Qwen3.6-35B-A3B` produced with NVIDIA TensorRT Model Optimizer via a multi-node ModelOpt quantizer. KV cache FP8. This is a quantized derivative and inherits the base model's license; see the base model card for usage. ## Quantization details (auto-generated) - **source model:** [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) - **qformat:** `nvfp4` **kv_cache:** `fp8` - **calibration:** `?` samples from `?` - **producer:** NVIDIA ModelOpt `?` - **generated:** ? _Before/after sample generation was skipped for this run (`SKIP_GENERATE=1`)._ ## Notes ### Quantized MTP/NEXTN draft head (research probe) Unlike a BF16-draft-head build, this checkpoint NVFP4-quantizes the MTP/NEXTN draft head (a full-attention + shared-expert-MoE block) and bakes FP8 KV scales onto its attention, so the SGLang p51 KV-scale loader exercises the MTP head. Because transformers drops the MTP layer at load and the multinode FSDP2 export carries no mtp.* at all, the head is produced by a post-export surgical requant that sources the raw BF16 mtp.* from the original checkpoint and NVFP4-packs them, mirroring each module's main-model analogue. All three gates have now been measured on GB10/sm121: the smoke-serve loads register-free with NEXTN active, GSM8K scores 96.5% (see the benchmarks table), and the draft head accepts 2.71 of a theoretical 3.00 tokens per step under load. What is still missing is the A/B against a BF16-MTP build, so the numbers below establish that the quantized draft works, not that it is free. ### MTP scales are heuristic, not calibrated The NVFP4 weights of the draft head are data-free and bit-identical to a calibrated run, but its W4A4 input_scale is a generous-amax heuristic and its FP8 KV k_scale/v_scale are borrowed from the last main-model full-attention layer. Correctness is unaffected (spec-decoding verifies every draft token); a poor draft only lowers acceptance/throughput. Measured: mean accept len 2.71 of a theoretical maximum of 3.00 (num_steps=2, num_draft_tokens=3), i.e. 90% of the ceiling, so whatever the heuristic scales cost is bounded by the remaining 10%. A BF16-MTP A/B would pin that number down exactly, but the headroom it could recover is small. ### Multimodal, vision tower kept BF16 Qwen3.6-35B-A3B carries a 27-layer vision tower. Calibration is text-only, so the vision stack is excluded from quantization and stays BF16 (avoiding the amax=0 degenerate-quant failure mode); this is why the model must be quantized via the multinode driver, whose quant policy applies the vision excludes. ### Hybrid-arch serving knobs Serving needs the mamba scheduler knobs to avoid a spec-v2-vs-radix-cache boot crash: mamba_scheduler_strategy extra_buffer plus SGLANG_ENABLE_SPEC_V2=1, along with the flashinfer attention backend and flashinfer_cutlass MoE/FP4-GEMM backends. NEXTN speculative decoding at num_steps=2 was the validated throughput sweet spot; higher step counts (3/4/5) decayed. ### Serving requires SGLang runtime patches (stock SGLang does not load this) Stock SGLang 0.5.15 crashes loading this checkpoint on GB10/sm121; it needs five launch-time source patches, shipped in the public dgxarley repo (https://github.com/vroomfondel/dgxarley) under roles/k8s_dgx/files/sglang_patches/ and applied by sglang_launch.sh: (1) p50 -- allow QUANTIZED attention for modelopt_fp4 (this is a uniform-W4A4 export that also quantizes the Gated-DeltaNet linear-attn, unlike NVIDIA's MoE-only NVFP4); (2) p51 -- load the main model's baked FP8 KV scales onto RadixAttention (remap ...self_attn.k_proj.k_scale -> ...attn.k_scale), else the full-attn layers default to scale 1.0; (3) p43 -- keep quant_config for the QUANTIZED MTP draft head, else SGLang assumes a BF16 MTP, builds it unquantized, and the fused-MoE loader narrows the unpacked intermediate dim over the NVFP4-packed expert weight ("start(0)+length(512) exceeds dimension size(256)"); (4) p44 -- apply the same KV-scale mapper in the MTP load path (qwen3_5_mtp.py has its own load_weights) so the draft attention loads its baked FP8 scales instead of the 1.0 sentinel; (5) the arch-independent sm121 CUTLASS-FP4 mma patch. p43/p44 are specific to a quantized MTP and are inert for a BF16-MTP checkpoint. Independently, config.json pins torch_dtype: bfloat16 (the Qwen3.6 source config omits it) so SGLang's --dtype auto resolves bf16; without it the unquantized in_proj_ba loads as fp16 and crashes at the CUDA-graph capture ("mat1 and mat2 ... BFloat16 != Half"). Validated end to end on GB10/sm121 (image 0.5.15.post1-sm121): loads register-free, NEXTN speculation active, coherent output, MTP draft attn loads its borrowed KV scales (k_scale 0.0398, byte-identical to the last full-attn layer, the requant borrow source). ## Benchmarks | Task | Metric | Value | Setup | Hardware | Date | Notes | | --- | --- | --- | --- | --- | --- | --- | | GSM8K | accuracy (strict == flexible) | 96.5% | chat + reasoning (qwen3 think), 4-shot, temperature 0.6, max_tokens 4096, N=200 (test split head) | 4x DGX Spark GB10 / sm121; SGLang 0.5.15.post1-sm121, NEXTN spec num_steps=2, flashinfer attention + flashinfer_cutlass MoE | 2026-07-25 | 193/200 correct, 0 request errors; strict == flexible on every item (no over-generation artifact). 3/200 came back empty on the generation-length cap, one of them verifiably needs ~8.8k reasoning tokens, so max_tokens 4096 is the binding limit rather than the model. Interim result: 95% CI [93.0%, 98.3%] at this N, and no BF16 / NVIDIA-reference A/B has been run yet, so this does not by itself establish that the quant is lossless. | | MTP/NEXTN speculative decoding | accept len (mean, max 3.00) / accept rate | 2.71 / 0.85 | read from the SGLang decode-batch logs of the GSM8K run above: 300 decode batches, num_steps=2, eagle_topk=1, num_draft_tokens=3, concurrency 16 | 4x DGX Spark GB10 / sm121; SGLang 0.5.15.post1-sm121, flashinfer attention + flashinfer_cutlass MoE | 2026-07-25 | 90% of the theoretical ceiling of 3.00, so roughly 2.7x fewer forward passes than non-speculative decode; 374 tok/s aggregate at concurrency 16, and 2.58 accept len single-stream. This is the quantized draft head with heuristic input_scale and borrowed FP8 KV scales. No BF16-MTP A/B was run, but the headroom a perfect draft could still recover is bounded by the missing 10%. |