JasonW2025/Qwen3.6-35B-A3B-HybridQuant-gptq-dv41-KVcal

Qwen3.6-35B-A3B HybridQuant (gptq-dv41) — KV-calibrated

Identical weights to the production quant-qwen36-gptq-dv41 HybridQuant artifact (NVFP4 W4A16 MLP + FP8 attention, GPTQ, diverse-v4.1 calibration). The base artifact declares kv_cache_quant_algo: FP8 but ships no k_scale/v_scale, so vLLM serves its fp8 KV cache at scale 1.0. This repo adds the missing scales, measured, not cast:

• amax captured with vLLM itself (running max over 64 diverse-4.2 calibration prompts, memory-profiler dummy pass excluded), on the 10 full-attention layers (the other 30 layers are linear-attention and hold no KV cache).
• baked as model-kvscales.safetensors: scale = amax x 1.25 / 448, k_scale 0.0378–0.0558, v_scale 0.0156–0.0513.
• verified at serve time as a PAIR: the unmodified base emits vLLM's Using KV cache scaling factor 1.0 for fp8_e4m3 warning; this artifact does not, with fp8 KV confirmed enabled in both. vLLM v0.25.1, GB10/sm_121a.

Honest scope: measured amax is far below the e4m3 max (448), so scale 1.0 was not clipping this model. The scales recover the low tail of the distribution from subnormal flush. No accuracy delta has been benchmarked; do not cite one from this card.

Serve with --kv-cache-dtype fp8 (or leave auto; the checkpoint declares FP8 KV).