JasonW2025/Qwopus3.6-35B-A3B-v1-PrismaQuant-5.06bit-vllm

Qwopus3.6-35B-A3B-v1 — PrismaQuant 5.06-bit (compressed-tensors, vLLM)

Mixed-precision quantization of Jackrong/Qwopus3.6-35B-A3B-v1 (Qwen3.6 MoE, 35B total / ~3B active, hybrid linear+full attention, in-checkpoint MTP head) produced with PrismaQuant (spark-improvements fork, streaming production-render pipeline).

• Selected point: 5.056 bpp, held-out last-token KL vs BF16 = 0.0428 (measured-KL kneedle over an 8-candidate Pareto band 4.70-5.06; runner-up 4.703 bpp had KL 0.0584)
• Mix (Linears): 253 BF16 / 106 NVFP4 / 42 MXFP8_E4M3; packed MoE experts NVFP4; MTP head preserved in BF16 (all 19 mtp.* tensors); visual encoder uniform NVFP4
• Calibration: diverse-v1 (256 rows x ~4096 tok; 40% prose / 20% code / 20% math / 20% multilingual), GPTQ + static act order + joint NVFP4 fused-sibling global scales
• Size: 25.9 GB (vs 72 GB BF16 source)

Serving (vLLM, tested on DGX Spark GB10 / Blackwell sm_121)

vllm serve JasonW2025/Qwopus3.6-35B-A3B-v1-PrismaQuant-5.06bit-vllm \
  --quantization compressed-tensors --kv-cache-dtype fp8 \
  --trust-remote-code --attention-backend flashinfer \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'