JasonW2025/Hy3-295B-A21B-HybridQuant-v1-vllm

Hy3-295B-A21B — HybridQuant v1 (NVFP4 W4A16 + FP8 attention), 4.86 bpp

A 166.77 GiB mixed-precision quantization of Hy3-295B-A21B (295 B total, ~21 B active, 192 experts, top-8 routing), exported with NVIDIA ModelOpt on the hy3-hybridquant branch and served on vLLM. Built and validated on a DGX Spark (GB10, sm_121a).

weights on disk	179,066,406,984 B = 166.77 GiB (17 shards)
average	4.86 bits/parameter
recipe	huggingface/hy_v3/ptq/w4a16_nvfp4_sharedbf16
quant_algo	MIXED_PRECISION, kv_cache_quant_algo: FP8
format histogram	FP8 × 320, W4A16_NVFP4 × 83
excluded	model.layers.80* (kept at higher precision)
producer	ModelOpt 20260705.dev17+gc5446d378 (branch hy3-hybridquant, clean tree)

Full provenance — source snapshot, recipe/config/calibration hashes, container digest and driver version — is shipped in provenance.json; the per-tensor dump is in .quant_summary.txt and the router statistics in router_stats.json.

Where the bytes are

Measured from the shard headers at export time:

component	size	tensors
always-active (attention, norms, shared)	8.84 GiB	1,702
routed experts	156.73 GiB	137,088
embeddings + lm_head	1.18 GiB	4
MTP	0.00 GiB	0

Active bytes per token ≈ 15.63 GiB (always 8.84 + experts 6.53 + lm_head 0.26). At an effective 210 GB/s that puts the bandwidth-bound decode ceiling at ≈ 13 tok/s with no speculative decoding — this is a capacity build for a single Spark, not a throughput build.

Calibration

PTQ, 256 sequences × 512 tokens from opencode-v1.jsonl (calib_sha16: c94dc70743c5cee7), run under ghcr.io/spark-arena/dgx-vllm-eugr-nightly with NVIDIA driver 580.159.03, max_memory = 0:12GiB, cpu:6GiB.

Serving (vLLM)

vllm serve JasonW2025/Hy3-295B-A21B-HybridQuant-v1-vllm \
  --trust-remote-code \
  --kv-cache-dtype fp8 --calculate-kv-scales \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --no-enable-prefix-caching

Caveats — please read

• KV scales are declared but not baked. hf_quant_config.json sets kv_cache_quant_algo: FP8, but no k_scale/v_scale tensors ship with this export. Serving therefore needs --calculate-kv-scales, which derives them from the first batch and is non-deterministic across sessions. Do not use this checkpoint for determinism-sensitive evaluation without baking scales first.
• Decode is bandwidth-bound at ~13 tok/s on a single GB10 by the active-bytes calculation above. The export's own verifier flags this against a 60 tok/s target.
• model.layers.80* is excluded from quantization; expect a precision discontinuity there if you re-derive per-layer statistics.
• Export self-check: VERIFY_EXPORT: PASS.

License

Inherits the base model's licence (tencent/Hy3). Confirm the upstream terms before redistribution.