--- license: mit base_model: - zai-org/GLM-5.2 - lukealonso/GLM-5.2-NVFP4 base_model_relation: quantized tags: - glm - moe - nvfp4 - nf3 - mxfp8 - blackwell - quantization --- # GLM-5.2-MXFP8-NVFP4-NF3-Hybrid > Ready-to-run v20 Compose profiles are included for four 96 GiB SM120 GPUs, > with validated context lengths from 400K through 775K using FP8 or calibrated > NVFP4 MLA KV. Use the pinned v20 image and included launch files because the > checkpoint's custom NF3 expert tensors require its native runtime support. The full 753B-parameter GLM-5.2, with all 256 routed experts per MoE layer, compressed into 340.9 GiB (366,021,469,560 bytes) without expert pruning. Sixty-four damage-selected expert slots per layer retain Luke Alonso's NVFP4 tensors; the other 192 use the healed NF3 format and its SM120 Grid188 kernel. Attention, dense, and shared-expert tensors remain BF16 in the checkpoint. Eligible linears are converted once to MXFP8 at load according to the selected serving profile. The weight tensors are v3.6 at immutable weight revision `68babde27a97a4c980c2494e830dd424975cd5a3`. They contain 184 safetensor shards of at most 2 GB. The reviewed release bundle adds only documentation, Compose files, the launcher shim, and the calibrated KV-scale sidecar; it does not rewrite those weight shards. ## Four v20 serving profiles | Profile | MTP | Public max-model-len | Shared-ruler KLD ↓ | Launch | |---|---:|---:|---:|---| | FP8 MLA KV | 0 | 400,000 | **0.12715 ± 0.00183** | `docker-compose.fp8.yml` | | FP8 MLA KV, shared-expert MXFP8 | 3 | **450,000** | **0.12821 ± 0.00306** | base FP8 + `docker-compose.fp8-450k-mtp3.yml` | | Calibrated NVFP4 KV, BF16 RoPE | 0 | 700,000 | **0.13234 ± 0.00226** | `docker-compose.nvfp4.yml` | | Calibrated NVFP4 KV, FP8 RoPE | 3 | **775,000** | **0.13386 ± 0.00544** | base NVFP4 + `docker-compose.nvfp4-775k-mtp3.yml` | All profiles use TP4/DCP4, A16 expert activations, lossless BF16 DCP transport, and InstantTensor `BUFFERED`. The FP8 profile uses the accepted v19 online-MXFP8 membership. The two NVFP4 capacity profiles additionally convert shared experts online; their KLD is therefore certified separately rather than borrowed from the FP8 profile. The final 775k profile allocated exactly 775,000 KV tokens and completed a token-counted 764,999-token prompt plus 16 generated tokens. It then completed a fresh request returning `POST CAP READY`, with zero engine restarts. It is a tight four-GPU pin and uses a small launcher shim plus two source patches applied only inside the ephemeral container. Read the KV-token count at startup and lower `MAX_MODEL_LEN` if your machine has less runtime headroom. An exact-stack ceiling canary also completed a 774,999-token prompt against a 785,000-token pool and passed post-cap reuse, but its measured free-memory floor was only 94 MiB per GPU. The portable release profile therefore remains 775,000 rather than advertising the brittle ceiling. The FP8-KV/MTP3 overlay allocated 450,255 KV tokens for a 450,000-token advertised limit. It completed a 449,935-token prompt with coherent generation and then a fresh 8k request, with zero restarts/OOMs and a 146 MiB measured free-memory floor. The explicit KV-byte pin avoids an automatically sized 467,627-token pool that booted but left only 6 MiB after lazy allocation. ## Accuracy GPQA-Diamond used temperature 1.0, top-p 0.95, and maximum reasoning effort. The v3.6 row is this exact checkpoint. | Precision | GPQA Diamond | |---|---:| | Official FP8 reference | 89.52 | | Full NVFP4 reference | 89.39 | | **This model, v3.6** | **88.89 (176/198)** | | Previous hybrid revision | 88.38 (175/198) | | REAP-594B prune, for contrast | 86.87 | KLD is a distribution-drift instrument, not an intelligence score. The table above remains the direct quality check. ## KLD method The reported KLD uses the current shared GLM-5.2 ruler: - source: `zai-org/GLM-5.2` BF16 reference logits from 2026-07-08; - one fixed 2,048-token Wikitext window, 2,047 scored positions; - full 154,880-token vocabulary; - direction `KL(BF16 reference || candidate)`; - TP4/DCP4, MTP0, A16, five fresh engine starts; - exact 78-character GLM IndexCache pattern. The final v20 review reruns each precision posture five times with a fresh engine and an isolated empty cache. KLD is target-only MTP0 even when the corresponding serving profile enables MTP3. The scale file is an audited, per-layer outer-scale artifact; it changes no checkpoint weights and adds no KV bytes. | Profile | Runs | KLD mean ± sample SD | Min | Max | |---|---:|---:|---:|---:| | FP8 MLA KV | 5 | 0.12715 ± 0.00183 | 0.12564 | 0.12958 | | **FP8 MLA KV, shared-expert MXFP8, direct MXFP8 BMM** | **5** | **0.12821 ± 0.00306** | **0.12318** | **0.13055** | | Calibrated NVFP4 KV, BF16 RoPE | 5 | 0.13234 ± 0.00226 | 0.12888 | 0.13484 | | Calibrated NVFP4 KV, FP8 RoPE, direct BMM off | 5 | 0.13630 ± 0.00641 | 0.13052 | 0.14671 | | **Calibrated NVFP4 KV, FP8 RoPE, direct MXFP8 BMM** | **5** | **0.13386 ± 0.00544** | **0.12769** | **0.14033** | The machine-readable aggregate is included as `KLD_MATRIX.json`. The published scale sidecar uses host-neutral provenance labels; its `max_abs` and `scales` arrays are byte-for-byte identical to the artifact used for every KLD receipt. `KLD_MATRIX.json` records both the receipt-artifact hash and a canonical numeric-payload hash. Do not compare these values with the old card's FP8-as-reference, top-8192 drift table. That was a different reference, support, and direction. ## Serve from the immutable v20 image Exact image: ```text voipmonitor/vllm:gilded-gnosis-v20-vllm0c79e41-sie603f74-fi801d57a-cu132-20260726 sha256:10261c7d65101c8aba2ce1fb59eabe73aff9d35eca5043b330cc0ce76d3c98d0 ``` A matched four-GPU A/B retained this July-26 image over the July-25 final image: the selected image measured 2,291/2,247 prompt tok/s at 64k/128k versus 2,264/2,217, and it contains the scale-file and launcher interfaces used by the reviewed profiles. A later policy A/B found that the July-26 helper's opt-in owner-top-k merge was the remaining prefill regression on this topology. The final 775k overlay disables that experimental route and retains the established replicated global-top-k oracle. Download without Xet: ```bash export HF_HUB_DISABLE_XET=1 hf download madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid \ --local-dir ./GLM-5.2-MXFP8-NVFP4-NF3-Hybrid cd ./GLM-5.2-MXFP8-NVFP4-NF3-Hybrid export MODEL_DIR=$PWD ``` Do not pin the old weight revision for this serving recipe: that historical commit predates the calibrated scale sidecar. After publication, pin the final release-bundle commit if an immutable metadata checkout is required. Choose one of the standalone Compose files included in this repository: ```bash # Roughly 400k context, lowest measured KLD. docker compose -f docker-compose.fp8.yml up -d # 450k/MTP3, FP8 KV with shared experts converted online to MXFP8. docker compose \ -f docker-compose.fp8.yml \ -f docker-compose.fp8-450k-mtp3.yml up -d # 700k quality-first long-context pin, calibrated NVFP4 KV and BF16 RoPE. docker compose -f docker-compose.nvfp4.yml up -d # 775k/MTP3 capacity profile, calibrated NVFP4 KV and FP8 RoPE. docker compose \ -f docker-compose.nvfp4.yml \ -f docker-compose.nvfp4-775k-mtp3.yml up -d ``` Wait for health before sending traffic: ```bash curl -fsS http://127.0.0.1:5001/health curl -fsS http://127.0.0.1:5001/v1/models ``` The two base files default to MTP0 so their advertised capacity is reproducible. The 450k and 775k overlays enable MTP3. Keep MTP disabled for target-only decode measurements or KLD work: ```bash MTP=0 docker compose -f docker-compose.fp8.yml up -d --force-recreate ``` The v20 helper owns DCP query split, CKV gather, bounded prefetch, InstantTensor, NCCL, graph sizing, sparse attention, and parsers for the two base profiles. Leave those controls on auto there. Both MTP3 overlays pin the measured validation policy to avoid a fresh-cache calibration cleanup defect and to bound their CKV workspace. The 775k NVFP4 overlay additionally pins both `DCP_TOPK_OWNER_MERGE=0` and `VLLM_DCP_TOPK_OWNER_MERGE=0`: the July-26 auto-policy otherwise enables an explicitly opt-in owner merge that regressed 64k/128k prompt throughput by about 19% on the measured topology. The overlays also apply the included MTP online-quant inheritance and sparse-indexer carry-fold patches before launch. No checkpoint tensor is changed. `F8_DMA=0` is intentional: it keeps DCP transport lossless. ### Why the Compose uses `ONLINE_QUANT=custom` The image's convenient `nf3-mxfp8` shorthand is a faster, more aggressive membership that also converts shared experts and fused q-a/kv-a projections. The FP8 profile uses this measured quality posture: ```json {"linear":{"weight":"mxfp8"},"ignore":["re:.*\\.fused_qkv_a_proj$","re:.*\\.q_a_proj$","re:.*kv_a_proj_with_mqa","re:.*\\.mlp\\.gate$","model.layers.78.eh_proj","lm_head"]} ``` The NVFP4 profiles use the same ignore list plus `"shared_experts":{"weight":"mxfp8"}` to recover the capacity required by their public pins. Their five-run KLD rows measure that exact membership. The NVFP4 Compose also passes `VLLM_NVFP4_MLA_SCALES_FILE` explicitly. The final v20 image contains scale support, but checkpoint-relative auto-discovery is not part of this release. The generic vLLM environment scanner may still label this extension variable unknown; the stock v20 MLA writer reads it directly. Runtime verification observed non-identity `latent_scale_identity=0` kernel signatures, proving that the calibrated values—not the identity fallback—reached the attention path. ## v20 performance Final review measurements on four RTX PRO 6000 Blackwell Max-Q GPUs: | KV profile | MTP | Max len | C1 decode, ctx 0 | C1 decode, 64k | C1 decode, 128k | Prefill 64k | Prefill 128k | Reported KV pool | |---|---:|---:|---:|---:|---:|---:|---:|---:| | FP8 | 0 | 400k | 48.0 | 46.9 | 47.3 | 2,069 | 2,073 | 435,316 | | FP8 | 3 | 180k | 89.7 | 86.6 | 88.7 | — | — | 218,096 | | FP8, shared-expert MXFP8 | 3 | **450k** | — | — | — | **2,172** | **2,060** | **450,255** | | NVFP4, BF16 RoPE | 0 | 700k | 48.8 | 47.8 | 48.3 | 1,994 | 2,027 | 733,272 | | NVFP4, BF16 RoPE | 3 | 360k | 100.4 | 100.8 | 97.9 | — | — | 441,364 | | NVFP4, FP8 RoPE | 0 | 775k | — | — | — | — | — | 842,057 | | **NVFP4, FP8 RoPE** | **3** | **775k** | **106.1** | **106.1** | **102.2** | **2,548** | **2,470** | **775,000** | The final 775k/MTP3 run measured a five-run coding peak of 121.6 tok/s mean / 124.3 tok/s maximum. The final prefill row is the second complete exact-token 64k/128k pass after warmup. The bounded carry-fold measured within about one percent of the unpatched owner-merge-off profile, while avoiding the stock fold's 112 MiB near-capacity allocation and engine OOM. The 450k FP8 row combines the capacity certification with a separate warmed, exact-token 64k/128k prefill measurement. The older 180k row retains the matched decode measurements. No unmeasured decode number is borrowed for the new shared-expert precision posture. `VLLM_B12X_ABSORB_BMM=1` is active in the certified 775k posture. Load ordering matters: after the 184-shard target load, v20 logged that it was serving MLA absorbed projections directly from the B12X MXFP8 pack. The same-checkpoint MTP inheritance patch gives layer 78 the identical online-MXFP8 membership, so the four-shard draft also uses the direct pack and no post-draft fallback is logged. The second patch selects SparkInfer's existing streaming carry fold for logical sparse-indexer output. Exact CUDA tests produced identical selected token sets at 131k and 524k. This removes page-table-sized transient candidate tensors that otherwise caused late OOMs even after a successful 775k boot. These are explicit runtime patches over the immutable image; this profile is not described as unmodified stock v20. ### High-concurrency status The table above intentionally reports C1 decode only. The July-26 v20 NF3 hybrid kernel has a known batch-dispatch crossover under investigation: Grid188 covers the smallest decode bucket and the optimized direct-top-k launch is currently bounded to eight routed rows; larger MTP verifier batches fall back to the packed route used for prefill. This does not affect correctness, KLD, advertised KV capacity, C1 decode, or the prefill results reported here, but it limits C2-C8 scaling. A lossless native dispatch fix is being prepared for an upstream runtime release; no unvalidated concurrency number or private kernel patch is included in this checkpoint release. ## Credits - Base model: [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) - NVFP4 tier: [lukealonso/GLM-5.2-NVFP4](https://huggingface.co/lukealonso/GLM-5.2-NVFP4) - v20 runtime and documentation: [local-inference-lab/rtx6kpro](https://github.com/local-inference-lab/rtx6kpro/blob/master/models/glm5.2_v20.md) - Shared KLD ruler: [GLM-5.2 KLD Evaluation](https://github.com/local-inference-lab/rtx6kpro/blob/master/benchmarks/glm52-kld-evaluation.md) Built and evaluated by madeby561.