Add model card: GPQA-Diamond 88.38 (parity with full precision), NF3-hybrid + mxfp8 tier

README.md

@@ -0,0 +1,70 @@
+---
+license: mit
+base_model:
+- zai-org/GLM-5.2
+- lukealonso/GLM-5.2-NVFP4
+base_model_relation: quantized
+tags:
+- glm
+- moe
+- nvfp4
+- nf3
+- mxfp8
+- quantization
+---
+
+# GLM-5.2-MXFP8-NVFP4-NF3-Hybrid
+
+> ⚠️ **Usable — read the hazards.** Loads ONLY through the custom serving image below
+> (in-house NF3 3-bit kernel — no official vLLM/inference stack runs it). Serves on
+> 4× 96GB sm120 GPUs at **~100k context** (118k max) — NOT the official model's 1M.
+> Not officially supported.
+
+**The full GLM-5.2 — 753B, all 256 experts per layer — on 4× 96GB GPUs. No pruning.**
+The high-traffic experts ride NVFP4 byte-exact from
+[lukealonso/GLM-5.2-NVFP4](https://huggingface.co/lukealonso/GLM-5.2-NVFP4) (credit to
+Luke for the best-in-class low-KLD 4-bit tier); the long tail is compressed with
+**NF3 — our in-house 3-bit expert format with a matching custom sm120 kernel**. The
+non-expert layers (attention / dense / shared) are **MXFP8** (OCP microscaling FP8,
+built from the BF16 original). ~327 GB total — the only GLM-5.2 that fits 4 cards
+with usable context.
+
+## Accuracy
+
+Same protocol as the reference cards (temperature 1.0, top_p 0.95, max reasoning
+effort). GPQA-Diamond is measured on **this exact checkpoint**; the remaining axes
+are not yet run.
+
+| Precision | GPQA Diamond | SciCode | IFBench | AA-LCR | τ²-Bench Telecom |
+|---|---|---|---|---|---|
+| baseline (FP8) | 89.52 | 49.85 | 74.95 | 69.38 | 97.9 |
+| NVFP4 (full) | 89.39 | 49.04 | 75.81 | 70.13 | 98.25 |
+| **this model** (326 GB, 4× 96GB) | **88.38** | *pending* | *pending* | *pending* | *pending* |
+| *REAP-594B (a prune, for contrast)* | *86.87* | *47.77* | — | — | — |
+
+**Read:** on GPQA-Diamond the hybrid scores **88.38 (175/198)** — within ~1 point
+(2 questions) of full-precision FP8/NVFP4, i.e. statistically tied at this sample
+size — while **beating the REAP-594B prune by ~1.5 points**. That's the whole point
+of the build: *keeping* all 256 experts at mixed precision holds parity where
+*pruning* experts measurably loses ground. Baselines: nvidia/GLM-5.2-NVFP4 card.
+
+## Output drift (KLD)
+
+A *sameness* measure vs the official FP8 (how far outputs move), not a quality
+ranking. Our re-quantization of the NF3 tail from the **BF16 original** (not an
+already-quantized checkpoint) put this at **KL ≈ 0.172** (JS 0.035, 1.05M prefill
+positions, top-8192 support) — the drift lives in confidence on near-ties; token
+argmax matches FP8 ~99.98% of the time. The MXFP8 non-expert tier is quality-
+equivalent to the prior FP8-tier build (identical experts; only the ~8-bit
+non-expert layers changed, to finer per-32 microscaling).
+
+## Serving
+
+Ready-to-run image with the NF3 kernel + hybrid loader baked in:
+[`madeby561/vllm-glm52-nvfp4-nf3-hybrid`](https://hub.docker.com/r/madeby561/vllm-glm52-nvfp4-nf3-hybrid).
+The `docker-compose.yml` in this repo is the exact serving configuration —
+`hf download` the weights, `MODEL_DIR=<dir> docker compose up`. Needs 4× 96GB sm120
+GPUs + ~64GB RAM. 118k context, MTP-5, DCP4.
+
+*By madeby561 — built to answer one question: can a 4-card box run the full model
+with no lobotomy? On GPQA-Diamond, the answer is yes.*