--- license: mit base_model: zai-org/GLM-5.3-Flash-BF16 base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers tags: - glm - glm-5 - glm5_next - tr3 - trellis - mcg - quantized - 8-bit - moe - reasoning - text-generation - fidelity - kl-divergence - exllamav3 --- # GLM-5.3-Flash-TR3-8bpw (K8) **The first 8-bit (K8) TR3/MCG trellis quantization of [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash)** — 321B-total / A18B MoE, `glm5_next` hybrid architecture. Routed experts and the MTP layer quantized at K8 (128-word trellis, MCG `0xCBAC1FED`); everything else (KDA linear-attention layers, DSA indexer, hyper-connections, routers, norms, embeddings, lm_head) **bit-exact native BF16**. 331.4 GB — within 1% of the official FP8 release's footprint, at 1.66× lower divergence. > ⚠️ **PRELIMINARY**: the number below is cold run 1 of 2 over the full sealed > panel. The second cold run (determinism check) is in flight; the sealed > two-run receipt replaces this section when it lands. ## Quality — full sealed panel, streaming lane | Model | Mean KLD (nats) | Top-1 | Size | Scope | |---|---:|---:|---:|---| | **This K8 (preliminary)** | **0.012384** | **96.78 %** | 331 GB | full panel, run 1/2 | | K6 (same lane) | 0.013715 | 96.56 % | 254 GB | full panel, run 1 | | [K6 (sealed, 5 runs)](https://huggingface.co/malaiwah/GLM-5.3-Flash-TR3-6bpw) | 0.013723 | — | 254 GB | full panel × 5 bitwise-identical | | Official FP8 | 0.020615 | 95.63 % | 328 GB | cross-stack, [receipt](https://huggingface.co/datasets/malaiwah/GLM-5.3-Flash-fidelity-suite-v1/blob/main/reports/fp8-on-brandon-panel.json) | | brandonmusic 4bpw | 0.024555 | — | 176 GB | full panel, his stack | | 0xSero Dione Q4 | 0.027263 | — | 188 GB | full panel, [our measurement](https://huggingface.co/datasets/malaiwah/GLM-5.3-Flash-fidelity-suite-v1/blob/main/reports/dione-q4-packed-kld.json) | | NVFP4 (his stack) | 0.060535 | — | ~180 GB | 1 window × 5 runs | Measured on brandonmusic's sealed 25-window / 51,175-position panel against his fp32 BF16 teacher logits, direction KLD(teacher ‖ student), fp64 accumulation. **Headline: at the same footprint as the official FP8 (331 vs 328 GB), K8 is 1.66× closer to the BF16 teacher.** Weight-space corroboration: with the intermediate-channel permutation undone, K8's shipped store is **13.2× tighter in NMSE** than K6's (3.505e-5 vs 4.624e-4, better in 30 of 30 sampled matrices). **Lane note.** This is the *streaming* lane (single GPU, ~$8/panel), not the 8×H200 sealed lane K6 was qualified on. The lanes were bridged on this exact panel: K6 measures 0.013715 streaming vs 0.013723 sealed — a **−8.5e-6** difference (0.06 %), fully explained by expert-combine order in bf16. Rows above from different lanes are labelled; do not mix them without that offset in mind. **Methodology note worth stealing.** A single-window comparison of these two rates is *statistically meaningless*: per-window KLD scatter has sd 1.73e-3 against a K6-vs-K8 effect of 1.22e-3. On one unlucky window (`window-0000`) K8 appeared *worse* than K6; over the full panel it wins decisively. Never quote a single-window KLD as a rate comparison — the full write-up is in [`k6/K8-ANOMALY.md`](https://github.com/malaiwah/glm53-flash-fidelity-suite/blob/main/k6/K8-ANOMALY.md). ## What this is (and is not) - **Codec:** EXL3-format TR3/MCG trellis (turboderp's [exllamav3](https://github.com/turboderp-org/exllamav3) kernels @ `c5d9c657`), through [brandonmusic's GLM-5.3 pipeline](https://github.com/brandonmmusic-max/glm-5.3-flash-exl3-4bpw) with a disclosed patch series. His published core admits K3/K4/K5, so **K8 is a declared rate extension** — our encoder was verified **byte-identical** to his sealed core across 120 encodes / 624 MiB / 0 differing bytes ([evidence](https://github.com/malaiwah/glm53-flash-fidelity-suite/blob/main/k6/fallback/closure-comparison.json), [issue #1](https://github.com/brandonmmusic-max/glm-5.3-flash-exl3-4bpw/issues/1)). - **NOT loadable by stock exllamav3/TabbyAPI** — no `glm5_next` architecture upstream. Serve via brandonmusic's custom Transformers TP runtime, or adapt the drivers in [k6/](https://github.com/malaiwah/glm53-flash-fidelity-suite/tree/main/k6). Note his runtime currently admits bits (4, 6); K8 serving needs the same 128-word admission our patch series adds. - **Topology-neutral checkpoint:** canonical unsharded tensors, `qualified_tp_sizes: []` — TP layout is a load-time decision. 331 GB fits TP4 × 96 GB (4× RTX 6000 Pro) with room for a 512K-context fp8 KV cache (~3.3 GiB, since only 11 layers are MLA). - **Parts-bin sibling:** encoded with the **same transform seed and calibration as [K6](https://huggingface.co/malaiwah/GLM-5.3-Flash-TR3-6bpw)**, so the two per-choice payload stores are mix-and-matchable — a K6K8 multi-precision build (e.g. K8 on `down_proj`, K6 on gate/up) is **offline CPU assembly, no GPU re-encode**. The parts-bin dataset follows. ## Provenance & disclosed deviations Pins: BF16 source `zai-org/GLM-5.3-Flash-BF16` (weights == `a6c167b6`); calibration = brandonmusic's published EP4 captures (sealed inventory `f56e9d62…` adopted verbatim). Deviations, all receipted: encoded on 4×H200 SM90 (his campaign attests 4×B200 SM100; fat `9.0;10.0` extension build), K4-KL gate satisfied via a disclosed bridge document carrying his real published K4 receipt hashes, measurement on the streaming lane at EP8 emulation with fp32 combine order. Materialization receipt: bits 8, `complete`, `main_and_mtp_complete`, `nonrouted_native_exact`, 331,449,761,784 logical bytes, 37,152 routed choices, 1,618 native tensors. ## Lineage on the Hub Z.ai published two sibling roots for this model and neither declares the other: [`zai-org/GLM-5.3-Flash`](https://huggingface.co/zai-org/GLM-5.3-Flash) (the **FP8** release, where most traffic lands) and [`zai-org/GLM-5.3-Flash-BF16`](https://huggingface.co/zai-org/GLM-5.3-Flash-BF16) (the **BF16** weights). This quant declares BF16 as its `base_model` because that is what it was actually quantized from — the FP8 release is a *sibling* quantization of the same model, not our source, and it is the baseline we measure against rather than build on. Quants that list FP8 as their base were genuinely made from the FP8 weights; the trees differ for real reasons. Related work on the same model, all measured on one panel in the [quant-fidelity registry](https://huggingface.co/datasets/malaiwah/quant-fidelity-registry): [brandonmusic 4bpw](https://huggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw), [0xSero Dione Q4](https://huggingface.co/0xSero/GLM-5.3-Flash-EXL3-Q4), [orcarouter MLX](https://huggingface.co/orcarouter/GLM-5.3-Flash-MLX). Collection: [GLM-5.3-Flash — measured quants & fidelity](https://huggingface.co/collections/malaiwah/glm-53-flash-measured-quants-and-fidelity-6a91f253e7107818359f37c8). ## Credits Base model by [Z.ai](https://huggingface.co/zai-org). Quantization pipeline, calibration captures, and teacher panel by [brandonmusic](https://huggingface.co/brandonmusic) — co-credited, see the [collaboration thread](https://huggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw/discussions/1). Trellis codec and kernels by [turboderp](https://github.com/turboderp-org/exllamav3). Every tool, patch, receipt and the full campaign log: [malaiwah/glm53-flash-fidelity-suite](https://github.com/malaiwah/glm53-flash-fidelity-suite). Comparable measurements across quants: [quant-fidelity-registry](https://huggingface.co/datasets/malaiwah/quant-fidelity-registry).