---
license: mit
base_model:
  - zai-org/GLM-5.2
base_model_relation: quantized
language:
  - ja
  - en
  - zh
tags:
  - quantization
  - vector-quantization
  - aqlm
  - mixture-of-experts
  - glm
  - vllm
pipeline_tag: text-generation
---

# GLM-5.2 — mixed-bit VQ (AQLM) ~1.86-bit

A **~180 GiB** quantization of **[GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)**
(744B Chinese-native reasoning MoE, MIT) that keeps Japanese/English/Chinese
**thinking-mode** quality at **~1.86 bit/weight**, using **vector quantization with
GPTQ error compensation (AQLM-style)** instead of scalar rounding.

Runs on **2× RTX PRO 6000 (sm_120 Blackwell, ~95 GiB each)** via vLLM.

## Why VQ

At the same size, scalar mixed-bit rounding loses too much at 1–2 bit. Replacing the
scalar codes with a **shared vector codebook + per-row error compensation** recovers most
of it (the two together are *super-additive* — neither alone is enough):

| Metric | Scalar mixed-bit (same ~180 GiB) | **This (VQ + compensation)** |
|---|---|---|
| Calibration KL (fake-quant, iso-size) | baseline | **−47 %** |
| Greedy arithmetic eval (JA/EN/ZH, terminate + correct) | 21/22 | **22/22** |
| 1-bit experts | collapse (KL ≈ 13) | **survive (KL ≈ 0.42)** |

> The −47 % KL is a fake-quant, iso-size comparison; the deployable end-to-end signal is
> the greedy eval (multi-digit multiplication and word problems in all three languages,
> including held-out items not in calibration).

## Serving

This is **not** a plug-and-play GGUF — it needs a matching sm_120 stack:

- **vLLM** with `GlmMoeDsaForCausalLM` + sm_120 kernels (reference: `jasl/vllm` PR-41834 sm12x preview).
- **transformers 5.12**.
- The **VQ serving plugin** from **[mmzz164/OneCompression @ `glm-serving-v1`](https://github.com/mmzz164/OneCompression)** — see [`example/glm-5.2/`](https://github.com/mmzz164/OneCompression/tree/glm-serving-v1/example/glm-5.2) for the launcher and full instructions.
- **2× ~95 GiB sm_120 GPUs**, EP=1 / TP=2 (VQ codes can't be tensor-parallel-sharded).

```bash
GLM_CKPT=/path/to/this/model bash start_glm_api_vq.sh   # OpenAI API :8001, served as "glm-5.2"
```

`MixedVQMoEMethod` is auto-selected from the `format:"vq"` markers in `quantization_config`.

## Performance

- **~16 tok/s** steady-state decode (single stream), **38×** over the eager dequant baseline
  (grouped Triton VQ-GEMM + CUDA graphs; the key win was fixing a shared-memory bank conflict
  in the codebook gather).
- **Context ≤ 4096** (dense MLA — sm_120 has no sparse-DSA forward kernel; dense is exact at
  ctx ≤ 2048 and validated functional, incl. >2048 needle retrieval, up to 4096).

## Allocation

Mixed **1/2/3-bit per expert** (≈ 15.8k @1-bit / 34.0k @2-bit / 7.8k @3-bit projections),
allocated by activation-frequency-aware AutoBit (arithmetic-routing tuned), then re-encoded
to VQ codes + a shared codebook per bit-width. Non-expert "spine" stays scalar 4-bit.

## Limitations

- sm_120-specific serving stack (not portable to plain wheels).
- Context capped at 4096 on this hardware (sm_120 sparse-attention kernel gap + VRAM); longer
  context needs smaller weights.
- Default serving template is **thinking-on at max reasoning effort** — thorough but verbose
  for casual chat (pass `chat_template_kwargs={"enable_thinking": false}` for direct answers).

## License & attribution

- This quantized model: **MIT**.
- Base **GLM-5.2**: **MIT**, © Zhipu AI — this is a derivative; all rights/attribution to upstream.
- Quantization/serving built on **OneCompression** (MIT, © Fujitsu Ltd.) and vLLM / transformers (Apache-2.0).