README.md

---
base_model: zai-org/GLM-5.2
base_model_relation: quantized
tags:
- gguf
- glm
- moe
- ds4
- ssd-streaming
---

# GLM-5.2 GGUF for ds4 (SSD streaming, CUDA)

This is a mirror of the official ds4 GGUF of [GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)
(743B MoE) built by antirez and published at
[antirez/GLM-5.2-GGUF](https://huggingface.co/antirez/GLM-5.2-GGUF) (bit-identical file, same
sha256). Credit for the quantization is his; this repo re-documents it with the full
per-tensor recipe below and pairs it with the CUDA/SSD-streaming usage notes. It is the file
used by the [ds4](https://github.com/antirez/ds4) inference engine, specifically the
[glm-local branch](https://github.com/giannisanni/neutronstar/tree/glm-local) which adds the CUDA port,
SSD expert streaming optimizations, and the first MTP speculative-decoding implementation for
GLM 5.2 on any backend.

The design target is machines that cannot hold the model in RAM at all: routed experts are
read from disk per token while the ~20 GiB of attention/shared weights stay resident. On an
RTX 4060 Ti 16GB + 30GB RAM + a Gen4 NVMe it decodes at ~0.4 tokens/s and climbs with disk
bandwidth. Yes, that is slow. It is also a 743B model answering on hardware worth less than
a mid-range gaming rig.

## Files

| File | Size | Routed experts (ffn_{gate,up,down}_exps) | Everything else |
|------|------|------------------------------------------|-----------------|
| GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf | 196.6 GiB | IQ2_XXS (layers 3-77, 225 tensors, 177 GiB); blk.78 MTP layer at Q2_K | Q8_0 attention/shared-expert/embeddings/output (872 tensors) + F32 norms (709 tensors), 19.6 GiB |

The MTP draft head (blk.78: full GLM layer + nextn eh_proj/enorm/hnorm/shared_head_norm) is
included in the main file. No separate draft gguf is needed: pass the same file to --mtp.

## Quantization recipe

The filename is the spec. In detail:

| Tensor class | Quant | Notes |
|---|---|---|
| blk.*.ffn_{gate,up,down}_exps (layers 3-77) | IQ2_XXS | routed experts, uniform on purpose: the streaming expert cache uses fixed-size slabs and the dp4a decode kernels read IQ2_XXS directly |
| blk.78.ffn_{gate,up,down}_exps | Q2_K | the MTP draft layer's experts; never runs in the main decode loop, only feeds speculation |
| blk.*.ffn_{gate,up,down}_shexp | Q8_0 | shared experts |
| blk.{0,1,2}.ffn_{gate,up,down} | Q8_0 | leading dense layers |
| blk.*.attn_q_a, attn_q_b, attn_kv_a_mqa, attn_k_b, attn_v_b, attn_output | Q8_0 | all MLA attention projections |
| blk.*.indexer.attn_q_b, indexer.attn_k | Q8_0 | DSA sparse-indexer projections |
| blk.*.indexer.proj, indexer.k_norm(+bias) | F32 | indexer scoring head |
| blk.*.ffn_gate_inp (router), blk.*.exp_probs_b (router bias) | F32 | learned router, kept exact |
| blk.78.nextn.eh_proj | Q8_0 | MTP embed/hidden fusion |
| blk.78.nextn.{enorm,hnorm,shared_head_norm} | F32 | MTP glue norms |
| token_embd.weight, output.weight | Q8_0 | embeddings and output head |
| all *_norm.weight | F32 | |

The motivation behind the asymmetry: the routed experts are the majority of the parameter
count but each individual expert handles only a fraction of tokens, so aggressive
quantization on them costs less in average quality than the same treatment of the router,
projections, or shared experts. Keeping the decision-making components at Q8_0 preserves
model behavior; crushing the experts buys the size.

SSD streaming adds a second reason: the Q8_0/F32 set (~20 GiB) is resident, so its bytes
are paid once in RAM, while the experts are read from disk again and again. Quantizing the
experts harder is a per-token bandwidth win; quantizing the resident set harder would save
only idle memory. The asymmetry follows the traffic, not just the parameter count.

## Usage

Needs the glm-local branch of ds4 (CUDA, sm_89 tested) and a fast NVMe. Expert reads are
O_DIRECT through io_uring; host RAM is used for a popularity (LFU) expert cache, so give it
whatever you can spare with DS4_CUDA_HOST_EXPERT_CACHE_GB.

```
git clone -b glm-local https://github.com/giannisanni/neutronstar
cd ds4 && make cuda CUDA_ARCH=sm_89
DS4_GLM_CUDA_UNSAFE=1 DS4_CUDA_HOST_EXPERT_CACHE_GB=7 DS4_CUDA_PARALLEL_FETCH_THREADS=16 \
./ds4 -m GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf \
  --cuda --ssd-streaming --ssd-streaming-cache-experts 64 \
  --ctx 4096 --tokens 400 --nothink -p "Tell me something surprising about Suriname."
```

Interactive chat: drop -p. MTP probe telemetry: add --mtp <same file> with DS4_MTP_PROBE=1
and DS4_MTP_STREAMING_UNSAFE=1.

Measured on RTX 4060 Ti 16GB / 30GB DDR5 / Gen4 x4 NVMe: prefill ~0.35 t/s, generation
~0.40 t/s with a 7 GiB host expert cache (~30% hit rate: the hottest 4% of experts serve
30% of lookups). The engine runs at ~89% of the PCIe link ceiling; a faster disk moves the
number almost linearly.

sha256: a49de64c5020432bdae23de36a423a9660a5621bc0db8d12b66bd8814b07fea0

## License

Inherits the upstream GLM-5.2 model license (zai-org). The quantization recipe and this
card: MIT.