---
license: mit
base_model: zai-org/GLM-5.2
pipeline_tag: text-generation
tags:
  - moe
  - reap
  - pruning
  - router-kd
  - gguf
  - llama.cpp
  - glm
  - glm-5.2
---

# GLM-5.2-REAP-504B-GGUF — BF16 + dynamic 2/3/4-bit

> GGUF builds of the **34%-expert-pruned, Router-KD-recovered GLM-5.2-504B** — for **llama.cpp**
> (CPU / Metal / CUDA). Includes the **first working GGUF of a GLM-5.2 DSA model**, made possible by
> a custom patch for its shared sparse-attention indexer.

The full-precision NVFP4 model lives at **[`0xSero/GLM-5.2-504B`](https://huggingface.co/0xSero/GLM-5.2-504B)** — start there for the model story, eval, and architecture. This card is about the **GGUF conversion** specifically.

---

## 🙏 Sponsor
Built on **8× NVIDIA B200 sponsored by [Lambda](https://lambda.ai)**. Thank you. 🙏

---

## Files

| file | bits | size | notes |
|---|---|---|---|
| `GLM-5.2-REAP-504B-BF16`        | 16   | ~933 GB | full precision — for fine-tuning / re-quantizing (23 shards) |
| `GLM-5.2-REAP-504B-Q4_K_XL`     | ~4.5 | ~325 GB | **recommended** — dynamic, highest usable quality (8 shards) |
| `GLM-5.2-REAP-504B-Q3_K_XL`     | ~3.5 | ~259 GB | dynamic, strong quality/size (6 shards) |
| `GLM-5.2-REAP-504B-Q2_K_XL`     | ~2.7 | ~111 GB | dynamic, smallest (5 shards) |

Files > 45 GB are split (`...-00001-of-000NN.gguf`); **point llama.cpp at the first shard** and it
loads the rest automatically.

## What "dynamic" means here

These are **not flat K-quants.** Each uses **per-tensor precision**: the parts that hurt most under
quantization are kept high while the bulk (routed experts) is pushed low:

- **token-embedding** → Q6_K, **output head** → Q8_0/Q6_K
- **attention** (MLA q/kv) → Q6_K / Q5_K / Q4_K (by tier)
- **shared expert** → Q8_0 / Q6_K
- **routed-expert down-projection** kept above gate/up (it's the sensitive one)

> **Not yet imatrix-calibrated.** A true importance-matrix pass needs the model *running* under
> llama.cpp, and our llama.cpp build is CPU-only while the GPUs were committed elsewhere. The
> per-tensor recipe still puts these well above naive quants; imatrix-calibrated builds may follow.

## The hard part — making GLM-5.2 loadable in llama.cpp

GLM-5.2 uses **DSA (DeepSeek-style sparse attention) with a *shared* indexer**: only ~1 layer in 4 is
a "full" indexer layer; the other **57 of 78 layers share** it and carry **no indexer weights of
their own** (`index_topk_freq: 4`). Stock llama.cpp demands an indexer tensor on **every** layer, so
the converted GGUF failed to load:

```
missing tensor 'blk.3.indexer.k_norm.weight'
```

**The fix (applied to these files):** we **filled each of the 57 shared layers' indexer tensors
(285 tensors total) by duplicating them from the nearest preceding full layer**, making the GGUF
loadable.

**The MTP / next-token-prediction layer (`blk.78`) is preserved** — kept at **Q6_K** in every quant
(the stock quantizer rejects the nextn layer's experts, so we pin them with explicit per-tensor
types). This keeps **self-speculative decoding** available in llama.cpp for this architecture.

**Verified:** the patched model loads and **generates coherent text** end-to-end in `llama-cli`.

> **Caveat (honest):** the sharing is *approximated* — the duplicated weights are recomputed per
> layer rather than the index being reused exactly as the original does. Output is coherent in
> testing, but this is not a bit-exact reproduction of the reference attention. Prefer `Q4_K_XL` or
> BF16 where fidelity matters most; the low-bit quants were not individually loop-tested.

## Run it

```bash
# loads all shards from the first one
llama-cli    -m GLM-5.2-REAP-504B-Q4_K_XL-00001-of-00008.gguf -p "Hello"
llama-server -m GLM-5.2-REAP-504B-Q4_K_XL-00001-of-00008.gguf --host 0.0.0.0 --port 8080
```

**Tips (measured at n=2000):** a **sampler guardrail** is the single best free anti-loop knob —
`--min-p 0.05 --repeat-penalty 1.05` → loop 7.2%→**4.9%**; `--repeat-penalty 1.10` → **2.3%**, i.e. it
**fully recovers** the pruning-induced looping (raw 7.2%; unpruned teacher 3.6%). Use 1.05 by default,
1.10 if you see loops (higher penalty can slightly over-suppress legitimate repetition in code). A
*"Be concise…"* system prompt separately halves output length (it shortens, doesn't de-loop).

## How the underlying model was made (short version)

REAP expert-pruning (`gate × ‖expert_output‖`) → keep 168/256 experts/layer (~504B, 34% pruned) →
gate-only **Router-KD** to the unpruned teacher (train 0.016% of params). On **n=2000** held-out real
prompts it loops at **7.2%** vs the unpruned teacher's **3.6%** — pruning roughly doubles the loop rate
(gate-KD recovers routing but not full termination; a knowledge-recovery LoRA is in progress to close
the gap). Full details + honest accounting on the [NVFP4 model card](https://huggingface.co/0xSero/GLM-5.2-504B).

---
*GGUF conversion + custom DSA-indexer patch. Compute sponsored by **[Lambda](https://lambda.ai)** — thank you. 🙏*