File size: 5,019 Bytes
84f360b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | ---
license: other
license_name: glm-5
license_link: https://huggingface.co/zai-org/GLM-5.1/blob/main/LICENSE
base_model: zai-org/GLM-5.1
tags:
- reap
- pruning
- moe
- expert-pruning
- glm
- unverified
- experimental
- do-not-use-in-production
library_name: transformers
pipeline_tag: text-generation
---
# ***CRITICAL WARNING: UNTESTED EXPERIMENTAL CHECKPOINT***
## **DO NOT USE THIS MODEL FOR ANYTHING SERIOUS.**
This checkpoint has **not been benchmarked, validated, or tested for coherence**.
It may produce **garbage, repetitive loops, incoherent text, or complete nonsense**.
Treat it as a **broken artifact until proven otherwise**.
---
## GLM-5.1 — 25% Expert Pruned (REAP)
This is a **25% expert-pruned** version of [`zai-org/GLM-5.1`](https://huggingface.co/zai-org/GLM-5.1) using the [REAP method](https://github.com/CerebrasResearch/reap) (Relative Expert Activation Pruning).
| Property | Value |
|----------|-------|
| Base model | `zai-org/GLM-5.1` |
| Architecture | `GlmMoeDsaForCausalLM` (MoE with Dynamic Sparse Attention) |
| Params before prune | 743.91B |
| Params after prune | ~555B |
| Parameter reduction | 25.4% |
| Routed experts per layer | 256 → 192 (removed 64) |
| Shared experts per layer | 1 (unchanged) |
| Active params/token | ~14B (top-8 routing preserved) |
| Precision | BF16 |
| Prune method | REAP (layerwise, refusal_contrast_reap, renorm) |
| Sparse MoE layers | 75 of 78 total (first 3 are dense) |
| Estimated max per-layer REAP signal loss | ~15.8% |
| Observation coverage | 6144/6999 packed batches, 7707/22000 samples (~35% of planned calibration) |
## Why This Might Be Broken
1. **Partial calibration data** — The saliency scores used to select experts for removal were computed from only ~35% of the planned 22,000-sample calibration corpus. Expert importance rankings may be inaccurate.
2. **No quality testing whatsoever** — Zero benchmarks have been run. No coherence check. No perplexity measurement. No human evaluation. The model could produce degenerate output for all we know.
3. **Aggressive prune ratio** — Prior experiments with GLM-family models at similar or higher prune ratios resulted in complete output collapse (repetitive text, broken reasoning, junk logits). The 50% checkpoint in particular is very likely broken based on prior GLM-5 evidence.
4. **DSA architecture sensitivity** — GLM-5.1 uses Dynamic Sparse Attention with learned indexer weights. The interaction between pruned expert routing and the DSA indexer has not been validated.
5. **refusal_contrast_reap without preserve guards** — The pruning was done using `refusal_contrast_reap` selection without `preserve_super` or `preserve_outlier` guardrails, which in prior GLM-5 experiments led to output collapse at high prune ratios.
## What This Is Useful For
- **Research only.** Specifically:
- Studying REAP expert saliency patterns in GLM-5.1
- Comparing prune-ratio robustness across architectures
- Running your own coherence/benchmark evaluations
- Investigating MoE collapse behavior
## How to Load
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"0xSero/GLM-5.1-555B-A14B-REAP",
device_map="auto",
torch_dtype=torch.bfloat16,
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("0xSero/GLM-5.1-555B-A14B-REAP", trust_remote_code=True)
# IMPORTANT: GLM-5.1 is a thinking/chat model. Use the chat template.
messages = [{"role": "user", "content": "Hello"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
out = model.generate(inputs.to(model.device), max_new_tokens=128)
print(tokenizer.decode(out[0]))
```
## Pruning Method
[REAP](https://github.com/CerebrasResearch/reap) (Relative Expert Activation Pruning) removes MoE experts by measuring their relative activation patterns during a calibration pass. Experts with the lowest saliency scores (combined REAP signal + frequency weighting) are removed layer-by-layer, keeping `top-8` routing unchanged so the active-parameter budget per token stays the same.
## Sibling Checkpoints
| Prune % | Total Params | Experts/layer | HuggingFace |
|---------|-------------|--------------|-------------|
| 25% | ~555B | 192/256 | [`0xSero/GLM-5.1-555B-A14B-REAP`](https://huggingface.co/0xSero/GLM-5.1-555B-A14B-REAP) |
| 40% | 455B | 154/256 | [`0xSero/GLM-5.1-444B-A14B-REAP`](https://huggingface.co/0xSero/GLM-5.1-444B-A14B-REAP) |
| 50% | ~367B | 128/256 | [`0xSero/GLM-5.1-367B-A14B-REAP`](https://huggingface.co/0xSero/GLM-5.1-367B-A14B-REAP) |
**All three are untested. The 25% checkpoint is the most likely to be coherent.**
## Citation
If you use this checkpoint, cite the [REAP paper](https://github.com/CerebrasResearch/reap) and clearly note that this is an unverified experimental artifact.
---
**Last updated:** 2026-04-14
**Status:** UNVERIFIED / UNTESTED / EXPERIMENTAL / LIKELY DEGENERATE
|