README.md

# Mini GLM 5.2 Active-MTP Debug Checkpoint, FP8

This is a small GLM-5.2-style checkpoint built for inference plumbing and kernel
debugging, not for model quality. This directory is the FP8 weight-quantized
variant of the active-MTP full-dimension checkpoint.

The base model is a 14-layer mini GLM-MoE-DSA model trained for readable English.
It has been upscaled to GLM-5.2-compatible tensor dimensions and includes one
appended MTP/EAGLE next-token layer for SGLang speculative decoding.

The FP8 export follows the public GLM-5.2-FP8 checkpoint convention:

- eligible 2D linear weights are stored as `torch.float8_e4m3fn`
- each quantized weight has a sibling `*_scale_inv` tensor
- scale tensors are FP32 with 128x128 block granularity
- embeddings, norms, router gates, DSA `weights_proj`, and MTP glue
  projections/norms remain unquantized

## Intended Use

- Debug SGLang GLM-MoE-DSA inference without loading the full GLM 5.2 model.
- Exercise DSA, MoE, attention, TP sharding, FP8 KV cache, and EAGLE/MTP paths.
- Validate kernel changes against a checkpoint that is small in layer count but
  keeps GLM-5.2-like shapes.

This model is expected to produce readable but repetitive English. It is not a
general-purpose language model.

## MTP Details

The appended MTP layer is active and nonzero. Its attention and MoE tensors were
trained, rather than zeroed out, so this checkpoint can exercise the MTP decoder
attention/MoE path during serving.

The MTP trainer uses SGLang EAGLE nextn alignment:

```text
hidden[t] + token[t+1] -> token[t+2]
```

For rollout step `k`:

```text
input token = token[t+k+1]
target      = token[t+k+2]
```

This is intentionally not the standard LM shift
`hidden[t] + token[t] -> token[t+1]`.

## Compatibility

### SGLang

This directory is intended for SGLang serving with GLM-MoE-DSA, TileLang DSA,
FP8 KV cache, and EAGLE speculative decoding.

Example shape of the serving command:

```bash
source ./ENV_RUN.sh
CUDA_VISIBLE_DEVICES=6,7 HIP_VISIBLE_DEVICES=6,7 sglang serve \
  --model-path /root/inference-v2/mini-glm-5.2/v1c-upscale/checkpoints/v1b-mtp-active-decoder-serveddata-align-1k-full-original-dims-tiled-sglang-fp8 \
  --tp 2 \
  --trust-remote-code \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --kv-cache-dtype fp8_e4m3 \
  --chunked-prefill-size 4096 \
  --max-total-tokens 4096 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --disable-custom-all-reduce \
  --disable-shared-experts-fusion \
  --cuda-graph-backend-decode disabled \
  --cuda-graph-backend-prefill disabled
```

### Hugging Face Transformers

Use the BF16/full-dimension checkpoint for Hugging Face Transformers CPU/GPU
reference inference. This FP8 directory is intended for SGLang's GLM FP8 loader.
The FP8 tensors match the GLM-5.2-FP8 safetensors layout, but generic HF
Transformers inference may not dequantize this custom GLM-MoE-DSA FP8 format.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "/root/inference-v2/mini-glm-5.2/v1c-upscale/checkpoints/v1b-mtp-active-decoder-serveddata-align-1k-full-original-dims-tiled-sglang"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)
```

HF inference uses only the base 14-layer causal LM. The appended MTP tensors are
reported as unexpected keys because HF Transformers' `glm_moe_dsa` model does
not implement the MTP/EAGLE nextn path.

## Training Summary

The MTP layer was trained from copied active decoder weights with the corrected
SGLang EAGLE alignment on a small SGLang-served token dataset.

Final offline served-data metrics for the source MTP checkpoint:

| Metric | Value |
|---|---:|
| accept@0 | 0.9295 |
| accept@1 | 0.8760 |
| accept@2 | 0.8292 |
| average accepted draft tokens | 2.6346 |
| data loss | 0.7798 |

These are offline metrics on the small served-token dataset and should be treated
as a debugging signal, not a model-quality benchmark.

## Example Outputs

The model is only expected to produce barely passable English. Repetition,
generic phrasing, weak reasoning, and topic drift are normal.

Observed SGLang FP8 raw-completion sample:

Prompt:

```text
A small language model is useful for testing inference code because
```

Response:

```text
it is an independent scientific method. However, the application of the
language is considered as a social, practical approach. It has a wide range of
specific characteristics and may be used for clinical practice.
```

The text is readable but semantically loose; that is the expected quality bar
for this plumbing checkpoint.

These examples show the intended quality bar: readable text for plumbing tests,
not useful answers.

### Example 1

Prompt:

```text
The quick brown fox
```

Observed HF base-model response:

```text
The quick brown fox is a very good thing to do. It is a good thing to do.
```

### Example 2

Prompt:

```text
Write one short paragraph about a library in a small town.
```

Expected style of response:

```text
The library is a quiet place in the center of the town. People come there to
read books, ask questions, and sit at the tables in the afternoon. It is not a
large building, but it is useful for the people who live nearby.
```

### Example 3

Prompt:

```text
Explain why rain falls from clouds.
```

Expected style of response:

```text
Rain falls from clouds when water in the air becomes heavy enough to fall down.
The cloud is made of small drops, and those drops can join together. When they
become too heavy, they fall as rain.
```

### Example 4

Prompt:

```text
Continue this sentence: The old machine started slowly because
```

Expected style of response:

```text
The old machine started slowly because the parts were worn and the motor needed
time to move. It made a small sound, then a louder sound, and finally began to
work again.
```

The responses above are coherent enough to inspect inference behavior, token
flow, speculative decoding, and kernel changes. They should not be used to judge
instruction following or factual accuracy.