Instructions to use brandonmusic/GLM-5.3-Flash-tr3-4bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use brandonmusic/GLM-5.3-Flash-tr3-4bpw with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="brandonmusic/GLM-5.3-Flash-tr3-4bpw")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("brandonmusic/GLM-5.3-Flash-tr3-4bpw")
model = AutoModelForMultimodalLM.from_pretrained("brandonmusic/GLM-5.3-Flash-tr3-4bpw", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use brandonmusic/GLM-5.3-Flash-tr3-4bpw with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "brandonmusic/GLM-5.3-Flash-tr3-4bpw"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brandonmusic/GLM-5.3-Flash-tr3-4bpw",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw

SGLang

How to use brandonmusic/GLM-5.3-Flash-tr3-4bpw with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "brandonmusic/GLM-5.3-Flash-tr3-4bpw" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brandonmusic/GLM-5.3-Flash-tr3-4bpw",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "brandonmusic/GLM-5.3-Flash-tr3-4bpw" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brandonmusic/GLM-5.3-Flash-tr3-4bpw",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use brandonmusic/GLM-5.3-Flash-tr3-4bpw with Docker Model Runner:
```
docker model run hf.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw
```

brandonmusic commited on Aug 27

Commit

767b027

verified ·

1 Parent(s): 4739eb1

Document SM120 v34 KLD fix and MTP5 runtime

Browse files

Files changed (1) hide show

README.md +73 -4

README.md CHANGED Viewed

@@ -1,6 +1,6 @@
 # GLM-5.3-Flash-EXL3-4bpw
-Source: `zai-org/GLM-5.3-Flash-BF16@a6c167b62691b2bac901344b65cb651a70f53e43`. All routed experts including MTP45 are uniform four-bit EXL3/TR3 MCG; non-routed tensors retain their official native dtype. The custom TP2 runtime is qualified by actual-runtime BF16-teacher KLD plus byte-identical rank outputs and multi-token generation; its stricter decoded raw-logit parity diagnostic remains failed.
 Five-cold-run mean teacher-to-student KLD: `0.024554564250` over 51,175 sealed causal positions per run. Actual TP2 runtime qualification-window KLD: `0.022750847878` over 2,047 positions (both gates: mean KLD < 0.06). This checkpoint requires the included custom Transformers TP2 adapter and is not a stock vLLM/ExLlamaV3 compatibility claim.
@@ -39,11 +39,80 @@ which records every payload path, size, and SHA-256. The 25 final windows remain
 qualification-only and are excluded from fitting and expert selection.
 ## Minimal TP2 launch
-Note this is NOT an optimize launch nearly at all.  I woudl recommend trying https://github.com/chriswritescode-dev/glm-5.3-flash-sm120 image referenced
-here.  that use the MLA which compresses the latent before you get to kv_b_proj.  Getting that running you might get 1 million kv cache.
-the below run time is probably closer to 32k kv cache max.
 Use Transformers 5.16.1, clone ExLlamaV3 at commit `c5d9c657966ffeeaa9353f0cc899f18629da4a13`, compile its CUDA extension, then run:
 ```bash
 PYTHONPATH=runtime/src torchrun --standalone --nproc-per-node=2 runtime/scripts/run_glm53_custom_tp_runtime.py --model . --exllamav3-source /path/to/exllamav3 --prompt 'Hello'
 ```

 # GLM-5.3-Flash-EXL3-4bpw
+Source: `zai-org/GLM-5.3-Flash-BF16@a6c167b62691b2bac901344b65cb651a70f53e43`. All routed experts including MTP45 are uniform four-bit EXL3/TR3 MCG; non-routed tensors retain their official native dtype. The original custom Transformers TP2 runtime and the dedicated SM120 vLLM image below are qualified separately against the same BF16 teacher evidence.
 Five-cold-run mean teacher-to-student KLD: `0.024554564250` over 51,175 sealed causal positions per run. Actual TP2 runtime qualification-window KLD: `0.022750847878` over 2,047 positions (both gates: mean KLD < 0.06). This checkpoint requires the included custom Transformers TP2 adapter and is not a stock vLLM/ExLlamaV3 compatibility claim.
 qualification-only and are excluded from fitting and expert selection.
 ## Minimal TP2 launch
+The historical Transformers runtime below is an evidence/qualification path,
+not the optimized daily-driver launch. Use the SM120 container in the next
+section for serving.
 Use Transformers 5.16.1, clone ExLlamaV3 at commit `c5d9c657966ffeeaa9353f0cc899f18629da4a13`, compile its CUDA extension, then run:
 ```bash
 PYTHONPATH=runtime/src torchrun --standalone --nproc-per-node=2 runtime/scripts/run_glm53_custom_tp_runtime.py --model . --exllamav3-source /path/to/exllamav3 --prompt 'Hello'
 ```
+## SM120 TP2 daily-driver image
+Docker Hub: [`verdictai/glm53-flash-exl3-k4`](https://hub.docker.com/r/verdictai/glm53-flash-exl3-k4)
+- Version: `r19-sm120-tp2-v34`
+- Immutable digest: `sha256:98df6f97cf5a40513a4bc8deda6a61e5c741430e7e53a71434e2bf22b4e0aa89`
+- Hardware qualified: 2x RTX PRO 6000 Blackwell (SM120), TP2
+- Daily-driver mode: NVFP4 MLA KV, DCP1, CUDA graphs, MTP5, 499,968-token maximum context
+- Alternate modes: FP8 MLA KV and DCP2
+- Sampling defaults from the model generation config: temperature `1.0`, top-p `0.95`
+The image contains a dedicated vLLM/B12X overlay for GLM-5.3's hybrid linear/full-attention architecture, NoPE MLA, EXL3 K4 routed experts, and the MTP layer. It is not a claim that the checkpoint runs in upstream stock vLLM.
+### Corrected actual-runtime KLD
+The exact 2,048-token `final-0000` qualification window was captured with TP2,
+DCP1, eager execution, `fp8_ds_mla`, no MTP, and full-vocabulary float32
+runtime logits, then compared to the sealed BF16 teacher in float64 chunks.
+| Metric | Corrected SM120 v34 | Rented B200 custom TP2 | Offline K4 |
+|---|---:|---:|---:|
+| Mean teacher KLD | **0.024864241526** | 0.022750847878 | 0.031831601179 |
+| Top-1 agreement | **0.940400586224** | 0.9384 | — |
+The earlier local KLD near `0.10` was a runtime scale-decoding defect, not a
+routing-quality result. The cache writer stores GLM's four calibrated
+per-token, per-128-channel scales as arbitrary FP32 values (`amax / 448`). The
+SM120 FlashInfer reader was left at `kv_scale_format="auto"`, which interprets
+inline scales using the DeepSeek-v3.2 power-of-two convention. v34 explicitly
+selects `arbitrary_fp32` in the GLM NoPE adapter. The corrected first-64-row KLD
+is `0.1918669499`; rows 64 onward are `0.0194743407`, and the whole-window
+result reproduces the independently observed server range.
+The KLD report is published in this repo at
+`runtime-results/v34/fp8-final-0000-kld-report.json`.
+### MTP5 measured decode
+MTP5 is enabled by default with probabilistic rejection sampling. On the NVFP4
+CUDA-graph path, zero-context concurrency-1 decode improved from 81.9 to
+89.0 tokens/s (+8.7%). A sustained sample accepted 2,113 of 6,195 draft tokens
+(34.1%); acceptance is workload-dependent. The server allocates exactly
+499,968 KV tokens at the published settings—500,000 crosses one additional
+cache block.
+### Docker Compose
+Download `runtime/compose.sm120-tp2.yaml` from this repo, set the model path if
+needed, and run:
+```bash
+GLM53_MODEL_PATH=/absolute/path/to/GLM-5.3-Flash-EXL3-4bpw \
+  docker compose -f compose.sm120-tp2.yaml up -d
+```
+### Serve script
+The published `runtime/serve-glm53-sm120-tp2.sh` defaults to GPUs 0,1, port
+8012, NVFP4/DCP1/MTP5, and the immutable image digest:
+```bash
+chmod +x serve-glm53-sm120-tp2.sh
+MODEL=/absolute/path/to/GLM-5.3-Flash-EXL3-4bpw ./serve-glm53-sm120-tp2.sh
+```
+Use `CACHE=fp8_ds_mla`, `DCP=2`, or `MTP_TOKENS=0` for controlled variants.