Instructions to use malaiwah/GLM-5.3-Flash-TR3-8bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use malaiwah/GLM-5.3-Flash-TR3-8bpw with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="malaiwah/GLM-5.3-Flash-TR3-8bpw")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("malaiwah/GLM-5.3-Flash-TR3-8bpw")
model = AutoModelForMultimodalLM.from_pretrained("malaiwah/GLM-5.3-Flash-TR3-8bpw", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Trellis

How to use malaiwah/GLM-5.3-Flash-TR3-8bpw with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use malaiwah/GLM-5.3-Flash-TR3-8bpw with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "malaiwah/GLM-5.3-Flash-TR3-8bpw"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "malaiwah/GLM-5.3-Flash-TR3-8bpw",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/malaiwah/GLM-5.3-Flash-TR3-8bpw

SGLang

How to use malaiwah/GLM-5.3-Flash-TR3-8bpw with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "malaiwah/GLM-5.3-Flash-TR3-8bpw" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "malaiwah/GLM-5.3-Flash-TR3-8bpw",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "malaiwah/GLM-5.3-Flash-TR3-8bpw" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "malaiwah/GLM-5.3-Flash-TR3-8bpw",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use malaiwah/GLM-5.3-Flash-TR3-8bpw with Docker Model Runner:
```
docker model run hf.co/malaiwah/GLM-5.3-Flash-TR3-8bpw
```

malaiwah commited on 5 days ago

Commit

b5ef443

verified ·

1 Parent(s): 1e5c97a

SEALED: 0.012384191 over the full panel, two bitwise-identical cold runs

Browse files

Files changed (2) hide show

README.md +33 -34
receipts/stream-k8-kld.json +26 -0

README.md CHANGED Viewed

@@ -31,43 +31,42 @@ MTP layer quantized at K8 (128-word trellis, MCG `0xCBAC1FED`); everything else
 embeddings, lm_head) **bit-exact native BF16**. 331.4 GB — within 1% of the
 official FP8 release's footprint, at 1.66× lower divergence.
-> ⚠️ **PRELIMINARY**: the number below is cold run 1 of 2 over the full sealed
-> panel. The second cold run (determinism check) is in flight; the sealed
-> two-run receipt replaces this section when it lands.
-## Quality — full sealed panel, streaming lane
-| Model | Mean KLD (nats) | Top-1 | Size | Scope |
-|---|---:|---:|---:|---|
-| **This K8 (preliminary)** | **0.012384** | **96.78 %** | 331 GB | full panel, run 1/2 |
-| K6 (same lane) | 0.013715 | 96.56 % | 254 GB | full panel, run 1 |
-| [K6 (sealed, 5 runs)](https://huggingface.co/malaiwah/GLM-5.3-Flash-TR3-6bpw) | 0.013723 | — | 254 GB | full panel × 5 bitwise-identical |
-| Official FP8 | 0.020615 | 95.63 % | 328 GB | cross-stack, [receipt](https://huggingface.co/datasets/malaiwah/GLM-5.3-Flash-fidelity-suite-v1/blob/main/reports/fp8-on-brandon-panel.json) |
-| brandonmusic 4bpw | 0.024555 | — | 176 GB | full panel, his stack |
-| 0xSero Dione Q4 | 0.027263 | — | 188 GB | full panel, [our measurement](https://huggingface.co/datasets/malaiwah/GLM-5.3-Flash-fidelity-suite-v1/blob/main/reports/dione-q4-packed-kld.json) |
-| NVFP4 (his stack) | 0.060535 | — | ~180 GB | 1 window × 5 runs |
-Measured on brandonmusic's sealed 25-window / 51,175-position panel against his
-fp32 BF16 teacher logits, direction KLD(teacher ‖ student), fp64 accumulation.
-**Headline: at the same footprint as the official FP8 (331 vs 328 GB), K8 is
-1.66× closer to the BF16 teacher.** Weight-space corroboration: with the
-intermediate-channel permutation undone, K8's shipped store is **13.2× tighter
-in NMSE** than K6's (3.505e-5 vs 4.624e-4, better in 30 of 30 sampled matrices).
-**Lane note.** This is the *streaming* lane (single GPU, ~$8/panel), not the
-8×H200 sealed lane K6 was qualified on. The lanes were bridged on this exact
-panel: K6 measures 0.013715 streaming vs 0.013723 sealed — a **−8.5e-6**
-difference (0.06 %), fully explained by expert-combine order in bf16. Rows
-above from different lanes are labelled; do not mix them without that offset in
-mind.
 **Methodology note worth stealing.** A single-window comparison of these two
 rates is *statistically meaningless*: per-window KLD scatter has sd 1.73e-3
-against a K6-vs-K8 effect of 1.22e-3. On one unlucky window (`window-0000`) K8
-appeared *worse* than K6; over the full panel it wins decisively. Never quote a
-single-window KLD as a rate comparison — the full write-up is in
-[`k6/K8-ANOMALY.md`](https://github.com/malaiwah/glm53-flash-fidelity-suite/blob/main/k6/K8-ANOMALY.md).
 ## What this is (and is not)

 embeddings, lm_head) **bit-exact native BF16**. 331.4 GB — within 1% of the
 official FP8 release's footprint, at 1.66× lower divergence.
+## Quality — SEALED, full panel, two bitwise-identical cold runs
+**Mean KLD(teacher ‖ K8) = 0.012384191023436866** over the full sealed panel
+(25 windows / 51,175 positions), **two cold runs producing identical means to
+the last digit** (`bitwise_deterministic: true`). Quality gate passed.
+Receipt: [`receipts/stream-k8-kld.json`](receipts/stream-k8-kld.json).
+| Model | Mean KLD (nats) | Size | Lane |
+|---|---:|---:|---|
+| **This K8** | **0.012384** | 331 GB | streaming, 2 runs |
+| [K6](https://huggingface.co/malaiwah/GLM-5.3-Flash-TR3-6bpw) | 0.013715 | 254 GB | streaming, 2 runs |
+| K6 (sealed 8×H200, 5 runs) | 0.013723 | 254 GB | sealed EP8 |
+| Official FP8 | 0.020615 | 328 GB | cross-stack |
+| brandonmusic 4bpw | 0.024555 | 176 GB | his stack |
+| 0xSero Dione Q4 | 0.027263 | 188 GB | [our measurement](https://huggingface.co/datasets/malaiwah/GLM-5.3-Flash-fidelity-suite-v1/blob/main/reports/dione-q4-packed-kld.json) |
+| NVFP4 | 0.060535 | ~180 GB | his stack, 1 window |
+**At the same footprint as the official FP8 release (331 vs 328 GB), K8 is
+1.66× closer to the BF16 teacher** — and 1.11× closer than K6 at 30 % more
+bytes. Weight-space corroboration: with the intermediate-channel permutation
+undone, K8's shipped store is **13.2× tighter in NMSE** than K6's (3.505e-5 vs
+4.624e-4, better in 30 of 30 sampled matrices).
+**Lane disclosure.** Measured on the single-GPU *streaming* lane (~$6/model),
+not the 8×H200 sealed lane. The lanes were bridged on this exact panel: K6
+reads 0.013714889 streaming vs 0.013723385 sealed — **−8.5e-6 (0.06 %)**, with
+the worst single window differing by 2.9e-4. The streaming receipt sets
+`publishable_as_reproduction: false` because a different expert-combine order
+is an independent measurement that agrees closely, not a bitwise reproduction.
 **Methodology note worth stealing.** A single-window comparison of these two
 rates is *statistically meaningless*: per-window KLD scatter has sd 1.73e-3
+against a K6-vs-K8 effect of 1.22e-3. On one unlucky window (`window-0000`)
+K8 appeared *worse* than K6; over the full panel it wins decisively. Never
+quote a single-window KLD as a rate comparison —
+[full write-up](https://github.com/malaiwah/glm53-flash-fidelity-suite/blob/main/k6/K8-ANOMALY.md).
 ## What this is (and is not)

receipts/stream-k8-kld.json ADDED Viewed

	@@ -0,0 +1,26 @@

+{
+  "bitwise_deterministic": true,
+  "cold_run_count": 2,
+  "cold_run_deviation": "2 cold runs, not 5 (budget; disclosed)",
+  "distinct_tokenwise_kld_sha256": [
+    "763bc4a56a371e11a0f96469885b920deb6acb2c7c576d1268fb0907577f0942"
+  ],
+  "kld_report_sha256": [
+    "965566f1d62e75051612045ff88424e50c46a6d1abfecfb92b04af78e8bf3798",
+    "d223567460e4c22cccc432b082cf74bfbc7339628b13db823fd0e0590f2f20bb"
+  ],
+  "measured_mean_kld": 0.012384191023436866,
+  "profile": "k8-tp4",
+  "quality_gate": {
+    "metric": "mean_tokenwise_kld",
+    "threshold_lt": 0.06
+  },
+  "quality_gate_passed": true,
+  "run_means": [
+    0.012384191023436866,
+    0.012384191023436866
+  ],
+  "schema": "malaiwah.glm53-k8-packed-kld-summary.v1",
+  "student_label": "uniform-k8",
+  "teacher_receipt_sha256": "2ae08117c3d4247f747b2a9a889b68e1a06387b788d56a0bf23bb950c77bc5a5"
+}