Instructions to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700")
model = AutoModelForMultimodalLM.from_pretrained("djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700

SGLang

How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with Docker Model Runner:
```
docker model run hf.co/djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700
```

GLM-5.3-Flash-RFA-RFI8-8xR9700 / README.md

djdeniro

Update README.md

1dc5d39 verified about 1 month ago

preview code

Raw History Blame

5.88 kB

	---
	license: mit
	base_model: zai-org/GLM-5.3-Flash
	pipeline_tag: image-text-to-text
	library_name: transformers
	tags:
	- glm
	- glm-5
	- multimodal
	- vision-language
	- moe
	- sparse-attention
	- mla
	- quantization
	- rfa
	- rfi
	- rocm
	- rdna4
	- gfx1201
	- 4-bit
	---

	<div align="center">

	# GLM-5.3-Flash · RFA + RFI8 composite quant (8× R9700 / RDNA4)

	A self-quantized derivative of [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash), tuned for 8× AMD Radeon R9700 (gfx1201 / RDNA4)

	[![Model size](https://img.shields.io/badge/params-321B%20(18B%20active)-blue)](https://huggingface.co/zai-org/GLM-5.3-Flash)
	[![bpw](https://img.shields.io/badge/bpw-4.93-ff69b4)](#quantization)
	[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
	[![Serving stack](https://img.shields.io/badge/serving%20stack-tcclaviger%2Fvllm-00a2a2)](https://hub.docker.com/r/tcclaviger/vllm)
	[![GPU](https://img.shields.io/badge/GPU-RDNA4%20(gfx1201)-ED1C24)](https://www.amd.com/en/products/processors/desktops/radeon.html)

	</div>

	---

	## Model card

	This is a composite-quantized checkpoint of Z.ai's GLM-5.3-Flash — the first natively
	multimodal model in the GLM-5 series (321B total / ~18B active parameters). It was quantized from
	the official FP8 release using the `tcclaviger/vllm` composite quantizer and is intended to be
	served with vLLM on RDNA4 (AMD Radeon R9700) hardware.

	> 📦 Companion serving repo: [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) — the ROCm/RDNA4 overlay + quantization recipe.

	### Benchmark

	\| Configuration \| GPQA Diamond \| Correct \| Empty \| Answered \|
	\|---------------\|---------\|---\|---\|------\|
	\| GLM-5.3-Flash (Z-AI API) \| 80.8% \| 76 \| 6 \| 94 \|
	\| GLM-5.3-Flash-RFA-RFI8 \| 85.1% \| 80 \| 6 \| 94 \|

	#### Details

	- Reasoning Effort: high
	- Max Output Len: 32k Tokens


	*Total Context Limit for each task in test 32k, means 6x tasks use more than 32k output tokens*


	---

	## Table of contents

	- [Attribution & credits](#attribution--credits)
	- [Quantization](#quantization)
	- [Quick start](#quick-start)
	- [Model details](#model-details)
	- [Multimodal policy](#multimodal-policy)
	- [Known limitations](#known-limitations)
	- [License](#license)

	---

	## Attribution & credits

	\| Component \| Credit \|
	\|-----------\|--------\|
	\| Base model \| [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) (Z.ai, MIT) \|
	\| Quantizer & quant kernels (RFA / RFI) \| [`tcclaviger/vllm:latest`](https://hub.docker.com/r/tcclaviger/vllm) (IronLLM Labs) \|
	\| RDNA4 port + quant recipe \| [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) (this project's overlay) \|

	The composite RFA + RFI8 quantization scheme, its kernels, and the serving runtime are provided
	by `tcclaviger/vllm` (IronLLM Labs). This checkpoint is the result of applying a quantization
	recipe (archspec + source patches + kda-remap, in the companion repo) on top of that stack, and a
	port of the GLM-5.3-Flash architecture to the RDNA4 serving path.

	---

	## Quantization

	The checkpoint mixes three schemes in one `quant_method: "rfi"` composite:

	\| Scheme \| Bits \| Applied to \| Stored size \|
	\|--------\|------\|------------\|-------------\|
	\| RFA \| 4.5 bpw \| MoE routed experts (42 layers × 288 experts) \| ~171.3 GB \|
	\| RFI8 \| 8 bpw \| attention / shared-expert / dense linears \| ~7.8 GB \|
	\| BF16 / FP32 \| 16 / 32 bpw \| embeddings, vision tower, norms, MTP layer \| ~18.7 GB \|

	### Average bits-per-weight

	```
	bpw = (total safetensors bytes × 8) / total parameters
	= (197,843,715,288 × 8) / 321,342,220,638
	= 4.9254 ≈ 4.93 bpw
	```

	\| Metric \| Value \|
	\|--------\|-------\|
	\| Total parameters \| 321,342,220,638 (~321.3B) \|
	\| On-disk size \| 197.8 GB · 25 safetensors shards \|
	\| Average bpw \| 4.9254 ≈ 4.93 \|
	\| vs. FP8 source \| 0.60× (197.8 GB vs 328.3 GB) \|
	\| vs. BF16 \| 0.31× (197.8 GB vs 642.7 GB) \|

	---

	## Quick start

	```bash
	docker pull tcclaviger/vllm:latest

	git clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay

	docker run --rm --tty --ipc=host --shm-size=128g \
	--device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
	-v /path/to/GLM-5.3-Flash-RFA-RFI8-8xR9700:/models:ro \
	-v "$PWD/overlay":/overlay:ro \
	--entrypoint bash tcclaviger/vllm:latest \
	-c "/overlay/apply_overlay.sh && exec vllm serve /models \
	--served-model-name glm53-flash --trust-remote-code --quantization rfi \
	--tensor-parallel-size 8 --gpu-memory-utilization 0.95 \
	--max-model-len 190080 --max-num-seqs 4 --kv-cache-dtype auto"
	```

	---

	## Model details

	\| Property \| Value \|
	\|----------\|-------\|
	\| Architecture \| `Glm5NextForConditionalGeneration` \|
	\| Layers \| 45 = 34 KDA (linear attention) + 11 DSA (sparse-MLA) \|
	\| Routed experts \| 288 (top-8) + 1 shared expert \|
	\| Extra \| mHC hyper-connections, 1 nextn MTP draft layer, native vision tower \|
	\| Context (bf16 KV) \| 190,080 tokens \|

	---

	## Multimodal policy

	Images are resized with aspect ratio preserved, clamped to min 384×384 / max 1280×1280, and
	fed with a min/max image-token budget. The model accepts image and video inputs natively.

	---

	## Known limitations

	- MTP is disabled in the reference serving config (drafter KV-group blocker).
	- Serve with bf16 KV (`--kv-cache-dtype auto`) — fp8 KV with runtime scale calibration is
	broken on this architecture (garbage scales from the uninitialized KDA recurrent state).
	- Chat needs `reasoning_effort="low"` — the default Reasoning Effort Max spends 16k+ tokens
	thinking before producing content on long generations.

	---

	## License

	MIT. Base model © Z.ai (zai-org), MIT license. Quantizer & runtime © IronLLM Labs
	([tcclaviger/vllm](https://hub.docker.com/r/tcclaviger/vllm)).