Instructions to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700")
model = AutoModelForMultimodalLM.from_pretrained("djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700

SGLang

How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with Docker Model Runner:
```
docker model run hf.co/djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700
```

djdeniro commited on 20 days ago

Commit

7c20f79

verified ·

1 Parent(s): 1dc5d39

Upload README.md with huggingface_hub

Browse files

Files changed (1) hide show

README.md +26 -7

README.md CHANGED Viewed

@@ -60,6 +60,18 @@ served with **vLLM** on **RDNA4** (AMD Radeon R9700) hardware.
 ***Total Context Limit for each task in test 32k, means 6x tasks use more than 32k output tokens***
 ---
@@ -125,15 +137,20 @@ docker pull tcclaviger/vllm:latest
 git clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay
 docker run --rm --tty --ipc=host --shm-size=128g \
   --device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
   -v /path/to/GLM-5.3-Flash-RFA-RFI8-8xR9700:/models:ro \
   -v "$PWD/overlay":/overlay:ro \
   --entrypoint bash tcclaviger/vllm:latest \
-  -c "/overlay/apply_overlay.sh && exec vllm serve /models \
       --served-model-name glm53-flash --trust-remote-code --quantization rfi \
-      --tensor-parallel-size 8 --gpu-memory-utilization 0.95 \
-      --max-model-len 190080 --max-num-seqs 4 --kv-cache-dtype auto"
 ```
 ---
@@ -146,7 +163,7 @@ docker run --rm --tty --ipc=host --shm-size=128g \
 | Layers | 45 = 34 KDA (linear attention) + 11 DSA (sparse-MLA) |
 | Routed experts | 288 (top-8) + 1 shared expert |
 | Extra | mHC hyper-connections, 1 nextn MTP draft layer, native vision tower |
-| Context (bf16 KV) | 190,080 tokens |
 ---
@@ -159,9 +176,11 @@ fed with a min/max image-token budget. The model accepts image and video inputs
 ## Known limitations
-- **MTP is disabled** in the reference serving config (drafter KV-group blocker).
-- **Serve with bf16 KV** (`--kv-cache-dtype auto`) — fp8 KV with runtime scale calibration is
-  broken on this architecture (garbage scales from the uninitialized KDA recurrent state).
 - **Chat needs `reasoning_effort="low"`** — the default Reasoning Effort Max spends 16k+ tokens
   thinking before producing content on long generations.

 ***Total Context Limit for each task in test 32k, means 6x tasks use more than 32k output tokens***
+### Serving performance (8× R9700, FY2026-09 production config)
+| Scenario | Throughput |
+|----------|------------|
+| Decode, batch size 1, MTP OFF | ~34–39 tok/s |
+| Decode, batch size 1, **MTP spec=3** | **~82–88 tok/s** |
+| Aggregate, 4 concurrent, MTP spec=3 | ~155 tok/s |
+| Context window (fp8 KV) | 300,000 tokens |
+MTP speculative decoding: mean acceptance length ~3.7–3.9 of 4 draft tokens,
+average draft acceptance 91–97% (live engine metrics, GPQA-style prompts).
 ---
 git clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay
+# current production config (MTP spec=3, fp8 KV, 300k context)
 docker run --rm --tty --ipc=host --shm-size=128g \
   --device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
   -v /path/to/GLM-5.3-Flash-RFA-RFI8-8xR9700:/models:ro \
   -v "$PWD/overlay":/overlay:ro \
   --entrypoint bash tcclaviger/vllm:latest \
+  -c "/overlay/apply_overlay.sh && GLM5_NEXT_MTP_PROPOSER=1 exec vllm serve /models \
       --served-model-name glm53-flash --trust-remote-code --quantization rfi \
+      --tensor-parallel-size 8 --gpu-memory-utilization 0.9575 \
+      --max-model-len 300000 --max-num-seqs 4 --max-num-batched-tokens 2048 \
+      --kv-cache-dtype fp8 \
+      --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \
+      --enable-prefix-caching --distributed-executor-backend mp \
+      --compilation-config '{\"cudagraph_capture_sizes\":[1,2,4,8,16],\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_copy_inputs\":true}'"
 ```
 ---
 | Layers | 45 = 34 KDA (linear attention) + 11 DSA (sparse-MLA) |
 | Routed experts | 288 (top-8) + 1 shared expert |
 | Extra | mHC hyper-connections, 1 nextn MTP draft layer, native vision tower |
+| Context (fp8 KV) | 300,000 tokens |
 ---
 ## Known limitations
+- **fp8 KV without runtime calibration** — serve with `--kv-cache-dtype fp8` and **scales fixed
+  at 1.0**. Do not enable `--calculate-kv-scales`: runtime calibration on the profile dummy-run
+  produces garbage scales from the uninitialized KDA recurrent state (details in the
+  [serving repo](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700)).
+- The 300k context / MTP spec=3 config presumes the VRAM headroom of the 256 GB 8× R9700 node.
 - **Chat needs `reasoning_effort="low"`** — the default Reasoning Effort Max spends 16k+ tokens
   thinking before producing content on long generations.