license: mit
base_model: zai-org/GLM-5.3-Flash-BF16
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags:
- exl3
- mixture-of-experts
- dgx-spark
- multimodal
GLM-5.3-Flash EXL3 TR3 2.0bpw
This is a selectively compressed GLM-5.3-Flash checkpoint validated on one NVIDIA DGX Spark. It serves text, structured tool calls, images, native video, and a 200k-token request from one GB10 GPU with CUDA graphs enabled.
The routed experts use the EXL3 K2 tier. The 2.0bpw name describes that
routed-expert tier; it is not the storage rate of every tensor. Routing,
attention, embeddings, vision, and other retained components are preserved in
their recorded formats.
Artifact
| Property | Value |
|---|---|
| Base model | zai-org/GLM-5.3-Flash-BF16 |
| Base revision | a6c167b62691b2bac901344b65cb651a70f53e43 |
| Weight files | 133 root-level safetensors shards |
| Weight bytes | 111,352,026,456 |
| Indexed tensors | 583,090 |
| Routed-expert tier | EXL3 K2, 2.0 bpw |
| Structural manifest SHA-256 | 501641b947fa56afc9ed098bdc034e59c2bd229d4fa1a1d7b80e3727f10c8ba3 |
The checkpoint includes an unused MTP companion layer (889 tensors, 14,865,185,408 bytes). The accepted baseline does not load that layer. It is retained for provenance and future runtime work; enabling it requires a new memory and acceptance run.
Measured one-Spark baseline
Tested on one DGX Spark / one GB10 GPU with the custom vLLM EXL3 loader, TP1, FP8 target KV cache, 204,800 configured context, prefix caching disabled, and full-decode CUDA graphs.
| Gate | Result |
|---|---|
| Live model load | 89.89 GiB |
| Live KV capacity | 1,638,400 tokens on the final running instance (8.00x configured context) |
| Text identity | Exact final content and natural stop passed |
| Structured tools | Parsed function name and JSON arguments passed |
| Images | 4/4 paired synthetic fixtures passed exact JSON checks |
| Native video | 2/2 paired synthetic fixtures passed exact JSON checks |
| 200k context | 200,012 server-reported prompt tokens; exact 4/4 retrieval at 5%, 35%, 65%, and 95%; natural stop |
| Multilingual smoke | Arabic, Chinese, and Polish constraint-following prompts passed |
The 200k retrieval took 498.40 seconds end to end, including 474.04 seconds to the first streamed token. Its client-side decode estimate was 9.28 tokens/s. No prefix-cache hit was used.
Bounded exact-answer speed cells measured:
| Input text tokens | Incoming concurrency | Wall time | Client decode estimate |
|---|---|---|---|
| 1,024 | 1 | 14.88 s | 9.16 tok/s |
| 32,768 | 1 | 84.45 s | 9.47 tok/s |
| 131,072 | 1 | 306.10 s | 9.28 tok/s |
| 1,024 | 2 | 29.84 s total | 9.34 tok/s mean |
These are client estimates from usage counts and stream timing, not isolated kernel timings. The server was intentionally configured for one active sequence, so two incoming requests queued rather than decoding concurrently.
Important limitations
- The requested 25–50 tok/s decode target was not met. The accepted baseline is consistently about 9.3 tok/s.
- One simple Python task spent the full 600-second wall deadline generating reasoning and returned no final content. This is retained as a failed responsiveness/code-quality probe.
- Exact held-out KLD against the BF16 base has not been measured for this exact 2.0-bpw artifact. Historical results from other prunes are not claimed here.
- The visual checks are small controlled fixtures, not a broad vision-quality benchmark.
- DFlash2 acceleration was investigated but is not part of this release. In the tested vLLM build, draft metadata inherited the target MLA KV format and failed initialization. Context was not reduced and eager mode was not used as a workaround.
Runtime
This checkpoint needs the accompanying rank-stacked EXL3 runtime. Stock vLLM does not understand this artifact layout. Use the pinned ARM64 container and launch recipe from the public reproduction repository linked in this model's repository metadata. Mount weights read-only. CUDA graphs are required.
Method and attribution
- Z.AI: GLM-5.3-Flash base model and tokenizer.
- TurboDerp / ExLlamaV3: EXL3 format and encoder lineage.
- Brandon M. Music: MIT-licensed GLM-5.2 EXL3/TR3 predecessor and
reproducibility methods. The permitted reference revision is
f79c9167690ca705e877ae4dc55a841d1aae1247; later GLM-5.3 artifacts were not used. - MiaAI Lab: GLM-5.3-Flash DGX Spark runtime lineage.
- Cerebras Research: REAP observation and expert-selection lineage.
Private prompts, credentials, hostnames, addresses, and fleet logs are not included in this repository.