vcruz305/GLM-5.3-Flash-EXL3-K2-spark-vllm

GLM-5.3-Flash EXL3 K2 runtime wheels (DGX Spark / GB10)

Prebuilt wheels for serving vcruz305/GLM-5.3-Flash-EXL3-K2 with vLLM on one NVIDIA DGX Spark (GB10, SM121).

These exist so nobody has to compile vLLM to run this pack. Installing takes minutes and needs no CUDA toolchain.

Recipe and scripts: https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe

K-pool tail bug: fixed in this build (2026-08-30)

Wheels published before 2026-08-30 wrote GLM-5.3's sparse-MLA K-pool tail cache out of bounds on long generations: engine death, or silent corruption of a neighbouring layer's sparse-attention index. Root cause: hybrid models never passed positions to the tail metadata builder, so its one-block mapping was skipped. Not EXL3-specific; every GLM-5.3 build on this vLLM lineage had it. This build carries the fix and was validated at 65,536 context with CUDA graphs on (8,192-token generation to completion, engine alive). Details: docs/KPOOL_TAIL_BUG.md in the recipe repo. If you installed earlier wheels, reinstall with bash scripts/install_prebuilt.sh.

The plugin in this build also reads quantization_config.layer_bits for mixed-K checkpoints such as vcruz305/GLM-5.3-Flash-EXL3-K2K3-mix.

What is in here

This is one runtime, not two. vLLM is the inference engine. ExLlamaV3 is here as a CUDA kernel library, not as a second server:

Wheel	Role
vllm	the runtime: scheduler, KV cache, attention, OpenAI API, Glm5Next
exllamav3	CUDA kernels that decode the 2-bit EXL3 trellis weights
glm53_exl3_vllm_plugin	registers exl3 as a vLLM quantization method

The pack stores its 288 routed experts as 2-bit EXL3 trellis tensors, which is not a format PyTorch reads natively. The plugin calls exactly two symbols from the compiled extension, exllamav3_ext.exl3_moe and exllamav3_ext.exl3_moe_max_concurrency, and imports no Python from the exllamav3 package at all. ExLlamaV3's own generator and model code are not loaded and not used.

Why these wheels and not stock vLLM

Stock vLLM cannot serve this checkpoint. It has neither the exl3 quantization method nor the Glm5Next architecture, and no flag turns them on:

• Glm5Next support lives in a fork, not in vllm-project/vllm main
• EXL3 routed-expert loading comes from an out-of-tree plugin

pip install vllm will appear to work and then fail after you have downloaded 91 GiB of weights. Use these wheels instead.

The vLLM wheel is built from the pinned fork revision with the recipe's fixes already applied, so there is nothing left to patch after install:

• SM121 NoPE sparse-MLA path
• GLM-5.3 EAGLE3 auxiliary-state taps
• DFlash2 selective draft quantization guard

Install

git clone https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
cd GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
python scripts/preflight.py        # ~1 second, tells you what is missing
bash scripts/install_prebuilt.sh   # fetches these wheels and installs them

Manually, if you prefer:

python3.12 -m venv ~/venvs/glm53-exl3-local
~/venvs/glm53-exl3-local/bin/python -m pip install \
  --index-url https://download.pytorch.org/whl/cu130 \
  torch==2.13.0+cu130 torchvision==0.28.0+cu130
hf download vcruz305/GLM-5.3-Flash-EXL3-K2-spark-vllm --local-dir ~/glm53-runtime-wheels
~/venvs/glm53-exl3-local/bin/python -m pip install ~/glm53-runtime-wheels/*.whl
~/venvs/glm53-exl3-local/bin/python -m pip install --pre "flashinfer-python==0.6.18rc10"

FlashInfer JIT-compiles its kernels on this box, so the CUDA 13 toolkit's nvcc must be on PATH when the server starts (export PATH=/usr/local/cuda-13.0/bin:$PATH). Without it vLLM's has_flashinfer() returns False, the only sparse-MLA backend for GB10 is rejected, and engine init fails with No valid attention backend found for cuda. The recipe's serve_one_spark.sh adds the path itself and preflight.py checks for it.

Expected pip warning

pip reports a conflict on install and it is expected:

vllm ... requires flashinfer-python==0.6.17, but you have flashinfer-python 0.6.18rc10

0.6.18rc10 is the version every measurement in this recipe was taken on. Do not "fix" it by downgrading to 0.6.17; that silently leaves the measured runtime.

Compatibility, and it is narrow

These carry compiled CUDA extensions. They are not portable:

Requirement	Value
Architecture	aarch64 (ARM64)
GPU	GB10, compute capability 12.1 (SM121)
Python	3.12 (cp312)
PyTorch	2.13.0+cu130
CUDA	13.0

On any other combination, build from source with scripts/install_local_runtime.sh in the recipe repo.

PyTorch and FlashInfer are deliberately not bundled. Both are published wheels whose distribution channels change independently of this recipe.

Provenance

Component	Source
vLLM	ZJY0516/vllm @ 878631b6079d2cf9fb80830ef9cb41b43aded098, plus the recipe patches and the K-pool tail fix
ExLlamaV3	turboderp-org/exllamav3 @ 17bc3923259ffd48aab742edd261a0ca45d55459 (1.4.4)
EXL3 plugin	runtime/exl3_plugin in the recipe repo

Verify what you installed:

python scripts/preflight.py

Not affiliated

Independent community engineering. Not affiliated with or endorsed by Z.ai, NVIDIA, or the vLLM project. Weights are not redistributed here; pull them from the pack repo and respect the GLM-5.3-Flash license. vLLM, ExLlamaV3 and FlashInfer carry their own licenses.