Document selectable MTP3 and DFlash2 profiles

Adds directly runnable multimodal DFlash2, language-only DFlash2, and language-only built-in MTP3 options with the measured KV-token capacity difference.

README.md

@@ -19,9 +19,23 @@ tags:
 # GLM-5.3-Flash TR3 4bpw — current SM120 runtime
 
 This is the uniform-K4 EXL3/TR3 routed-expert checkpoint for GLM-5.3-Flash.
-The current daily-driver runtime is v84: TP2/EP2/DCP2, calibrated NVFP4 MLA
-KV, DFlash2-7, CUDA graphs, and working image input on two SM120 GPUs. It is a
-custom vLLM/B12X build and is not compatible with stock upstream vLLM.
+The current v84 runtime supports three explicit TP2/EP2/DCP2 profiles on two
+SM120 GPUs: multimodal DFlash2, language-only DFlash2, and language-only MTP3.
+All use calibrated NVFP4 MLA KV and CUDA graphs. This is a custom vLLM/B12X
+build and is not compatible with stock upstream vLLM.
+
+## Pick a serving profile
+
+| Goal | Launcher | Extra checkpoint | Measured KV tokens |
+|---|---|---|---:|
+| Images plus fastest measured C1 decode | `compose.sm120-tp2.yaml` | DFlash2-7 | 129,473 |
+| Text-only DFlash2 decode | `compose.sm120-tp2-language-only-dflash2.yaml` | DFlash2-7 | 184,619 |
+| **Text-only capacity/default** | **`compose.sm120-tp2-language-only.yaml`** | **none; built-in MTP3** | **1,376,256** |
+
+The MTP3 option means the model's built-in MTP head only: it does not load or
+mount the external DFlash checkpoint. Choose DFlash2 when its modest C1 decode
+gain matters more than resident context/concurrency; choose MTP3 for the normal
+text-only daily driver.
 
 ## Run the current image
 
@@ -91,6 +105,20 @@ GPU_DEVICES=0,1 \
 ./serve-glm53-sm120-tp2-language-only.sh
 ```
 
+To keep DFlash2 while disabling vision, use the separate speed-first launcher:
+
+```bash
+curl -L -o compose.sm120-tp2-language-only-dflash2.yaml \
+  https://raw.githubusercontent.com/brandonmmusic-max/glm-5.3-flash-exl3-4bpw/main/runtime/compose.sm120-tp2-language-only-dflash2.yaml
+
+GLM53_MODEL_PATH=/absolute/path/to/GLM-5.3-Flash-tr3-4bpw \
+GLM53_DFLASH_PATH=/absolute/path/to/GLM-5.3-Flash-DFlash2 \
+docker compose -f compose.sm120-tp2-language-only-dflash2.yaml up -d
+```
+
+Its standalone equivalent is
+[`serve-glm53-sm120-tp2-language-only-dflash2.sh`](runtime/serve-glm53-sm120-tp2-language-only-dflash2.sh).
+
 The language-only alias points to the same tested v84 code digest:
 
 ```text
@@ -109,7 +137,10 @@ Measured capacity on the same two 96 GB GPUs at 300 W each:
 Turning vision off raises the DFlash KV token pool by 42.6%. The much larger
 7.45x language-only gain comes from using the built-in MTP head instead of
 keeping the external DFlash2-7 model resident. It is not a vision-only gain.
-The language-only profile does not accept image inputs.
+The language-only profiles do not accept image inputs. The reported KV-token
+pool is total allocated capacity, not a promise that every request can use the
+entire pool; the configured per-request ceiling and scheduler concurrency still
+apply.
 
 ## Current measured results
 

runtime/compose.sm120-tp2-language-only-dflash2.yaml

@@ -0,0 +1,84 @@
+services:
+  glm53-flash-language-only-dflash2:
+    image: verdictai/glm53-flash-exl3-k4:r19-sm120-tp2-ep2-dcp2-v84-language-only@sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692
+    container_name: glm53-flash-exl3-k4-language-only-dflash2
+    init: true
+    ipc: host
+    shm_size: 32gb
+    restart: unless-stopped
+    ports:
+      - "${GLM53_PORT:-8012}:${GLM53_PORT:-8012}"
+    environment:
+      VLLM_ENGINE_READY_TIMEOUT_S: "3600"
+      VLLM_B12X_GLM_NOPE_NVFP4: "1"
+      VLLM_NVFP4_MLA_DYNAMIC_SCALE: "0"
+      VLLM_NVFP4_MLA_SCALES_FILE: /opt/glm53/calibration/glm53_nvfp4_mla_outer_scales_mtp_power2_v2.json
+      VLLM_EXL3_PREFILL_BLOCK_M: "128"
+      VLLM_EXL3_PREFILL_TRELLIS: "1"
+      B12X_GL53_ROUTE128_WIDE: "1"
+      B12X_GL53_ROUTE128_HYBRID_TAIL: "1"
+      VLLM_USE_B12X_DCP_A2A: "1"
+      VLLM_ENABLE_PCIE_ALLREDUCE: "1"
+      VLLM_PCIE_ALLREDUCE_BACKEND: cpp
+      KV_FP8_ROPE: "0"
+      OMP_NUM_THREADS: "2"
+      NCCL_IB_DISABLE: "1"
+      NCCL_P2P_LEVEL: "4"
+    volumes:
+      - "${GLM53_MODEL_PATH:?set GLM53_MODEL_PATH to the EXL3 checkpoint}:/model:ro"
+      - "${GLM53_DFLASH_PATH:?set GLM53_DFLASH_PATH to incoai/GLM-5.3-Flash-DFlash2}:/draft:ro"
+      - "${GLM53_CACHE_PATH:-./glm53-vllm-cache}:/cache"
+    command:
+      - serve
+      - /model
+      - --served-model-name
+      - GLM-5.3-Flash-EXL3-4bpw
+      - --host
+      - 0.0.0.0
+      - --port
+      - "${GLM53_PORT:-8012}"
+      - --language-model-only
+      - --tensor-parallel-size
+      - "2"
+      - --enable-expert-parallel
+      - --decode-context-parallel-size
+      - "2"
+      - --dcp-comm-backend
+      - a2a
+      - --dtype
+      - bfloat16
+      - --load-format
+      - safetensors
+      - --moe-backend
+      - b12x
+      - --attention-backend
+      - B12X_MLA_SPARSE
+      - --kv-cache-dtype
+      - nvfp4_ds_mla
+      - --max-model-len
+      - "98304"
+      - --max-num-batched-tokens
+      - "2072"
+      - --max-num-seqs
+      - "4"
+      - --gpu-memory-utilization
+      - "0.986"
+      - --enable-chunked-prefill
+      - --no-enable-prefix-caching
+      - --generation-config
+      - /model
+      - --reasoning-parser
+      - glm45
+      - --tool-call-parser
+      - glm47
+      - --enable-auto-tool-choice
+      - --disable-custom-all-reduce
+      - --speculative-config
+      - '{"method":"dflash","model":"/draft","num_speculative_tokens":7,"draft_tensor_parallel_size":2,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","attention_backend":"TRITON_ATTN","kv_cache_dtype":"auto"}'
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ["${GLM53_GPU_0:-0}", "${GLM53_GPU_1:-1}"]
+              capabilities: [gpu]

runtime/serve-glm53-sm120-tp2-language-only-dflash2.sh

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+IMAGE="${IMAGE:-verdictai/glm53-flash-exl3-k4:r19-sm120-tp2-ep2-dcp2-v84-language-only@sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692}"
+MODEL="${MODEL:?set MODEL to the local EXL3 checkpoint directory}"
+DFLASH_MODEL="${DFLASH_MODEL:?set DFLASH_MODEL to the local incoai/GLM-5.3-Flash-DFlash2 directory}"
+GPU_DEVICES="${GPU_DEVICES:-0,1}"
+PORT="${PORT:-8012}"
+NAME="${NAME:-glm53-flash-exl3-k4-language-only-dflash2}"
+CACHE_PATH="${GLM53_CACHE_PATH:-${PWD}/glm53-vllm-cache}"
+
+mkdir -p "${CACHE_PATH}"
+
+exec docker run --rm --name "${NAME}" \
+  --init --gpus "\"device=${GPU_DEVICES}\"" --ipc=host --shm-size 32g \
+  -p "${PORT}:${PORT}" \
+  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
+  -e VLLM_B12X_GLM_NOPE_NVFP4=1 \
+  -e VLLM_NVFP4_MLA_DYNAMIC_SCALE=0 \
+  -e VLLM_NVFP4_MLA_SCALES_FILE=/opt/glm53/calibration/glm53_nvfp4_mla_outer_scales_mtp_power2_v2.json \
+  -e VLLM_EXL3_PREFILL_BLOCK_M=128 \
+  -e VLLM_EXL3_PREFILL_TRELLIS=1 \
+  -e B12X_GL53_ROUTE128_WIDE=1 \
+  -e B12X_GL53_ROUTE128_HYBRID_TAIL=1 \
+  -e VLLM_USE_B12X_DCP_A2A=1 \
+  -e VLLM_ENABLE_PCIE_ALLREDUCE=1 \
+  -e VLLM_PCIE_ALLREDUCE_BACKEND=cpp \
+  -e KV_FP8_ROPE=0 \
+  -e OMP_NUM_THREADS=2 \
+  -e NCCL_IB_DISABLE=1 \
+  -e NCCL_P2P_LEVEL=4 \
+  -v "${MODEL}:/model:ro" \
+  -v "${DFLASH_MODEL}:/draft:ro" \
+  -v "${CACHE_PATH}:/cache" \
+  "${IMAGE}" serve /model \
+  --served-model-name GLM-5.3-Flash-EXL3-4bpw \
+  --host 0.0.0.0 --port "${PORT}" \
+  --language-model-only \
+  --tensor-parallel-size 2 \
+  --enable-expert-parallel \
+  --decode-context-parallel-size 2 \
+  --dcp-comm-backend a2a \
+  --dtype bfloat16 \
+  --load-format safetensors \
+  --moe-backend b12x \
+  --attention-backend B12X_MLA_SPARSE \
+  --kv-cache-dtype nvfp4_ds_mla \
+  --max-model-len 98304 \
+  --max-num-batched-tokens 2072 \
+  --max-num-seqs 4 \
+  --gpu-memory-utilization 0.986 \
+  --enable-chunked-prefill \
+  --no-enable-prefix-caching \
+  --generation-config /model \
+  --reasoning-parser glm45 \
+  --tool-call-parser glm47 \
+  --enable-auto-tool-choice \
+  --disable-custom-all-reduce \
+  --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":7,"draft_tensor_parallel_size":2,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","attention_backend":"TRITON_ATTN","kv_cache_dtype":"auto"}' \
+  "$@"