Hugging Face's logo

gorbatjovy
/
glm-5.3-flash-ablit-nvfp4-DFlash2-patched

glm
nvfp4
vllm
speculative-decoding
dgx-spark
abliterated
Model card Files
xet
 Community

GLM-5.3-Flash abliterated NVFP4 + DFlash2 — 2× DGX Spark deployment recipe

This repo contains no weights. It is the configuration, patches and verification harness for serving the abliterated NVFP4 GLM-5.3-Flash across two DGX Sparks (GB10, TP=2) with speculative decoding, vision, 500K context and a working prefix cache.

Weights and drafter live in their own repos — see Components.

What this adds over the upstream recipe

upstream here effect
prefix cache 0% hit rate 96.7% reuse @100K see below
KV_BYTES 3–4.5 GiB 6 GiB pool 414K → 777K tokens
CTX 204,800 500,000 1.55× concurrency at that pin
drafter stock incoai fc-recalibrated recovers acceptance on an abliterated target
RoCEv2 GID hardcoded index auto-detected per rank survives GID table reordering
model + drafter paths /var/tmp staging ~/.cache/huggingface no disposable dir in the load path

1. The prefix-cache fix

Out of the box this stack logs enable_prefix_caching=True and then scores zero cache hits — every chat turn re-prefills the whole conversation (~77 s on a 100K session). Two vLLM coordinator bugs cause it:

  1. dflash counts as use_eagle(), but GLM never sets is_eagle_group (that annotator is DeepseekV4-only). vLLM's conservative fallback flags every KV group as EAGLE, and MLA drops its last scheduler-aligned block.
  2. The DFlash2 SlidingWindow group participates in the hybrid min(); after an EAGLE one-block pop it re-aligns down by a full scheduler page, wiping a longer MLA hit.

kv_cache_coordinator.hybrid-apc.py flags only the drafter's SlidingWindowSpec group and stops that group shrinking curr_hit_length. It deliberately never touches KpoolTailManager — wrong indexer tail state is fatal, and KpoolTail has already opted out of prefix caching.

Measured with a needle-in-context correctness test, not just counters:

prompt cold hits warm hits reuse needle
8K 0 4,608 57% ✅
100K 0 96,768 96.7% ✅

Expected log line after boot:

hybrid APC groups: [('MLAAttentionSpec', [0], 'FullAttentionManager', False),
                    ('MambaSpec', [2,3,4,5], 'MambaManager', False),
                    ('SlidingWindowSpec', [6], 'SlidingWindowManager', True)]
eagle_group_ids=[6]

Only group 6 flagged. The KpoolTailManager warning still appears and is expected — it means hits are block-aligned (4608) rather than fine-grained.

Origin: ported from overlay/patch_hybrid_prefix_hit.py in MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks, which solved it first for the EXL3 stack. Both run the same vLLM build (0.1.dev20051+g487ecf187); all four patch anchors matched exactly once, and the patch fails closed if the coordinator drifts.

2. Per-rank RoCEv2 GID auto-detection

A hardcoded NCCL_IB_GID_INDEX is a time bomb: GID tables reorder across reboots and link events, and the correct index often differs per node. When it goes stale the boot dies ~40 s in, before any weights load:

NCCL WARN Call to ibv_modify_qp failed with 22 Invalid argument, on dev rocep1s0f0:1
RuntimeError: NCCL error: unhandled system error

Observed on this pair after a reboot — index 2 became RoCE v1 on the head and an all-zero entry on the worker, while the launcher demanded NCCL_IB_ROCE_VERSION_NUM=2:

gid2 gid3 gid4
head 10.10.10.1 ::ffff:10.10.10.1 RoCE v1 ::ffff:10.10.10.1 v2 ✅ —
worker 10.10.10.2 all zeros ❌ ::ffff:10.10.10.2 v1 ::ffff:10.10.10.2 v2 ✅

The launcher now picks, per rank, the index whose GID carries that node's own ::ffff:<ip> with type RoCE v2, printing e.g. gid: rank=0 ip=10.10.10.1 dev=rocep1s0f0 -> RoCEv2 gid index 3 (auto). Override with GID_INDEX=N; it fails fast with the inspection commands if no RoCEv2 GID exists.

3. Everything resolves from the HF cache

Earlier revisions staged weights into /var/tmp/glm-5.3-flash-ablit-nvfp4 and the drafter into /var/tmp/models/. /var/tmp is a disposable directory -- the HF cache is the durable original -- so the launcher now mounts ~/.cache/huggingface at /hf (read-only) and resolves each repo's snapshots/<rev> from refs/main at launch.

Mounting the whole hub dir is required: snapshot entries are symlinks (snapshots/<rev>/x -> ../../blobs/<sha>), so a narrower mount leaves them dangling inside the container.

The chat template is not in the weights repo, so it is bind-mounted separately from this recipe to /opt/glm53/.

Override with HF_CACHE, MODEL_REPO, DRAFT_REPO.

Components

what where
weights (abliterated NVFP4) drowzeys/keys-GLM-5.3-Flash-NVFP4-ablit-l15-45-anchorstock
drafter (fc recalibrated) gorbatjovy/GLM-5.3-Flash-DFlash2-ablit-fc
deployment scripts this repo

Files

file purpose
prep-and-launch.sh entry point — verifies livelock mitigations, drops page cache, launches both ranks
launch-glm53-ablit-tp2-dflash2.sh per-rank launcher (worker rank 1 first, then head rank 0)
fleet.sh two-node control: up / down / status / wait / logs
kv_cache_coordinator.hybrid-apc.py patched vLLM coordinator; bind-mounted read-only over the container's copy
chat_template_ablit_mm.jinja the abliterated repo's template + an enable_thinking branch; must sit beside the weights
apc_test.py prefix-cache correctness + hit test (needle in context)

Usage 

./prep-and-launch.sh     # start both ranks (~15 min cold load, 90.67 GiB/rank)
./fleet.sh wait
./fleet.sh status
./fleet.sh down

Verify the prefix cache actually works:

GLM53_BASE=http://127.0.0.1:8000 python3 apc_test.py
GLM53_BASE=http://127.0.0.1:8000 APC_TARGET=100000 python3 apc_test.py

A rising hit count with a wrong answer is worse than no cache — which is why this checks the model still recalls a unique needle from the cached prefix, not just the counters.

Hard-won operational notes

  • Never exceed a 6 GiB KV pin. Above ~6 GiB/rank the reservation can succeed while physical backing is absent; the first real touch during warmup kills the worker with exit code None — no exception, no NVRM line, no oom-kill. 5.5 and 6.0 GiB verified stable.
  • --max-num-batched-tokens must stay ≥ 2048, or GLM-5.3-Flash segfaults in warmup (the DSA indexer's index_topk is 2048).
  • Always launch via prep-and-launch.sh. This stack can hit a GB10 UVM livelock on cold load — an unrecoverable spin needing a physical power cycle. The script verifies swap is present and vm.swappiness=0 on both ranks and drops the page cache immediately before the load.
  • CUDA graphs are flat on TP2 (mean ≈ −0.3%), so --enforce-eager stays. They pay at TP4, where each rank holds ¼ of the model and launch overhead is a larger share of a shorter step.
  • k=7 below concurrency 4, k=5 above it. On a 288-expert MoE, verify compute grows super-linearly in k. Judge by mean accepted length, not acceptance ratio — the ratio rises simply because you stopped drafting the low-probability tail.
  • The KV pool scales with context, not just the pin: 6 GiB gives 554,865 tokens at CTX=204,800 but 776,666 at CTX=500,000.
  • Thinking defaults to ON at Reasoning Effort: Max. With a small max_tokens the budget can be spent entirely on reasoning, returning an empty reply. Clients control it per request: chat_template_kwargs {"enable_thinking": false} or {"reasoning_effort": "low"|"high"}.

Measured on 2× DGX Spark (GB10), TP=2

Cold prefill (unique salt per rung, TTFT = first content token):

prompt tok/s
~12K 833
~16K 937
~100K 1,519
~256K 1,463
~300K 1,510

Decode: structured 67.5 tok/s (0.978 acceptance / 6.84 accepted per step), prose 28.7 (0.352 / 2.47). Vision verified (image described + embedded text read back).

Prose is entropy-bound rather than a tuning problem — see the per-domain analysis in the drafter repo.

Credits

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support