Qwen3.8-Flash-Next-W4A16-Pruned-294E

A ready-to-use, load-profiled expert-pruned build of Qwen3.8-Flash-Next: 512 → 294 experts per layer, -43% expert memory, quality held at baseline (quiz + agentic bug-hunt gates). For single-GPU serving where the full model's expert pool is the bandwidth bottleneck.

Derived from Qwen/Qwen3.8-Flash-Next (via Intel/Qwen3.8-Flash-Next-W4A16-AutoRound). Released under the Qwen Community License 1.0.

Quick start

# any SGLang build with Flash-Next support; 80GB+ GPU recommended
sglang launch_server --model-path <this-repo> \
  --ple-offload-embedding --moe-a2a-backend none \
  --linear-attn-prefill-backend triton --linear-attn-decode-backend triton \
  --mamba-ssm-dtype bfloat16 --context-length 262144 \
  --mem-fraction-static 0.93 --max-running-requests 96

Notes:

• The 47.7GB PLE embedding stays in host pinned memory (--ple-offload-embedding); PCIe traffic is <0.1% at steady state.

Size & memory requirements

item	size
repo download	98 GB (23 shards + PLE)
weights on GPU	~44 GB (experts int4 ~15GB + dense/GDN/BF16 ~29GB)
PLE table (host pinned)	47.7 GB
host RAM (load peak)	~110 GB free needed during startup; 128 GB+ recommended

Minimum GPU to serve:

GPU	verdict
48 GB	does not fit (44GB weights + CUDA context ≈ 48.4GB, no room for KV)
64 GB	minimum viable — small KV pool (~150K tokens), tight
80 GB	comfortable (~500K token KV pool)
96 GB	full config — 928K token KV pool, concurrent 71 (our setup)

• On large expert batches the Marlin repack path can OOM at load (int32 staging + delayed GC). If that hits, apply the gc.collect() after process_weights_after_loading patch in gptq_kernels.py.
• This keep-map is tuned for coding-agent + long-structured-output traffic. If your workload is very different (e.g. translation, math -heavy), expect some quality drift — the toolkit below lets you prune for your own profile instead.

Why

Flash-Next routes top-k=10 of 512 experts per token, so single-stream speed is independent of the expert pool size — but under concurrency the pool is read almost fully every step (bandwidth-bound). Shrinking the pool while keeping your workload's hot experts cuts per-step traffic proportionally.

config	single-stream	w32	w64 (tok/s)	quality gate
W4A16 full (512E)	110	—	~1000	baseline
256E, stale profile	110	1448	2264	FAIL (logic errors)
288E, mixed profile	110	1375	2124	pass (variance band)
294E (this repo)	110	1375	2171	pass + bug-hunt suite

Measured on a single RTX Pro 6000 Blackwell (96GB), SGLang, cuda graphs on, --ple-offload-embedding.

Method (3 steps)

1. Profile — run the server with --expert-distribution-recorder-mode stat and feed it a corpus that mirrors your real traffic (feed_profile.py). Dump gives per-layer expert routing counts.
2. Slice — slice_experts.py --keep <map.json> performs key-level surgery: packed GPTQ tensors are never unpacked (experts are whole keys), router gate rows are reindexed, config.num_experts rewritten. ~40 min on NVMe, no GPU required.
3. Validate — two gates before trusting a prune:
   • capability quiz (~30 mixed questions incl. domains you care about),
   • agentic bug-hunt (give the model a real repo with a real known bug; compare the found-defect set against ground truth).

The one lesson that matters

Your profiling corpus must cover your target load's task shapes. We hit this three times:

prune	profile corpus	failure mode
p256	day-to-day agent traffic only	math/logic experts starved → basic reasoning errors
p288	+ hand-written math/logic prompts	JSON-heavy long outputs degraded (missing } in structured output, 1/8 vs 0/20 full)
p294	+ JSON-structured compression tasks	clean on all gates (0/14 malformed, reasoning intact)

Routing statistics are cheap to collect — rerun the profile whenever your traffic mix shifts, and re-check both gates. The keep maps shipped here are tuned for coding-agent + long-structured-output traffic; a different workload should regenerate its own (tail layers L42–47 are where structured output and reasoning experts concentrate, see keep_maps/json_suspect_summary.json).

Contents

• scripts/slice_experts.py — checkpoint surgery (keep-map → pruned dir)
• scripts/feed_profile.py — corpus → router distribution dump
• scripts/json_task_prompts.json — 24 structured-output probe prompts (the class of traffic that exposed p288's blind spot)
• keep_maps/keep288.json, keep_maps/keep294.json — our maps (agent + math/logic [+ JSON] profiles), expert IDs per layer
• keep_maps/json_suspect_summary.json — per-layer routed-traffic share of pruned experts under JSON load (the evidence that located p288's failure)

Runtime notes (SGLang)

• W4A16 MoE on large batches triggers a memory spike in the Marlin repack path (int32 staging buffers + delayed GC); if you OOM at load, apply the gc.collect() after process_weights_after_loading patch (see repo docs).
• QSA (quantized sparse attention) + --enable-mixed-chunk requires the pending fix for decode-row prefix alignment; without mixed-chunk you are fine.
• --ple-offload-embedding keeps the 47.7GB PLE table in host pinned memory (PCIe traffic is <0.1% at steady state — non-issue).

Attribution & license

Chain: Qwen/Qwen3.8-Flash-Next → Intel/Qwen3.8-Flash-Next-W4A16-AutoRound → this toolkit's method. This repo ships scripts and keep maps only (no weights). Qwen Community License 1.0 — see LICENSE.