diffbot/DeepSeek-V4.1-Flash-EXL3-2.0bpw-2x-RTX-PRO-6000

DeepSeek-V4.1-Flash EXL3 2.0 bpw for 2× RTX PRO 6000 Blackwell

A 2.0 bpw EXL3 quantization of DeepSeek-V4.1-Flash's routed experts that serves on two RTX PRO 6000 Blackwell cards (sm_120, 96 GB each) with vLLM at tensor-parallel 2. The recipe/ folder has everything needed to run it: sm_120 fixes for vLLM and FlashInfer, a faster EXL3 MoE kernel, and image build, serving, benchmark and quantization scripts. It adapts sfxnz's 2× DGX Spark recipe to x86 workstation Blackwell.

Benchmarks

Two RTX PRO 6000 Blackwell Max-Q at their 300 W cap, PCIe, TP2, 8 GiB KV cache, 512K context window, CUDA graphs and DSpark speculative decoding. A ~2.5 GB/GPU side process stayed resident during every run. recipe/bench/bench-sbs.py sends non-streaming requests with a synthetic prompt; decode speed is the difference between a 400-token and a 16-token reply on a warm prefix cache.

Configuration	Decode, 2K, 1 stream	Decode, 2K, 4 streams	Prefill, 46K, 1 stream	Decode, 46K, 1 stream	Prefill, 46K, 4 streams	GSM8K-200
sm_120 prefill fixes only (eager, Engram on NVMe)			1,805	31.5		97.5%
+ CUDA graphs, Engram in pinned RAM			4,026	68.0		97.5%
+ Marlin dense GEMMs	101.6	210.2	3,804	90.5	7,617	97.0%
+ multi-row prefill MoE kernel	98.9	219.7	4,091	96.7	8,180	98.0%
+ flat decode MoE scheduler (default)	113.3	277.2	4,044	116.6	8,085	98.5%
+ hybrid dense GEMM (opt-in)	114.1	274.4	4,260	114.9	8,503	95.5%

Throughput is tokens per second, aggregate across streams. Quality of this pack with thinking off: GSM8K-200 97.5% and HumanEval 91.5%; the GSM8K column is a single run per configuration.

On a 40-minute run of a real agentic coding workload, the default configuration served 184 requests and 18.5M prompt tokens. 87% of prompt tokens came from the prefix cache, decode ran at 240–254 tok/s across 3–4 streams, and there were no errors or preemptions.

MoE kernel alone

Per-call time of ExLlamaV3's exl3_moe against recipe/kernels/xmoe at this model's per-rank shapes (384 experts, hidden 5120, intermediate 1152, 2 bpw). Decode with DSpark at one stream issues 6-token calls.

Tokens per call	Stock	xmoe	Speedup, uniform routing	Speedup, skewed routing
6	523.6 µs	327.1 µs	1.60×	1.95×
12	771.3 µs	564.0 µs	1.37×	1.58×
24	1,482 µs	1,066 µs	1.39×	1.45×
48	2,502 µs	1,999 µs	1.25×	1.32×
1,542	9.00 ms	5.98 ms	1.51×
4,096	20.4 ms	14.3 ms	1.42×

In the server, the MoE share of a decode step fell from 16.1 ms to 8.9 ms, and the whole step from 28.9 ms to 23.5 ms. Full tables, profiles and Nsight Compute numbers are in recipe/results/RESULTS.md.

What's in this repo

Files	Contents
model-00003 to model-00042	The 40 MoE layers. Routed experts re-encoded as EXL3, 2 bits per weight, MCG codebook; attention and dense weights unchanged (FP8).
model-00001, -00002, -00043	Vision aligner, embeddings and LM head, unchanged.
model-00044 to model-00046	DSpark draft layers, unchanged; their experts stay at source precision.
model-00047, -00048	Engram tables of layers 1 and 14, unchanged (203 GB).
config.json	Adds quantization_config: quant_method: exl3, 2 bits, MCG; non-routed weights stay DeepSeek's FP8.
recipe/	Image build, serving, kernel, benchmark and quantization scripts.

The tokenizer, encoding/, inference/, evaluation/, assets/, the tech report and LICENSE are copied from DeepSeek's release. The download is 358 GB. Quantization was uncalibrated, with greedy trellis search at beam 16, the settings from sfxnz's recipe.

Quick start

You need two sm_120 GPUs with 96 GB each, a CUDA 13 driver, Docker with the NVIDIA container toolkit, and about 300 GB of free host RAM (the Engram tables are pinned, ~190 GiB).

hf download diffbot/DeepSeek-V4.1-Flash-EXL3-2.0bpw-2x-RTX-PRO-6000 --local-dir ~/models/DeepSeek-V4.1-Flash-EXL3-2.0bpw
cd ~/models/DeepSeek-V4.1-Flash-EXL3-2.0bpw/recipe
docker/build.sh                      # serving image dsv41-flash-exl3-sm120
kernels/build.sh                     # the xmoe MoE kernel, ~15 s
API_KEY=change-me serve/serve.sh     # OpenAI-compatible API on :8000, model deepseek-v4.1-flash

Smoke tests against the running server:

export API_KEY=change-me
python3 bench/needle-test.py 200 && python3 bench/test-toolcall.py

recipe/serve/serve.sh documents every knob, and recipe/README.md explains each sm_120 fix and the kernel design. To rebuild the pack from DeepSeek's checkpoint instead, use recipe/quantize/quantize.sh.

Limitations

• Text only. Vision needs a FlashInfer sm_120 decode dispatch for top-k 1152.
• The quality numbers come from GSM8K-200 and HumanEval probes, not a full evaluation.
• With four 46K prompts prefilling at once, decode for the other streams drops to about 30 tok/s aggregate.
• The serving stack is pinned to the vLLM deepseekv41-flash-0909 day-0 image, ExLlamaV3 5be88657 and vllm-exl3 d3cfd394.

Credits and license

Built on work by DeepSeek, sfxnz (the V4.1 EXL3 recipe), turboderp (ExLlamaV3), vcruz305 (vllm-exl3), and the FlashInfer and vLLM teams. The weights keep DeepSeek's MIT license (LICENSE). The code in recipe/ is MIT (recipe/LICENSE); see recipe/THIRD_PARTY_NOTICES.md for third-party code.