Runs on one RTX 5090 (31.8 GiB) + 125.7 GiB RAM via a llama.cpp fork: GGUF, report and numbers
If you want to run this model on a single consumer card: I ported it to a llama.cpp fork and measured what limits it. It runs exactly — logits within the reference implementation's own fp8 rounding floor — at 5.1 tokens/s on new content and 21 tokens/s on content already resident in the cache, on an RTX 5090 (31.8 GiB of VRAM) with 125.7 GiB of RAM and a PCIe 5 NVMe.
How. The 269 GiB of routed experts stream from disk through a VRAM cache (18 GiB) and a pinned RAM tier (72 GiB); the 189 GiB engram tables stay memory-mapped on disk and are read by the host. Nothing fits in RAM + VRAM and nothing has to. The routed experts are a lossless repack of the released MXFP4 blocks (verified byte for byte), attention and dense weights are dequantized from fp8, and the engram tables travel as their raw fp8 bytes.
Where the token goes: 20 % compute, 26 % PCIe, 54 % NVMe. The disk is at 43 % utilisation; the limit is that only ~3.8 disk requests exist in flight per layer, because a layer's requests can only leave after its router has run. Ceiling with zero disk misses: 6.2 tokens/s; with everything resident, 21.
What didn't pay (measured, not guessed): a prefetch oracle with perfect knowledge of future routing gives +30 % but only 5 tokens ahead — at the depths a layer predictor can reach it loses; an n-gram draft is −33 %; the model's own DSpark draft head hits 51–97 % acceptance and is still neutral, because on a disk-bound machine a verification step pays for the union of experts of the block; encoder-only prefill isn't exact below 2 432 tokens. Also measured flat: more VRAM cache (13 → 22 GiB), a larger RAM tier (72 → 88 GiB, actually slower), and a smarter eviction policy (LRU against the optimal Belady policy: 0 % margin). A faster NVMe I did not test; the disk sits at 43 % utilisation, so I don't expect it to help.
One real defect found on the way: V4 normalises each attention head of q after wq_b; V4.1 does not. The inherited line never raised an error (norm 181.019 = √(64×512) gave it away).
- GGUF with the engram tables (502 GB, 11 shards): https://huggingface.co/JigSawPT/DeepSeek-V4.1-Flash-GGUF
- Draft head GGUF (8 GB): https://huggingface.co/JigSawPT/DeepSeek-V4.1-Flash-DSpark-GGUF
- Fork branch: https://github.com/JigSawPT/llama.cpp/tree/dsv41-porte
- Report, tools, raw results (MIT, DOI 10.5281/zenodo.22711369): https://github.com/JigSawPT/deepseek-v41-flash-on-5090
- llama.cpp discussion: https://github.com/ggml-org/llama.cpp/discussions/28766
Command:
llama-server -m DeepSeek-V4.1-Flash-MXFP4-engram-00001-of-00011.gguf -ngl 99 -c 8192 --moe-stream --moe-stream-cache 18 --moe-stream-l2 72 --reasoning off
Caveats: the GGUF runs only on the fork (upstream llama.cpp has no V4.1 runtime yet, and its open conversion PR stores the engram differently); Linux is untested on this branch; build instructions (Windows, CUDA 13.0) are in the branch README. Thanks to DeepSeek for the model and the reference implementation, which made the exactness checks possible.
Thanks!
Regarding: "20 % compute, 26 % PCIe, 54 % NVMe."
What does it mean exactly ? 26% of the time is spent on weights transfer to the GPU via PCIE-5.0 x16 ?
"Ceiling with zero disk misses: 6.2 tokens/s"
So i.e. You mean with 100% memory misses?
If 54% is NVMe reads , then it’s the bottleneck, right? What if engram tables would be stored on a separate NVMe or model layers split between two or more disks?