JigSawPT/DeepSeek-V4.1-Flash-GGUF

DeepSeek-V4.1-Flash, GGUF with the engram tables

DeepSeek-V4.1-Flash (552B MoE, 40 layers, 384 routed experts, hyper-connections, CSA2 sparse attention, a 189 GiB n-gram conditional memory) converted for llama.cpp without touching the released precision: the routed experts are a lossless repack of the released MXFP4 blocks, attention and dense weights are dequantized from fp8 and stored as Q8_0/BF16, and the two engram tables travel as their raw fp8 bytes with their scales. 502 GB in 11 shards.

It runs only on the dsv41-porte branch of JigSawPT/llama.cpp. Upstream llama.cpp has no runtime for V4.1 yet; its open conversion PR (#28696) stores the engram tables differently, so this file is not interchangeable with files from that converter. Reconciling the two is part of the plan to upstream the branch.

What it needs, and what it gives

Measured on one RTX 5090 (31.8 GiB of VRAM) with 125.7 GiB of RAM and the file on a PCIe 5 NVMe:

	decode	time to first token
new content (our benchmark: 4 prompts of mixed content x 3 rounds)	5.1 tokens/s	7.7 s
resident content (the same prompt again)	21.4 tokens/s	0.26 s

The experts stream from disk through a VRAM cache (18 GiB) and a pinned host tier (72 GiB); the engram tables are memory-mapped and read 48 rows per token. Nothing here fits in RAM + VRAM and nothing has to. The ceiling of this architecture on this machine is 6.2 tokens/s without any disk miss; 21 tokens/s needs the working set on the card.

Exactness against the reference implementation: logit correlation 0.9967 at 1 401 tokens, equal to the port against itself across two runs (0.9959). The remaining gap sits at the rounding floor of the reference's own fp8 arithmetic. Details, method and every negative result: the report.

Run

llama-server -m DeepSeek-V4.1-Flash-MXFP4-engram-00001-of-00011.gguf -ngl 99 -c 8192 ^
  --moe-stream --moe-stream-cache 18 --moe-stream-l2 72 --reasoning off ^
  --host 127.0.0.1 --port 8080

All 11 shards go in the same directory; point -m at the first. --moe-stream-cache has a minimum of 18 slots per layer (13 GiB). --moe-stream-l2 above 72 GiB on a 125.7 GiB machine is slower: it steals page cache from the engram tables. --moe-stream-io-threads 1 makes runs bit-for-bit reproducible at 3.6 instead of 4.3 tokens/s. Chat mode (--reasoning off) is the measured configuration; thinking mode at temperature 0 loops on vague prompts.

The draft head is published separately (DeepSeek-V4.1-Flash-DSpark-GGUF); on this hardware it is neutral on mixed content.

Files

shard	GB	contents
-00001-of-00011	0.01	metadata, small tensors
-00002-of-00011	98.3	engram table, layer 3 (fp8, raw)
-00003-of-00011	3.1
-00004-of-00011	98.3	engram table, layer 13 (fp8, raw)
-00005 .. -00010	45.7–46.4 each	routed experts (MXFP4), attention, norms
-00011-of-00011	27.1

SHA256SUMS.txt covers all eleven. The 299 GB variant without the engram tables is not published: zeroing the memory is the exact identity of the module, but the model that comes out does not produce the released model's outputs.

How it was made

python convert_hf_to_gguf.py <DeepSeek-V4.1-Flash> --outtype bf16 --engram --outfile DeepSeek-V4.1-Flash-MXFP4-engram.gguf
llama-gguf-split --split --split-max-size 48G DeepSeek-V4.1-Flash-MXFP4-engram.gguf DeepSeek-V4.1-Flash-MXFP4-engram

From the released checkpoint, on the same branch. The MXFP4 repack was verified block by block (480/480 identical); the split was verified by loading the first shard and comparing the answer with the monolithic file.

Credits

DeepSeek for the model and the reference implementation (MIT). nibor1896 for Crow, whose expert-streaming patch series the branch builds on (MIT). ggml-org/llama.cpp. Engineering assisted by Claude (Anthropic).