smalinin/DeepSeek-V4.1-Flash-GGUF

DeepSeek-V4.1-Flash GGUF

GGUF conversions and mixed-precision quantizations of deepseek-ai/DeepSeek-V4.1-Flash for the experimental DeepSeek-V4.1 runtime in smalinin/llama.cpp, branch my_build_deepseek41.

This is DeepSeek-V4.1-Flash (DeepseekV41ForCausalLM), not DeepSeek-V4-Flash-0731. The architecture includes sparse attention, hyper-connections, MoE layers, and two very large n-gram Engram lookup tables.

These files require the linked my_build_deepseek41 branch. Compatibility with stock upstream llama.cpp is not claimed. The GGUFs in this repository are text-only: the vision tower is not included.

Available and planned files

Variant	Repository state	First shard	Effective whole-model BPW	Total size
Q2_K backbone + Q5_K Engram	Available	Q2_K-Q5/DeepSeek-V4.1-Flash-EngramQ5-Q2_K-00001-of-00010.gguf	3.58	335.382 GB
IQ3_XS backbone + Q8_0 Engram	Available	IQ3_XS-Q8/DeepSeek-V4.1-Flash-IQ3_XS-Q8_0_Engram-imatrix-00001-of-00010.gguf	4.64	434.024 GB
MXFP4 conversion + Q8_0 Engram	Available	MXFP4/DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf	5.43	507.954 GB

Download every shard of the selected variant and pass only shard 00001-of-00010 to llama-server; the remaining shards are discovered automatically.

Precision and size breakdown

Variant	Primary backbone format	Nominal format BPW	Backbone and other data	Engram format	Engram BPW	Two Engram tables	Total	Effective total BPW
Q2_K + Q5_K Engram	Q2_K	2.625	200.210 GB	Q5_K	5.50	135.172 GB	335.382 GB / 312.35 GiB	3.58
IQ3_XS + Q8_0 Engram	IQ3_XS	3.30	225.122 GB	Q8_0	8.50	208.902 GB	434.024 GB / 404.22 GiB	4.64
MXFP4 + Q8_0 Engram	MXFP4	4.00	299.052 GB	Q8_0	8.50	208.902 GB	507.954 GB / 473.07 GiB	5.43

The size split is exact for the completed local ten-shard artifacts. “Two Engram tables” means only blk.1.engram_embd.weight and blk.14.engram_embd.weight. “Backbone and other data” is the total shard size minus those two tensor payloads, so it also includes embeddings, output and auxiliary tensors, metadata, alignment, and per-shard overhead.

The nominal BPW column describes the named storage format, not every tensor in the file. These are deliberately hybrid models: numerically sensitive tensors remain in higher precision, and K-quant presets may select different types for different matrices. Consequently, effective whole-model BPW is the useful number for comparing final download sizes.

The MXFP4 conversion is also mixed precision, not a pure 4-bit model and not a Q8-equivalent model. Its 1,046 tensors are stored as 120 MXFP4, 332 Q8_0, 65 BF16, and 529 F32 tensors. Both Engram tables are Q8_0.

Which variant should I use?

• Q2_K + Q5_K Engram is the smallest release. It is the practical choice when storage, RAM, or aggregate GPU memory is the limiting factor.
• IQ3_XS + Q8_0 Engram is the planned middle tier. It uses the calibration importance matrix and preserves both Engram tables in Q8_0.
• MXFP4 + Q8_0 Engram is the conversion source and highest-precision tier in this repository. It avoids a second backbone quantization pass, but is substantially larger.

Lower BPW can increase repetition or otherwise reduce output quality. Evaluate the variants on your own prompts; the format name alone is not a quality guarantee.

Reproducible conversion

The source checkpoint used here is:

deepseek-ai/DeepSeek-V4.1-Flash@dba1be0a40aa45a94ad051997016db3960a90277

The accepted MXFP4 artifact was converted with the DeepSeek-V4.1 converter at smalinin/llama.cpp commit 4c7025b52339fec48d8e8bdfcea905db80130b9c. The examples below use local paths only as placeholders; change them for your system.

0. Build the required branch

git clone --branch my_build_deepseek41 \
  https://github.com/smalinin/llama.cpp.git
cd llama.cpp

cmake -S . -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j \
  --target llama-server llama-quantize llama-imatrix llama-gguf-split

python3 -m pip install -r requirements.txt

Use the CUDA/toolchain settings appropriate for your system. The commands in the following sections assume this checkout is assigned to LLAMA_CPP.

1. Create the MXFP4 GGUF

Install the converter requirements from the linked branch first, then run:

LLAMA_CPP=/path/to/llama.cpp
HF_MODEL=/path/to/DeepSeek-V4.1-Flash
OUT=/path/to/output

mkdir -p "$OUT/tmp"

TMPDIR="$OUT/tmp" \
/usr/bin/time -v python3 "$LLAMA_CPP/convert_hf_to_gguf.py" \
  "$HF_MODEL" \
  --outtype auto \
  --use-temp-file \
  --outfile "$OUT/DeepSeek-V4.1-Flash-MXFP4.gguf"

--outtype auto invokes the architecture-specific precision policy: supported official block-FP4 expert weights are written as MXFP4, while tensors requiring another representation are kept as Q8_0, BF16, or F32.

The conversion needs substantial temporary disk space and RAM. The reference run had approximately 193 GiB peak RSS and took about 2 h 59 min on the build machine.

2. Split the MXFP4 file

"$LLAMA_CPP/build/bin/llama-gguf-split" \
  --split \
  --split-max-size 48G \
  "$OUT/DeepSeek-V4.1-Flash-MXFP4.gguf" \
  "$OUT/DeepSeek-V4.1-Flash-MXFP4"

This produced ten shards. A single GGUF tensor cannot be split across files, so a shard containing one of the approximately 104.45 GB Q8_0 Engram tables can exceed the requested 48G limit.

Importance matrix

Both additional quants use the same 262,144-token calibration corpus and the published imatrix-dsv41-262144.gguf. The accepted run used 512 chunks:

"$LLAMA_CPP/build/bin/llama-imatrix" \
  --model "$OUT/DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf" \
  --file /path/to/calibration.txt \
  --output "$OUT/imatrix-dsv41-262144.gguf" \
  --output-format gguf \
  --ctx-size 512 \
  --batch-size 512 \
  --ubatch-size 128 \
  --parallel 1 \
  --chunks 512 \
  --output-frequency 512 \
  --save-frequency 128 \
  --no-ppl \
  --parse-special \
  --no-escape \
  --n-gpu-layers auto \
  --split-mode layer \
  --fit on \
  --fit-ctx 512 \
  --fit-target 2048 \
  --load-mode mmap \
  --lazy-mode auto \
  --n-cpu-moe 0 \
  --flash-attn on

Quantization commands

The quantizer reads the complete MXFP4 split set through its first shard. --keep-split preserves the shard layout. The output path below is a base GGUF name; the quantizer creates numbered shards.

Q2_K backbone + Q5_K Engram

"$LLAMA_CPP/build/bin/llama-quantize" \
  --allow-requantize \
  --keep-split \
  --max-buffer-size 8192 \
  --imatrix "$OUT/imatrix-dsv41-262144.gguf" \
  --tensor-type 'engram_embd\.weight=q5_k' \
  "$OUT/DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf" \
  "$OUT/DeepSeek-V4.1-Flash-EngramQ5-Q2_K.gguf" \
  Q2_K 20

IQ3_XS backbone + Q8_0 Engram

"$LLAMA_CPP/build/bin/llama-quantize" \
  --allow-requantize \
  --keep-split \
  --max-buffer-size 8192 \
  --imatrix "$OUT/imatrix-dsv41-262144.gguf" \
  --token-embedding-type q5_k \
  --tensor-type 'engram_embd\.weight=q8_0' \
  "$OUT/DeepSeek-V4.1-Flash-MXFP4-00001-of-00010.gguf" \
  "$OUT/DeepSeek-V4.1-Flash-IQ3_XS-Q8_0_Engram-imatrix.gguf" \
  IQ3_XS 20

The final positional argument is the worker-thread count; tune it for your machine. --allow-requantize is required because the source GGUF is already a mixed-precision conversion.

Running with llama-server

The following is the tested six-GPU profile for the Q2_K + Q5_K Engram release. Adjust the GPU order, paths, thread counts, and fit target for your hardware.

/path/to/llama.cpp/build/bin/llama-server \
  --model /path/to/Q2_K-Q5/DeepSeek-V4.1-Flash-EngramQ5-Q2_K-00001-of-00010.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 64000 \
  --batch-size 2048 \
  --ubatch-size 256 \
  --parallel 1 \
  --threads 4 \
  --n-gpu-layers auto \
  --split-mode layer \
  --fit on \
  --fit-ctx 64000 \
  --fit-target 1024 \
  --load-mode mmap \
  --lazy-mode auto \
  --flash-attn on \
  --n-cpu-moe 0 \
  --no-warmup \
  --no-context-shift \
  --jinja \
  --chat-template-kwargs '{"reasoning_effort":80,"enable_thinking":true}' \
  --reasoning-format deepseek \
  --no-reasoning-preserve \
  --no-prefill-assistant

Important launch parameters

Parameter	Purpose
--ctx-size 64000	Allocates a 64K-token context for the single server slot. A larger capacity increases cache and graph memory.
--batch-size 2048	Maximum logical prompt batch. It mainly affects prompt processing.
--ubatch-size 256	Physical micro-batch used to execute prompt work. Reduce it if graph memory is too high.
--parallel 1	Uses one server slot. With multiple slots, the configured context is divided among them.
--n-gpu-layers auto --split-mode layer	Lets the custom runtime place layers across the visible GPUs.
--fit on --fit-ctx 64000 --fit-target 2048	Runs the placement fitter for the requested context while targeting approximately 2 GiB of free VRAM per GPU.
--load-mode mmap --lazy-mode auto	Memory-maps the very large GGUF and allows the runtime to keep suitable tensors, notably Engram tables, lazily backed by host storage.
--flash-attn on	Enables the supported Flash Attention path.
--n-cpu-moe 0	Requests GPU placement for all MoE layers. Change this only when the model does not fit.
--no-warmup	Skips startup warm-up. Remove this option if you prefer to pay warm-up cost before the first real request.
--no-context-shift	Disables rolling context reuse. A request that exceeds the configured context must be rejected or shortened instead of silently shifting it.
--jinja --chat-template-file ...	Uses the supplied DeepSeek-V4.1 chat template. The template file is also included in this repository.
reasoning_effort	Template value from 1 to 100. Higher values request more thorough and usually longer reasoning; they do not change model weights.
--reasoning-format deepseek	Parses DeepSeek reasoning into the server's reasoning field.
--no-reasoning-preserve	Does not preserve extracted reasoning when assistant history is reconstructed.
--no-prefill-assistant	Disables assistant-message prefill.

The two Engram tables are intentionally compatible with mmap/lazy host-backed operation; total GGUF size is therefore not the same as required VRAM. Actual RAM and VRAM requirements still depend on context length, batch sizes, GPU topology, and placement selected by the fitter.

Optional DSpark sidecar

DeepSeek-V4.1-Flash-DSpark-AUTO.gguf is an experimental three-stage MTP speculative-decoding sidecar (approximately 7.97 GB). It is not enabled in the recommended command above: on the tested system it accepted draft tokens correctly but reduced end-to-end generation speed. If you want to experiment, add the following options and measure on your own workload:

  --spec-type draft-dspark \
  --spec-draft-model /path/to/DeepSeek-V4.1-Flash-DSpark-AUTO.gguf \
  --spec-draft-n-max 5 \
  --gpu-layers-draft auto

Scope and limitations

• Text generation is supported; the checkpoint's vision tower is not mapped into these GGUF files.
• The native one-million-token training context has not been claimed as a validated deployment configuration here. The documented production profile uses a 64K context.
• Q2_K is an aggressive quantization. Repetition or quality loss at this size is a model-quantization limitation, not necessarily a runtime failure.
• Performance depends heavily on memory placement, PCIe topology, context length, page-cache state, and the number and type of GPUs. No universal tokens-per-second figure is claimed by this card.

Samples

Q2_K + Q5_K Engram

IQ3_XS + Q8_0 Engram

MXFP4 Higher-precision sample

Acknowledgements

The conversion and runtime work builds on llama.cpp, the official DeepSeek checkpoint and reference implementation, and early DeepSeek-V4.1 conversion and runtime work by vcruz305 and JigSawPT.