Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw

DeepSeek-V4.1-Flash · EXL3 · 2.9bpw

EXL3 quantization of deepseek-ai/DeepSeek-V4.1-Flash, converted with exllamav3 v1.4.2.

This is not a drop-in replacement for the original FP8 checkpoint. The Engram embedding tables are not in this repo — you still need shards 47 and 48 from the base model (~95 GiB each).

A slightly denser sibling lives at Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-3.0bpw.

Base model	deepseek-ai/DeepSeek-V4.1-Flash
Format	EXL3 v1.4.2, codebook mul1, out_scales=always
Average bpw	2.90 (decoder; --hq)
Head bits	6
MTP / DSpark bits	4 (mtp_bits)
Size	~197 GB, 39 safetensor shards
Calibration	128 rows × 2048 cols (262144 tokens), workload trace cal_trace_dsv41_flash_workload.json
DSpark	In-checkpoint (dspark_block_size=5, 128 draft experts / top-3, layers 37–39)
Vision	Included (tower left native)
Engram tables	Not quantized. Use original shards 47 and 48

Engram tables (required)

EXL3 only quantized the Engram wkv linears. The FP8 row tables stay in the original DeepSeek tree:

• model-00047-of-00048.safetensors (~95 GiB)
• model-00048-of-00048.safetensors (~95 GiB)

Point the runtime at that directory. This card ships engram_table_dir as null on purpose — do not leave a machine-local path in config.json.

vLLM:

--quantization exl3 \
--hf-overrides '{"engram_table_dir":"/path/to/DeepSeek-V4.1-Flash"}'

A slimmer overlay is enough: hardlink or copy only shards 47–48 (plus an embed-only index if your stack wants one). You do not need the other 46 original shards once this EXL3 tree is loaded.

Download

hf download Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw \
  --local-dir ./DeepSeek-V4.1-Flash-EXL3-2.9bpw

hf download deepseek-ai/DeepSeek-V4.1-Flash \
  --include "model-00047-of-00048.safetensors" \
  --include "model-00048-of-00048.safetensors" \
  --local-dir ./DeepSeek-V4.1-Flash

Runtime

This checkpoint is meant for stacks that load EXL3 (trellis / suh / svh / mul1), not vanilla Transformers weight loaders.

• vLLM with EXL3 + DeepSeek-V4.1 support (--quantization exl3, DeepSeek V4.1 tokenizer / tool / reasoning parsers). Native DSpark: --speculative-config '{"method":"dspark","num_speculative_tokens":3}'.
• ExLlamaV3 / TabbyAPI once the architecture is supported in the build you are running.

Official sampling from the base card:

Parameter	Value
temperature	1.0
top_p	0.95
context	up to 1M tokens

Thinking defaults on. Disable per request with chat-template kwargs (enable_thinking=false) when you want a non-reasoning smoke.

KV cache is native V4.1 FP4 (~890 B/token). Do not force an unrelated fp8 KV dtype from other recipes.

Conversion

python convert.py \
  -i DeepSeek-V4.1-Flash \
  -o DeepSeek-V4.1-Flash-EXL3-2.9bpw \
  -w /path/to/work \
  -b 2.9 \
  -hb 6 \
  -hq \
  --codebook mul1 \
  -cd cal_trace_dsv41_flash_workload.json

Final reported decoder bitrate: 2.90 bpw (--hq enabled). MTP tensors were quantized at 4 bpw. Routed experts are a 2/3 mix (mostly K=3, a slice at K=2); attention / shared / Engram wkv sit higher (typically 4–6). Indexer wk is 8 bpw. Embeddings stay bf16.

Base model

DeepSeek-V4.1-Flash is a multimodal MoE (552B backbone, 8B/16B activated per token in prefill/decode) with CED, CSA2, Engram memory, and in-checkpoint DSpark. Architecture, evals, and prompt format live on the original model card and tech report. Scores there are for the unquantized model, not this EXL3 build.

License

MIT, inherited from deepseek-ai/DeepSeek-V4.1-Flash. Quantization adds no extra restrictions.

Citation

@misc{deepseekai2026deepseekv41flash,
      title={DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression},
      author={DeepSeek-AI},
      year={2026},
}

Quantized by Mia's AI Lab. Format and tooling: exllamav3 (turboderp).