prometheusAIR/DeepSeek-V4-Flash-Vision-Exp-GGUF

DeepSeek-V4-Flash-Vision-Exp — GGUF (multimodal, imatrix)

Imatrix GGUFs of deepseek-ai/DeepSeek-V4-Flash-Vision-Exp that actually do vision — four rungs from 66.9 to 108.7 GiB. It also ships the importance matrix all four rungs were built with.

⚠️ Build llama.cpp from master — 2026-09-02 or later

Upstream support landed in two halves on 2026-09-02: #28133 (vision tower) and #28154 (image-token routing). You need both. Release build b10763 carries only the first and will refuse to load these files; build master at or after 9400c894. No patch is needed any more — see Why a recent build.

Files

Four rungs, all built from the same importance matrix, which is included.

rung	size	bits/weight¹	notes
IQ1_M	66.9 GiB	~1.9	smallest; most aggressive, least validated
IQ2_XXS	78.8 GiB	~2.2	fits one 96 GB card at full 1M context
IQ2_S	95.8 GiB	~2.7
IQ3_XXS	108.7 GiB	~3.1	highest quality here

Shard naming is DeepSeek-V4-Flash-Vision-Exp-<RUNG>-0000{1..4}-of-00004.gguf.

also included	size
mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf	890 MiB	vision tower — required for images, works with any rung
DeepSeek-V4-Flash-Vision-Exp-imatrix.gguf	470 MB	the importance matrix all four were built with

¹ Effective average over the whole file, not the expert type alone. The rung name is the dominant expert type; every rung mixes several — see Recipes. Note general.file_type reads Q6_K in all of them: that field records the base type passed to llama-quantize, and cannot express per-tensor overrides.

Will this run on my GPU?

Weights only — add ~13 GiB for KV/compute at a full 1M context, or far less at shorter contexts. Keep the vision tower on CPU with --no-mmproj-offload unless you have room; it is a one-off cost per image (466M params over ≤384 visual tokens).

your VRAM	rung
96 GB (1×)	IQ2_XXS at full 1M ctx (measured 96.9 GB peak), or IQ2_S/IQ3_XXS at reduced ctx
80 GB (1×)	IQ1_M, or IQ2_XXS with a reduced --ctx-size
64 GB (1×)	IQ1_M at reduced ctx
128–192 GB (2×)	IQ3_XXS comfortably
less, or CPU	any rung with --n-cpu-moe / -ngl tuning; CPU-only works, slowly

Build

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
# vision support merged 2026-09-02; no tagged release contains all of it yet,
# so build master (or check out 9400c894 or later explicitly)
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-mtmd-cli

Run

Substitute your chosen rung for IQ2_XXS below.

./build/bin/llama-server \
  --model DeepSeek-V4-Flash-Vision-Exp-IQ2_XXS-00001-of-00004.gguf \
  --mmproj mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf --no-mmproj-offload \
  -ngl 999 --ctx-size 1048576 -fa on -b 2048 -ub 512 -np 1 --no-mmap --jinja

-ub 512 is needed to reach 1M context on a 96 GB card. -ub 2048 roughly doubles prefill but adds ~4.2 GB of compute buffer, which will not fit alongside a speculative draft; drop the draft if you want the larger ubatch.

Quick check with llama-mtmd-cli:

./build/bin/llama-mtmd-cli \
  -m DeepSeek-V4-Flash-Vision-Exp-IQ2_XXS-00001-of-00004.gguf \
  --mmproj mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf \
  --image photo.jpg -p "What is in this image?"

Why a recent build

DeepSeek-V4-Flash-Vision carries a second MoE router bias, ffn.gate.bias_vl, used only for image tokens. The reference implementation selects experts with

scores + torch.where(input_ids >= vocab_size, bias_vl, bias)

and on the three hash-routed layers image tokens abandon the tid2eid lookup entirely for a learned top-k over scores + bias_vl.

That is the whole reason for the version floor, and it shows up as two distinct failures on an older build:

1. The tensor is not in the architecture yet. These files keep it, as blk.N.exp_probs_b_vl.bias, so any llama.cpp before #28154 reports wrong number of tensors; expected 1371, got 1328 and refuses to load. Note that converters which simply drop bias_vl load anywhere — and route image tokens by text preferences. Measured across all 43 layers the two biases are uncorrelated (mean Pearson r = −0.026), so they encode genuinely different expert choices.

2. #28133 on its own aborts on the first image. An image arrives as an embeddings batch, and llm_graph_input_embd::set_input only writes the token tensor when ubatch->token is set — so the hash layers gather tid2eid[t_inp_tokens] from uninitialised memory and hit GGML_ASSERT(i01 >= 0 && i01 < ne01). #28133 touches no file under src/; #28154 is the half that fixes it.

#28154 adds FFN_EXP_PROBS_B_VL to the deepseek4 architecture and selects the bias per batch. Because llama.cpp delivers an image as its own homogeneous embeddings batch — every position in it is an image token, markers included — that is exact for the batches mtmd produces, not an approximation.

Recipes

All four share the same non-expert treatment: attention, shared experts and dense at Q6_K; token_embd and output at Q8_0; routers (ffn_gate_inp, exp_probs_b, exp_probs_b_vl) at BF16/F32. They differ only in the routed experts:

rung	ffn_gate_exps, ffn_up_exps	ffn_down_exps blk 0–9, 33–42	ffn_down_exps blk 10–32
IQ1_M	IQ1_M	IQ2_XS	IQ2_XXS
IQ2_XXS	IQ2_XXS	IQ3_XXS	IQ2_XS
IQ2_S	IQ2_S	IQ3_S	IQ3_XXS
IQ3_XXS	IQ3_XXS	IQ3_S	IQ3_S

ffn_down gets the higher rung on the first ten and last ten blocks because that is where quantization damage concentrates.

The importance matrix covers 606,208 tokens (74 × 8192-token chunks) with full expert coverage — no expert went uncalibrated, which matters for a 256-expert MoE.

FP8 source weights were dequantized to BF16, not Q8_0 — BF16 represents the E4M3 significand exactly once the power-of-two E8M0 block scale is applied, so this avoids a second lossy step on 365 attention and shared-expert tensors.

Known limitations

• In-span attention visibility is brand new and unmeasured here. The reference lets tokens inside an [IMAGE_START, IMAGE_END] span see each other bidirectionally and feeds that to the DFlash sparse-attention indexer. Upstream implements it in #28154 (swa_full_non_causal), which landed after the throughput figures below were taken on a build that fell back to the causal window. Larger images, multiple images and long contexts remain unmeasured either way.
• No KL-divergence ladder has been measured. The rungs are ordered by bits, which is the expected ordering, but this repo makes no measured quality claim between them. IQ1_M in particular is aggressive and unvalidated behaviourally.
• Testing so far is single-image. Multi-image and interleaved prompts are untested.
• This is an experimental upstream model, and its llama.cpp support merged on 2026-09-02. Expect the runtime side to keep moving.

Verification

Re-checked 2026-09-02 against stock llama.cpp master (9cc33944f) with no patch of any kind. Both locally-held rungs answer image prompts correctly, on the CPU backend and with CUDA offload. Prompts used DeepSeek's own example images.

rung	image	answer
IQ2_XXS	carrots.jpeg	"Four orange carrots with green stems are arranged in a pile against a white background."
IQ2_XXS	corn.jpeg	"This image displays three ears of fresh corn on a plain white background, with one ear of corn partially husked to reveal its bright yellow kernels."
IQ1_M	carrots.jpeg	"The image depicts a group of four fresh, orange carrots with green stems, arranged in a pile on a white background."

Throughput was measured earlier on a single RTX PRO 6000 Blackwell (96 GB), fully resident at 1M context, with a Q2_K DSpark speculative draft — on the pre-merge patched build, so treat it as indicative rather than a measurement of the code above:

profile	prefill	decode
image, -ub 512, Q2_K DSpark draft	135.8 tok/s	129.4 tok/s (63.9% draft acceptance)
text, same	177.9	140.9 (68.2%)

License

MIT, inherited from the base model.