---
license: other
license_name: deepseek
license_link: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
base_model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
base_model_relation: quantized
library_name: exllamav3
pipeline_tag: image-text-to-text
tags:
- exl3
- exllamav3
- deepseek_v4
- moe
- mixed-precision
- vllm
- blackwell
- lna-lab
language:
- en
- ja
- zh
---

# DSV4-Flash-Vision — EXL3 MixedK · D2 · K2×3 · Dense6 (380K context on 8 × 16 GB)

A serving-tuned derivative of [vcruz305/DSV4-Flash-Vision-EXL3-MixedK](https://huggingface.co/vcruz305/DSV4-Flash-Vision-EXL3-MixedK)
(DeepSeek-V4-Flash-Vision-Exp, 305B MoE, vision). Everything that was still BF16 or fp8 in the original has been carved into EXL3
so that a **380K-context, 4-stream, DSpark3-speculative** seat fits on eight RTX PRO 2000 16 GB under vLLM.
Baked and measured by **Lna-Lab** (YUKI) on 2026-09-03; house model of SAZANAMI ("オオタニ").

Recipe, patches and the serve script: **github.com/lna-lab/dsv4-carve** (mirrored in `lna-lab/` here).

## Composition

| Part | Here | Original MixedK |
|---|---|---|
| Routed experts | EXL3 2/3-bit mixed; layers 3, 21, 41 → K2 (wrldsuksgo2mars K2-v1) | EXL3 2/3-bit |
| MTP draft experts (3 layers, DSpark) | EXL3 K2 (wrldsuksgo2mars K2.2-D2-v1) | fp8 |
| Attention + shared experts (790 linears) | **EXL3 6-bit**, calibrated (250 × 2048) | BF16 |
| lm_head / embeddings / indexer weights_proj | BF16 (unchanged) | BF16 |

86 GB on disk, 51 shards. Per GPU at TP8: 12.0 GiB weights → **396,656 tokens** of fp8 KV at `--max-model-len 389120`.

## Measured (vLLM 0.28.1rc1.dev337 + vllm-exl3 0.2.3, TP8, CUDA graphs, RTX PRO 2000 × 8)

| | |
|---|---|
| ppl wikitext-2 (512 × 16) | **6.7159** — original MixedK 6.6271 (+1.34 %); 4-bit dense was +3.8 % and rejected |
| single stream, no speculation | en 42.1 / ja 42.0 / code 41.9 tok/s |
| single stream, DSpark3 (3 MTP tokens) | en 62 / ja 55 / code 86 tok/s |
| 4 streams, DSpark3, aggregate | en 136 / ja 116 / code 186 tok/s |
| needle at 166K tokens | found; TTFT 167 s (prefill ≈ 1.0k tok/s), decode at that depth 37 tok/s |
| vision | works (shapes, colours, text read correctly) |

## 2026-09-04 update — house kernel **lna2** for the routed experts

Same weights, same pack. The routed-experts decode kernel was replaced by a bespoke one (`lna-lab/lna2/`): resident expert teams with a dynamic ticket scheduler outside, small-R inner loops (1–8 rows per expert) inside — the incumbent's M=16 tile wastes 15/16 of its MMA when each expert sees ~1 row. Adopted as the house kernel by Ken on 2026-09-04.

| (TP8, DSpark3, CUDA graphs, RTX PRO 2000 × 8) | exllamav3 `exl3_moe` | **lna2** |
|---|---|---|
| routed kernel per launch | 232.6 µs | **104.0 µs** (2.24×) |
| single stream code / en / ja | 86 / 60 / 56 tok/s | **108.6 / 78.6 / 71.1** |
| 4 streams code / en / ja | 125–179 / 96 / 83 | 193.8 / 146.4 / 129.0 |
| KV window @ `--gpu-memory-utilization 0.97` | 396,656 tok | 395,069 |
| needle @166k | found, TTFT 167 s | found, **119.6 s** |
| ppl (wikitext-2, 4k ctx, speculation off, matched control) | 4.7630 | 4.7647 |

Launch = the recipe above plus `NATIVE_SO=lna-lab/lna2/vllm_exl3_c.cpython-312-x86_64-linux-gnu.so MOE_KERNEL=lna2` and `-e VLLM_EXL3_MOE_STRICT=1`; the seat log must show `LNA2 MoE kernel ACTIVE` on every rank (the first delivery silently fell back to the incumbent and "passed" its gates — hence strict mode). Desk parity 171 cases (rel ≤ 1.64e-3, bitwise repeatable); a null-decode variant of the kernel pins the card at its measured 270 GB/s cold-read ceiling, so the remaining ceiling is the trellis decoder itself. Canon: `LNA-CANON.md`. Design reviews and every gate report: `lna-lab/orders/`. Code: github.com/lna-lab/dsv4-carve.

## You need the patches

Stock exllamav3 1.4.5 / vllm-exl3 0.2.3 will **not** serve this pack: dense EXL3 linears in attention trigger (a) the cooperative-GEMM
autotuner inside CUDA-graph capture and (b) a two-stream deadlock between vLLM's shared-experts stream and the routed EXL3 kernels
(both use exllamav3's per-device lock buffer). The `lna-lab/` folder carries the patched sources, the vLLM patches, the prebuilt
extension for sm_120 (`exllamav3_ext.cpython-312-x86_64-linux-gnu.so`, torch cu13) and the serve script; the README there explains each.
Minimum: `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1`, `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128`, aux streams off, `--kv-cache-dtype fp8`.

```bash
AUX_STREAMS=0 MODEL=. UTIL=0.97 MAXLEN=389120 BT=512 SEQS=4 \
SPEC='{"method":"dspark","num_speculative_tokens":3}' bash lna-lab/serve-dsv4-tp8.sh
```

Thinking is on by default; pass `chat_template_kwargs: {"thinking": false}` or a large `max_tokens`.

## Files

- `model-*.safetensors`, `model.safetensors.index.json`, `config.json` (`quantization_config.non_routed_exl3` lists the 6-bit dense modules), tokenizer, `generation_config.json`
- `lna-lab/` — Dockerfile, serve/bench/ppl scripts, `tools/` (overlay + dense bake), `patches/`, prebuilt extension, upstream model card

## Credits and license

- Base weights: DeepSeek (MIT). This pack follows the DeepSeek model license like the upstream MixedK.
- MixedK, vllm-exl3 and its DSV4 patches: [vcruz305](https://huggingface.co/vcruz305) · K2 packs: [wrldsuksgo2mars](https://huggingface.co/wrldsuksgo2mars) · ExLlamaV3: turboderp · vLLM
- Bake, patches, measurements: Lna-Lab / YUKI with Ken ([@Tono_Ken3](https://x.com/Tono_Ken3))