lna-lab/README.md

# dsv4-carve — DeepSeek-V4-Flash-Vision (305B) EXL3 on 8 × 16 GB, 380K context, DSpark3

Lna-Lab's serving recipe for [vcruz305/DSV4-Flash-Vision-EXL3-MixedK](https://huggingface.co/vcruz305/DSV4-Flash-Vision-EXL3-MixedK)
carved down until a **380K-context, 4-stream, speculative (DSpark3) seat** fits on eight RTX PRO 2000 (16 GB, Blackwell sm_120)
under vLLM. Adopted as the house model of SAZANAMI on 2026-09-03 ("オオタニ").

Weights (86 GB): `sakamakismile/DSV4-Flash-Vision-EXL3-MixedK-D2-K2x3-Dense6`.

## What is in the pack

| Part | Format | Source |
|---|---|---|
| Routed experts (43 layers) | EXL3 2/3 bit mixed, layers 3/21/41 replaced by K2 | vcruz305 MixedK + wrldsuksgo2mars K2-v1 (`tools/layer_overlay.py`) |
| MTP draft experts (3 layers) | EXL3 K2 (was fp8) | wrldsuksgo2mars K2.2-D2-v1 (`tools/mtp_overlay.py`) |
| Attention + shared experts (790 dense linears) | **EXL3 6 bit** (was BF16), calibrated 250 × 2048 | baked here (`tools/dense_bake.py`) |
| KV cache | fp8 (≈17 KB/token incl. draft) | vLLM |

Per-GPU (TP8): weights 12.0 GiB, KV pool **396,656 tokens** at `--max-model-len 389120`.

## Measured (2026-09-03, TP8, CUDA graphs on)

| | tok/s |
|---|---|
| ppl (wikitext-2, 512 × 16) | **6.7159** (Cruz original 6.6271, +1.34 %) |
| single stream, no speculation | en 42.1 / ja 42.0 / code 41.9 |
| single stream, DSpark3 | en 62 / ja 55 / code 86 |
| 4 streams, DSpark3 (aggregate) | en 136 / ja 116 / code 186 |
| needle at 166K tokens | found; TTFT 167 s (prefill ≈ 1.0k tok/s), decode at depth 37 |

2 streams cost the same wall time as 4 (graph capture sizes + EXL3 small-batch bucketing): run 1 or 4.

## The three traps (all fixed in this repo)

1. **`dense_bake.py --merge` stamped `bits: 4` into `config.json` for a 6-bit bake** → shape mismatch at load. Merge now reads bits from `work/args.json`.
2. **exllamav3's cooperative-GEMM autotuner runs inside CUDA-graph capture** (`coop_autotune.cu:464`, "operation not permitted when stream is capturing"). Patch in `recipe-lna/exllamav3/exl3_gemm.cu.lna`: skip tuning while `cudaStreamIsCapturing`, fall back to the static heuristic; plus the plugin pre-tunes decode row counts (1,2,4,8,16) right after weight load (`LNA_EXL3_PREWARM_ROWS`).
3. **Decode deadlock, GPU 100 % on all ranks.** vLLM's FusedMoE overlaps *shared experts on a second stream* for ≤ 256 tokens. With shared experts also EXL3, two cooperative kernels share exllamav3's per-device lock buffer (`DevCtx::get_locks`) from two streams and spin forever. Diagnosed with `CUDA_LAUNCH_BLOCKING=1` (no hang → concurrency) and py-spy. Fix: **`VLLM_DISABLE_SHARED_EXPERTS_STREAM=1`** (default in `serve-dsv4-tp8.sh`).

## 2026-09-04 — house kernel **lna2** (routed experts), adopted by Ken

The routed-experts decode kernel was replaced by a bespoke one (`recipe-lna/lna2/`, the "Ferrari" doctrine: one kernel for one model on one machine).
Resident expert teams + dynamic tickets (no grid-wide barrier) outside, small-R inner loops (rows per expert 1–8) inside.

| | exllamav3 `exl3_moe` | **lna2** |
|---|---|---|
| routed kernel, per launch | 232.6 µs | **104.0 µs** (2.24×) |
| single stream code / en / ja | 86 / 60 / 56 tok/s | **108.6 / 78.6 / 71.1** |
| 4 streams code / en / ja | 125–179 / 96 / 83 | 193.8 / 146.4 / 129.0 |
| KV window @ util 0.97 | 396,656 tok | 395,069 |
| needle @166k | found, TTFT 167 s | found, **119.6 s** |
| ppl (wikitext, 4k, spec off, matched) | 4.7630 | 4.7647 |

Launch: as above plus `NATIVE_SO=recipe-lna/lna2/vllm_exl3_c.cpython-312-x86_64-linux-gnu.so MOE_KERNEL=lna2` and `-e VLLM_EXL3_MOE_STRICT=1` (fail loudly instead of silently falling back — the first delivery of this kernel *did* silently fall back and "passed" its gates; the seat log must show `LNA2 MoE kernel ACTIVE` on every rank).
Not adopted: the dense-linear launch fusion (`LNA_EXL3_DENSE_GROUP`, T0 passed its gate at 1.21× on the dense segment but T1 is still being fixed) and the first barrier-based attempt (`lna`, correct but not faster). Full story, gates and reviews: `orders/reports/`, canon `docs/LNA-CANON.md`.

## Serving

Image: `Dockerfile` (vllm/vllm-openai:nightly + vLLM 0.28.1rc1.dev337 wheel + exllamav3 1.4.5 built from source with the house patches + vllm-exl3 0.2.3 + `recipe-lna/patch_*.py`). The rebuilt extension and plugin are mounted over the image at run time (`EXT_SO`, `PLUGIN_SRC`) so the image never needs re-baking.

```bash
export EXT_SO=/path/to/exllamav3_ext.cpython-312-x86_64-linux-gnu.so   # built from recipe-lna/exllamav3/*.lna
export PLUGIN_SRC=/path/to/vllm_exl3                                   # vllm-exl3 0.2.3 + recipe-lna/vllm_exl3_exl3.py.lna
AUX_STREAMS=0 IMAGE=dsv4-dense \
NCCL_EXTRA="-e VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128 -e VLLM_DISABLE_DSV4_MEGAMOE_SHARED_EXPERT_FUSION=1" \
MODEL=/path/to/DSV4-Flash-Vision-EXL3-MixedK-D2-K2x3-Dense6 \
UTIL=0.97 MAXLEN=389120 BT=512 SEQS=4 SPEC='{"method":"dspark","num_speculative_tokens":3}' \
bash serve-dsv4-tp8.sh          # OpenAI API on 127.0.0.1:8899, model name DSV4-Flash
```

Notes: thinking is on by default and eats the budget (700 tokens → empty answer); send `chat_template_kwargs: {"thinking": false}` or a large `max_tokens`. Vision works. `--ulimit core=0` and cache mounts are set because a crashed seat once wrote 27 GB of core dumps into the container layer.

## Layout

- `serve-dsv4-tp8.sh` — the seat. `bench-dsv4.py` / `bench-streams.py` (usage-based token count; SSE chunk counting undercounts with speculation) / `ppl-vllm.py` / `prof-summary.py`.
- `tools/` — `mtp_overlay.py`, `layer_overlay.py`, `dense_bake.py` (+ `README-mtp-overlay.md`).
- `recipe-lna/` — vLLM/exllamav3/plugin patches applied in the image; `exllamav3/*.lna` are the patched source files.
- `orders/` — the work orders given to the craftspeople (codex "Luna", Explore agents). `docs/` — campaign canon (`PLAN.md`) and design notes.
- `exl3-tune-cache/` — autotuner disk cache for the 8 × RTX PRO 2000.

## Credits

vcruz305 (MixedK, vllm-exl3 patches), wrldsuksgo2mars (K2 packs), turboderp (exllamav3), DeepSeek (weights, MIT), vLLM.
Bake, patches and measurements: Lna-Lab / YUKI with Ken ([@Tono_Ken3](https://x.com/Tono_Ken3)). License for this repo: MIT; the weights follow the DeepSeek model license.