DSV4-Flash-Vision EXL3 MixedK D2-K2x3-Dense6 + lna-lab recipe (2026-09-03)

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+lna-lab/exllamav3_ext.cpython-312-x86_64-linux-gnu.so filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,79 @@
+---
+license: other
+license_name: deepseek
+license_link: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
+base_model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
+base_model_relation: quantized
+library_name: exllamav3
+pipeline_tag: image-text-to-text
+tags:
+- exl3
+- exllamav3
+- deepseek_v4
+- moe
+- mixed-precision
+- vllm
+- blackwell
+- lna-lab
+language:
+- en
+- ja
+- zh
+---
+
+# DSV4-Flash-Vision — EXL3 MixedK · D2 · K2×3 · Dense6 (380K context on 8 × 16 GB)
+
+A serving-tuned derivative of [vcruz305/DSV4-Flash-Vision-EXL3-MixedK](https://huggingface.co/vcruz305/DSV4-Flash-Vision-EXL3-MixedK)
+(DeepSeek-V4-Flash-Vision-Exp, 305B MoE, vision). Everything that was still BF16 or fp8 in the original has been carved into EXL3
+so that a **380K-context, 4-stream, DSpark3-speculative** seat fits on eight RTX PRO 2000 16 GB under vLLM.
+Baked and measured by **Lna-Lab** (YUKI) on 2026-09-03; house model of SAZANAMI ("オオタニ").
+
+Recipe, patches and the serve script: **github.com/lna-lab/dsv4-carve** (mirrored in `lna-lab/` here).
+
+## Composition
+
+| Part | Here | Original MixedK |
+|---|---|---|
+| Routed experts | EXL3 2/3-bit mixed; layers 3, 21, 41 → K2 (wrldsuksgo2mars K2-v1) | EXL3 2/3-bit |
+| MTP draft experts (3 layers, DSpark) | EXL3 K2 (wrldsuksgo2mars K2.2-D2-v1) | fp8 |
+| Attention + shared experts (790 linears) | **EXL3 6-bit**, calibrated (250 × 2048) | BF16 |
+| lm_head / embeddings / indexer weights_proj | BF16 (unchanged) | BF16 |
+
+86 GB on disk, 51 shards. Per GPU at TP8: 12.0 GiB weights → **396,656 tokens** of fp8 KV at `--max-model-len 389120`.
+
+## Measured (vLLM 0.28.1rc1.dev337 + vllm-exl3 0.2.3, TP8, CUDA graphs, RTX PRO 2000 × 8)
+
+| | |
+|---|---|
+| ppl wikitext-2 (512 × 16) | **6.7159** — original MixedK 6.6271 (+1.34 %); 4-bit dense was +3.8 % and rejected |
+| single stream, no speculation | en 42.1 / ja 42.0 / code 41.9 tok/s |
+| single stream, DSpark3 (3 MTP tokens) | en 62 / ja 55 / code 86 tok/s |
+| 4 streams, DSpark3, aggregate | en 136 / ja 116 / code 186 tok/s |
+| needle at 166K tokens | found; TTFT 167 s (prefill ≈ 1.0k tok/s), decode at that depth 37 tok/s |
+| vision | works (shapes, colours, text read correctly) |
+
+## You need the patches
+
+Stock exllamav3 1.4.5 / vllm-exl3 0.2.3 will **not** serve this pack: dense EXL3 linears in attention trigger (a) the cooperative-GEMM
+autotuner inside CUDA-graph capture and (b) a two-stream deadlock between vLLM's shared-experts stream and the routed EXL3 kernels
+(both use exllamav3's per-device lock buffer). The `lna-lab/` folder carries the patched sources, the vLLM patches, the prebuilt
+extension for sm_120 (`exllamav3_ext.cpython-312-x86_64-linux-gnu.so`, torch cu13) and the serve script; the README there explains each.
+Minimum: `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1`, `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128`, aux streams off, `--kv-cache-dtype fp8`.
+
+```bash
+AUX_STREAMS=0 MODEL=. UTIL=0.97 MAXLEN=389120 BT=512 SEQS=4 \
+SPEC='{"method":"dspark","num_speculative_tokens":3}' bash lna-lab/serve-dsv4-tp8.sh
+```
+
+Thinking is on by default; pass `chat_template_kwargs: {"thinking": false}` or a large `max_tokens`.
+
+## Files
+
+- `model-*.safetensors`, `model.safetensors.index.json`, `config.json` (`quantization_config.non_routed_exl3` lists the 6-bit dense modules), tokenizer, `generation_config.json`
+- `lna-lab/` — Dockerfile, serve/bench/ppl scripts, `tools/` (overlay + dense bake), `patches/`, prebuilt extension, upstream model card
+
+## Credits and license
+
+- Base weights: DeepSeek (MIT). This pack follows the DeepSeek model license like the upstream MixedK.
+- MixedK, vllm-exl3 and its DSV4 patches: [vcruz305](https://huggingface.co/vcruz305) · K2 packs: [wrldsuksgo2mars](https://huggingface.co/wrldsuksgo2mars) · ExLlamaV3: turboderp · vLLM
+- Bake, patches, measurements: Lna-Lab / YUKI with Ken ([@Tono_Ken3](https://x.com/Tono_Ken3))

config.json

@@ -0,0 +1,1178 @@
+{
+  "architectures": [
+    "DeepseekV4ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "expert_dtype": "fp4",
+  "hc_eps": 1e-06,
+  "hc_mult": 4,
+  "hc_sinkhorn_iters": 20,
+  "head_dim": 512,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "index_head_dim": 128,
+  "index_n_heads": 64,
+  "index_topk": 512,
+  "initializer_range": 0.02,
+  "max_position_embeddings": 1048576,
+  "model_type": "deepseek_v4",
+  "moe_intermediate_size": 2048,
+  "n_routed_experts": 256,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 6,
+  "num_hidden_layers": 43,
+  "num_hash_layers": 3,
+  "num_key_value_heads": 1,
+  "num_nextn_predict_layers": 3,
+  "o_groups": 8,
+  "o_lora_rank": 1024,
+  "q_lora_rank": 1024,
+  "qk_rope_head_dim": 64,
+  "quantization_config": {
+    "quant_method": "exl3",
+    "bits": 2,
+    "codebook": "mcg",
+    "head_bits": 16,
+    "non_routed_dtype_policy": "bf16_as_stored",
+    "scope": "dsv4_routed_experts_only",
+    "serving_reader_qualified": false,
+    "version": "0.0.43",
+    "layer_bits": {
+      "13": 3,
+      "22": 3,
+      "28": 3,
+      "43": 2,
+      "44": 2,
+      "45": 2
+    },
+    "non_routed_quantization": {
+      "activation_scheme": "dynamic",
+      "fmt": "e4m3",
+      "quant_method": "fp8",
+      "scale_fmt": "ue8m0",
+      "weight_block_size": [
+        128,
+        128
+      ]
+    },
+    "non_routed_exl3": {
+      "codebook": "mcg",
+      "layers": {
+        "language_model.model.layers.0.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.0.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.0.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.0.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.0.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.0.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.1.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.1.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.1.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.1.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.1.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.1.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.10.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.10.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.10.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.10.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.10.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.10.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.10.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.10.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.10.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.11.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.11.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.11.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.11.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.11.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.11.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.11.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.12.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.12.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.12.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.12.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.12.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.12.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.12.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.12.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.12.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.13.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.13.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.13.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.13.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.13.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.13.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.13.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.14.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.14.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.14.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.14.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.14.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.14.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.14.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.14.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.14.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.15.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.15.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.15.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.15.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.15.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.15.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.15.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.16.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.16.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.16.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.16.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.16.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.16.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.16.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.16.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.16.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.17.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.17.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.17.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.17.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.17.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.17.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.17.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.18.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.18.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.18.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.18.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.18.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.18.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.18.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.18.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.18.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.19.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.19.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.19.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.19.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.19.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.19.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.19.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.2.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.2.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.2.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.2.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.2.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.2.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.2.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.2.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.2.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.20.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.20.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.20.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.20.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.20.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.20.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.20.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.20.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.20.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.21.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.21.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.21.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.21.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.21.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.21.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.21.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.22.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.22.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.22.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.22.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.22.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.22.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.22.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.22.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.22.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.23.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.23.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.23.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.23.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.23.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.23.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.23.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.24.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.24.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.24.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.24.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.24.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.24.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.24.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.24.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.24.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.25.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.25.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.25.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.25.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.25.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.25.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.25.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.26.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.26.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.26.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.26.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.26.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.26.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.26.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.26.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.26.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.27.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.27.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.27.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.27.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.27.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.27.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.27.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.28.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.28.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.28.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.28.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.28.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.28.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.28.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.28.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.28.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.29.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.29.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.29.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.29.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.29.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.29.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.29.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.3.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.3.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.3.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.3.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.3.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.3.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.3.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.30.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.30.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.30.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.30.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.30.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.30.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.30.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.30.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.30.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.31.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.31.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.31.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.31.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.31.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.31.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.31.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.32.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.32.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.32.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.32.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.32.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.32.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.32.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.32.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.32.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.33.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.33.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.33.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.33.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.33.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.33.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.33.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.34.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.34.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.34.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.34.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.34.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.34.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.34.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.34.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.34.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.35.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.35.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.35.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.35.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.35.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.35.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.35.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.36.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.36.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.36.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.36.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.36.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.36.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.36.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.36.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.36.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.37.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.37.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.37.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.37.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.37.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.37.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.37.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.38.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.38.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.38.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.38.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.38.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.38.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.38.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.38.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.38.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.39.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.39.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.39.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.39.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.39.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.39.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.39.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.4.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.4.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.4.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.4.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.4.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.4.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.4.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.4.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.4.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.40.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.40.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.40.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.40.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.40.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.40.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.40.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.40.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.40.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.41.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.41.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.41.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.41.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.41.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.41.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.41.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.42.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.42.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.42.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.42.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.42.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.42.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.42.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.42.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.42.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.5.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.5.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.5.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.5.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.5.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.5.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.5.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.6.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.6.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.6.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.6.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.6.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.6.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.6.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.6.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.6.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.7.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.7.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.7.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.7.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.7.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.7.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.7.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.8.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.8.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.8.attn.indexer.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.8.attn.indexer.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.8.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.8.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.8.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.8.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.8.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.9.attn.compressor.fused_wkv_wgate": {
+          "bits": 6
+        },
+        "language_model.model.layers.9.attn.fused_wqa_wkv": {
+          "bits": 6
+        },
+        "language_model.model.layers.9.attn.wo_a": {
+          "bits": 6
+        },
+        "language_model.model.layers.9.attn.wo_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.9.attn.wq_b": {
+          "bits": 6
+        },
+        "language_model.model.layers.9.ffn.shared_experts.down_proj": {
+          "bits": 6
+        },
+        "language_model.model.layers.9.ffn.shared_experts.gate_up_proj": {
+          "bits": 6
+        }
+      }
+    }
+  },
+  "rms_norm_eps": 1e-20,
+  "rope_scaling": {
+    "beta_fast": 32,
+    "beta_slow": 1,
+    "factor": 16,
+    "original_max_position_embeddings": 65536,
+    "type": "yarn"
+  },
+  "rope_theta": 10000,
+  "routed_scaling_factor": 1.5,
+  "scoring_func": "sqrtsoftplus",
+  "sliding_window": 128,
+  "swiglu_limit": 10.0,
+  "tie_word_embeddings": false,
+  "topk_method": "noaux_tc",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "5.0.0",
+  "use_cache": true,
+  "vocab_size": 129280,
+  "compress_rope_theta": 160000,
+  "compress_ratios": [
+    0,
+    0,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    0,
+    0,
+    0
+  ],
+  "dspark_block_size": 5,
+  "dspark_noise_token_id": 128799,
+  "dspark_target_layer_ids": [
+    40,
+    41,
+    42
+  ],
+  "dspark_markov_rank": 256,
+  "vision_n_layers": 32,
+  "vision_dim": 1024,
+  "vision_n_heads": 16,
+  "vision_inter_dim": 2816,
+  "vision_patch_size": 14,
+  "vision_rope_theta": 10000.0,
+  "vision_downsample_ratio": 3,
+  "vision_max_n_token": 384,
+  "vision_min_pixels": 147456,
+  "vision_max_wh_ratio": 8
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "do_sample": true,
+  "temperature": 1.0,
+  "top_p": 1.0,
+  "transformers_version": "4.46.3"
+}

lna-lab/Dockerfile

@@ -0,0 +1,14 @@
+# lna-lab/vllm-exl3:dsv4 — vLLM nightly dev337 (vision class) + exllamav3 1.4.5 (compiled ext, sm_120) + vllm-exl3 + recipe patches
+FROM vllm/vllm-openai:nightly
+ENV TORCH_CUDA_ARCH_LIST="12.0" MAX_JOBS=32 CPATH=/usr/local/lib/python3.12/dist-packages/nvidia/cu13/include LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/nvidia/cu13/lib
+RUN pip install --no-deps "https://wheels.vllm.ai/27a94d1ce4e3fc100c4732439ccec10f8246a804/vllm-0.28.1rc1.dev337%2Bg27a94d1ce-cp38-abi3-manylinux_2_28_x86_64.whl" 2>&1 | tail -2 \
+ && python3 -c "import vllm,os;print(vllm.__version__); print(os.listdir(os.path.join(os.path.dirname(vllm.__file__),'models','deepseek_v4','nvidia')))"
+COPY exllamav3-src /opt/exllamav3
+RUN pip install ninja 2>&1 | tail -1 && pip install --no-build-isolation --no-deps -v /opt/exllamav3 2>&1 | grep -i -E "error|Successfully" | tail -5 \
+ && python3 -c "import torch, exllamav3_ext; print('exllamav3_ext OK')"
+COPY vllm-exl3 /opt/vllm-exl3
+RUN pip install --no-deps /opt/vllm-exl3
+COPY recipe/scripts /opt/recipe
+COPY recipe-lna /opt/recipe-lna
+RUN python3 /opt/recipe/patch_dsv4_stock028.py && python3 /opt/recipe/patch_dsv4_vl_stream_load.py && python3 /opt/recipe/patch_dsv4_vl_sm120_wide_swa.py \
+ && python3 /opt/recipe-lna/patch_dsv4_dense_exl3.py && python3 /opt/recipe-lna/patch_dsv4_attention_compressor_exl3.py && python3 /opt/recipe-lna/patch_dsv4_aux_streams_env.py

lna-lab/PLAN.md

@@ -0,0 +1,74 @@
+# 鉱脈 dsv4-carve — DSV4-Flash-Vision EXL3 MixedK を TP=8 (16GB×8) で DSpark3 + 長文脈が同居する形に彫る
+総監督: ユキ（雲）。開戦 2026-09-03。ケン「DSpark3 が入る隙間を削りで作る。-c128 ではなく（笑）」。
+職人: GLM（Z.ai）／Luna（codex gpt-5.6-luna xhigh）。
+
+## 北極星
+TP=8 **単流**（ケン 09-03 18:10「単流は目をつぶる」）で **DSpark3 が乗り、文脈 128k**（ケン決定）、decode は素を割らない。
+画像（vision）は落とさない。無検閲版（ablit）は素で型が固まってから同じ型で。
+
+## 実測の基点（2026-09-03、像 lna-lab/vllm-exl3:dsv4 = vLLM dev337 + exllamav3 1.4.5 + vllm-exl3 0.2.3）
+- 札 15.48 GiB、util 0.96 → 14.86 使用可。重み+非torch 13.18、活性の山 0.73（BT 512）、graph 0.04、KV 1.12 = 69k tok（32k×2.1）
+- decode 単流 en/ja/code 40 tok/s（CUDA graph sizes [1,2,4]）。eager 12-16。プリフィル 1.3k tok/s。画像 OK。
+- draft 3 層を足すと 15.2/枚 → KV 0 → OOM。NCCL_BUFFSIZE 絞りは効かない（13.18 のまま）。
+- ディスク家計簿（/枚）: routed 8.67 / attention BF16 1.18 / draft experts fp8 1.20 / vision 0.87(複製疑い) / shared 0.25 / embed 0.12 / head 0.12 / draft他 0.12
+
+## 職人と発注
+- T2 draft overlay（Luna/codex）: DATA1/vllm-exl3-lab/orders/T2-mtp-overlay.md。出所 = wrldsuksgo2mars K2.2-D2（draft が EXL3 K2、shard 2/11 に集中、≈2.4GB を range-read）
+- T3 attention 自家焼き（GLM、第二段）: 元 checkpoint の置き場が先
+
+## 必要な隙間
+64k+draft: 2.9 GiB／128k+draft: 4.7 GiB（KV fp8 ≈ 28 KB/token、1.82 GiB/64k）
+
+## 彫る順（効き × 易しさ）
+1. ~~vision tower を TP 分割~~ → dev337 で既に分割済（QKV/Row parallel）。家計簿の 0.87 は誤り、実は ≈0.11/枚。削れない
+2. **draft experts fp8 → EXL3 2bit**（−0.9）— 出所: 自家焼き（exllamav3 convert -mb 2）or 外の EXL3 pack から overlay。plugin は mtp_experts="exl3" 既定で読める
+3. **attention/shared BF16 → EXL3 4bit overlay**（−1.0）— 世界に EXL3 版 attention の pack は無い（D2 も BF16）→ **自家焼き**が要る（元 checkpoint ≈300GB、置き場: DATA1 115G/stripe 101G では足りず→NAS 経由か掃除）。exllamav3 の計測パスを非 expert 線形だけに絞る道具が要る。第二段。⚠ bf16_shards は TP>1 不可 → 融合線形は全 shard EXL3 に
+4. 活性 BT 512→384（−0.2）※画像 1 枚 387 token が下限
+5. 3bit 層 6 つ → 2bit（−0.4）— 最後。ppl の門つき
+6. 1トークンの家計簿（段ごとの issue_ms）で decode 40 を守る／伸ばす
+
+## 門
+- 各段: 正気三言語 + 画像テストカード + decode ≥ 38 tok/s + KV トークン数（ログ行）を記録。
+- 投機: spec-OFF と同一 greedy 出力 + 受理長（/metrics）。[[speculation-gate-doctrine]]
+- 化粧合格禁止。実測と ESTIMATE を書き分ける。
+
+## 台本
+DATA1/vllm-exl3-lab/{Dockerfile, serve-dsv4-tp8.sh, bench-dsv4.py, recipe/, vllm-exl3/}。口 :8899。GPU 0,1,2,3,5,7,8,9。GPU4/6 は触らない。
+
+## 09-03 18:10 中間実測
+- 🎖️ DSpark3 が TP8 に乗った（D2 overlay + VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128、16k）: en 67/ja 55/code 70 tok/s、素 45。受理長 ≈2.1。画像 OK。
+- 128k への残り: KV ≈2.1 GiB 必要、空き 0.51 → **+1.6 GiB/枚**。削り代: T3 3bit→K2 −0.4／attention EXL3 自家焼き −0.85〜1.0／BT・util −0.3／(未確認) fp4 KV で KV 半減。
+- 決定性: この建て付けは greedy でも走行間不一致 → ビット一致は門に使えない。品質は ppl/実食。
+
+## 一週間の段取り（ケン 09-03 18:15「一週間かけてもいい。精度を保ったまま実用域へ」「コードを読んで無駄時間を詰めるスタイル」）
+北極星: **TP=8 単流 DSpark3 付き 128k、精度は素の pack と同等（ppl 門）、decode ≥ 素**。
+- **段 A（〜09-04）: 32k**。T3 layer_overlay で 3bit 層 6 つ → K2（−0.4 GiB）。donor shard 7 本を DATA1/.tmp/k2v1-donor に落として local 経路で組む。門: ppl（vLLM prompt_logprobs で自前計器を作る、同一文・同一 ctx で pack 間比較）＋三言語正気＋画像。
+- **段 B（09-05〜07）: attention/shared の EXL3 化（−0.85〜1.0）→ 64k〜128k**。世界に無いので自家焼き。元 = NAS models-cold/DeepSeek-V4-Flash-Vision-Exp（157G）。置き場は掃除待ち（ケン決裁）。方式は exllamav3 の量子化器（quantize_linears_single）を非 expert 線形だけに当て、前段の forward は D2 pack の EXL3 experts で回す「部分焼き」。⚠ plugin の non_routed_exl3 は TP>1 で bf16_shards 不可 → 融合線形は全 shard EXL3。lm_head は plugin 未対応（0.12 だけなので後回し）。
+- **段 C（並行）: 1 トークンの家計簿**。vLLM の torch profiler で段ごとの時間（attention/indexer/MoE/all-reduce/lm_head/draft）を取り、CUDA graph の外に漏れている起動律速を潰す。目安: 素 45 → 50+、DSpark3 70 → 80+。
+- **段 D: 128k 実走**。長文プロンプト（100k）で TTFT・decode・受理長。KV 2.1 GiB の確保を確認。
+- 撤退線: ppl が素 pack より 2% 超悪化する削りは採らない（数字は段 A の計器で確定）。
+- **ppl 計器（09-03 18:50 確定）**: `DATA1/vllm-exl3-lab/ppl-vllm.py <port> wiki.test.raw 512 16`、計測席 = draft 無し pack・spec OFF・util 0.95・max-len 4096・BT 1024（prompt_logprobs は語彙 129k の logits を rank0 に要求するので満杯席では OOM）。**基準 = 元 pack 6.6271**（7,215 tok）。撤退線 2% → 6.76。
+- **家計簿（09-03 19:00、torch profiler、rank0、DSpark3 付き 64tok ≈29 手）**: 密 BF16 GEMM **51%**（cuBLAS が sm120 核を持たず sm80 WMMA 16×16 核に落ちる、11,686 発 × 43-50µs、一手 ≈21.8 ms、帯域計算なら ≈3 ms）／EXL3 expert GEMM 29.5%（exl3_moe_kernel 282µs × 1120）／NCCL 7.3%／indexer・norm・attention 各 ≤1%。→ **段 B（attention EXL3 化）は容量と速度の両方の本丸**。計器: prof-summary.py、席は PROFILE='{"profiler":"torch","torch_profiler_dir":"/lab/prof"}' で立て /start_profile /stop_profile。
+- **段 A 実測（09-03 19:25）**: K2×6（3,13,21,22,28,41 を K2）= ppl **6.8845（+3.9%）** vs 元 6.6271 → 撤退線 2% 超、**丸ごとは不採用**。容量は 13.78→13.22 GiB/枚、KV 0.49→**1.05 GiB（64.5k tok）**、DSpark3 decode en 66/ja 58/code 69、受理長 2.2/1.8/2.3。→ 層ごとの感度掃引（t3-sens.sh、一層ずつ K2 → ppl）で痛くない層だけ採る。
+- 道具の傷（直した）: layer_overlay の shard 書き換えに (名前,meta) のリストを渡していて何も落ちていなかった（verify も見逃し）。verify に「書き換え shard に落とした名前が残っていない」を追加。教訓: **verify は「残したものが同じ」だけでなく「消したものが無い」も見る**。
+- 段 B 設計メモ: docs/T4-dense-partial-bake-design.md（VariantSafetensorsCollection で EXL3 experts を差し、密線形だけ校正付きで焼く）。
+- 🎖️ **段 A 着地（09-03 20:45）**: 層別感度 3/13/21/22/28/41 = +0.41/+1.06/+0.67/+0.89/+1.17/+0.66%。**K2×3（3,21,41）= ppl 6.7041（+1.16%）**、重み 13.50 GiB、KV 0.77 GiB = 47k tok → **32k + DSpark3**（en 60/ja 59/code 68、受理長 2.1/1.9/2.2）。給仕 pack = `DATA1/DSV4-Flash-Vision-EXL3-MixedK-D2-K2x3`、計測 pack = `…-MixedK-K2x3`。六層版は不採用（+3.9%）。
+- 128k までの残り（ESTIMATE）: KV 2.1 GiB 必要 vs 0.77 → +1.35 GiB。段 B の attention EXL3（−1.1、wo_a 込み）+ BT/util（−0.3）でちょうど。段 B は T4（Luna 製作中）+ plugin/vLLM 改変（docs/T5）。
+- **段 B 着手（09-03 夜）**: 元 checkpoint を DATA1 に写し（157G）。T4 `tools/dense_bake.py`（Luna）= exllamav3 の校正付き量子化で密線形 790 本（43 層 × 15〜20）だけを 4bit EXL3 に。GPU 10,11 で焼き中（作業場 DATA1/.tmp/t4-dense-work、層ごとに ckpt、`--resume`）。私が直した傷: 層鍵の正規表現・target の順序比較・inference_mode。
+- T6（Luna）: plugin に実効 TP（disable_tp/Replicated）と rank 局所 `wo_a.slice.{rank}`、vLLM 側は recipe-lna/patch_dsv4_dense_exl3.py（compressor quant_config・_o_proj EXL3 分岐）。像 **lna-lab/vllm-exl3:dsv4-dense**（旧 :dsv4 は温存）。給仕台本は IMAGE= で切替。
+- 教訓（今日三度目）: `pgrep -f`/`pkill -f` のパターンが自分の bash -c に入る → 自殺（exit 144）。**pid で殺す**。docker build のタグは最初から別名で。
+- 段 B 焼き（09-03 16:46〜）: 8 枚並列（10 枚は peer 上限で不可 → [[exllamav3-tp-peer-limit]]）、層 0→1 が 77 秒、1 層 66 MB、wo_a 4bit proxy_err ≈0.001。完了後は t4-after.sh が自動で merge（K2x3 計測 pack / D2-K2x3 給仕 pack）→ ppl → 64k/128k DSpark3 席まで回す（ログ DATA1/.tmp/t4-after.log）。
+- **段 B 初点灯（09-03 18:30）**: 焼き 43 層 68 分（8 枚）。merge は 43 shard 書き直しで 66G（元 checkpoint は NAS へ退去して場所を作った）。**密 EXL3 で重み 13.50 → 10.56 GiB/枚（−2.9）**。vLLM 側で三つ直した: (1) config の prefix は `language_model.model.layers.N`／`ffn.shared_experts`（Vision クラス）、(2) wo_a EXL3 分岐の出力は fp32→bf16、(3) compressor の kv_score は `torch.mm(weight.T)` 直叩き → EXL3 shard 分岐（recipe-lna/patch_dsv4_attention_compressor_exl3.py）。CUDA graph + 多ストリーム有効だと warmup で無言ハング（GPU 100% spin）、eager + `VLLM_MULTI_STREAM_GEMM_TOKEN_THRESHOLD=0` で正気 OK。切り分けと 64k/128k を t4-after4.sh で実施中。
+- 段 B ppl（09-03 18:29、eager）: 密 4bit = **6.8789（基準 +3.8%、K2×3 比 +2.6%）** → 線越え。余裕 2.9 GiB があるので attention/shared を 6bit で���き直す（元 checkpoint を NAS から再写し）。graph ON + 多ストリーム OFF は起動するが生成で 500（要 traceback）。
+- **128k の席が立った（09-03 19:00、密 4bit pack、graph ON・多ストリーム OFF・DSpark3）**: 重み 11.51 GiB/枚（draft 込み）、**KV 396k〜420k トークン（128k の 3 倍）**。ただし生成が「無言→EngineCore TimeoutError」で落ちる（graph ON でも eager でも。12 トークンの補完は通る、思考付きの長い生成で落ちる）。多ストリーム ON は warmup で固まる。容疑: 小バッチ経路 `LinearEXL3.forward` → `ext.BC_LinearEXL3.run_alloc`（rows ≤ 144）— TP8 の 8 プロセス×多モジュールでの静的状態か同期。exllamav3_ext を読んで特定する。
+- 6bit 焼き直し開始（09-03 19:05、8 枚、work=.tmp/t4-dense-work-b6）。4bit の ppl は 6.8789（+3.8%）なので不採用、6bit で線内を狙う。
+- 6bit 焼き完了（09-03 20:22、43 層 RC=0、trellis 幅 96）。merge が config に bits=4 を書いていて（--bits 未指定＝既定）読み込みで形不一致 → 設定を 6 に直し、dense_bake merge は work/args.json の bits を既定にするよう修正（cd3b686）。席の連鎖は .tmp/t4-after6.sh / t4-after6.log で張り直し（20:33）。
+- 6bit 席の二つ目の傷（09-03 20:45）: 密 EXL3 の小バッチ GEMM が CUDA graph 捕捉中に**未チューンの形**で自動チューナへ入り `coop_autotune.cu:464` "operation not permitted when stream is capturing" で全 rank 死亡。4bit で通ったのは密が BF16 だった頃のキャッシュ形と重なっていたため。直し=①exl3_gemm.cu: 捕捉中は `cudaStreamIsCapturing` を見てチューンせず静的ヒューリスティックへ（mgemm も同条件）②プラグイン: 重み読込直後に行数 1,2,4,8,16 で事前実行（LNA_EXL3_PREWARM_ROWS）③像は焼き直さず .so と plugin を DATA1 から差し込む（serve の EXT_SO / PLUGIN_SRC）。a131603。
+- ⚠️ルート盤事故（09-03 20:50）: 落ちた席のコンテナ書込層が 27.6GB（コアダンプ/キャッシュ）でルート 0 バイト。docker data-root がルートに在る構造問題。対策=serve に --ulimit core=0・/root/.cache と .tilelang を DATA1 へ。**data-root の DATA1 移設はケン決裁待ち**。
+- ★decode ハングの真因（09-03 21:40、py-spy＋CUDA_LAUNCH_BLOCKING=1 で確定）: CUDA_LAUNCH_BLOCKING=1 では止まらない＝同時走行の衝突。vLLM FusedMoE は ≤256 トークンで **shared experts を別ストリームに重ねる**（VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD=256）。密焼きで shared experts も EXL3 協調カーネルになり、routed の mgemm と同じ DevCtx ロック領域を別ストリームから同時に叩いて永久 spin（GPU 100%）。プリフィル(1024行)で平気・decode で止まる症状と一致。対策= VLLM_DISABLE_SHARED_EXPERTS_STREAM=1（serve 既定）。
+- 🎖️**128k 単流 DSpark3 到達（09-03 21:30）**: pack=`DSV4-Flash-Vision-EXL3-MixedK-D2-K2x3-Dense6`（68G）。KV **324,519 tok**（128k×2.48）、重み 12.0 GiB/枚。ppl **6.7159（+1.34%）**。投機なし 4k: en 42.1/ja 42.0/code 41.9。**DSpark3 128k: en 54-56 / ja 53 / code 62 tok/s**（受理 2.0/1.8/2.2）。32k BF16 密より約 1 割低い（shared 別ストリーム OFF＋密 EXL3 の小バッチ経路、要 profiler）。席の起動= serve-dsv4-tp8.sh（EXT_SO/PLUGIN_SRC 差し込み・AUX_STREAMS=0・SHARED_STREAM_OFF 既定・UTIL 0.97・MAXLEN 131072・BT 512・SEQS 1・SPEC dspark 3）。Dense4 と 4bit qtensors は削除。
+- **-c 307,200 実測（09-03 21:45）**: 同 pack・同席で MAXLEN=307200 → KV **394,570 tok**（1.28×）、短文 DSpark3 en 58.6/ja 53.1/code 62.6。**針テスト 166k tok**（実測 prompt_tokens、84 万字、針は 37% 位置）: 正答 `sazanami-hotaru-2026`、TTFT **167 s**（プリフィル ≈1.0k tok/s、実測から算出）、その深さの decode **37.4 tok/s**。作文テスト 7 本（思考オフ）品質良、誤字 1（冼）。思考オンは max_tokens を大きく。Vision OK。
+- **多流実測（09-03 21:55、-c 307,200・SEQS=4・DSpark3、bench-streams.py=usage 計数・metrics 突合一致）**: en 1流 62.4 / 2流 35×2=69 / 4流 35×4=**136** ・ ja 55 / 62 / **116** ・ code 86 / 96 / **186** tok/s。★2 流と 4 流の壁時計がほぼ同じ＝2 流の一手が 4 流と同コスト（graph 捕捉サイズと EXL3 小バッチの丸め。2 流は損、4 流が甘い）。⚠️SSE チャンク数で数えると投機で過小（1流 28 に見えた）。
+- 🎖️**制式採用（09-03 22:00 ケン決裁「俺達のショウヘイオオタニがやってきた」）**: 制式構成 = -c 307,200・SEQS 4・DSpark3。レシピは Models/RECIPES.md。残: docker data-root 移設（先）→ 常設化（systemd/DSH）→ HF＋GitHub lna-lab 公開（private 先行）→ プリフィル彫刻・8 流。
+- -c 389,120（380K）・SEQS 4 でも起動: KV **396,656 tok**（1.02×）、4 流 en 132 / code 141 合計（128 tok 短走）。制式は **-c 389,120・4 流**に更新（KV は共用プール、1 本 380k or 4 本 ≈99k ずつ）。
+- DATA1: ds4 222G と焼き作業場 4 つ（ケン決裁）を削除、533G 空き。ds4 の履歴は DATA2/Lna-Lab/archive。

lna-lab/README-dsv4-carve.md

@@ -0,0 +1,65 @@
+# dsv4-carve — DeepSeek-V4-Flash-Vision (305B) EXL3 on 8 × 16 GB, 380K context, DSpark3
+
+Lna-Lab's serving recipe for [vcruz305/DSV4-Flash-Vision-EXL3-MixedK](https://huggingface.co/vcruz305/DSV4-Flash-Vision-EXL3-MixedK)
+carved down until a **380K-context, 4-stream, speculative (DSpark3) seat** fits on eight RTX PRO 2000 (16 GB, Blackwell sm_120)
+under vLLM. Adopted as the house model of SAZANAMI on 2026-09-03 ("オオタニ").
+
+Weights (private mirror, 86 GB): `sakamakismile/DSV4-Flash-Vision-EXL3-MixedK-D2-K2x3-Dense6`.
+
+## What is in the pack
+
+| Part | Format | Source |
+|---|---|---|
+| Routed experts (43 layers) | EXL3 2/3 bit mixed, layers 3/21/41 replaced by K2 | vcruz305 MixedK + wrldsuksgo2mars K2-v1 (`tools/layer_overlay.py`) |
+| MTP draft experts (3 layers) | EXL3 K2 (was fp8) | wrldsuksgo2mars K2.2-D2-v1 (`tools/mtp_overlay.py`) |
+| Attention + shared experts (790 dense linears) | **EXL3 6 bit** (was BF16), calibrated 250 × 2048 | baked here (`tools/dense_bake.py`) |
+| KV cache | fp8 (≈17 KB/token incl. draft) | vLLM |
+
+Per-GPU (TP8): weights 12.0 GiB, KV pool **396,656 tokens** at `--max-model-len 389120`.
+
+## Measured (2026-09-03, TP8, CUDA graphs on)
+
+| | tok/s |
+|---|---|
+| ppl (wikitext-2, 512 × 16) | **6.7159** (Cruz original 6.6271, +1.34 %) |
+| single stream, no speculation | en 42.1 / ja 42.0 / code 41.9 |
+| single stream, DSpark3 | en 62 / ja 55 / code 86 |
+| 4 streams, DSpark3 (aggregate) | en 136 / ja 116 / code 186 |
+| needle at 166K tokens | found; TTFT 167 s (prefill ≈ 1.0k tok/s), decode at depth 37 |
+
+2 streams cost the same wall time as 4 (graph capture sizes + EXL3 small-batch bucketing): run 1 or 4.
+
+## The three traps (all fixed in this repo)
+
+1. **`dense_bake.py --merge` stamped `bits: 4` into `config.json` for a 6-bit bake** → shape mismatch at load. Merge now reads bits from `work/args.json`.
+2. **exllamav3's cooperative-GEMM autotuner runs inside CUDA-graph capture** (`coop_autotune.cu:464`, "operation not permitted when stream is capturing"). Patch in `recipe-lna/exllamav3/exl3_gemm.cu.lna`: skip tuning while `cudaStreamIsCapturing`, fall back to the static heuristic; plus the plugin pre-tunes decode row counts (1,2,4,8,16) right after weight load (`LNA_EXL3_PREWARM_ROWS`).
+3. **Decode deadlock, GPU 100 % on all ranks.** vLLM's FusedMoE overlaps *shared experts on a second stream* for ≤ 256 tokens. With shared experts also EXL3, two cooperative kernels share exllamav3's per-device lock buffer (`DevCtx::get_locks`) from two streams and spin forever. Diagnosed with `CUDA_LAUNCH_BLOCKING=1` (no hang → concurrency) and py-spy. Fix: **`VLLM_DISABLE_SHARED_EXPERTS_STREAM=1`** (default in `serve-dsv4-tp8.sh`).
+
+## Serving
+
+Image: `Dockerfile` (vllm/vllm-openai:nightly + vLLM 0.28.1rc1.dev337 wheel + exllamav3 1.4.5 built from source with the house patches + vllm-exl3 0.2.3 + `recipe-lna/patch_*.py`). The rebuilt extension and plugin are mounted over the image at run time (`EXT_SO`, `PLUGIN_SRC`) so the image never needs re-baking.
+
+```bash
+export EXT_SO=/path/to/exllamav3_ext.cpython-312-x86_64-linux-gnu.so   # built from recipe-lna/exllamav3/*.lna
+export PLUGIN_SRC=/path/to/vllm_exl3                                   # vllm-exl3 0.2.3 + recipe-lna/vllm_exl3_exl3.py.lna
+AUX_STREAMS=0 IMAGE=dsv4-dense \
+NCCL_EXTRA="-e VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128 -e VLLM_DISABLE_DSV4_MEGAMOE_SHARED_EXPERT_FUSION=1" \
+MODEL=/path/to/DSV4-Flash-Vision-EXL3-MixedK-D2-K2x3-Dense6 \
+UTIL=0.97 MAXLEN=389120 BT=512 SEQS=4 SPEC='{"method":"dspark","num_speculative_tokens":3}' \
+bash serve-dsv4-tp8.sh          # OpenAI API on 127.0.0.1:8899, model name DSV4-Flash
+```
+
+Notes: thinking is on by default and eats the budget (700 tokens → empty answer); send `chat_template_kwargs: {"thinking": false}` or a large `max_tokens`. Vision works. `--ulimit core=0` and cache mounts are set because a crashed seat once wrote 27 GB of core dumps into the container layer.
+
+## Layout
+
+- `serve-dsv4-tp8.sh` — the seat. `bench-dsv4.py` / `bench-streams.py` (usage-based token count; SSE chunk counting undercounts with speculation) / `ppl-vllm.py` / `prof-summary.py`.
+- `tools/` — `mtp_overlay.py`, `layer_overlay.py`, `dense_bake.py` (+ `README-mtp-overlay.md`).
+- `recipe-lna/` — vLLM/exllamav3/plugin patches applied in the image; `exllamav3/*.lna` are the patched source files.
+- `orders/` — the work orders given to the craftspeople (codex "Luna", Explore agents). `docs/` — campaign canon (`PLAN.md`) and design notes.
+- `exl3-tune-cache/` — autotuner disk cache for the 8 × RTX PRO 2000.
+
+## Credits
+
+vcruz305 (MixedK, vllm-exl3 patches), wrldsuksgo2mars (K2 packs), turboderp (exllamav3), DeepSeek (weights, MIT), vLLM.
+Bake, patches and measurements: Lna-Lab / YUKI with Ken ([@Tono_Ken3](https://x.com/Tono_Ken3)). License for this repo: MIT; the weights follow the DeepSeek model license.

lna-lab/UPSTREAM-vcruz305-README.md

@@ -0,0 +1,51 @@
+---
+license: other
+license_name: deepseek
+license_link: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
+base_model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
+base_model_relation: quantized
+library_name: exllamav3
+tags: [exl3, trellis, deepseek_v4, moe, mixed-precision, dgx-spark]
+---
+
+# DeepSeek-V4-Flash-Vision-Exp — EXL3 MixedK
+
+Sensitivity-allocated mixed-precision EXL3 quantization of
+[deepseek-ai/DeepSeek-V4-Flash-Vision-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp):
+a K2 base with the most quantization-sensitive MoE layers upgraded to K3.
+~95 GiB — fits one DGX Spark.
+
+## Strategy
+Full K2 base (~2.0 bpw experts) with **K3 (~3.0 bpw) on layers 21, 22, 13,
+41, 28, 3**, selected by a per-layer K2-vs-K3 proxy-error scan. K is
+self-describing per tensor via trellis shape, so the pack needs no metadata
+changes — kernels dispatch per layer at load.
+
+| Layer | K2->K3 gain |
+|---|---|
+| 21 | 0.081 |
+| 22 | 0.081 |
+| 13 | 0.081 |
+| 41 | 0.081 |
+| 28 | 0.081 |
+| 3  | 0.081 |
+
+The landscape is **near-flat** (~0.3% spread across all 43 layers), so this
+top-6 is a noise-level ordering of near-ties, not a set of distinct outliers.
+Whether mixed-K measurably beats uniform K2 is pending PPL/KLD.
+
+## Source, format, serving
+Identical to the [K2 base](https://huggingface.co/vcruz305/DSV4-Flash-Vision-EXL3-K2):
+mixed-format source (MXFP4 experts + FP8 attention) dequanted to BF16 before
+trellis (4-bit -> 2-bit double quant on experts); MTP/DSpark drafter kept at
+source MXFP4 verbatim. vLLM EXL3 overlay, one Spark, text-only serve.
+
+## Limitations
+Not boot-tested; mixed-format load (EXL3 + MXFP4 drafter) unverified; quality
+gain over K2 not yet measured (flat landscape suggests it may be small);
+DeepSeek license (`other`). See the K2 card for full detail.
+
+## Related
+- [K2 base](https://huggingface.co/vcruz305/DSV4-Flash-Vision-EXL3-K2)
+- [k3-delta + evidence](https://huggingface.co/vcruz305/dsv4-flash-vision-exl3-k3-delta)
+- [ablit MixedK](https://huggingface.co/vcruz305/DSV4-Flash-Vision-ablit-EXL3-MixedK) — uncensored variant

lna-lab/bench-dsv4.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+"""tok/s + DSpark acceptance meter for the vLLM DSV4 seat. usage: bench-dsv4.py [port] [max_tokens]"""
+import json, sys, time, urllib.request, re
+port = sys.argv[1] if len(sys.argv) > 1 else "8899"; N = int(sys.argv[2]) if len(sys.argv) > 2 else 256
+U = f"http://127.0.0.1:{port}"
+P = {"en": "Explain in detail how a transformer language model generates text, step by step.",
+     "ja": "日本の四季それぞれの特徴と、季節ごとの代表的な行事や食べ物について詳しく説明してください。",
+     "code": "Write a Python class implementing an LRU cache with get/put and O(1) operations, with docstrings and a small test."}
+def metrics():
+    try: t = urllib.request.urlopen(U + "/metrics", timeout=10).read().decode()
+    except Exception: return {}
+    out = {}
+    for k in ("vllm:spec_decode_num_accepted_tokens_total", "vllm:spec_decode_num_drafts_total", "vllm:spec_decode_num_draft_tokens_total"):
+        m = re.findall(r"^" + re.escape(k) + r"(?:\{[^}]*\})?\s+([\d.e+]+)", t, re.M)
+        if m: out[k.split("num_")[1].replace("_total","")] = sum(float(x) for x in m)
+    return out
+for name, prompt in P.items():
+    body = {"model": "DSV4-Flash", "messages": [{"role": "user", "content": prompt}], "max_tokens": N, "temperature": 0, "stream": True, "stream_options": {"include_usage": True}}
+    m0 = metrics(); t0 = time.time(); first = None; toks = 0; text = ""
+    req = urllib.request.Request(U + "/v1/chat/completions", data=json.dumps(body).encode(), headers={"content-type": "application/json"})
+    with urllib.request.urlopen(req, timeout=600) as r:
+        for line in r:
+            line = line.decode().strip()
+            if not line.startswith("data:") or line.endswith("[DONE]"): continue
+            d = json.loads(line[5:])
+            if d.get("usage"): toks = d["usage"]["completion_tokens"]
+            ch = d.get("choices") or []
+            if ch and (ch[0]["delta"].get("content") or (ch[0]["delta"].get("reasoning_content") or ch[0]["delta"].get("reasoning"))):
+                if first is None: first = time.time()
+                text += ch[0]["delta"].get("content") or ""
+    wall = time.time() - t0; m1 = metrics()
+    acc = ""
+    if m1.get("drafts") and m0 is not None:
+        dr = m1["drafts"] - m0.get("drafts", 0); ac = m1["accepted_tokens"] - m0.get("accepted_tokens", 0)
+        if dr: acc = f" accept_len={1 + ac / dr:.2f} (accepted {ac:.0f}/{m1['draft_tokens'] - m0.get('draft_tokens', 0):.0f} drafted)"
+    print(f"{name}: tokens={toks} wall={wall:.2f}s TTFT={(first or t0) - t0:.2f}s decode={toks / (wall - ((first or t0) - t0) + 1e-9):.2f} tok/s{acc}")
+    print("   ", repr(text[:100]))

lna-lab/bench-streams.py

@@ -0,0 +1,29 @@
+#!/usr/bin/env python3
+"""Multi-stream decode meter for the DSV4 seat: per-stream and aggregate tok/s (streaming, wall-clock)."""
+import json, sys, time, threading, urllib.request
+port = sys.argv[1] if len(sys.argv) > 1 else "8899"; N = int(sys.argv[2]) if len(sys.argv) > 2 else 256
+streams_list = [int(x) for x in (sys.argv[3] if len(sys.argv) > 3 else "1,2,4").split(",")]
+P = {"en": "Write a detailed essay about the history of the printing press and its effects on European society.",
+     "ja": "日本の四季それぞれの風物詩と、それが文学に与えた影響について詳しく説明してください。",
+     "code": "Write a complete Python implementation of an LRU cache with tests and explanations."}
+def one(prompt, out):
+    body = {"model": "DSV4-Flash", "messages": [{"role": "user", "content": prompt}], "max_tokens": N, "temperature": 0.0, "stream": True, "stream_options": {"include_usage": True}, "chat_template_kwargs": {"thinking": False}}
+    t0 = time.time(); first = None; n = 0
+    r = urllib.request.urlopen(urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", data=json.dumps(body).encode(), headers={"Content-Type": "application/json"}), timeout=3600)
+    for line in r:
+        line = line.decode().strip()
+        if not line.startswith("data:") or line.endswith("[DONE]"): continue
+        j = json.loads(line[5:])
+        if j.get("usage") and j["usage"].get("completion_tokens"): n = j["usage"]["completion_tokens"]; last = time.time(); continue
+        ch = j.get("choices") or []
+        d = ch[0].get("delta", {}) if ch else {}
+        if d.get("content") or d.get("reasoning_content") or d.get("reasoning"):
+            if first is None: first = time.time()
+            last = time.time()
+    out.append((first - t0, n, last - first))
+for lang, prompt in P.items():
+    for s in streams_list:
+        res = []; th = [threading.Thread(target=one, args=(prompt, res)) for _ in range(s)]
+        t = time.time(); [x.start() for x in th]; [x.join() for x in th]; wall = time.time() - t
+        tot = sum(n for _, n, _ in res); per = [n / dt for _, n, dt in res]
+        print(f"{lang}: streams={s} tokens={tot} wall={wall:.2f}s TTFT={max(f for f,_,_ in res):.2f}s agg={tot/wall:.1f} tok/s per_stream={min(per):.1f}-{max(per):.1f}", flush=True)

lna-lab/exllamav3_ext.cpython-312-x86_64-linux-gnu.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:175c099f07950496da3f5faa40e32d78eab94228ba5a5961770f8d4605691bf4
+size 356940600

lna-lab/patches/exl3.py.lna

@@ -0,0 +1,389 @@
+from __future__ import annotations
+import torch
+from ...model.config import Config
+from .exl3_lib.quantize import preapply_had_l, preapply_had_r, had_k, had_n
+from ...ext import exllamav3_ext as ext
+from ...util.tensor import g_tensor_cache
+import os
+from ...util import profile_opt
+
+AUTO_RECONSTRUCT_THRESHOLD = 144
+MAX_RECONSTRUCT_SLICE_N = 32768
+RECONSTRUCT_SLICE_GRANULARITY_N = 128
+
+no_fused_reconstruct = os.environ.get("EXL3_NO_FUSED_RECONSTRUCT", "0") != "0"
+
+class LinearEXL3:
+
+    quant_type: str = "exl3"
+
+    def __init__(
+        self,
+        config: Config | None,
+        in_features: int,
+        out_features: int,
+        scale: torch.Tensor | None = None,
+        su: torch.Tensor | None = None,
+        sv: torch.Tensor | None = None,
+        suh: torch.Tensor | None = None,
+        svh: torch.Tensor | None = None,
+        trellis: torch.Tensor | None = None,
+        mcg: torch.Tensor | None = None,
+        mul1: torch.Tensor | None = None,
+        bias: torch.Tensor | None = None,
+        out_dtype: torch.dtype | None = None,
+        transformers_fix: bool = False,
+        key: str | None = None
+    ):
+        assert scale is None, "scale is no longer used"
+        assert su is not None or suh is not None, "either su (packed) or suh (unpacked) is required"
+        assert sv is not None or svh is not None, "either sv (packed) or svh (unpacked) is required"
+        assert trellis is not None, "trellis is required"
+        if su is not None: assert su.dtype == torch.int16, "su is wrong datatype"
+        if sv is not None: assert sv.dtype == torch.int16, "sv is wrong datatype"
+        if suh is not None: assert suh.dtype == torch.half, "suh is wrong datatype"
+        if svh is not None: assert svh.dtype == torch.half, "svh is wrong datatype"
+        assert trellis.dtype == torch.int16, "trellis is wrong datatype"
+        assert len(trellis.shape) == 3, "trellis must have dim = 3"
+
+        if bias is not None and bias.dtype == torch.float: bias = bias.to(torch.half)
+
+        # Not a Module subclass, so the config-or-NullConfig default doesn't apply here; TP imports pass
+        # config=None and forward() reads config.infer_params
+        if config is None:
+            from ...model.config import NullConfig
+            config = NullConfig()
+        self.config = config
+        self.transformers_fix = transformers_fix
+        self.key = key
+
+        # self.scale = scale.item()
+        self.su = None
+        self.sv = None
+        self.suh = suh if suh is not None else self.unpack_bf(su)
+        self.svh = svh if svh is not None else self.unpack_bf(sv)
+        self.trellis = trellis
+        self.K = trellis.shape[-1] // 16
+        self.in_features = in_features
+        self.out_features = out_features
+        self.bias = bias
+        self.swap_device = None
+        self.out_dtype = out_dtype
+        self.default_out_dtype = out_dtype or torch.half
+
+        self.mcg_tensor = mcg
+        self.mul1_tensor = mul1
+        self.mcg = self.mcg_tensor is not None
+        self.mul1 = self.mul1_tensor is not None
+
+        self._fused_reconstruct = None
+        self.bsz1_xh_args = (self.trellis.device, (1, self.in_features), self.out_dtype)
+        self.bc = ext.BC_LinearEXL3(
+            self.trellis,
+            self.suh,
+            self.svh,
+            self.K,
+            self.bias,
+            self.mcg,
+            self.mul1,
+            torch.empty((1, self.in_features), dtype = self.out_dtype, device = self.trellis.device)  # LNA-LAB: per-linear scratch (was a shared cache entry → races across streams)
+        )
+
+
+    def unload(self):
+        # g_tensor_cache.drop(*self.bsz1_xh_args)
+        pass
+
+
+    def get_tensors(self, key: str):
+        return {
+            f"{key}.{subkey}": tensor.contiguous()
+            for subkey, tensor in [
+                ("su", self.su),
+                ("sv", self.sv),
+                ("suh", self.suh),
+                ("svh", self.svh),
+                ("trellis", self.trellis),
+                ("bias", self.bias),
+                ("mcg", self.mcg_tensor),
+                ("mul1", self.mul1_tensor),
+            ] if tensor is not None
+        }
+
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        params: dict,
+        out_dtype: torch.dtype | None = None,
+    ) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
+
+        if "ovr" in params:
+            ovr = params["ovr"]
+            if self.key in ovr and ovr[self.key].inner is not self:
+                return ovr[self.key].forward(x, params, out_dtype)
+
+        # The EXL3 kernels read x as contiguous rows; a strided view (e.g. a head-group slice
+        # of a wider tensor) would be silently misread as interleaved garbage. Producers are
+        # responsible for contiguity (a silent copy here would hide a hot-path inefficiency
+        # and break CUDA-graph address stability)
+        assert x.is_contiguous(), f"LinearEXL3 {self.key}: non-contiguous input {tuple(x.shape)}"
+
+        reconstruct = params.get("reconstruct")
+        if not reconstruct:
+            rows = x.numel() // x.shape[-1]
+            if rows <= AUTO_RECONSTRUCT_THRESHOLD or self.config.infer_params.no_reconstruct:
+                dtype = out_dtype or self.default_out_dtype
+                return self.bc.run_alloc(x, self.out_features, dtype == torch.float)
+
+        return self.reconstruct_hgemm(x, out_dtype)
+
+
+    def unpack_bf(self, bitfield: torch.Tensor):
+        # For some reason this operation causes a GPU assert on Transformers. Running on CPU seems to fix it
+        device = bitfield.device
+        if self.transformers_fix:
+            bitfield = bitfield.cpu()
+
+        # (Only used for full reconstruct and loading old models, not during inference)
+        bitfield = bitfield.view(torch.uint16).to(torch.int)
+        masks = (1 << torch.arange(16)).to(bitfield.device)
+        expanded = (bitfield.unsqueeze(-1) & masks) > 0
+        expanded = expanded.flatten()
+        # NOT torch.where with CPU scalar tensors: that path misses the device guard when the
+        # condition lives on a non-current device (observed on torch 2.11.0+cu130) — the kernel
+        # launches on the current device's context, faults there, silently zero-fills the output
+        # and leaves every other device in the process unusable. Map bool -> {-1, +1} arithmetically
+        expanded = 1.0 - expanded.to(torch.float16) * 2.0
+        return expanded.contiguous().to(device)
+
+
+    def reconstruct_hgemm(self, x: torch.Tensor, out_dtype):
+
+        shape = x.shape
+        rows = x.numel() // shape[-1]
+        out_shape = shape[:-1] + (self.out_features,)
+        x = x.view(rows, self.in_features)
+        y = torch.empty(out_shape, dtype = out_dtype or self.default_out_dtype, device = x.device)
+
+        y_ = y.view(rows, self.out_features)
+
+        # Fused path: reconstruct emits ORIGINAL-basis weights (both Hadamards + sign
+        # vectors folded into the memory-bound reconstruct kernel), so the gemm runs on the
+        # raw input and the standalone input/output had_r_128 launches disappear (~14% of
+        # long-chunk prefill GPU time). Requires 128-divisible dims (always true for EXL3
+        # tensors: both sides are had-transformed at quant time)
+        if self._fused_reconstruct is None:
+            self._fused_reconstruct = (
+                self.in_features % 128 == 0 and self.out_features % 128 == 0
+                and not no_fused_reconstruct
+            )
+
+        # The fused kernel costs ~4x plain reconstruct (k*n-proportional) while the saved
+        # had launches scale with rows*(k+n); breakeven is rows ~400-900 across shapes
+        use_fused = self._fused_reconstruct and rows >= 1024
+
+        if use_fused:
+            xh = x
+        else:
+            xh = torch.empty_like(x)
+            ext.had_r_128(x, xh, self.suh, None, 1.0)
+
+        if self.out_features <= MAX_RECONSTRUCT_SLICE_N:
+            w = torch.empty((self.in_features, self.out_features), dtype = torch.half, device = self.trellis.device)
+            if use_fused:
+                ext.reconstruct_had_slice(w, self.trellis, self.suh, self.svh, self.K, self.mcg, self.mul1, 0)
+            else:
+                ext.reconstruct(w, self.trellis, self.K, self.mcg, self.mul1)
+            ext.hgemm(xh, w, y_)
+        else:
+            numel_ = self.in_features * MAX_RECONSTRUCT_SLICE_N
+            w_ = torch.empty((numel_,), dtype = torch.half, device = self.trellis.device)
+            for n_start in range(0, self.out_features, MAX_RECONSTRUCT_SLICE_N):
+                n_end = min(n_start + MAX_RECONSTRUCT_SLICE_N, self.out_features)
+                numel = self.in_features * (n_end - n_start)
+                w = w_[:numel].view(self.in_features, n_end - n_start)
+                if use_fused:
+                    ext.reconstruct_had_slice(
+                        w, self.trellis, self.suh, self.svh[n_start:], self.K, self.mcg, self.mul1, n_start)
+                else:
+                    ext.reconstruct_slice(w, self.trellis, self.K, self.mcg, self.mul1, n_start)
+                ext.hgemm(xh, w, y_[:, n_start:n_end])
+
+        if not use_fused:
+            ext.had_r_128(y_, y_, None, self.svh, 1.0)
+
+        if self.bias is not None:
+            y += self.bias
+        return y
+
+
+    def get_inner_weight_tensor(self, n_offset: int = 0, n_features: int | None = None):
+        w = torch.empty((self.in_features, self.out_features), dtype = torch.half, device = self.trellis.device)
+        ext.reconstruct(w, self.trellis, self.K, self.mcg, self.mul1)
+        return w
+
+
+    def get_weight_tensor(self):
+        # suh = self.unpack_bf(self.su).unsqueeze(1)
+        suh = self.unpack_bf(self.su).unsqueeze(1) if self.su else self.suh.unsqueeze(1)
+        svh = self.unpack_bf(self.sv).unsqueeze(0) if self.sv else self.svh.unsqueeze(0)
+        w = self.get_inner_weight_tensor()
+        w = preapply_had_l(w, had_k)
+        w *= suh
+        w = preapply_had_r(w, had_n)
+        w *= svh
+        # w *= self.scale
+        return w
+
+
+    def get_bias_tensor(self) -> torch.Tensor | None:
+        return self.bias
+
+
+    # Swap tensors to CPU (to free some space while quantizing)
+    def swap_cpu(self):
+        if self.swap_device is not None:
+            return
+        self.swap_device = self.trellis.device
+        if self.su is not None: self.su = self.su.cpu()
+        if self.sv is not None: self.sv = self.sv.cpu()
+        if self.suh is not None: self.suh = self.suh.cpu()
+        if self.svh is not None: self.svh = self.svh.cpu()
+        if self.trellis is not None: self.trellis = self.trellis.cpu()
+        if self.bias is not None: self.bias = self.bias.cpu()
+
+
+    def unswap_cpu(self):
+        if self.swap_device is None:
+            return
+        if self.su is not None: self.su = self.su.to(self.swap_device)
+        if self.sv is not None: self.sv = self.sv.to(self.swap_device)
+        if self.suh is not None: self.suh = self.suh.to(self.swap_device)
+        if self.svh is not None: self.svh = self.svh.to(self.swap_device)
+        if self.trellis is not None: self.trellis = self.trellis.to(self.swap_device)
+        if self.bias is not None: self.bias = self.bias.to(self.swap_device)
+        self.swap_device = None
+
+
+    def tp_export(self, plan, producer):
+        return {
+            "cls": LinearEXL3,
+            "in_features": self.in_features,
+            "out_features": self.out_features,
+            "suh": producer.send(self.suh),
+            "svh": producer.send(self.svh),
+            "trellis": producer.send(self.trellis),
+            "bias": producer.send(self.bias),
+            "mcg": producer.send(self.mcg_tensor),
+            "mul1": producer.send(self.mul1_tensor),
+            "out_dtype": self.out_dtype,
+        }
+
+
+    @staticmethod
+    def tp_import_split(local_context, exported, plan, split):
+        consumer = local_context["consumer"]
+        device = local_context["device"]
+        id_suh = exported["suh"]
+        id_svh = exported["svh"]
+        id_trellis = exported["trellis"]
+        id_bias = exported["bias"]
+        mcg = consumer.recv(exported["mcg"], cuda = True)
+        mul1 = consumer.recv(exported["mul1"], cuda = True)
+
+        if split is not None:
+            split_out, first, last = split
+        else:
+            split_out, first, last = True, 0, exported["out_features"]
+
+        if split_out:
+            suh = consumer.recv(id_suh, cuda = True)
+            svh = consumer.recv(id_svh, cuda = True, slice_dim = 0, first = first, last = last)
+            trellis = consumer.recv(id_trellis, cuda = True, slice_dim = 1, first = first // 16, last = last // 16)
+            bias = consumer.recv(id_bias, cuda = True, slice_dim = 0, first = first, last = last)
+            in_features = exported["in_features"]
+            out_features = last - first
+        else:
+            suh = consumer.recv(id_suh, cuda = True, slice_dim = 0, first = first, last = last)
+            svh = consumer.recv(id_svh, cuda = True)
+            trellis = consumer.recv(id_trellis, cuda = True, slice_dim = 0, first = first // 16, last = last // 16)
+            bias = consumer.recv(id_bias, cuda = True) if first == 0 else None
+            in_features = last - first
+            out_features = exported["out_features"]
+
+        module = LinearEXL3(
+            config = None,
+            in_features = in_features,
+            out_features = out_features,
+            scale = None,
+            su = None,
+            sv = None,
+            suh = suh,
+            svh = svh,
+            trellis = trellis,
+            mcg = mcg,
+            mul1 = mul1,
+            bias = bias,
+            out_dtype = exported["out_dtype"],
+        )
+        return module
+
+
+    @staticmethod
+    def tp_import_split_3(local_context, exported, plan, split_0, split_1, split_2, dbg = False):
+        return LinearEXL3.tp_import_split_n(local_context, exported, plan, [split_0, split_1, split_2], dbg)
+
+
+    @staticmethod
+    def tp_import_split_n(local_context, exported, plan, splits, dbg = False):
+        consumer = local_context["consumer"]
+        device = local_context["device"]
+        id_suh = exported["suh"]
+        id_svh = exported["svh"]
+        id_trellis = exported["trellis"]
+        id_bias = exported["bias"]
+        mcg = consumer.recv(exported["mcg"], cuda = True)
+        mul1 = consumer.recv(exported["mul1"], cuda = True)
+
+        svh_ = []
+        trellis_ = []
+        bias_ = []
+        in_features = 0
+        out_features = 0
+
+        for split in splits:
+            assert split is not None
+            split_out, first, last = split
+            assert split_out
+
+            suh = consumer.recv(id_suh, cuda = True)
+            svh = consumer.recv(id_svh, cuda = True, slice_dim = 0, first = first, last = last)
+            trellis = consumer.recv(id_trellis, cuda = True, slice_dim = 1, first = first // 16, last = last // 16)
+            bias = consumer.recv(id_bias, cuda = True, slice_dim = 0, first = first, last = last)
+            in_features = exported["in_features"]
+            out_features += last - first
+            svh_.append(svh)
+            trellis_.append(trellis)
+            bias_.append(bias)
+
+        svh = torch.cat(svh_, dim = 0)
+        trellis = torch.cat(trellis_, dim = 1)
+        bias = torch.cat(bias_, dim = 0) if bias_[0] is not None else None
+
+        module = LinearEXL3(
+            config = None,
+            in_features = in_features,
+            out_features = out_features,
+            scale = None,
+            su = None,
+            sv = None,
+            suh = suh,
+            svh = svh,
+            trellis = trellis,
+            mcg = mcg,
+            mul1 = mul1,
+            bias = bias,
+            out_dtype = exported["out_dtype"],
+        )
+        return module

lna-lab/patches/exl3_gemm.cu.lna

@@ -0,0 +1,703 @@
+#include <cuda_fp16.h>
+#include "exl3_gemm.cuh"
+
+#include <c10/cuda/CUDAGuard.h>
+#include <ATen/cuda/CUDAContext.h>
+#include <cooperative_groups.h>
+namespace cg = cooperative_groups;
+#include "../util.h"
+#include "../util.cuh"
+#include "exl3_gemm_kernel.cuh"
+#include "exl3_kernel_map.cuh"
+#include "exl3_devctx.cuh"
+#include "exl3_gemv.cuh"
+#include "exl3_gemv_int8.cuh"
+#include "coop_autotune.cuh"
+#include <atomic>
+#include <cstdio>
+#include <set>
+#include <vector>
+
+int exl3_gemm_tilesize_k_g[] = {EXL3_GEMM_TILESIZE_K};
+int exl3_gemm_tilesize_n_g[] = {EXL3_GEMM_TILESIZE_N};
+int exl3_gemm_blockdim_g[] = {EXL3_GEMM_BLOCKDIM};
+
+/*
+EXL3 matmul, A @ B -> C
+
+- A: row-major A tensor, shape (m, k), dtype float16, contiguous
+- B: EXL3-quantized B tensor, shape (k//16, n//16, 16*K), dtype uint16
+- C: empty row-major C tensor, shape (m, n), dtype float16 or float32, contiguous. Does not need to be zero-initialized
+- suh: optional, packed input scales/flips, shape (k//16), dtype float16
+- A_had: required if suh given, may be reference to A, temporary storage for input transform, size and dtype as A
+- svh: optional, packed output scales/flips, shape (n//16), dtype float16
+
+limitations:
+- k % 16 == 0
+- n % 128 == 0
+*/
+
+std::set<void*> kernel_attr_set[MAX_DEVICES] = {};
+
+uint64_t roundup_pow2(uint64_t x)
+{
+    if (x == 0) return 1;
+    x--;
+    x |= x >> 1;
+	x |= x >> 2;
+	x |= x >> 4;
+	x |= x >> 8;
+	x |= x >> 16;
+	x |= x >> 32;
+    return x + 1;
+}
+
+
+// LNA-LAB: the autotuner times candidates with cudaEvent/cudaStreamSynchronize, which is
+// forbidden while the stream is being captured into a CUDA graph. If a shape reaches here
+// untuned during capture, fall back to the static heuristic instead of asserting.
+static inline bool lna_stream_is_capturing(cudaStream_t stream)
+{
+    cudaStreamCaptureStatus st = cudaStreamCaptureStatusNone;
+    if (cudaStreamIsCapturing(stream, &st) != cudaSuccess) { (void) cudaGetLastError(); return false; }
+    return st != cudaStreamCaptureStatusNone;
+}
+
+uint64_t gemm_autotune_hash
+(
+    int size_m,
+    int size_k,
+    int size_n,
+    int K,
+    bool c_fp32,
+    int device,
+    int cc,
+    int max_num_sms,
+    int cb
+)
+{
+    uint64_t h = 1469598103934665603ull;
+    auto mix = [&] (uint64_t v)
+    {
+        h ^= v;
+        h *= 1099511628211ull;
+    };
+    mix((uint64_t) MIN(roundup_pow2(size_m), 16));
+    mix((uint64_t) size_k);
+    mix((uint64_t) size_n);
+    mix((uint64_t) K);
+    mix(c_fp32 ? 1ull : 0ull);
+    mix((uint64_t) device);
+    mix((uint64_t) cc);
+    mix((uint64_t) max_num_sms);
+    mix((uint64_t) cb);
+    return h;
+}
+
+uint64_t mgemm_autotune_hash
+(
+    int size_m,
+    int size_k,
+    int size_n,
+    int K,
+    bool c_fp32,
+    int device,
+    int cc,
+    int max_num_sms,
+    int cb,
+    int bszm_in,
+    int bszm_out
+)
+{
+    uint64_t h = gemm_autotune_hash(size_m, size_k, size_n, K, c_fp32, device, cc, max_num_sms, cb);
+    auto mix = [&] (uint64_t v)
+    {
+        h ^= v;
+        h *= 1099511628211ull;
+    };
+    mix((uint64_t) MIN(bszm_in, 24));
+    mix((uint64_t) MIN(bszm_out, 24));
+    return h;
+}
+
+int exl3_gemm_gr
+(
+    const at::Tensor& A,
+    const at::Tensor& B,
+    at::Tensor& C,
+    const c10::optional<at::Tensor>& suh,
+    const c10::optional<at::Tensor>& A_had,
+    const c10::optional<at::Tensor>& svh,
+    int force_shape_idx,
+    bool mcg,
+    bool mul1,
+    int force_num_sms,
+    Graph* graph
+)
+{
+    const at::cuda::OptionalCUDAGuard device_guard(A.device());
+    cudaStream_t stream = graph ? graph->capture_stream : at::cuda::getCurrentCUDAStream().stream();
+
+    TORCH_CHECK_DIM(B, 3);
+    TORCH_CHECK_SHAPES(A, -1, B, 0, 16);
+    TORCH_CHECK_SHAPES(C, -1, B, 1, 16);
+    // TORCH_CHECK_SHAPES(A, 0, C, 0, 1);
+    TORCH_CHECK_DTYPE(A, kHalf);
+    TORCH_CHECK_DTYPE(B, kShort);
+    bool c_fp32 = C.dtype() == at::kFloat;
+    if (!c_fp32) TORCH_CHECK_DTYPE(C, kHalf);
+
+    // Get SU, optionally
+    const half* suh_ptr = (const half*) OPTPTR(suh);
+    half* A_had_ptr = nullptr;
+    if (suh_ptr)
+    {
+        // TORCH_CHECK_SHAPES(suh.value(), 0, A, 1, 1);
+        A_had_ptr = (half*) OPTPTR(A_had);
+        // TORCH_CHECK(A_had_ptr, "Must supply A_had with suh");
+        // TORCH_CHECK_SHAPES_FULL(A_had.value(), A);
+    }
+
+    // Get SV, optionally
+    const half* svh_ptr = (const half*) OPTPTR(svh);
+    // if (svh_ptr)
+        // TORCH_CHECK_SHAPES(svh.value(), 0, B, 1, 16);
+
+    // Device properties
+    int device;
+    cudaGetDevice(&device);
+    int num_sms = force_num_sms ? force_num_sms : DevCtx::instance().get_num_sms(device);
+    int cc = DevCtx::instance().get_cc(device);
+    int* locks = DevCtx::instance().get_locks(device);
+
+    // Dispatch
+    int K = B.size(2) / 16;
+    const half* A_ptr = (const half*) A.data_ptr();
+    const uint16_t* B_ptr = (const uint16_t*) B.data_ptr();
+    void* C_ptr = (void*) C.data_ptr();
+
+    int size_m = 1;
+    int dim = A.dim();
+    for (int d = 0; d < dim - 1; ++d) size_m *= A.size(d);
+    int size_k = A.size(-1);
+    int size_n = B.size(1) * 16;
+
+    // Select kernel
+    TORCH_CHECK(!(mcg && mul1), "Specified both mcg and mul1")
+    int cb = 0;
+    if (mcg) cb = 1;
+    if (mul1) cb = 2;
+
+    // Experimental fused int8-activation GEMV path (EXL3_INT8_GEMV=1) for mul1 tensors. Rows are
+    // processed as successive GEMV launches, so this is only sensible for small m (the reconstruct
+    // threshold keeps m <= 144 in practice). Not graph-capturable yet; graphed callers fall through
+    // to the regular kernel.
+    if (mul1 && exl3_gemv_int8_enabled())
+    {
+        if (exl3_gemv_int8(A, B, C, suh, A_had, svh, stream, graph))
+            return 0;
+    }
+
+    int block_dim;
+    int shape_idx;
+    fp_exl3_gemm_kernel kernel;
+
+    void* kernelArgs[] =
+    {
+        (void*)& A_ptr,
+        (void*)& B_ptr,
+        (void*)& C_ptr,
+        (void*)& size_m,
+        (void*)& size_k,
+        (void*)& size_n,
+        (void*)& locks,
+        (void*)& suh_ptr,
+        (void*)& A_had_ptr,
+        (void*)& svh_ptr
+    };
+
+    auto add_graph_args = [&](void* kernel_ptr)
+    {
+        if (graph)
+        {
+            graph->record_param(kernel_ptr, GP_gemm_A, 0);
+            graph->record_param(kernel_ptr, GP_gemm_B_trellis, 1);
+            graph->record_param(kernel_ptr, GP_gemm_C, 2);
+            graph->record_param(kernel_ptr, GP_gemm_B_suh, 7);
+            graph->record_param(kernel_ptr, GP_gemm_A_had, 8);
+            graph->record_param(kernel_ptr, GP_gemm_B_svh, 9);
+            graph->record_param(kernel_ptr, GP_end, 0);
+        }
+    };
+
+    // QTIP-style GEMV path for small m (exl3_gemv_kernel.cuh). Same kernel arguments, so graph
+    // recording is identical; falls through to the regular kernel when the heuristic declines
+    if (force_shape_idx <= 0 && force_num_sms <= 0)
+    {
+        void* gemv_kernel = nullptr;
+        if (exl3_gemv_try_launch
+        (
+            kernelArgs, size_m, size_k, size_n, K, cb, c_fp32,
+            suh_ptr && A_had_ptr && svh_ptr,
+            device, stream, &gemv_kernel, false
+        ))
+        {
+            add_graph_args(gemv_kernel);
+            cuda_check(cudaPeekAtLastError());
+            return 90;
+        }
+    }
+
+    bool autotune = force_shape_idx <= 0 && force_num_sms <= 0;
+    if (autotune)
+    {
+        uint64_t autotune_key = gemm_autotune_hash(MAX(size_m, 2), size_k, size_n, K, c_fp32, device, cc, num_sms, cb);
+        CoopAutotuneLaunch tuned;
+        if (CoopKernelAutotuner::launch_locked(autotune_key, kernelArgs, SMEM_MAX, stream, &tuned))
+        {
+            add_graph_args((void*) tuned.kernel);
+            cuda_check(cudaPeekAtLastError());
+            return tuned.tag;
+        }
+        if (lna_stream_is_capturing(stream))
+        {
+            static std::atomic<int> warned{0};
+            if (warned.fetch_add(1) == 0)
+                fprintf(stderr, "[exl3_gemm] LNA-LAB: untuned shape m=%d k=%d n=%d K=%d during CUDA graph capture; using static heuristic (pre-warm to tune)\n", size_m, size_k, size_n, K);
+            autotune = false;
+        }
+      }
+      if (autotune)
+      {
+        uint64_t autotune_key = gemm_autotune_hash(MAX(size_m, 2), size_k, size_n, K, c_fp32, device, cc, num_sms, cb);
+        CoopAutotuneLaunch tuned;
+        std::vector<CoopAutotuneCandidate> candidates;
+        for (int candidate_shape_idx = 1; candidate_shape_idx <= EXL3_GEMM_NUM_SHAPES; ++candidate_shape_idx)
+        {
+            if (!exl3_gemm_shape_compat(candidate_shape_idx, size_m, size_k, size_n, K)) continue;
+
+            fp_exl3_gemm_kernel candidate_kernel = get_gemm_kernel_ptr(K, candidate_shape_idx, c_fp32, cb);
+            if (!candidate_kernel) continue;
+
+            int tilesize_k = exl3_gemm_tilesize_k_g[candidate_shape_idx];
+            int tilesize_n = exl3_gemm_tilesize_n_g[candidate_shape_idx];
+            int max_slices = MAX(size_k / tilesize_k * size_n / tilesize_n, 1);
+            int max_candidate_sms = MAX(MIN(max_slices, num_sms), 1);
+
+            candidates.push_back
+            ({
+                (void*) candidate_kernel,
+                exl3_gemm_blockdim_g[candidate_shape_idx],
+                max_candidate_sms,
+                1,
+                max_candidate_sms,
+                candidate_shape_idx
+            });
+        }
+        TORCH_CHECK(!candidates.empty(), "exl3_gemm autotune: no compatible kernel shapes");
+
+        tuned = CoopKernelAutotuner::launch(autotune_key, candidates, kernelArgs, SMEM_MAX, stream, (size_t) size_k * size_n);
+        if (graph)
+        add_graph_args((void*) tuned.kernel);
+        cuda_check(cudaPeekAtLastError());
+        return tuned.tag;
+    }
+
+    kernel = select_exl3_gemm_kernel
+    (
+        cc, size_m, size_k, size_n, K, c_fp32,
+        force_shape_idx, &block_dim, &shape_idx,
+        &num_sms, cb
+    );
+    if (!kernel) return 0;
+
+    // Launch
+    if (kernel_attr_set[device].find((void*) kernel) == kernel_attr_set[device].end())
+    {
+        cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, SMEM_MAX);
+        kernel_attr_set[device].insert((void*) kernel);
+        cuda_check(cudaPeekAtLastError());
+    }
+    cudaLaunchCooperativeKernel
+    (
+        (void*) kernel,
+        num_sms,
+        block_dim,
+        kernelArgs,
+        SMEM_MAX,
+        stream
+    );
+    add_graph_args((void*) kernel);
+
+    cuda_check(cudaPeekAtLastError());
+    return shape_idx;
+}
+
+int exl3_gemm
+(
+    const at::Tensor& A,
+    const at::Tensor& B,
+    at::Tensor& C,
+    const c10::optional<at::Tensor>& suh,
+    const c10::optional<at::Tensor>& A_had,
+    const c10::optional<at::Tensor>& svh,
+    int force_shape_idx,
+    bool mcg,
+    bool mul1,
+    int force_num_sms
+)
+{
+    return exl3_gemm_gr
+    (
+        A,
+        B,
+        C,
+        suh,
+        A_had,
+        svh,
+        force_shape_idx,
+        mcg,
+        mul1,
+        force_num_sms,
+        nullptr
+    );
+}
+
+/*
+EXL3 batched/multi-matrix matmul.
+
+This is not a conventional batched A @ B. B, suh and svh are CUDA int64
+tensors containing device addresses (one address per quantized matrix), rather
+than the matrix data themselves. Entry q of each table describes one linear:
+
+    B[q]   -> EXL3 trellis, logically (k / 16, n / 16, 16 * K) uint16
+    suh[q] -> packed input scales/flips, logically (k / 16) float16
+    svh[q] -> packed output scales/flips, logically (n / 16) float16
+
+A is contiguous float16 [a_batches, m, k], C is contiguous float16 or
+float32 [c_batches, m, n], and A_had is float16 scratch with room for every
+active matrix. The kernel applies the input Hadamard transform into A_had,
+performs the selected EXL3 matmul, then applies the output transform.
+
+The active matrix/output slot j selects q = indices[j] when indices is given,
+or q = j otherwise. This supports the following modes:
+
+- Multiple inputs and outputs: A[j] @ B[q] -> C[j].
+- One input, multiple outputs: when a_batches == 1, A[0] is broadcast and
+  transformed separately for each selected B[q], producing C[j]. This is used
+  for e.g. fused gate/up projections and MoE expert fan-out.
+- Indexed matrices: indices is a contiguous int64 [*, num_indices] tensor;
+  the kernel reads its first num_indices entries as q values. Negative indices
+  skip that slot.
+- Weighted MoE reduction: weights is a float16 tensor parallel to indices.
+  Each transformed result is multiplied by weights[j], then all active C[j]
+  are summed into C[0]. C therefore also serves as per-expert scratch; only
+  C[0] is the reduced result.
+- Expert-range filtering: with min_index >= 0, selections outside
+  [min_index, max_index) are removed and retained indices are rebased by
+  min_index. This allows B/suh/svh to be local pointer tables for an expert
+  shard. At num_tokens == 1 the retained indices (and their weights) are
+  compacted; at num_tokens > 1 out-of-range slots are instead masked to -1 in
+  place, preserving the per-token slot groups the final reduction depends on
+  (and, with bszm_in > 1, the slot -> input-row correspondence).
+
+Without weights, every active C[j] is a separate output. The active slot count
+is max(a_batches, c_batches), capped to num_indices when indices is present.
+
+Limitations: k must be divisible by 16 and n by 128. Range filtering supports
+at most 128 slots (the kernel's index-compaction capacity).
+*/
+
+int exl3_mgemm_gr
+(
+    const at::Tensor& A,
+    const at::Tensor& B,
+    at::Tensor& C,
+    const at::Tensor& suh,
+    const at::Tensor& A_had,
+    const at::Tensor& svh,
+    const c10::optional<at::Tensor>& indices,
+    const c10::optional<at::Tensor>& weights,
+    int K,
+    int force_shape_idx,
+    bool mcg,
+    bool mul1,
+    int min_index,
+    int max_index,
+    int force_num_sms,
+    Graph* graph,
+    int num_tokens,
+    const c10::optional<at::Tensor>& size_n_list,
+    const c10::optional<at::Tensor>& c_ptrs
+)
+{
+    const at::cuda::OptionalCUDAGuard device_guard(A.device());
+    cudaStream_t stream = graph ? graph->capture_stream : at::cuda::getCurrentCUDAStream().stream();
+
+    // num_tokens > 1 with expert-range filtering (min_index >= 0) uses position-preserving
+    // masking in the kernel (out-of-range slots marked -1 in place) instead of index
+    // compaction, so the grouped reduction's fixed per-token slot runs stay intact
+
+    TORCH_CHECK_DTYPE(A, kHalf);
+    TORCH_CHECK_DTYPE(B, kLong);
+    TORCH_CHECK_DTYPE(suh, kLong);
+    TORCH_CHECK_DTYPE(svh, kLong);
+    bool c_fp32 = C.dtype() == at::kFloat;
+    if (!c_fp32) TORCH_CHECK_DTYPE(C, kHalf);
+    TORCH_CHECK_DIM(A, 3);
+    TORCH_CHECK_DIM(B, 1);
+    TORCH_CHECK_DIM(suh, 1);
+    TORCH_CHECK_DIM(svh, 1);
+    TORCH_CHECK_DIM(C, 3);
+
+    TORCH_CHECK_SHAPES(A, 1, C, 1, 1);
+    TORCH_CHECK_SHAPES(B, 0, suh, 0, 1);
+    TORCH_CHECK_SHAPES(B, 0, svh, 0, 1);
+
+    int bsz = A.size(1);
+    int bszm_in = A.size(0);
+    int bszm_out = C.size(0);
+
+    // Per-matrix output widths/pointers (uniform-width callers pass neither): C then only
+    // provides the dtype and the max width (locks/shape sizing); outputs go to c_ptrs
+    const int* size_n_list_ptr = nullptr;
+    void** c_list_ptr = nullptr;
+    if (size_n_list)
+    {
+        TORCH_CHECK(c_ptrs, "exl3_mgemm: size_n_list requires c_ptrs");
+        TORCH_CHECK_DTYPE(size_n_list.value(), kInt);
+        TORCH_CHECK_DTYPE(c_ptrs.value(), kLong);
+        TORCH_CHECK(num_tokens == 1 && min_index < 0 && !weights,
+                    "exl3_mgemm: per-matrix widths incompatible with multi-token/filtering/weights");
+        size_n_list_ptr = (const int*) size_n_list.value().data_ptr();
+        c_list_ptr = (void**) c_ptrs.value().data_ptr();
+        bszm_out = (int) c_ptrs.value().size(0);
+    }
+    int bszm = MAX(bszm_in, bszm_out);
+
+    // The kernel writes one hadamard-transformed input slab PER MATRIX (A_had + j * m * k);
+    // an undersized scratch is silent OOB corruption (found the hard way)
+    TORCH_CHECK(A_had.numel() >= (int64_t) bszm * A.size(1) * A.size(2),
+                "exl3_mgemm: A_had must hold bszm * m * k elements");
+
+    const int64_t* indices_ptr = (const int64_t*) OPTPTR(indices);
+    const half* weights_ptr = (const half*) OPTPTR(weights);
+
+    if (indices)
+    {
+        TORCH_CHECK_DIM(indices.value(), 2);
+        int num_indices = indices.value().size(1);
+        TORCH_CHECK(num_indices <= bszm_in || num_indices <= bszm_out, "mgemm: too many indices for tensor batch");
+        if (bszm_in > num_indices) bszm_in = num_indices;
+        if (bszm_out > num_indices) bszm_out = num_indices;
+    }
+
+    if (weights)
+    {
+        TORCH_CHECK_DIM(weights.value(), 2);
+    }
+
+    int size_m = A.size(1);
+    int size_k = A.size(2);
+    int size_n = C.size(2);
+
+    // Device properties
+    int device;
+    cudaGetDevice(&device);
+    int total_sms = DevCtx::instance().get_num_sms(device);
+    int num_sms = force_num_sms ? force_num_sms : total_sms;
+    int cc = DevCtx::instance().get_cc(device);
+    int* locks = DevCtx::instance().get_locks(device);
+
+    // Dispatch
+    const half* A_ptr = (const half*) A.data_ptr();
+    const uintptr_t* B_ptr_ptr = (const uintptr_t*) B.data_ptr();
+    void* C_ptr = (void*) C.data_ptr();
+    const half* A_had_ptr = (const half*) A_had.data_ptr();
+    const uintptr_t* suh_ptr_ptr = (const uintptr_t*) suh.data_ptr();
+    const uintptr_t* svh_ptr_ptr = (const uintptr_t*) svh.data_ptr();
+
+    // Select kernel
+    TORCH_CHECK(!(mcg && mul1), "Specified both mcg and mul1")
+    int cb = 0;
+    if (mcg) cb = 1;
+    if (mul1) cb = 2;
+
+    int shape_idx;
+    int block_dim;
+    fp_exl3_mgemm_kernel kernel;
+    int concurrency;
+
+    void* kernelArgs[] =
+    {
+        (void*)& A_ptr,
+        (void*)& B_ptr_ptr,
+        (void*)& C_ptr,
+        (void*)& size_m,
+        (void*)& size_k,
+        (void*)& size_n,
+        (void*)& locks,
+        (void*)& suh_ptr_ptr,
+        (void*)& A_had_ptr,
+        (void*)& svh_ptr_ptr,
+        (void*)& indices_ptr,
+        (void*)& weights_ptr,
+        (void*)& bszm_in,
+        (void*)& bszm_out,
+        (void*)& min_index,
+        (void*)& max_index,
+        (void*)& num_tokens,
+        (void*)& size_n_list_ptr,
+        (void*)& c_list_ptr
+    };
+
+    auto add_graph_args = [&](void* kernel_ptr)
+    {
+        if (graph)
+        {
+            graph->record_param(kernel_ptr, GP_mgemm_A, 0);
+            graph->record_param(kernel_ptr, GP_mgemm_C, 2);
+            graph->record_param(kernel_ptr, GP_mgemm_indices, 10);
+            graph->record_param(kernel_ptr, GP_mgemm_weights, 11);
+            graph->record_param(kernel_ptr, GP_end, 0);
+        }
+    };
+
+    bool autotune = force_shape_idx <= 0 && force_num_sms <= 0;
+    if (autotune)
+    {
+        uint64_t autotune_key = mgemm_autotune_hash
+        (
+            size_m, size_k, size_n, K, c_fp32, device, cc, total_sms, cb, bszm_in, bszm_out
+        );
+
+        CoopAutotuneLaunch tuned;
+        if (CoopKernelAutotuner::launch_locked(autotune_key, kernelArgs, SMEM_MAX, stream, &tuned))
+        {
+            add_graph_args((void*) tuned.kernel);
+            cuda_check(cudaPeekAtLastError());
+            return tuned.tag;
+        }
+        if (!graph && !lna_stream_is_capturing(stream))
+        {
+            std::vector<CoopAutotuneCandidate> candidates;
+            for (int candidate_shape_idx = 1; candidate_shape_idx <= EXL3_GEMM_NUM_SHAPES; ++candidate_shape_idx)
+            {
+                if (!exl3_gemm_shape_compat(candidate_shape_idx, size_m, size_k, size_n, K)) continue;
+
+                fp_exl3_mgemm_kernel candidate_kernel = get_mgemm_kernel_ptr(K, candidate_shape_idx, c_fp32, cb);
+                if (!candidate_kernel) continue;
+
+                int tilesize_k = exl3_gemm_tilesize_k_g[candidate_shape_idx];
+                int tilesize_n = exl3_gemm_tilesize_n_g[candidate_shape_idx];
+                int max_slices = MAX(size_k / tilesize_k * size_n / tilesize_n, 1);
+                int max_candidate_sms = MAX(MIN(max_slices, total_sms), 1);
+
+                candidates.push_back
+                ({
+                    (void*) candidate_kernel,
+                    exl3_gemm_blockdim_g[candidate_shape_idx],
+                    max_candidate_sms,
+                    bszm,
+                    total_sms,
+                    candidate_shape_idx
+                });
+            }
+            TORCH_CHECK(!candidates.empty(), "exl3_mgemm autotune: no compatible kernel shapes");
+
+            tuned = CoopKernelAutotuner::launch(autotune_key, candidates, kernelArgs, SMEM_MAX, stream, (size_t) size_k * size_n * bszm);
+            add_graph_args((void*) tuned.kernel);
+
+            // DBGI10(size_m, size_k, size_n, K, bszm_in, bszm_out, tuned.tag, tuned.block_dim, tuned.num_sms, tuned.concurrency);
+
+            cuda_check(cudaPeekAtLastError());
+            return tuned.tag;
+        }
+    }
+
+    kernel = select_exl3_mgemm_kernel
+    (
+        cc, size_m, size_k, size_n, K, c_fp32,
+        force_shape_idx, &block_dim, &shape_idx,
+        &num_sms, cb, bszm_in, bszm_out
+    );
+    int tilesize_k = exl3_gemm_tilesize_k_g[shape_idx];
+    int tilesize_n = exl3_gemm_tilesize_n_g[shape_idx];
+    int tiles = MAX(size_k / tilesize_k * size_n / tilesize_n, 1);
+    num_sms = tiles;
+    if (num_sms * bszm > total_sms) num_sms = MAX(total_sms / bszm, 1);
+    if (num_sms <= total_sms && tiles / num_sms > 48) num_sms = MIN(total_sms, num_sms * 2);
+    concurrency = MIN(total_sms / num_sms, bszm);
+
+    // DBGI10(size_m, size_k, size_n, K, bszm_in, bszm_out, shape_idx, block_dim, num_sms, concurrency);
+
+    // Launch bigger grid if possible
+    dim3 block_grid(num_sms, 1, concurrency);
+
+    // Launch
+    if (kernel_attr_set[device].find((void*) kernel) == kernel_attr_set[device].end())
+    {
+        cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, SMEM_MAX);
+        kernel_attr_set[device].insert((void*) kernel);
+    }
+
+    cudaLaunchCooperativeKernel
+    (
+        (void*) kernel,
+        block_grid,
+        block_dim,
+        kernelArgs,
+        SMEM_MAX,
+        stream
+    );
+    add_graph_args((void*) kernel);
+
+    cuda_check(cudaPeekAtLastError());
+    return shape_idx;
+}
+
+int exl3_mgemm
+(
+    const at::Tensor& A,
+    const at::Tensor& B,
+    at::Tensor& C,
+    const at::Tensor& suh,
+    const at::Tensor& A_had,
+    const at::Tensor& svh,
+    const c10::optional<at::Tensor>& indices,
+    const c10::optional<at::Tensor>& weights,
+    int K,
+    int force_shape_idx,
+    uint32_t mcg_mult,
+    uint32_t mul1_mult,
+    int min_index,
+    int max_index,
+    int force_num_sms,
+    int num_tokens,
+    const c10::optional<at::Tensor>& size_n_list,
+    const c10::optional<at::Tensor>& c_ptrs
+)
+{
+    return exl3_mgemm_gr
+    (
+        A,
+        B,
+        C,
+        suh,
+        A_had,
+        svh,
+        indices,
+        weights,
+        K,
+        force_shape_idx,
+        mcg_mult,
+        mul1_mult,
+        min_index,
+        max_index,
+        force_num_sms,
+        nullptr,
+        num_tokens,
+        size_n_list,
+        c_ptrs
+    );
+}

lna-lab/patches/patch_dsv4_attention_compressor_exl3.py

@@ -0,0 +1,62 @@
+#!/usr/bin/env python3
+"""LNA-LAB: DSV4 attention.py reads compressor.fused_wkv_wgate.weight directly (torch.mm). When the
+module was built by vllm-exl3 (no .weight, has _exl3_linears), run the EXL3 shards instead.
+Idempotent, exact anchors, backup attention.py.orig-lna2."""
+import shutil, sys
+from pathlib import Path
+
+HELPER = '''
+def _lna_kv_score(module, hidden_states):
+    """LNA-LAB: fused_wkv_wgate as bf16 weight (torch.mm) or as EXL3 shards (vllm-exl3)."""
+    linears = getattr(module, "_exl3_linears", None)
+    if linears:
+        x = hidden_states.reshape(-1, hidden_states.shape[-1]).to(torch.float16).contiguous()
+        outs = [lin.forward(x, {}, out_dtype=torch.float32) for lin in linears]
+        out = outs[0] if len(outs) == 1 else torch.cat(outs, dim=-1)
+        return out.reshape(*hidden_states.shape[:-1], out.shape[-1])
+    return torch.mm(hidden_states, module.fused_wkv_wgate.weight.T, out_dtype=torch.float32)
+
+'''
+EDITS = [
+    ('''                return torch.mm(
+                    hidden_states,
+                    compressor.fused_wkv_wgate.weight.T,
+                    out_dtype=torch.float32,
+                )
+''', '''                return _lna_kv_score(compressor.fused_wkv_wgate, hidden_states)  # LNA-LAB
+'''),
+    ('''                return torch.mm(
+                    hidden_states,
+                    indexer.compressor.fused_wkv_wgate.weight.T,
+                    out_dtype=torch.float32,
+                )
+''', '''                return _lna_kv_score(indexer.compressor.fused_wkv_wgate, hidden_states)  # LNA-LAB
+'''),
+]
+
+def main():
+    root = Path(sys.argv[1]) if len(sys.argv) > 1 else None
+    if root is None:
+        import vllm
+        root = Path(vllm.__file__).resolve().parent
+    f = root / "models" / "deepseek_v4" / "attention.py"
+    text = f.read_text()
+    if "_lna_kv_score" in text:
+        print(f"{f}: already patched"); return 0
+    for old, new in EDITS:
+        if text.count(old) != 1:
+            raise SystemExit(f"{f}: expected 1 anchor, found {text.count(old)}: {old[:60]!r}")
+        text = text.replace(old, new)
+    # helper: insert before the first class definition
+    idx = text.index("\nclass ")
+    text = text[:idx] + "\n" + HELPER + text[idx:]
+    # the helper only reads .weight in the fallback path; keep it after fused_wkv_wgate module refs
+    text = text.replace("return torch.mm(hidden_states, module.fused_wkv_wgate.weight.T, out_dtype=torch.float32)",
+                        "return torch.mm(hidden_states, module.weight.T, out_dtype=torch.float32)")
+    bak = f.with_suffix(f.suffix + ".orig-lna2")
+    if not bak.exists(): shutil.copy2(f, bak)
+    f.write_text(text); compile(text, str(f), "exec"); print(f"{f}: patched")
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())

lna-lab/patches/patch_dsv4_aux_streams_env.py

@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+"""LNA-LAB: let LNA_DSV4_AUX_STREAMS=0 disable DSV4's attention aux CUDA streams (EXL3 dense GEMMs are
+cooperative kernels sharing one device lock buffer; running them concurrently on aux streams deadlocks).
+Idempotent exact-anchor patch of nvidia/model.py, backup model.py.orig-lna3."""
+import shutil, sys
+from pathlib import Path
+OLD = "        aux_stream_list = [torch.cuda.Stream() for _ in range(3)]\n"
+NEW = ("        import os as _lna_os  # LNA-LAB\n"
+       "        aux_stream_list = (None if _lna_os.environ.get(\"LNA_DSV4_AUX_STREAMS\", \"1\") == \"0\"\n"
+       "                           else [torch.cuda.Stream() for _ in range(3)])  # LNA-LAB: EXL3 dense needs sequential GEMMs\n")
+def main():
+    import vllm
+    f = Path(vllm.__file__).resolve().parent / "models" / "deepseek_v4" / "nvidia" / "model.py"
+    t = f.read_text()
+    if "LNA_DSV4_AUX_STREAMS" in t: print(f"{f}: already patched"); return 0
+    if t.count(OLD) != 1: raise SystemExit(f"{f}: expected 1 anchor, found {t.count(OLD)}")
+    b = f.with_suffix(f.suffix + ".orig-lna3"); b.exists() or shutil.copy2(f, b)
+    t = t.replace(OLD, NEW); compile(t, str(f), "exec"); f.write_text(t); print(f"{f}: patched"); return 0
+if __name__ == "__main__": sys.exit(main())

lna-lab/patches/patch_dsv4_dense_exl3.py

@@ -0,0 +1,238 @@
+#!/usr/bin/env python3
+"""Patch the DSV4 vLLM fork to consume the dense EXL3 overlay at TP=8.
+
+The image keeps vLLM's model files private, so this recipe is deliberately an
+exact-anchor patch.  Every edit is idempotent, syntax-checked, and backed up as
+``*.orig-lna`` before it is written.
+
+Usage: python recipe-lna/patch_dsv4_dense_exl3.py [path/to/site-packages/vllm]
+"""
+
+from __future__ import annotations
+
+import ast
+import shutil
+import sys
+from pathlib import Path
+
+
+# LNA-LAB: the common compressor class is instantiated for both attention and
+# indexer compressors, so this one constructor anchor covers both instances.
+COMPRESSOR_ANCHOR = (
+    "            bias=False,\n"
+    "            return_bias=False,\n"
+    "            quant_config=None,\n"
+    "            disable_tp=True,\n"
+)
+COMPRESSOR_PATCH = (
+    "            bias=False,\n"
+    "            return_bias=False,\n"
+    "            # LNA-LAB: dense EXL3 must receive the real quant config.\n"
+    "            quant_config=vllm_config.quant_config,\n"
+    "            disable_tp=True,\n"
+)
+
+
+OPROJ_SIGNATURE = (
+    "    def _o_proj(self, o: torch.Tensor, positions: torch.Tensor)"
+    " -> torch.Tensor:\n"
+)
+OPROJ_BASELINE_ANCHOR = OPROJ_SIGNATURE + "        return deep_gemm_fp8_o_proj(\n"
+
+# This is the exact prefix emitted by recipe/scripts/patch_dsv4_stock028.py;
+# accepting it keeps this patch composable with the existing serving recipe.
+OPROJ_STOCK028_ANCHOR = (
+    OPROJ_SIGNATURE
+    + "        if self.wo_a.weight.dtype != torch.float8_e4m3fn:\n"
+    + "            # bf16 wo_a (packs that keep non-routed weights unquantized):\n"
+    + "            # the fp8 einsum path needs block scales that do not exist, so\n"
+    + "            # use the Triton inverse-RoPE + bf16 einsum reference instead.\n"
+    + "            from vllm.v1.attention.ops.rocm_aiter_mla_sparse import (\n"
+    + "                rocm_inv_rope_einsum,\n"
+    + "            )\n"
+    + "\n"
+    + "            z = rocm_inv_rope_einsum(\n"
+    + "                self.rotary_emb,\n"
+    + "                o,\n"
+    + "                positions,\n"
+    + "                self.rope_head_dim,\n"
+    + "                self.n_local_groups,\n"
+    + "                self.o_lora_rank,\n"
+    + "                self.wo_a,\n"
+    + "            )\n"
+    + "            return self.wo_b(z.flatten(1))\n"
+    + "        return deep_gemm_fp8_o_proj(\n"
+)
+
+
+OPROJ_COMMON = (
+    OPROJ_SIGNATURE
+    # LNA-LAB: rank-local EXL3 wo_a is a regular one-group LinearEXL3 call.
+    + "        exl3_linears = getattr(self.wo_a, \"_exl3_linears\", None)\n"
+    + "        if exl3_linears is not None:\n"
+    + "            if self.n_local_groups != 1:\n"
+    + "                raise NotImplementedError(\n"
+    + "                    \"EXL3 wo_a requires TP=8 (one local group); \"\n"
+    + "                    f\"got n_local_groups={self.n_local_groups}\"\n"
+    + "                )\n"
+    + "            if len(exl3_linears) != 1 or exl3_linears[0] is None:\n"
+    + "                raise RuntimeError(\n"
+    + "                    \"EXL3 wo_a has no rank-local LinearEXL3 instance\"\n"
+    + "                )\n"
+    + "            # LNA-LAB: use the same inverse GPT-J RoPE convention as the\n"
+    + "            # Triton rocm_inv_rope_einsum reference before the EXL3 GEMM.\n"
+    + "            try:\n"
+    + "                from vllm.v1.attention.ops.rocm_aiter_mla_sparse import (\n"
+    + "                    _fused_inverse_rope_gptj,\n"
+    + "                )\n"
+    + "                o_ref = _fused_inverse_rope_gptj(\n"
+    + "                    o, positions, self.rotary_emb.cos_sin_cache,\n"
+    + "                    self.rope_head_dim,\n"
+    + "                )\n"
+    + "            except ImportError:\n"
+    + "                from vllm.v1.attention.ops.rocm_aiter_mla_sparse import (\n"
+    + "                    _apply_inv_rope_ref,\n"
+    + "                )\n"
+    + "                o_ref = _apply_inv_rope_ref(\n"
+    + "                    self.rotary_emb, o, positions, self.rope_head_dim\n"
+    + "                )\n"
+    + "            b = o_ref.shape[0]\n"
+    + "            exl3_linear = exl3_linears[0]\n"
+    + "            z = exl3_linear.forward(\n"
+    + "                o_ref.reshape(b, -1).contiguous().half(),\n"
+    + "                {},\n"
+    + "                out_dtype=torch.float32,\n"
+    + "            )\n"
+    + "            z = z.to(torch.bfloat16).reshape(b, 1, self.o_lora_rank)  # LNA-LAB: EXL3 GEMM emits fp32/fp16\n"
+    + "            return self.wo_b(z.flatten(1))\n"
+    + "        wo_a_weight = getattr(self.wo_a, \"weight\", None)\n"
+    + "        if wo_a_weight is None:\n"
+    + "            raise RuntimeError(\n"
+    + "                \"DSV4 wo_a has neither dense weight nor EXL3 linears\"\n"
+    + "            )\n"
+    + "        if wo_a_weight.dtype != torch.float8_e4m3fn:\n"
+    + "            # LNA-LAB: preserve the existing bf16 reference path.\n"
+    + "            from vllm.v1.attention.ops.rocm_aiter_mla_sparse import (\n"
+    + "                rocm_inv_rope_einsum,\n"
+    + "            )\n"
+    + "\n"
+    + "            z = rocm_inv_rope_einsum(\n"
+    + "                self.rotary_emb,\n"
+    + "                o,\n"
+    + "                positions,\n"
+    + "                self.rope_head_dim,\n"
+    + "                self.n_local_groups,\n"
+    + "                self.o_lora_rank,\n"
+    + "                self.wo_a,\n"
+    + "            )\n"
+    + "            return self.wo_b(z.flatten(1))\n"
+    + "        return deep_gemm_fp8_o_proj(\n"
+)
+
+
+# LNA-LAB: AutoWeightsLoader maps the pre-fusion bake names to wo_a.tensors;
+# integer ids are then consumed by Exl3LinearMethod as the physical TP rank.
+WO_A_MAPPING_ANCHOR = (
+    '            ("compressor.fused_wkv_wgate", "compressor.wkv", 0),\n'
+    '            ("compressor.fused_wkv_wgate", "compressor.wgate", 1),\n'
+    "        ]\n"
+)
+WO_A_MAPPING_PATCH = (
+    '            ("compressor.fused_wkv_wgate", "compressor.wkv", 0),\n'
+    '            ("compressor.fused_wkv_wgate", "compressor.wgate", 1),\n'
+    "            # LNA-LAB: each pre-fusion wo_a slice is one rank-local shard.\n"
+    '            ("attn.wo_a", "attn.wo_a.slice.0", 0),\n'
+    '            ("attn.wo_a", "attn.wo_a.slice.1", 1),\n'
+    '            ("attn.wo_a", "attn.wo_a.slice.2", 2),\n'
+    '            ("attn.wo_a", "attn.wo_a.slice.3", 3),\n'
+    '            ("attn.wo_a", "attn.wo_a.slice.4", 4),\n'
+    '            ("attn.wo_a", "attn.wo_a.slice.5", 5),\n'
+    '            ("attn.wo_a", "attn.wo_a.slice.6", 6),\n'
+    '            ("attn.wo_a", "attn.wo_a.slice.7", 7),\n'
+    "        ]\n"
+)
+
+
+def patch_file(path: Path, anchor: str, patched: str, expect: int) -> str:
+    text = path.read_text(encoding="utf-8")
+    have = text.count(patched)
+    if have == expect:
+        return "already patched"
+    if have:
+        raise SystemExit(f"{path}: partially patched ({have}/{expect}); refusing to guess")
+    count = text.count(anchor)
+    if count != expect:
+        raise SystemExit(f"{path}: expected {expect} anchor(s), found {count}")
+    backup = path.with_name(path.name + ".orig-lna")
+    if not backup.exists():
+        shutil.copy2(path, backup)
+    new_text = text.replace(anchor, patched)
+    ast.parse(new_text)
+    path.write_text(new_text, encoding="utf-8")
+    return f"patched x{expect} (backup {backup.name})"
+
+
+def patch_flashinfer(path: Path) -> str:
+    text = path.read_text(encoding="utf-8")
+    marker = "exl3_linears = getattr(self.wo_a, \"_exl3_linears\", None)"
+    if text.count(marker) == 2:
+        return "already patched"
+    if marker in text:
+        raise SystemExit(f"{path}: partially patched; refusing to guess")
+    baseline_count = text.count(OPROJ_BASELINE_ANCHOR)
+    stock_count = text.count(OPROJ_STOCK028_ANCHOR)
+    if baseline_count == 2 and stock_count == 0:
+        new_text = text.replace(OPROJ_BASELINE_ANCHOR, OPROJ_COMMON)
+        expected = "baseline"
+    elif stock_count == 2 and baseline_count == 0:
+        new_text = text.replace(OPROJ_STOCK028_ANCHOR, OPROJ_COMMON)
+        expected = "stock028-patched"
+    else:
+        raise SystemExit(
+            f"{path}: expected two matching _o_proj methods; "
+            f"baseline={baseline_count}, stock028={stock_count}"
+        )
+    backup = path.with_name(path.name + ".orig-lna")
+    if not backup.exists():
+        shutil.copy2(path, backup)
+    ast.parse(new_text)
+    path.write_text(new_text, encoding="utf-8")
+    return f"patched x2 ({expected}; backup {backup.name})"
+
+
+def main() -> int:
+    if len(sys.argv) > 1:
+        root = Path(sys.argv[1]).expanduser().resolve()
+    else:
+        import vllm
+
+        root = Path(vllm.__file__).resolve().parent
+    model_dir = root / "models" / "deepseek_v4"
+    nvidia_dir = model_dir / "nvidia"
+    targets = {
+        "compressor.py": model_dir / "compressor.py",
+        "flashinfer_sparse.py": nvidia_dir / "flashinfer_sparse.py",
+        "model.py": nvidia_dir / "model.py",
+    }
+    missing = [str(path) for path in targets.values() if not path.is_file()]
+    if missing:
+        print("missing DSV4 vLLM file(s):", ", ".join(missing), file=sys.stderr)
+        return 2
+
+    try:
+        print("compressor.py:", patch_file(
+            targets["compressor.py"], COMPRESSOR_ANCHOR, COMPRESSOR_PATCH, 1
+        ))
+        print("flashinfer_sparse.py:", patch_flashinfer(targets["flashinfer_sparse.py"]))
+        print("model.py:", patch_file(
+            targets["model.py"], WO_A_MAPPING_ANCHOR, WO_A_MAPPING_PATCH, 1
+        ))
+    except (OSError, SystemExit) as exc:
+        print(exc, file=sys.stderr)
+        return 1
+    print("DSV4_DENSE_EXL3_PATCH_OK")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

lna-lab/patches/vllm_exl3_exl3.py.lna

@@ -0,0 +1,1528 @@
+# SPDX-License-Identifier: Apache-2.0
+"""EXL3/MCG trellis quantization for GLM-5.3-Flash routed experts.
+
+Checkpoint ABI used by this pack:
+  quant_method=exl3, codebook=mcg, scope=glm53_routed_experts_only
+  per expert matrix: trellis (int16) + suh/svh (fp16) + mcg (int32 marker)
+
+Non-routed tensors stay native (UnquantizedLinearMethod). Experts never
+expand to a persistent BF16 weight; LinearEXL3 / exllamav3_ext runs the
+trellis GEMM. TP=2 shards gate/up column-wise and down row-wise; the MoE
+runner all-reduces the combined output.
+"""
+
+from __future__ import annotations
+
+import importlib
+import os
+from typing import TYPE_CHECKING, Any
+
+import re
+import torch
+import torch.nn.functional as F
+from torch.nn.parameter import Parameter
+
+from vllm.logger import init_logger
+from vllm.model_executor.layers.fused_moe.config import FusedMoEQuantConfig
+from vllm.model_executor.layers.fused_moe.fused_moe_method_base import (
+    FusedMoEMethodBase,
+)
+from vllm.model_executor.layers.linear import (
+    LinearBase,
+    LinearMethodBase,
+    UnquantizedLinearMethod,
+)
+from vllm.model_executor.layers.quantization.base_config import QuantizationConfig
+from vllm.model_executor.layers.quantization import register_quantization_config
+from vllm.model_executor.utils import set_weight_attrs
+
+if TYPE_CHECKING:
+    from vllm.model_executor.layers.fused_moe.routed_experts import RoutedExperts
+    from vllm.model_executor.layers.fused_moe.runner.shared_experts import (
+        SharedExperts,
+    )
+
+# Under the "vllm." hierarchy so vLLM's logging config actually emits these
+# INFO lines; a bare module name is dropped and the load log shows nothing.
+logger = init_logger("vllm." + __name__)
+
+MCG_MULTIPLIER = 0xCBAC1FED
+MCG_MARKER_SIGNED_INT32 = -877912083
+MUL1_MULTIPLIER = 0x83DCD12D
+MUL1_MARKER_SIGNED_INT32 = -2082680531
+EXL3_SUFFIXES = ("trellis", "suh", "svh", "mcg", "mul1")
+_WO_A_SLICE_RE = re.compile(r"(?:^|\.)slice\.(?P<rank>[0-9]+)(?:\.|$)")
+SWIGLU_LIMIT_DEFAULT = 10.0
+TEMP_ROWS_FUSED = 2048
+MOE_ACT_SILU = 0
+# Shared fused scratch: decode is sequential across layers.
+_FUSED_TEMP_CACHE: dict[tuple, tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]] = {}
+
+
+def _narrow_tp(tensor: torch.Tensor, dim: int, tp_rank: int, tp_size: int) -> torch.Tensor:
+    if tp_size <= 1:
+        return tensor
+    size = int(tensor.shape[dim])
+    if size % tp_size:
+        raise ValueError(
+            f"EXL3 TP shard: dim {dim} size {size} is not divisible by tp={tp_size}"
+        )
+    chunk = size // tp_size
+    return tensor.narrow(dim, chunk * tp_rank, chunk).contiguous()
+
+
+def _effective_tp_for_layer(
+    layer: torch.nn.Module, global_tp_size: int, global_tp_rank: int
+) -> tuple[int, int]:
+    """Return the TP geometry that the checkpoint tensor actually uses."""
+    # LNA-LAB: replicated/disable_tp layers must not be narrowed by global TP.
+    try:
+        from vllm.model_executor.layers.linear import ReplicatedLinear
+    except ImportError:  # pragma: no cover - only old vLLM builds lack this class.
+        is_replicated = layer.__class__.__name__ == "ReplicatedLinear"
+    else:
+        is_replicated = isinstance(layer, ReplicatedLinear)
+    if getattr(layer, "disable_tp", False) or is_replicated:
+        return 1, 0
+    return int(global_tp_size), int(global_tp_rank)
+
+
+def _wo_a_slice_rank(loaded_shard_id: str | int | None) -> int | None:
+    """Decode a model-loader shard id or a literal ``wo_a.slice.N`` name."""
+    # LNA-LAB: dense-bake names carry the group rank; model.py maps them to ints.
+    if isinstance(loaded_shard_id, int):
+        return loaded_shard_id
+    if loaded_shard_id is None:
+        return None
+    if str(loaded_shard_id).isdigit():
+        return int(str(loaded_shard_id))
+    match = _WO_A_SLICE_RE.search(str(loaded_shard_id))
+    return int(match.group("rank")) if match else None
+
+
+def _is_wo_a_layer(layer: torch.nn.Module) -> bool:
+    # LNA-LAB: vLLM sets is_bmm after construction, so prefix is the fallback.
+    prefix = str(getattr(layer, "prefix", ""))
+    return bool(
+        _prefix_has_suffix(prefix, "attn.wo_a")
+        or (getattr(layer, "is_bmm", False) and prefix.endswith(".wo_a"))
+    )
+
+
+def shard_exl3_col(loaded: torch.Tensor, suffix: str, tp_rank: int, tp_size: int) -> torch.Tensor:
+    """Gate/up: trellis dim 1 and svh dim 0 are column-parallel."""
+    if suffix == "trellis":
+        return _narrow_tp(loaded, 1, tp_rank, tp_size)
+    if suffix == "svh":
+        return _narrow_tp(loaded, 0, tp_rank, tp_size)
+    return loaded.contiguous()
+
+
+def shard_exl3_row(loaded: torch.Tensor, suffix: str, tp_rank: int, tp_size: int) -> torch.Tensor:
+    """Down: trellis dim 0 and suh dim 0 are row-parallel."""
+    if suffix == "trellis":
+        return _narrow_tp(loaded, 0, tp_rank, tp_size)
+    if suffix == "suh":
+        return _narrow_tp(loaded, 0, tp_rank, tp_size)
+    return loaded.contiguous()
+
+
+def _install_exllamav3_namespace() -> None:
+    """Validate that the native ExLlamaV3 package and extension are importable."""
+    import exllamav3_ext  # noqa: F401  — compiled extension must exist
+
+    # ExLlamaV3 1.4+ imports cleanly as a regular package and its LinearEXL3
+    # constructor relies on the real NullConfig/InferParams implementation.
+    # Namespace stubs used by much older builds hide those classes and fail only
+    # after a full checkpoint load, so deliberately exercise the normal import.
+    importlib.import_module("exllamav3.modules.quant.exl3")
+
+
+def load_linear_exl3_cls():
+    _install_exllamav3_namespace()
+    return importlib.import_module("exllamav3.modules.quant.exl3").LinearEXL3
+
+
+def make_linear_exl3(
+    trellis: torch.Tensor,
+    suh: torch.Tensor,
+    svh: torch.Tensor,
+    mcg: torch.Tensor | None = None,
+    mul1: torch.Tensor | None = None,
+    *,
+    out_dtype: torch.dtype = torch.float16,
+):
+    """Build a LinearEXL3 over already-sharded packed tensors. No BF16 expand."""
+    cls = load_linear_exl3_cls()
+    return cls(
+        config=None,
+        in_features=int(suh.numel()),
+        out_features=int(svh.numel()),
+        trellis=trellis.contiguous(),
+        suh=suh.contiguous(),
+        svh=svh.contiguous(),
+        mcg=mcg.contiguous() if mcg is not None else None,
+        mul1=mul1.contiguous() if mul1 is not None else None,
+        out_dtype=out_dtype,
+        transformers_fix=True,
+    )
+
+
+def execute_exl3_linear(
+    x: torch.Tensor,
+    trellis: torch.Tensor,
+    suh: torch.Tensor,
+    svh: torch.Tensor,
+    mcg: torch.Tensor | None = None,
+    mul1: torch.Tensor | None = None,
+    *,
+    out_dtype: torch.dtype = torch.float32,
+) -> torch.Tensor:
+    """Real EXL3 expert GEMM entry (LinearEXL3 / exllamav3_ext)."""
+    inner = make_linear_exl3(trellis, suh, svh, mcg, mul1, out_dtype=torch.float16)
+    return inner.forward(x.contiguous().half(), {}, out_dtype=out_dtype)
+
+
+def fused_moe_enabled() -> bool:
+    return os.environ.get("EXL3_FUSED_MOE", "1") != "0"
+
+
+def load_exllamav3_ext():
+    import exllamav3_ext
+
+    return exllamav3_ext
+
+
+def _exl3_moe_accepts_num_active(fn) -> bool:
+    try:
+        import inspect
+
+        if "num_active" in inspect.signature(fn).parameters:
+            return True
+    except (TypeError, ValueError):
+        pass
+    doc = getattr(fn, "__doc__", None) or ""
+    return "num_active" in doc or "arg29" in doc or doc.count("arg") >= 30
+
+
+def pin_exl3_expert_map(
+    layer: torch.nn.Module, device: torch.device
+) -> torch.Tensor | None:
+    """Move expert_map onto `device` once. CUDA graph capture forbids a CPU→GPU copy."""
+    emap = getattr(layer, "expert_map", None)
+    if emap is None:
+        return None
+    if emap.device != device or emap.dtype != torch.long:
+        layer.expert_map = emap.to(device=device, dtype=torch.long)
+    return layer.expert_map
+
+
+def map_topk_to_local(
+    ids: torch.Tensor,
+    n_local: int,
+    expert_map: torch.Tensor | None,
+) -> torch.Tensor:
+    """ids (T, K) global expert ids → local ids, invalid/non-local → n_local sentinel.
+
+    `expert_map` must already live on `ids.device` (see pin_exl3_expert_map).
+    """
+    flat = ids.reshape(-1)
+    if expert_map is None:
+        invalid = (flat < 0) | (flat >= n_local)
+        return torch.where(invalid, flat.new_full(flat.shape, n_local), flat)
+    if expert_map.device != flat.device or expert_map.dtype != torch.long:
+        raise RuntimeError(
+            "EXL3 expert_map is not pinned to the hidden-state device; "
+            "call pin_exl3_expert_map before fused apply (CUDA graphs forbid the copy)"
+        )
+    n_global = int(expert_map.numel())
+    safe = flat.clamp(min=0, max=max(n_global - 1, 0))
+    mapped = expert_map[safe] if n_global else flat.new_full(flat.shape, n_local)
+    invalid = (flat < 0) | (flat >= n_global) | (mapped < 0) | (mapped >= n_local)
+    return torch.where(invalid, flat.new_full(flat.shape, n_local), mapped)
+
+
+def apply_exl3_python_loop(
+    x2d: torch.Tensor,
+    ids: torch.Tensor,
+    weights: torch.Tensor,
+    inners: list[dict[str, Any]],
+    expert_map: torch.Tensor | None,
+    limit: float,
+    *,
+    only_experts: set[int] | None = None,
+    out: torch.Tensor | None = None,
+) -> torch.Tensor:
+    """Unique-expert LinearEXL3 loop. `only_experts` is local ids (fat-expert fallback)."""
+    tokens, hidden = x2d.shape
+    if out is None:
+        out = torch.zeros(tokens, hidden, dtype=torch.float32, device=x2d.device)
+    unique = torch.unique(ids)
+    for raw in unique.tolist():
+        e_raw = int(raw)
+        if e_raw < 0:
+            continue
+        e = e_raw
+        if expert_map is not None:
+            mapped = int(expert_map[e].item()) if expert_map.numel() > e else e
+            if mapped < 0:
+                continue
+            e = mapped
+        if e >= len(inners):
+            continue
+        if only_experts is not None and e not in only_experts:
+            continue
+        token_idx, k_pos = (ids == int(raw)).nonzero(as_tuple=True)
+        h = x2d.index_select(0, token_idx)
+        pack = inners[e]
+        gate = pack["gate"].forward(h.contiguous().half(), {}, out_dtype=torch.float32)
+        up = pack["up"].forward(h.contiguous().half(), {}, out_dtype=torch.float32)
+        act = F.silu(gate.clamp(max=limit)) * up.clamp(min=-limit, max=limit)
+        down = pack["down"].forward(act.contiguous().half(), {}, out_dtype=torch.float32)
+        scale = weights[token_idx, k_pos].unsqueeze(-1).to(dtype=torch.float32)
+        out.index_add_(0, token_idx, down * scale)
+    return out
+
+
+def build_exl3_fused_state(layer: torch.nn.Module, inners: list[dict[str, Any]]) -> None:
+    """Pointer tables + fused temps, once after load. No per-token alloc."""
+    import exllamav3_ext
+
+    device = layer.w13_trellis.device
+    n_exp = len(inners)
+    hidden = int(layer._exl3_hidden_size)
+    intermediate = int(layer._exl3_intermediate_local)
+
+    def _ptrs(which: str, attr: str) -> torch.Tensor:
+        return torch.tensor(
+            [int(getattr(pack[which], attr).data_ptr()) for pack in inners],
+            dtype=torch.int64,
+            device=device,
+        )
+
+    layer._exl3_ptrs = {
+        "gate_trellis": _ptrs("gate", "trellis"),
+        "gate_suh": _ptrs("gate", "suh"),
+        "gate_svh": _ptrs("gate", "svh"),
+        "up_trellis": _ptrs("up", "trellis"),
+        "up_suh": _ptrs("up", "suh"),
+        "up_svh": _ptrs("up", "svh"),
+        "down_trellis": _ptrs("down", "trellis"),
+        "down_suh": _ptrs("down", "suh"),
+        "down_svh": _ptrs("down", "svh"),
+    }
+    idx = int(device.index) if device.index is not None else 0
+    concurrency = int(exllamav3_ext.exl3_moe_max_concurrency(idx))
+    if concurrency < 1:
+        concurrency = 1
+    key = (str(device), hidden, intermediate, concurrency)
+    temps = _FUSED_TEMP_CACHE.get(key)
+    if temps is None:
+        temps = (
+            torch.empty((concurrency, TEMP_ROWS_FUSED, hidden), dtype=torch.float16, device=device),
+            torch.empty((concurrency, TEMP_ROWS_FUSED, hidden), dtype=torch.float16, device=device),
+            torch.empty((concurrency, TEMP_ROWS_FUSED, intermediate), dtype=torch.float16, device=device),
+            torch.empty((concurrency, TEMP_ROWS_FUSED, intermediate), dtype=torch.float16, device=device),
+        )
+        _FUSED_TEMP_CACHE[key] = temps
+    layer._exl3_fused_temps = temps
+    layer._exl3_fused_concurrency = concurrency
+    layer._exl3_k = int(layer._exl3_bits)
+
+
+def apply_exl3_fused_moe(
+    x2d: torch.Tensor,
+    ids: torch.Tensor,
+    weights: torch.Tensor,
+    layer: torch.nn.Module,
+    inners: list[dict[str, Any]],
+    expert_map: torch.Tensor | None,
+    limit: float,
+) -> torch.Tensor:
+    """One exl3_moe launch per layer. Experts with count > 128 fall back to LinearEXL3."""
+    import exllamav3_ext
+
+    tokens, hidden = x2d.shape
+    n_exp = len(inners)
+    ptrs = getattr(layer, "_exl3_ptrs", None)
+    temps = getattr(layer, "_exl3_fused_temps", None)
+    if not ptrs or temps is None:
+        raise RuntimeError("EXL3 fused pointer tables were not built after weight load")
+
+    local = map_topk_to_local(ids, n_exp, expert_map)
+    topk = int(ids.shape[-1])
+    flat_token = torch.arange(tokens, device=x2d.device, dtype=torch.long).repeat_interleave(topk)
+    flat_weight = weights.reshape(-1).to(dtype=torch.float16)
+    order = local.argsort()
+    token_sorted = flat_token[order]
+    weight_sorted = flat_weight[order]
+    # scatter_add stays on GPU. torch.bincount can host-stage and break CUDA graphs.
+    expert_count = torch.zeros(n_exp + 1, dtype=torch.long, device=local.device)
+    expert_count.scatter_add_(
+        0, local.long(), torch.ones(local.shape, dtype=torch.long, device=local.device)
+    )
+    out = torch.zeros(tokens, hidden, dtype=torch.float32, device=x2d.device)
+    xh = x2d.contiguous().half()
+
+    counts = expert_count[:n_exp]
+
+    if tokens > TEMP_ROWS_FUSED and bool((counts > TEMP_ROWS_FUSED).any().item()):
+        logger.info_once("EXL3 fat-chunk slicing ACTIVE (tokens=%d)" % tokens)
+        # Deep-context prefill chunks can route more than TEMP_ROWS_FUSED rows
+        # to a single expert. The fused kernel covers at most TEMP_ROWS_FUSED
+        # rows per expert, and the old fallback reconstructed whole experts
+        # per chunk, stalling prefill by orders of magnitude past ~160k
+        # context (the ">163k hang"). Within a slice of <= TEMP_ROWS_FUSED
+        # tokens no expert can exceed TEMP_ROWS_FUSED rows (each token adds at
+        # most one row per expert), so re-run the fused path per slice.
+        # Prefill-only: decode batches are at most the largest capture size,
+        # far below TEMP_ROWS_FUSED, and never reach this host sync.
+        for s in range(0, tokens, TEMP_ROWS_FUSED):
+            e = min(s + TEMP_ROWS_FUSED, tokens)
+            out[s:e] = apply_exl3_fused_moe(
+                x2d[s:e], ids[s:e], weights[s:e], layer, inners, expert_map, limit
+            )
+        return out
+    fn = exllamav3_ext.exl3_moe
+    # -1 = unknown active count: max-concurrency grid, no .item() host sync.
+    n_active_host = -1 if _exl3_moe_accepts_num_active(fn) else None
+
+    k = int(getattr(layer, "_exl3_k", 4))
+    args = (
+        xh,
+        out,
+        expert_count,
+        token_sorted,
+        weight_sorted,
+        temps[0],
+        temps[1],
+        temps[2],
+        temps[3],
+        MOE_ACT_SILU,
+        k,
+        k,
+        k,
+        ptrs["gate_trellis"],
+        ptrs["gate_suh"],
+        ptrs["gate_svh"],
+        ptrs["up_trellis"],
+        ptrs["up_suh"],
+        ptrs["up_svh"],
+        ptrs["down_trellis"],
+        ptrs["down_suh"],
+        ptrs["down_svh"],
+        True,
+        False,
+        True,
+        False,
+        True,
+        False,
+        float(limit),
+    )
+    if n_active_host is not None:
+        fn(*args, n_active_host)
+    else:
+        fn(*args)
+
+    if tokens > TEMP_ROWS_FUSED:
+        fat = (counts > TEMP_ROWS_FUSED).nonzero(as_tuple=False).view(-1)
+        if fat.numel():
+            apply_exl3_python_loop(
+                x2d,
+                ids,
+                weights,
+                inners,
+                expert_map,
+                limit,
+                only_experts=set(int(i) for i in fat.tolist()),
+                out=out,
+            )
+    return out
+
+
+def apply_exl3_experts(
+    x: torch.Tensor,
+    topk_ids: torch.Tensor,
+    topk_weights: torch.Tensor,
+    layer: torch.nn.Module,
+    *,
+    limit: float = SWIGLU_LIMIT_DEFAULT,
+    fused: bool | None = None,
+) -> torch.Tensor:
+    """Shipped routed-expert apply. `fused=None` honors EXL3_FUSED_MOE."""
+    inners = getattr(layer, "_exl3_inners", None)
+    if not inners:
+        raise RuntimeError("EXL3 experts were not built after weight load")
+    tokens, hidden = x.shape[-2], x.shape[-1]
+    x2d = x.reshape(tokens, hidden)
+    ids = topk_ids.reshape(tokens, -1).to(torch.long)
+    weights = topk_weights.reshape(tokens, -1)
+    expert_map = pin_exl3_expert_map(layer, x2d.device)
+    have_ptrs = bool(getattr(layer, "_exl3_ptrs", None))
+    if fused is True and not have_ptrs:
+        raise RuntimeError("EXL3 fused apply requested but pointer tables are missing")
+    use_fused = (fused_moe_enabled() if fused is None else bool(fused)) and have_ptrs
+    if use_fused:
+        try:
+            import exllamav3_ext
+
+            use_fused = hasattr(exllamav3_ext, "exl3_moe")
+        except Exception:
+            use_fused = False
+    if use_fused:
+        out = apply_exl3_fused_moe(x2d, ids, weights, layer, inners, expert_map, limit)
+        layer._exl3_last_apply = "fused"
+    else:
+        out = apply_exl3_python_loop(x2d, ids, weights, inners, expert_map, limit)
+        layer._exl3_last_apply = "loop"
+    return out.to(dtype=x.dtype)
+
+
+def _suffix_from_mapped_name(weight_name: str) -> str:
+    tail = weight_name.rsplit(".", 1)[-1]
+    for suffix in EXL3_SUFFIXES:
+        if tail == suffix or tail.endswith("_" + suffix):
+            return suffix
+    raise ValueError(f"not an EXL3 packed name: {weight_name}")
+
+
+def _prefix_has_suffix(prefix: str, suffix: str) -> bool:
+    """Module-path suffix match: "self_attn.o_proj" matches
+    "model.layers.3.self_attn.o_proj" but not "...cross_attn.o_proj_x"."""
+    return prefix == suffix or prefix.endswith("." + suffix)
+
+
+@register_quantization_config("exl3")
+class Exl3Config(QuantizationConfig):
+    """Routed-experts-only EXL3/MCG. Dense / shared / attention stay native."""
+
+    def __init__(
+        self,
+        bits: int = 4,
+        codebook: str = "mcg",
+        scope: str = "glm53_routed_experts_only",
+        **kwargs: Any,
+    ) -> None:
+        super().__init__()
+        self.bits = int(bits)
+        self.codebook = str(codebook)
+        self.scope = str(scope)
+        # Optional per-layer override, e.g. {"42": 3, "27": 3}. Layers absent
+        # from the map use `bits`. This is how a mixed-K checkpoint (K2 base
+        # with K3 delta layers) declares itself; the trellis tensors for those
+        # layers are shaped for their own K and would fail the load shape
+        # check under the base K.
+        raw_layer_bits = kwargs.pop("layer_bits", None) or {}
+        self.layer_bits: dict[int, int] = {
+            int(k): int(v) for k, v in dict(raw_layer_bits).items()
+        }
+        for layer_idx, layer_k in self.layer_bits.items():
+            if layer_k not in (2, 3, 4, 5, 6):
+                raise ValueError(
+                    f"unsupported EXL3 bits={layer_k} for layer {layer_idx}"
+                )
+        # Non-routed dense linear config: optional {"modules": [...], "bits": K, "layer_bits": {...},
+        # "codebook": "mcg"|"mul1", "layers": {prefix: {"bits": K, "bf16_shards": [...]}, ...}}
+        raw_nr_exl3 = kwargs.pop("non_routed_exl3", None) or {}
+        self.non_routed_exl3: dict[str, Any] = dict(raw_nr_exl3) if raw_nr_exl3 else {}
+        # Validate non-routed bits if present
+        nr_bits = self.non_routed_exl3.get("bits")
+        if nr_bits is not None and nr_bits not in (2, 3, 4, 5, 6):
+            raise ValueError(f"unsupported non_routed_exl3 bits={nr_bits}")
+        nr_layer_bits = self.non_routed_exl3.get("layer_bits", {})
+        for suffix, k in (nr_layer_bits or {}).items():
+            if k not in (2, 3, 4, 5, 6):
+                raise ValueError(
+                    f"unsupported non_routed_exl3 bits={k} for suffix {suffix}"
+                )
+        # Validate non-routed layers dict: each value is {"bits": K[, "bf16_shards": [...]]}
+        nr_layers = self.non_routed_exl3.get("layers", {})
+        for prefix, layer_cfg in (nr_layers or {}).items():
+            if not isinstance(layer_cfg, dict):
+                raise ValueError(
+                    f"non_routed_exl3 layers[{prefix}] must be a dict, got {type(layer_cfg)}"
+                )
+            layer_bits = layer_cfg.get("bits")
+            if layer_bits is not None and layer_bits not in (2, 3, 4, 5, 6):
+                raise ValueError(
+                    f"unsupported non_routed_exl3 layers[{prefix}] bits={layer_bits}"
+                )
+        # Validate non-routed codebook
+        nr_codebook = self.non_routed_exl3.get("codebook", "mcg")
+        if nr_codebook not in ("mcg", "mul1"):
+            raise ValueError(
+                f"unsupported non_routed_exl3 codebook={nr_codebook!r}; must be 'mcg' or 'mul1'"
+            )
+        self.raw_config = dict(kwargs)
+        if self.codebook not in ("mcg", "mul1"):
+            raise ValueError(
+                f"unsupported codebook={self.codebook!r}; must be 'mcg' or 'mul1'"
+            )
+        if self.bits not in (2, 3, 4, 5, 6):
+            raise ValueError(f"unsupported EXL3 bits={self.bits}")
+
+    def get_name(self) -> str:
+        return "exl3"
+
+    _LAYER_RE = re.compile(r"(?:^|\.)layers\.(\d+)(?:\.|$)")
+
+    def bits_for_prefix(self, prefix: str) -> int:
+        """Per-layer K: `layer_bits` entry for this layer, else the base K."""
+        if not self.layer_bits:
+            return self.bits
+        m = self._LAYER_RE.search(prefix or "")
+        if m is None:
+            return self.bits
+        return self.layer_bits.get(int(m.group(1)), self.bits)
+
+    def _matches_non_routed_exl3(self, prefix: str) -> bool:
+        """Check if prefix matches non_routed_exl3: either layers dict keys or modules list."""
+        if not self.non_routed_exl3:
+            return False
+        # Check if prefix is a key in the layers dict
+        layers = self.non_routed_exl3.get("layers", {})
+        if layers and prefix in layers:
+            return True
+        # Fall back to suffix matching on modules list
+        modules = self.non_routed_exl3.get("modules", [])
+        if not modules:
+            return False
+        return any(_prefix_has_suffix(prefix, m) for m in modules)
+
+    def _bits_for_non_routed(self, prefix: str) -> int:
+        """Get K bits for non_routed_exl3 layer, checking layers dict first, then suffix form."""
+        if not self.non_routed_exl3:
+            return self.bits
+        # Check layers dict first
+        layers = self.non_routed_exl3.get("layers", {})
+        if layers and prefix in layers:
+            layer_cfg = layers[prefix]
+            if "bits" in layer_cfg:
+                return int(layer_cfg["bits"])
+            return int(self.non_routed_exl3.get("bits", self.bits))
+        # Fall back to suffix matching
+        modules = self.non_routed_exl3.get("modules", [])
+        matched_suffix = None
+        for suffix in modules:
+            if _prefix_has_suffix(prefix, suffix):
+                matched_suffix = suffix
+                break
+        if matched_suffix is None:
+            return self.bits
+        # Check layer_bits override for this suffix
+        layer_bits = self.non_routed_exl3.get("layer_bits", {})
+        if matched_suffix in layer_bits:
+            return int(layer_bits[matched_suffix])
+        # Fall back to non_routed_exl3 bits or base bits
+        return int(self.non_routed_exl3.get("bits", self.bits))
+
+    def _bf16_shards_for(self, prefix: str) -> list[int]:
+        """Get bf16 shard indices for a non_routed_exl3 layer from the layers dict."""
+        if not self.non_routed_exl3:
+            return []
+        layers = self.non_routed_exl3.get("layers", {})
+        if layers and prefix in layers:
+            layer_cfg = layers[prefix]
+            return list(layer_cfg.get("bf16_shards", []))
+        return []
+
+    def get_supported_act_dtypes(self) -> list[torch.dtype]:
+        return [torch.bfloat16, torch.float16, torch.float32]
+
+    @classmethod
+    def get_min_capability(cls) -> int:
+        # LinearEXL3 uses CUDA >= Ampere; GB10 is SM121.
+        return 80
+
+    @staticmethod
+    def get_config_filenames() -> list[str]:
+        return ["quantization_config.json"]
+
+    @classmethod
+    def from_config(cls, config: dict[str, Any]) -> "Exl3Config":
+        skip = {
+            "bits",
+            "codebook",
+            "scope",
+            "quant_method",
+            # Some packs ship a large per-tensor ledger here; keep it off the config object.
+            "tensor_storage",
+            "non_routed_exl3",
+            "non_routed_quantization",
+            "mtp_experts",
+            "mtp_experts_start_layer",
+        }
+        inst = cls(
+            bits=int(config.get("bits", 4)),
+            codebook=str(config.get("codebook", "mcg")),
+            scope=str(config.get("scope", "glm53_routed_experts_only")),
+            non_routed_exl3=config.get("non_routed_exl3"),
+            **{k: v for k, v in config.items() if k not in skip},
+        )
+        # __init__ swallows unknown kwargs; stash the delegation dict explicitly.
+        inst.non_routed_quantization = config.get("non_routed_quantization")
+        # "bf16_as_stored": dense linears are BF16 tensors; never delegate them
+        # (the delegate still serves source-format MTP experts).
+        inst.non_routed_dtype_policy = str(config.get("non_routed_dtype_policy", ""))
+        # Mixed-format packs: draft/MTP blocks appended past the main stack can
+        # keep their experts in the source format (e.g. MXFP4). Declare
+        # mtp_experts: "source" plus mtp_experts_start_layer: <first draft
+        # layer index>; those layers delegate to non_routed_quantization.
+        inst.mtp_experts = str(config.get("mtp_experts", "exl3"))
+        inst.mtp_experts_start_layer = config.get("mtp_experts_start_layer")
+        return inst
+
+    @classmethod
+    def override_quantization_method(
+        cls,
+        hf_quant_cfg: dict[str, Any],
+        user_quant: str | None,
+        hf_config: Any = None,
+    ) -> str | None:
+        method = str((hf_quant_cfg or {}).get("quant_method", "")).lower()
+        if method == "exl3":
+            return "exl3"
+        return None
+
+    def get_quant_method(self, layer: torch.nn.Module, prefix: str):
+        from vllm.model_executor.layers.fused_moe.routed_experts import RoutedExperts
+
+        if isinstance(layer, RoutedExperts):
+            # Draft/MTP blocks construct with plain layers.N prefixes (the
+            # mtp_block name appears only in parameter paths), so gate by
+            # declared layer index, never by name.
+            if getattr(self, "mtp_experts", "exl3") == "source":
+                _start = getattr(self, "mtp_experts_start_layer", None)
+                _lm = re.search(r"layers\.(\d+)\.", prefix)
+                if _start is not None and _lm and int(_lm.group(1)) >= int(_start):
+                    d = self._non_routed_delegate()
+                    if d is not None:
+                        dm = d.get_quant_method(layer, prefix)
+                        if dm is not None:
+                            return dm
+            return Exl3MoEMethod(
+                layer.moe_config, self, bits=self.bits_for_prefix(prefix)
+            )
+        if isinstance(layer, LinearBase):
+            # Check if this LinearBase should use non_routed_exl3
+            if self._matches_non_routed_exl3(prefix):
+                bits = self._bits_for_non_routed(prefix)
+                return Exl3LinearMethod(self, bits=bits)
+            if getattr(self, "non_routed_dtype_policy", "") == "bf16_as_stored":
+                return UnquantizedLinearMethod()
+            d = self._non_routed_delegate()
+            if d is not None:
+                m = d.get_quant_method(layer, prefix)
+                if m is not None:
+                    return m
+            return UnquantizedLinearMethod()
+        return None
+
+    def _non_routed_delegate(self):
+        # Packs that keep non-routed weights in the official source format
+        # (e.g. DeepSeek block-FP8) declare it under
+        # ``quantization_config.non_routed_quantization``; delegate those
+        # layers to the matching quant method so arch-specific fp8 forward
+        # paths get real scale tensors. Absent key = unquantized (GLM).
+        if not hasattr(self, "_nr_delegate_cached"):
+            self._nr_delegate_cached = None
+            nrq = getattr(self, "non_routed_quantization", None)
+            if isinstance(nrq, dict) and nrq.get("quant_method"):
+                from vllm.model_executor.layers.quantization import (
+                    get_quantization_config,
+                )
+                for name in ("deepseek_v4_fp8", str(nrq.get("quant_method"))):
+                    try:
+                        cls = get_quantization_config(name)
+                        self._nr_delegate_cached = cls.from_config(dict(nrq))
+                        break
+                    except Exception:
+                        continue
+        return self._nr_delegate_cached
+
+
+class Exl3MoEMethod(FusedMoEMethodBase):
+    """Packed MCG trellis experts: create/load packed tensors, LinearEXL3 apply."""
+
+    def __init__(
+        self, moe, quant_config: Exl3Config, bits: int | None = None
+    ) -> None:
+        super().__init__(moe)
+        self.quant_config = quant_config
+        # One method instance per RoutedExperts layer, so this is per-layer K.
+        self.bits = int(bits) if bits is not None else quant_config.bits
+        self._logged = False
+
+    def get_fused_moe_quant_config(self, layer: "RoutedExperts") -> FusedMoEQuantConfig | None:
+        return None
+
+    def create_weights(
+        self,
+        layer: "RoutedExperts",
+        num_experts: int,
+        hidden_size: int,
+        intermediate_size_per_partition: int,
+        params_dtype: torch.dtype,
+        **extra_weight_attrs,
+    ) -> None:
+        del params_dtype
+        if hidden_size % 16 or intermediate_size_per_partition % 16:
+            raise ValueError(
+                "EXL3 trellis tiles are 16-wide; "
+                f"hidden={hidden_size} intermediate_local={intermediate_size_per_partition}"
+            )
+        k_words = self.bits * 16
+        in_tiles = hidden_size // 16
+        out_tiles = intermediate_size_per_partition // 16
+
+        extra = {k: v for k, v in extra_weight_attrs.items() if k != "weight_loader"}
+
+        # w13_* : stacked [expert, {gate=0, up=1}, ...] so the stock
+        # expert_params_mapping (experts.w13_ + suffix) hits these names.
+        w13_trellis = Parameter(
+            torch.empty(
+                num_experts, 2, in_tiles, out_tiles, k_words, dtype=torch.int16
+            ),
+            requires_grad=False,
+        )
+        w13_suh = Parameter(
+            torch.empty(num_experts, 2, hidden_size, dtype=torch.float16),
+            requires_grad=False,
+        )
+        w13_svh = Parameter(
+            torch.empty(
+                num_experts, 2, intermediate_size_per_partition, dtype=torch.float16
+            ),
+            requires_grad=False,
+        )
+        w13_mcg = Parameter(
+            torch.empty(num_experts, 2, 1, dtype=torch.int32),
+            requires_grad=False,
+        )
+        w2_trellis = Parameter(
+            torch.empty(
+                num_experts, out_tiles, in_tiles, k_words, dtype=torch.int16
+            ),
+            requires_grad=False,
+        )
+        w2_suh = Parameter(
+            torch.empty(
+                num_experts, intermediate_size_per_partition, dtype=torch.float16
+            ),
+            requires_grad=False,
+        )
+        w2_svh = Parameter(
+            torch.empty(num_experts, hidden_size, dtype=torch.float16),
+            requires_grad=False,
+        )
+        w2_mcg = Parameter(
+            torch.empty(num_experts, 1, dtype=torch.int32),
+            requires_grad=False,
+        )
+
+        packed = {
+            "w13_trellis": w13_trellis,
+            "w13_suh": w13_suh,
+            "w13_svh": w13_svh,
+            "w13_mcg": w13_mcg,
+            "w2_trellis": w2_trellis,
+            "w2_suh": w2_suh,
+            "w2_svh": w2_svh,
+            "w2_mcg": w2_mcg,
+        }
+        for name, param in packed.items():
+            layer.register_parameter(name, param)
+            set_weight_attrs(param, extra)
+            param.weight_loader = self._load_exl3
+            param._exl3_owner = layer
+        if hasattr(layer, "w13_weight") or hasattr(layer, "w2_weight"):
+            raise RuntimeError("EXL3 create_weights must not allocate dense expert weights")
+
+        layer._exl3_hidden_size = hidden_size
+        layer._exl3_intermediate_local = intermediate_size_per_partition
+        layer._exl3_k_words = k_words
+        layer._exl3_bits = self.bits
+
+    def _load_exl3(
+        self,
+        param: Parameter,
+        loaded_weight: torch.Tensor,
+        weight_name: str,
+        shard_id: str = "w1",
+        expert_id: int = 0,
+        return_success: bool = False,
+    ) -> bool | None:
+        from vllm.distributed import (
+            get_tensor_model_parallel_rank,
+            get_tensor_model_parallel_world_size,
+        )
+
+        layer = param
+        # param is the Parameter; expert_id is already physical. Map to local
+        # via the owning module if present on the weight_loader closure... we
+        # look up from param's __dict__ after register. RoutedExperts.weight_loader
+        # maps global→local; glm5next calls *our* loader, so map here.
+        owner = getattr(param, "_exl3_owner", None)
+        if owner is not None:
+            local_id = owner._map_global_expert_id_to_local_expert_id(expert_id)
+            if local_id == -1:
+                return False if return_success else None
+            expert_id = local_id
+
+        tp_rank = get_tensor_model_parallel_rank()
+        tp_size = get_tensor_model_parallel_world_size()
+        suffix = _suffix_from_mapped_name(weight_name)
+        loaded = loaded_weight.detach().contiguous()
+
+        if shard_id in ("w1", "w3"):
+            shard_idx = 0 if shard_id == "w1" else 1
+            sharded = shard_exl3_col(loaded, suffix, tp_rank, tp_size)
+            dest = param.data[expert_id, shard_idx]
+        elif shard_id == "w2":
+            sharded = shard_exl3_row(loaded, suffix, tp_rank, tp_size)
+            dest = param.data[expert_id]
+        else:
+            raise ValueError(f"unknown EXL3 shard_id={shard_id}")
+
+        if tuple(dest.shape) != tuple(sharded.shape):
+            raise RuntimeError(
+                f"EXL3 load shape mismatch {weight_name} shard={shard_id} "
+                f"expert={expert_id}: dest {tuple(dest.shape)} != "
+                f"loaded {tuple(sharded.shape)}"
+            )
+        dest.copy_(sharded)
+        return True if return_success else None
+
+    def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
+        if not hasattr(layer, "w13_trellis"):
+            return
+        # Bind owner for any late loads; stitch LinearEXL3 handles.
+        for name in (
+            "w13_trellis",
+            "w13_suh",
+            "w13_svh",
+            "w13_mcg",
+            "w2_trellis",
+            "w2_suh",
+            "w2_svh",
+            "w2_mcg",
+        ):
+            getattr(layer, name)._exl3_owner = layer
+
+        mcg13 = layer.w13_mcg.reshape(-1)
+        mcg2 = layer.w2_mcg.reshape(-1)
+        if not torch.all(mcg13 == MCG_MARKER_SIGNED_INT32) or not torch.all(
+            mcg2 == MCG_MARKER_SIGNED_INT32
+        ):
+            raise RuntimeError(
+                "EXL3 mcg marker is not the MCG int32 0xCBAC1FED / "
+                f"{MCG_MARKER_SIGNED_INT32}; packed ABI mismatch"
+            )
+
+        n_exp = int(layer.w13_trellis.shape[0])
+        inners: list[dict[str, Any]] = []
+        for e in range(n_exp):
+            gate = make_linear_exl3(
+                layer.w13_trellis[e, 0],
+                layer.w13_suh[e, 0],
+                layer.w13_svh[e, 0],
+                layer.w13_mcg[e, 0],
+            )
+            up = make_linear_exl3(
+                layer.w13_trellis[e, 1],
+                layer.w13_suh[e, 1],
+                layer.w13_svh[e, 1],
+                layer.w13_mcg[e, 1],
+            )
+            down = make_linear_exl3(
+                layer.w2_trellis[e],
+                layer.w2_suh[e],
+                layer.w2_svh[e],
+                layer.w2_mcg[e],
+            )
+            inners.append({"gate": gate, "up": up, "down": down})
+        layer._exl3_inners = inners
+        fused_ok = False
+        fused_err = None
+        if fused_moe_enabled():
+            try:
+                import exllamav3_ext
+
+                if hasattr(exllamav3_ext, "exl3_moe"):
+                    build_exl3_fused_state(layer, inners)
+                    fused_ok = True
+                else:
+                    fused_err = "exllamav3_ext.exl3_moe missing"
+            except Exception as exc:
+                fused_err = repr(exc)
+                layer._exl3_ptrs = None
+        if not self._logged and self.bits != self.quant_config.bits:
+            logger.info(
+                "EXL3 per-layer K override: layer prefix %s uses bits=%d (base %d)",
+                getattr(layer, "layer_name", None) or getattr(layer, "prefix", "?"),
+                self.bits,
+                self.quant_config.bits,
+            )
+        if not self._logged:
+            if fused_ok:
+                logger.info(
+                    "EXL3 MCG trellis engaged for routed experts: bits=%s "
+                    "experts_local=%s hidden=%s intermediate_local=%s "
+                    "fused_moe=exl3_moe concurrency=%s "
+                    "(no BF16 expert reconstruct at load)",
+                    self.bits,
+                    n_exp,
+                    layer._exl3_hidden_size,
+                    layer._exl3_intermediate_local,
+                    getattr(layer, "_exl3_fused_concurrency", "?"),
+                )
+            else:
+                logger.info(
+                    "EXL3 MCG trellis engaged for routed experts: bits=%s "
+                    "experts_local=%s hidden=%s intermediate_local=%s "
+                    "fused_moe=python_loop (%s) "
+                    "(no BF16 expert reconstruct at load)",
+                    self.bits,
+                    n_exp,
+                    layer._exl3_hidden_size,
+                    layer._exl3_intermediate_local,
+                    fused_err or "EXL3_FUSED_MOE=0",
+                )
+            self._logged = True
+
+    def apply(
+        self,
+        layer: "RoutedExperts",
+        x: torch.Tensor,
+        topk_weights: torch.Tensor,
+        topk_ids: torch.Tensor,
+        shared_experts: "SharedExperts | None",
+        shared_experts_input: torch.Tensor | None,
+    ) -> torch.Tensor:
+        del shared_experts, shared_experts_input
+        limit = getattr(self.moe, "swiglu_limit", None) or SWIGLU_LIMIT_DEFAULT
+        return apply_exl3_experts(
+            x, topk_ids, topk_weights, layer, limit=float(limit)
+        )
+
+
+class Exl3LinearMethod(LinearMethodBase):
+    """Non-routed (dense) EXL3 linear method for QKV/MLP dense projections.
+
+    This method handles trellis/suh/svh/mcg parameters for non-routed dense
+    linear layers, building LinearEXL3 objects after weight loading and applying
+    them with proper TP slicing and shard concatenation.
+    """
+
+    def __init__(self, quant_config: Exl3Config, bits: int | None = None) -> None:
+        self.quant_config = quant_config
+        self.bits = int(bits) if bits is not None else quant_config.bits
+        self._logged = False
+
+    def create_weights(
+        self,
+        layer,
+        input_size_per_partition: int,
+        output_partition_sizes: list[int],
+        input_size: int,
+        output_size: int,
+        params_dtype: torch.dtype,
+        **extra_weight_attrs,
+    ) -> None:
+        from vllm.model_executor.layers.linear import (
+            ColumnParallelLinear,
+            RowParallelLinear,
+            QKVParallelLinear,
+            MergedColumnParallelLinear,
+        )
+        from vllm.distributed import (
+            get_tensor_model_parallel_rank,
+            get_tensor_model_parallel_world_size,
+        )
+
+        # Determine layer type and shard behavior
+        n_shards = len(output_partition_sizes)
+        is_row_parallel = isinstance(layer, RowParallelLinear)
+        is_col_parallel = isinstance(layer, ColumnParallelLinear)
+        is_qkv_parallel = isinstance(layer, QKVParallelLinear)
+        is_merged_col_parallel = isinstance(layer, MergedColumnParallelLinear)
+        global_tp_rank = get_tensor_model_parallel_rank()
+        global_tp_size = get_tensor_model_parallel_world_size()
+        effective_tp_size, effective_tp_rank = _effective_tp_for_layer(
+            layer, global_tp_size, global_tp_rank
+        )
+        is_wo_a = _is_wo_a_layer(layer)
+        # LNA-LAB: EXL3 wo_a is one rank-local group only; TP<8 needs a future
+        # multi-group path and must fail before an ambiguous partial load.
+        if is_wo_a and effective_tp_size != 8:
+            raise NotImplementedError(
+                "EXL3 wo_a requires TP=8 (one local group per rank); "
+                f"got effective tp_size={effective_tp_size}"
+            )
+
+        # For column-parallel: output dimension is sharded (each shard has different outputs)
+        # For row-parallel: input dimension is sharded (each shard has same input, different outputs)
+        if is_row_parallel:
+            # Input is partitioned across ranks, each rank gets full input height
+            in_per_partition = input_size_per_partition
+        else:
+            # Column-parallel or unsharded: each rank gets full input
+            in_per_partition = input_size_per_partition
+
+        # Get bf16_shards from config (may be empty)
+        bf16_shards = self.quant_config._bf16_shards_for(getattr(layer, "prefix", ""))
+        if bf16_shards:
+            # LNA-LAB: the guard follows effective, not process-global, TP.
+            if effective_tp_size > 1:
+                raise RuntimeError(
+                    "EXL3 bf16 shards are not supported with effective TP size > 1; "
+                    f"tp_size={effective_tp_size}"
+                )
+
+        # K words per shard
+        k_words = self.bits * 16
+
+        # Validate tile alignment for all shards
+        for i, out_size in enumerate(output_partition_sizes):
+            if in_per_partition % 16 or out_size % 16:
+                raise ValueError(
+                    f"EXL3 trellis tiles are 16-wide; "
+                    f"shard {i}: in={in_per_partition} out={out_size}"
+                )
+
+        in_tiles = in_per_partition // 16
+        out_tiles_list = [s // 16 for s in output_partition_sizes]
+        total_out_tiles = sum(out_tiles_list)
+
+        # Allocate fused trellis covering all shards (dim1 will be narrow per-shard)
+        trellis_param = Parameter(
+            torch.empty(in_tiles, total_out_tiles, k_words, dtype=torch.int16),
+            requires_grad=False,
+        )
+        # Per-shard suh (one per shard, each covers this rank's input partition)
+        suh_param = Parameter(
+            torch.empty(n_shards, in_per_partition, dtype=torch.float16),
+            requires_grad=False,
+        )
+        # Per-shard svh (one per shard, concatenated)
+        svh_param = Parameter(
+            torch.empty(sum(output_partition_sizes), dtype=torch.float16),
+            requires_grad=False,
+        )
+        # Per-shard mcg and mul1 markers (both registered, one will be nonzero)
+        mcg_param = Parameter(
+            torch.zeros(n_shards, 1, dtype=torch.int32),
+            requires_grad=False,
+        )
+        mul1_param = Parameter(
+            torch.zeros(n_shards, 1, dtype=torch.int32),
+            requires_grad=False,
+        )
+
+        # LNA-LAB: wo_a has no dense weight shim; _o_proj consumes _exl3_linears.
+        weight_param = None
+        if not is_wo_a:
+            # Staging parameter for bf16 shards: rows are concatenated bf16 weights
+            bf16_rows = sum(output_partition_sizes[i] for i in bf16_shards)
+            weight_param = Parameter(
+                torch.empty(bf16_rows, in_per_partition, dtype=params_dtype),
+                requires_grad=False,
+            )
+
+        layer.register_parameter("trellis", trellis_param)
+        layer.register_parameter("suh", suh_param)
+        layer.register_parameter("svh", svh_param)
+        layer.register_parameter("mcg", mcg_param)
+        layer.register_parameter("mul1", mul1_param)
+        if weight_param is not None:
+            layer.register_parameter("weight", weight_param)
+
+        # Custom weight loader
+        extra = {k: v for k, v in extra_weight_attrs.items() if k != "weight_loader"}
+        set_weight_attrs(trellis_param, extra)
+        set_weight_attrs(suh_param, extra)
+        set_weight_attrs(svh_param, extra)
+        set_weight_attrs(mcg_param, extra)
+        set_weight_attrs(mul1_param, extra)
+        if weight_param is not None:
+            set_weight_attrs(weight_param, extra)
+
+        # vLLM calls ``weight_loader(param, loaded_weight[, shard_id])`` and
+        # never passes the checkpoint name, so bind the tensor kind per param.
+        for suffix, p in (
+            ("trellis", trellis_param),
+            ("suh", suh_param),
+            ("svh", svh_param),
+            ("mcg", mcg_param),
+            ("mul1", mul1_param),
+        ):
+            p.weight_loader = self._make_weight_loader(
+                suffix,
+                n_shards,
+                output_partition_sizes,
+                is_row_parallel,
+                bf16_shards,
+                effective_tp_rank,
+                effective_tp_size,
+                is_wo_a,
+            )
+        if weight_param is not None:
+            weight_param.weight_loader = self._make_weight_loader(
+                "weight",
+                n_shards,
+                output_partition_sizes,
+                is_row_parallel,
+                bf16_shards,
+                effective_tp_rank,
+                effective_tp_size,
+                is_wo_a,
+            )
+
+        # Store metadata
+        layer._exl3_linear_n_shards = n_shards
+        layer._exl3_linear_output_partition_sizes = output_partition_sizes
+        layer._exl3_linear_input_size_per_partition = in_per_partition
+        layer._exl3_linear_is_row_parallel = is_row_parallel
+        layer._exl3_linear_is_qkv = is_qkv_parallel
+        layer._exl3_linear_is_merged = is_merged_col_parallel
+        layer._exl3_linear_bf16_shards = bf16_shards
+        # LNA-LAB: retain both process and effective geometry for diagnostics/loaders.
+        layer._exl3_linear_tp_size = effective_tp_size
+        layer._exl3_linear_tp_rank = effective_tp_rank
+        layer._exl3_linear_is_wo_a = is_wo_a
+
+    def _make_weight_loader(
+        self,
+        suffix,
+        n_shards,
+        output_partition_sizes,
+        is_row_parallel,
+        bf16_shards,
+        tp_rank,
+        tp_size,
+        is_wo_a,
+    ):
+        """Create a weight_loader closure for EXL3 linear parameters."""
+
+        def weight_loader(
+            param: Parameter,
+            loaded_weight: torch.Tensor,
+            loaded_shard_id: str | int | None = None,
+        ) -> None:
+            # Map shard_id to shard index
+            shard_idx = 0
+            if loaded_shard_id is not None:
+                if isinstance(loaded_shard_id, str):
+                    # "q", "k", "v" for QKV layers
+                    shard_map = {"q": 0, "k": 1, "v": 2}
+                    if loaded_shard_id in shard_map:
+                        shard_idx = shard_map[loaded_shard_id]
+                    elif is_wo_a and _wo_a_slice_rank(loaded_shard_id) is not None:
+                        # LNA-LAB: model.py supplies the wo_a group rank as the
+                        # shard id; the special loader below selects this rank.
+                        shard_idx = 0
+                    else:
+                        raise ValueError(
+                            f"unknown shard_id={loaded_shard_id} for EXL3 linear"
+                        )
+                elif isinstance(loaded_shard_id, int):
+                    # LNA-LAB: wo_a integer ids are group ranks, not local
+                    # fused-shard indices (there is exactly one local shard).
+                    shard_idx = 0 if is_wo_a else loaded_shard_id
+            if not is_wo_a and shard_idx >= n_shards:
+                raise ValueError(
+                    f"shard_idx={shard_idx} out of range for n_shards={n_shards}"
+                )
+
+            if is_wo_a:
+                # LNA-LAB: each wo_a.slice.N is already a complete local
+                # [group_width, o_lora_rank] tensor; never narrow it again.
+                slice_rank = _wo_a_slice_rank(loaded_shard_id)
+                if slice_rank is None:
+                    raise ValueError(
+                        "EXL3 wo_a loader requires a wo_a.slice.N shard id"
+                    )
+                if slice_rank != tp_rank:
+                    return
+                loaded = loaded_weight.detach().contiguous()
+                if suffix == "trellis":
+                    dest = param.data
+                elif suffix == "suh":
+                    dest = param.data[0]
+                elif suffix == "svh":
+                    dest = param.data
+                elif suffix in ("mcg", "mul1"):
+                    dest = param.data[0]
+                    if tuple(dest.shape) != (1,):
+                        raise RuntimeError(
+                            f"EXL3 wo_a {suffix} marker shape mismatch: "
+                            f"expected (1,) got {tuple(dest.shape)}"
+                        )
+                    if loaded.numel():
+                        dest[0] = int(loaded.item())
+                    return
+                else:
+                    raise ValueError(f"unknown EXL3 wo_a suffix={suffix}")
+                if tuple(dest.shape) != tuple(loaded.shape):
+                    raise RuntimeError(
+                        f"EXL3 wo_a load shape mismatch slice={slice_rank} "
+                        f"suffix={suffix}: dest {tuple(dest.shape)} != "
+                        f"loaded {tuple(loaded.shape)}"
+                    )
+                dest.copy_(loaded)
+                return
+
+            # Special handling for weight (bf16 staging) and markers
+            if suffix in ("weight", "mcg", "mul1"):
+                # Weight parameter: only load bf16 shards, discard EXL3 shards
+                if suffix == "weight":
+                    # Check shape matches the expected shard size
+                    expected_out = output_partition_sizes[shard_idx]
+                    expected_in = param.shape[1]
+                    loaded_shape = loaded_weight.shape
+                    if loaded_shape[0] != expected_out or (len(loaded_shape) > 1 and loaded_shape[1] != expected_in):
+                        # After TP narrowing, expect (shard_out, in)
+                        tp_sharded = _narrow_tp(loaded_weight, 0, tp_rank, tp_size) if is_row_parallel else _narrow_tp(loaded_weight, 1, tp_rank, tp_size) if not is_row_parallel else loaded_weight
+                        if tuple(tp_sharded.shape) != (expected_out, expected_in):
+                            raise RuntimeError(
+                                f"EXL3 weight load shape mismatch shard={shard_idx}: "
+                                f"expected ({expected_out},{expected_in}) but got {tuple(loaded_weight.shape)} "
+                                f"(after TP: {tuple(tp_sharded.shape)})"
+                            )
+                    # If this shard is in bf16_shards, copy; otherwise discard
+                    if shard_idx in bf16_shards:
+                        bf16_idx = bf16_shards.index(shard_idx)
+                        bf16_row_start = sum(output_partition_sizes[i] for i in bf16_shards[:bf16_idx])
+                        bf16_row_end = bf16_row_start + expected_out
+                        param.data[bf16_row_start:bf16_row_end].copy_(loaded_weight.detach())
+                    # else: discard this EXL3 shard's stale BF16 weight
+                    return
+                else:
+                    # Marker (mcg or mul1): store the value (will be 0 if marker not present)
+                    dest = param.data[shard_idx]
+                    if tuple(dest.shape) != (1,):
+                        raise RuntimeError(
+                            f"EXL3 {suffix} marker shape mismatch: expected (1,) got {tuple(dest.shape)}"
+                        )
+                    loaded_val = loaded_weight.detach().item() if loaded_weight.numel() > 0 else 0
+                    dest[0] = int(loaded_val)
+                    return
+
+            # Normal EXL3 suffix handling (trellis, suh, svh)
+            loaded = loaded_weight.detach().contiguous()
+
+            # Apply TP slicing based on layer type
+            if is_row_parallel:
+                # Row-parallel: input is sharded, trellis dim 0 and suh dim 0
+                sharded = shard_exl3_row(loaded, suffix, tp_rank, tp_size)
+            else:
+                # Column-parallel: output is sharded, trellis dim 1 and svh dim 0
+                sharded = shard_exl3_col(loaded, suffix, tp_rank, tp_size)
+
+            # Copy into the right location
+            if suffix == "trellis":
+                # Trellis is fused; narrow dim1 for this shard
+                out_tiles_start = sum(s // 16 for s in output_partition_sizes[:shard_idx])
+                out_tiles_end = out_tiles_start + output_partition_sizes[shard_idx] // 16
+                dest = param.data[:, out_tiles_start:out_tiles_end, :]
+            elif suffix == "suh":
+                # Suh per-shard
+                dest = param.data[shard_idx]
+            elif suffix == "svh":
+                # Svh is concatenated; slice for this shard
+                out_start = sum(output_partition_sizes[:shard_idx])
+                out_end = out_start + output_partition_sizes[shard_idx]
+                dest = param.data[out_start:out_end]
+            else:
+                raise ValueError(f"unknown EXL3 suffix={suffix}")
+
+            if tuple(dest.shape) != tuple(sharded.shape):
+                raise RuntimeError(
+                    f"EXL3 linear load shape mismatch shard={shard_idx} "
+                    f"suffix={suffix}: dest {tuple(dest.shape)} != "
+                    f"loaded {tuple(sharded.shape)}"
+                )
+            dest.copy_(sharded)
+
+        return weight_loader
+
+    def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
+        if not hasattr(layer, "trellis"):
+            return
+
+        # Get bf16 shards and verify exactly one marker per EXL3 shard
+        n_shards = int(layer._exl3_linear_n_shards)
+        output_sizes = layer._exl3_linear_output_partition_sizes
+        bf16_shards = getattr(layer, "_exl3_linear_bf16_shards", [])
+
+        mcg_vals = layer.mcg.reshape(-1)
+        mul1_vals = layer.mul1.reshape(-1)
+
+        for i in range(n_shards):
+            # Skip marker checks for bf16 shards - they don't use LinearEXL3
+            if i in bf16_shards:
+                continue
+            mcg_is_set = mcg_vals[i].item() != 0
+            mul1_is_set = mul1_vals[i].item() != 0
+            if mcg_is_set and mul1_is_set:
+                raise RuntimeError(
+                    f"EXL3 linear shard {i}: both mcg and mul1 markers are set; "
+                    f"exactly one codebook marker must be present"
+                )
+            if not mcg_is_set and not mul1_is_set:
+                raise RuntimeError(
+                    f"EXL3 linear shard {i}: neither mcg nor mul1 marker is set; "
+                    f"exactly one codebook marker must be present"
+                )
+            # Verify marker value
+            if mcg_is_set and mcg_vals[i].item() != MCG_MARKER_SIGNED_INT32:
+                raise RuntimeError(
+                    f"EXL3 linear shard {i}: mcg marker is {mcg_vals[i].item()}, "
+                    f"expected {MCG_MARKER_SIGNED_INT32}"
+                )
+            if mul1_is_set and mul1_vals[i].item() != MUL1_MARKER_SIGNED_INT32:
+                raise RuntimeError(
+                    f"EXL3 linear shard {i}: mul1 marker is {mul1_vals[i].item()}, "
+                    f"expected {MUL1_MARKER_SIGNED_INT32}"
+                )
+
+        # Build LinearEXL3 objects for EXL3 shards only (skip bf16 shards)
+        linears = []
+        for i in range(n_shards):
+            if i in bf16_shards:
+                # bf16 shards don't use LinearEXL3; store None as placeholder
+                linears.append(None)
+                continue
+            out_tiles_start = sum(s // 16 for s in output_sizes[:i])
+            out_tiles_end = out_tiles_start + output_sizes[i] // 16
+            trellis_shard = layer.trellis[:, out_tiles_start:out_tiles_end, :].contiguous()
+            suh_shard = layer.suh[i].contiguous()
+            svh_shard = layer.svh[
+                sum(output_sizes[:i]) : sum(output_sizes[: i + 1])
+            ].contiguous()
+            mcg_shard = layer.mcg[i].contiguous() if mcg_vals[i].item() != 0 else None
+            mul1_shard = layer.mul1[i].contiguous() if mul1_vals[i].item() != 0 else None
+
+            linear = make_linear_exl3(
+                trellis_shard, suh_shard, svh_shard, mcg_shard, mul1_shard, out_dtype=torch.float16
+            )
+            linears.append(linear)
+
+        layer._exl3_linears = linears
+
+        # LNA-LAB: pre-tune the EXL3 cooperative GEMM for the decode row counts before vLLM
+        # captures CUDA graphs. The exllamav3 autotuner times candidates with events and
+        # stream syncs, which is illegal during capture; warming the launch/disk cache here
+        # keeps capture on the tuned kernel instead of the static heuristic.
+        prewarm = os.environ.get("LNA_EXL3_PREWARM_ROWS", "1,2,4,8,16")
+        if prewarm:
+            try:
+                dev = linears[0].trellis.device if linears and linears[0] is not None else None
+                if dev is not None:
+                    for r in (int(v) for v in prewarm.split(",") if v.strip()):
+                        for lin in linears:
+                            if lin is not None:
+                                x = torch.zeros((r, int(lin.in_features)), dtype=torch.float16, device=dev)
+                                lin.forward(x, {}, out_dtype=torch.float32)
+                    torch.cuda.synchronize(dev)
+            except Exception as exc:  # pragma: no cover - diagnostic only
+                logger.warning("EXL3 prewarm skipped for %s: %s", getattr(layer, "prefix", "?"), exc)
+
+        # Keep bf16 weights if present, remove weight staging param if all loaded
+        if bf16_shards and hasattr(layer, "weight"):
+            bf16_rows = sum(output_sizes[i] for i in bf16_shards)
+            if bf16_rows > 0:
+                layer._exl3_bf16_weight = layer.weight.data.clone()
+            # Delete weight staging param only if it has rows; empty param stays as placeholder
+            if layer.weight.data.shape[0] > 0:
+                try:
+                    delattr(layer, "weight")
+                except Exception:
+                    pass
+        elif hasattr(layer, "weight"):
+            # No bf16 shards, delete the staging param
+            try:
+                delattr(layer, "weight")
+            except Exception:
+                pass
+
+        # Free fused parameters to avoid memory doubling
+        for param_name in ("trellis", "suh", "svh", "mcg", "mul1"):
+            if hasattr(layer, param_name):
+                try:
+                    delattr(layer, param_name)
+                except Exception:
+                    pass
+
+    def apply(
+        self,
+        layer,
+        x: torch.Tensor,
+        bias: torch.Tensor | None = None,
+    ) -> torch.Tensor:
+        linears = getattr(layer, "_exl3_linears", None)
+        if not linears:
+            raise RuntimeError("EXL3 linear layers were not built after weight load")
+
+        # x shape: (batch, in_features) or (batch, ..., in_features)
+        # Flatten to 2D: (rows, in_features)
+        orig_shape = x.shape
+        if len(orig_shape) > 2:
+            # Multi-dim input: flatten to (rows, in)
+            rows = 1
+            for d in orig_shape[:-1]:
+                rows *= d
+            x_2d = x.reshape(rows, orig_shape[-1])
+        else:
+            x_2d = x
+
+        # Cast to contiguous fp16 for EXL3 shards
+        x_fp16 = x_2d.to(torch.float16).contiguous()
+
+        # Get bf16 shards and weight if present
+        bf16_shards = getattr(layer, "_exl3_linear_bf16_shards", [])
+        bf16_weight = getattr(layer, "_exl3_bf16_weight", None)
+        output_sizes = layer._exl3_linear_output_partition_sizes
+        n_shards = len(linears)
+
+        # Run each shard in declared order
+        outputs = []
+        for i in range(n_shards):
+            if i in bf16_shards:
+                # BF16 shard: use dense linear
+                if bf16_weight is None:
+                    raise RuntimeError(
+                        f"EXL3 bf16 shard {i} but _exl3_bf16_weight is missing"
+                    )
+                bf16_idx = bf16_shards.index(i)
+                out_start = sum(output_sizes[j] for j in bf16_shards[:bf16_idx])
+                out_end = out_start + output_sizes[i]
+                w_shard = bf16_weight[out_start:out_end]
+                out = F.linear(x_2d, w_shard).to(dtype=torch.float32)
+                outputs.append(out)
+            else:
+                # EXL3 shard
+                linear = linears[i]
+                if linear is None:
+                    raise RuntimeError(f"EXL3 linear shard {i} is None")
+                out = linear.forward(x_fp16, {}, out_dtype=torch.float32)
+                outputs.append(out)
+
+        # Concatenate shards along output dimension
+        if len(outputs) > 1:
+            y = torch.cat(outputs, dim=1)
+        else:
+            y = outputs[0]
+
+        # Cast back to input dtype
+        y = y.to(dtype=x.dtype)
+
+        # Add bias if provided
+        if bias is not None:
+            y = y + bias
+
+        # Restore original shape
+        if len(orig_shape) > 2:
+            y = y.reshape(*orig_shape[:-1], y.shape[-1])
+
+        return y

lna-lab/ppl-vllm.py

@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+"""ppl meter through a vLLM OpenAI seat: prompt_logprobs over fixed text chunks. usage: ppl-vllm.py [port] [text_file] [ctx_tokens] [n_chunks]"""
+import json, math, sys, urllib.request
+port = sys.argv[1] if len(sys.argv) > 1 else "8899"
+path = sys.argv[2] if len(sys.argv) > 2 else "/run/media/tonoken3/DATA1/Models/wikitext-2-raw/wiki.test.raw"
+ctx = int(sys.argv[3]) if len(sys.argv) > 3 else 2048; nch = int(sys.argv[4]) if len(sys.argv) > 4 else 8
+U = f"http://127.0.0.1:{port}/v1/completions"
+text = open(path, encoding="utf-8", errors="ignore").read()
+chars = ctx * 4  # rough char budget per chunk; server truncates nothing, we just pick disjoint windows
+tot_nll = 0.0; tot_n = 0
+for i in range(nch):
+    chunk = text[i * chars:(i + 1) * chars]
+    body = {"model": "DSV4-Flash", "prompt": chunk, "max_tokens": 1, "temperature": 0, "prompt_logprobs": 0, "echo": False}
+    r = json.load(urllib.request.urlopen(urllib.request.Request(U, data=json.dumps(body).encode(), headers={"content-type": "application/json"}), timeout=900))
+    pl = r["choices"][0].get("prompt_logprobs") or []
+    lps = []
+    for tok in pl[1:ctx]:  # skip first token (no context); cap at ctx tokens
+        if tok:
+            lps.append(max(v["logprob"] if isinstance(v, dict) else v for v in tok.values()))
+    n = len(lps); nll = -sum(lps)
+    tot_nll += nll; tot_n += n
+    print(f"chunk {i}: tokens={n} ppl={math.exp(nll / max(n, 1)):.4f}", flush=True)
+print(f"PPL={math.exp(tot_nll / max(tot_n, 1)):.4f} tokens={tot_n} ctx={ctx} chunks={nch} file={path}")

lna-lab/serve-dsv4-tp8.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+# DSV4-Flash-Vision EXL3 MixedK on 8 experiment GPUs (TP=8) via lna-lab/vllm-exl3:dsv4. YUKI 2026-09-03
+# env: GPUS (default 0,1,2,3,5,7,8,9) PORT(8899) MAXLEN(65536) SPEC ('' or json) UTIL(0.90) NAME(dsv4)
+set -u
+GPUS=${GPUS:-0,1,2,3,5,7,8,9}; PORT=${PORT:-8899}; MAXLEN=${MAXLEN:-65536}; UTIL=${UTIL:-0.90}; NAME=${NAME:-dsv4}
+TP=$(echo $GPUS | tr ',' '\n' | wc -l)
+M=${MODEL:-/run/media/tonoken3/DATA1/DSV4-Flash-Vision-EXL3-MixedK}
+EXTRA=()
+[[ -n "${SPEC:-}" ]] && EXTRA+=(--speculative-config "$SPEC")
+docker rm -f $NAME >/dev/null 2>&1
+docker run --ulimit core=0 --cap-add=SYS_PTRACE -e VLLM_DISABLE_SHARED_EXPERTS_STREAM=${SHARED_STREAM_OFF:-1} -v /run/media/tonoken3/DATA1/.tmp/dsv4-cache/root-cache:/root/.cache -v /run/media/tonoken3/DATA1/.tmp/dsv4-cache/tilelang:/root/.tilelang ${EXT_SO:+-v $EXT_SO:/usr/local/lib/python3.12/dist-packages/$(basename ${EXT_SO:-x})} ${PLUGIN_SRC:+-v $PLUGIN_SRC:/usr/local/lib/python3.12/dist-packages/vllm_exl3} -d --name $NAME --gpus "\"device=$GPUS\"" --shm-size=16g --ipc=host \
+  -e NCCL_P2P_DISABLE=1 -e NCCL_CUMEM_ENABLE=0 -e EXLLAMAV3_TUNE_CACHE=/lab/exl3-tune-cache ${AUX_STREAMS:+-e LNA_DSV4_AUX_STREAMS=$AUX_STREAMS} ${NCCL_EXTRA:-} -e VLLM_NO_USAGE_STATS=1 -e DO_NOT_TRACK=1 \
+  -p 127.0.0.1:$PORT:8000 -v /run/media/tonoken3/DATA1:/run/media/tonoken3/DATA1 -v /run/media/tonoken3/DATA1/vllm-exl3-lab:/lab \
+  lna-lab/vllm-exl3:${IMAGE:-dsv4} \
+  $M --served-model-name DSV4-Flash --tensor-parallel-size $TP --quantization exl3 \
+  --max-model-len $MAXLEN --max-num-seqs ${SEQS:-4} --max-num-batched-tokens ${BT:-2048} \
+  --kv-cache-dtype ${KVDT:-fp8} --gpu-memory-utilization $UTIL ${EAGER:+--enforce-eager} --compilation-config "${COMPILE:-{\"cudagraph_capture_sizes\":[1,2,4]\}}" --disable-custom-all-reduce \
+  --no-enable-prefix-caching --trust-remote-code ${PROFILE:+--profiler-config "$PROFILE"} \
+  --enable-auto-tool-choice --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 "${EXTRA[@]}"
+echo "container $NAME on :$PORT (TP=$TP, spec=${SPEC:-off}); docker logs -f $NAME"

lna-lab/tools/dense_bake.py

@@ -0,0 +1,924 @@
+#!/usr/bin/env python3
+"""Calibrate and bake only the dense DSV4 linears into EXL3.
+
+The bake deliberately keeps the routed-expert tensors in the existing EXL3
+pack.  It uses the read-only exllamav3 checkout as a runtime dependency and
+does not copy or modify that checkout.  ``--dry-run`` only inspects config,
+model layout, and (when ``--merge`` is supplied) local safetensors headers.
+It never imports CUDA conversion kernels or writes a model pack.
+"""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+import os
+import re
+import struct
+import sys
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any, Iterable
+
+try:
+    from . import layer_overlay as overlay
+except ImportError:  # Running as ``python tools/dense_bake.py``.
+    import layer_overlay as overlay
+
+
+CONFIG_NAME = "config.json"
+INDEX_NAME = "model.safetensors.index.json"
+DENSE_SHARD = "model-dense-exl3.safetensors"
+EXL3_SUFFIXES = ("trellis", "suh", "svh", "mcg")
+EXL3_DTYPES = {"trellis": "I16", "suh": "F16", "svh": "F16", "mcg": "I32"}
+MCG_MARKER = -877912083
+DEFAULT_EXL3_SOURCE = "/run/media/tonoken3/DATA1/.tmp/exl3src"
+
+_LAYER_FILE_RE = re.compile(r"^layers\.(?P<layer>[0-9]+)\.safetensors$")
+_WO_A_RE = re.compile(r"^attn\.wo_a\.slice\.(?P<slice>[0-7])$")
+_DENSE_TAILS = {
+    "attn.wq_a",
+    "attn.wq_b",
+    "attn.wkv",
+    "attn.wo_b",
+    "attn.compressor.wkv",
+    "attn.compressor.wgate",
+    "attn.indexer.compressor.wkv",
+    "attn.indexer.compressor.wgate",
+    "attn.indexer.wq_b",
+    "ffn.shared_experts.w1",
+    "ffn.shared_experts.w2",
+    "ffn.shared_experts.w3",
+}
+
+
+def log(message: str) -> None:
+    print(message, flush=True)
+
+
+def dense_kind(key: str) -> str | None:
+    """Return ``attn`` or ``shared`` only for the ordered T4 target list."""
+    match = re.match(r"^layers\.[0-9]+\.(?P<tail>.+)$", key)
+    if match is None:
+        return None
+    tail = match.group("tail")
+    if tail in _DENSE_TAILS or _WO_A_RE.fullmatch(tail):
+        return "shared" if tail.startswith("ffn.shared_experts.") else "attn"
+    return None
+
+
+def is_dense_key(key: str) -> bool:
+    return dense_kind(key) is not None
+
+
+def dense_keys_for_layer(layer: int, keys: Iterable[str]) -> list[str]:
+    """Filter a module's recursive Linear keys, excluding experts, MTP, and non-linears."""
+    result = [key for key in keys if dense_kind(key) is not None and key.startswith(f"layers.{layer}.")]
+    result.sort()
+    wo_a = [key for key in result if ".attn.wo_a.slice." in key]
+    if wo_a and {int(key.rsplit(".", 1)[1]) for key in wo_a} != set(range(8)):
+        raise RuntimeError(f"layers.{layer}: wo_a must contain exactly slices 0..7, got {wo_a}")
+    return result
+
+
+def bits_for_key(key: str, bits: float, attn_bits: float | None, shared_bits: float | None) -> float:
+    kind = dense_kind(key)
+    if kind is None:
+        raise ValueError(f"not a T4 dense key: {key}")
+    value = float(attn_bits if kind == "attn" and attn_bits is not None else
+                  shared_bits if kind == "shared" and shared_bits is not None else bits)
+    return int(value) if value.is_integer() else value  # LNA-LAB: the quantizer shifts by K (must be int)
+
+
+def _schema_bits(value: float) -> int:
+    if not float(value).is_integer() or int(value) not in (2, 3, 4, 5, 6):
+        raise ValueError(f"vLLM non_routed_exl3 requires integer K in 2..6, got {value}")
+    return int(value)
+
+
+def _layer_number(key: str) -> int:
+    match = re.match(r"^layers\.([0-9]+)(?:\.|$)", key)  # LNA-LAB: module keys are bare "layers.N"
+    if match is None:
+        raise ValueError(f"not a language-layer key: {key}")
+    return int(match.group(1))
+
+
+def _parse_devices(value: str) -> list[int]:
+    try:
+        devices = [int(part.strip()) for part in value.split(",") if part.strip()]
+    except ValueError as exc:
+        raise ValueError(f"invalid --devices value: {value!r}") from exc
+    if not devices or any(device < 0 for device in devices) or len(set(devices)) != len(devices):
+        raise ValueError(f"--devices must be unique non-negative integers: {value!r}")
+    return devices
+
+
+def _runtime_import(*, dry_run: bool = False):
+    """Import the exact exllamav3 classes/functions used by the bake.
+
+    A dry-run can construct the Python model graph without the native extension.
+    The extension stub is never used by a real bake; it exists solely to make
+    the requested offline layout check possible on a CPU-only host.
+    """
+    source_root = os.environ.get("EXL3_SOURCE", DEFAULT_EXL3_SOURCE)
+    if not Path(source_root).is_dir():
+        raise RuntimeError(f"exllamav3 source checkout is missing: {source_root}")
+    if source_root not in sys.path:
+        sys.path.insert(0, source_root)
+
+    if dry_run and "exllamav3.ext" not in sys.modules:
+        class _ExtensionStub:
+            def __getattr__(self, _name):
+                return lambda *args, **kwargs: None
+
+        import types
+        ext_module = types.ModuleType("exllamav3.ext")
+        ext_module.exllamav3_ext = _ExtensionStub()
+        sys.modules["exllamav3.ext"] = ext_module
+
+    from exllamav3 import Config, Model, Tokenizer
+    from exllamav3.modules import Linear
+    from exllamav3.loader.safetensors import SafetensorsCollection, VariantSafetensorsCollection
+    from exllamav3.conversion.calibration_data import get_default_calibration
+    from exllamav3.conversion.convert_model import (
+        advance_state_parallel,
+        capture_module_parallel,
+        load_parallel_calib_modules,
+        quantize_linears_parallel,
+        quantize_linears_single,
+        load_tensor,
+        save_tensor,
+    )
+    return SimpleNamespace(
+        Config=Config,
+        Model=Model,
+        Tokenizer=Tokenizer,
+        Linear=Linear,
+        SafetensorsCollection=SafetensorsCollection,
+        VariantSafetensorsCollection=VariantSafetensorsCollection,
+        get_default_calibration=get_default_calibration,
+        advance_state_parallel=advance_state_parallel,
+        capture_module_parallel=capture_module_parallel,
+        load_parallel_calib_modules=load_parallel_calib_modules,
+        quantize_linears_parallel=quantize_linears_parallel,
+        quantize_linears_single=quantize_linears_single,
+        load_tensor=load_tensor,
+        save_tensor=save_tensor,
+    )
+
+
+def _load_variant_model(src: Path, experts: Path, runtime):
+    config = runtime.Config.from_directory(str(src))
+    expert_stc = runtime.SafetensorsCollection(str(experts))
+    variant = runtime.VariantSafetensorsCollection(config.stc)
+    # This is the T4 contract: only routed expert keys are overridden.
+    variant.add_stc(["layers.*.ffn.experts.*"], expert_stc)
+    config.stc = variant
+    model = runtime.Model.from_config(config)
+    # DSV4's architecture exposes this flag for calibration; make the T4
+    # invariant explicit in case a future architecture default changes.
+    model.calibration_all_experts = True
+    return config, model
+
+
+def _module_plan(model, linear_type) -> list[tuple[int, Any, list[Any]]]:
+    plan = []
+    for idx, module in enumerate(model.modules):
+        layer_match = re.fullmatch(r"layers\.([0-9]+)", getattr(module, "key", ""))
+        if layer_match is None:
+            continue
+        layer = int(layer_match.group(1))
+        linears = [m for m in module if isinstance(m, linear_type)]
+        keys = dense_keys_for_layer(layer, [m.key for m in linears])
+        by_key = {m.key: m for m in linears}
+        targets = [by_key[key] for key in keys]
+        if len(targets) == 0:
+            raise RuntimeError(f"{module.key}: no dense linears found")
+        plan.append((idx, module, targets))
+    if not plan:
+        raise RuntimeError("model has no layers.N modules")
+    return plan
+
+
+def dry_run_layout(src: Path, experts: Path) -> dict[str, Any]:
+    if not src.is_dir() or not experts.is_dir():
+        raise RuntimeError(f"dry-run inputs must be directories: {src}, {experts}")
+    runtime = _runtime_import(dry_run=True)
+    # Construct the real source layout with the D2 expert override, and also
+    # construct Config.from_directory(D2) explicitly: the latter is the
+    # complete experts-only config used for the offline pack sanity check.
+    config, model = _load_variant_model(src, experts, runtime)
+    d2_config = runtime.Config.from_directory(str(experts))
+    plan = _module_plan(model, runtime.Linear)
+    counts = {"attn": 0, "shared": 0}
+    for _, _, targets in plan:
+        for linear in targets:
+            counts[dense_kind(linear.key)] += 1
+    log(
+        f"DRY_RUN_LAYOUT architecture={getattr(config, 'architecture', 'unknown')} "
+        f"modules={len(model.modules)} layers={len(plan)} dense_linears={sum(counts.values())} "
+        f"attn={counts['attn']} shared={counts['shared']} routed_override=layers.*.ffn.experts.* "
+        f"d2_config={getattr(d2_config, 'architecture', 'unknown')}"
+    )
+    for _, module, targets in plan[:2]:
+        log(f"DRY_RUN_LAYER key={module.key} dense_linears={len(targets)}")
+    log(f"DRY_RUN_LAYER key={plan[-1][1].key} dense_linears={len(plan[-1][2])}")
+    return {"config": config, "model": model, "runtime": runtime, "plan": plan}
+
+
+def _load_module(module, device, *, load_slice=None, source=None, close=True):
+    defer = source is None and module.can_defer_load()
+    if defer:
+        module.config.stc.begin_deferred_load()
+    try:
+        kwargs = {}
+        if load_slice is not None:
+            kwargs["load_slice"] = load_slice
+        if source is not None:
+            kwargs["source"] = source
+        module.load(device, **kwargs)
+    finally:
+        if defer:
+            module.config.stc.end_deferred_load()
+    if close:
+        module.config.stc.close()
+
+
+def _capture_serial(model, module, state, original_ids, quant_preserves, bad_rows, num_ref_states=5):
+    capture_h = {}
+    ref_states = {}
+
+    def get_preserve(i, params):
+        params.update(quant_preserves[i])
+        params["quant_preserve"] = quant_preserves[i]
+
+    def put_preserve(i, params):
+        quant_preserves[i] = params["quant_preserve"]
+
+    for i in range(len(state)):
+        if i in bad_rows:
+            continue
+        params = {
+            "attn_mode": "flash_attn_nc",
+            "capture": capture_h,
+            "activate_all_experts": model.calibration_all_experts,
+            "input_ids": original_ids[i],
+        }
+        get_preserve(i, params)
+        result = module.forward(module.prepare_for_device(state[i], params), params)
+        put_preserve(i, params)
+        if i < num_ref_states:
+            if model.calibration_all_experts:
+                params = {"attn_mode": "flash_attn_nc", "input_ids": original_ids[i]}
+                get_preserve(i, params)
+                result = module.forward(module.prepare_for_device(state[i], params), params)
+                put_preserve(i, params)
+            if result.isfinite().all().item():
+                ref_states[i] = result.cpu()
+            else:
+                bad_rows.add(i)
+                log(f"WARNING non-finite reference state row={i}; excluded")
+    for item in capture_h.values():
+        item["H_swap_device"] = item["H"].device
+        item["H"] = item["H"].cpu()
+    return capture_h, ref_states, get_preserve, put_preserve
+
+
+def _advance_serial(model, module, state, original_ids, quant_preserves, ref_states, bad_rows, have_linears):
+    error = 0.0
+    measured = 0
+
+    def get_preserve(i, params):
+        params.update(quant_preserves[i])
+        params["quant_preserve"] = quant_preserves[i]
+
+    def put_preserve(i, params):
+        quant_preserves[i] = params["quant_preserve"]
+
+    for i in range(len(state)):
+        if i in bad_rows:
+            continue
+        params = {"attn_mode": "flash_attn_nc", "input_ids": original_ids[i]}
+        state[i] = module.prepare_for_device(state[i], params)
+        if i < 5:
+            get_preserve(i, params)
+            result = module.forward(state[i], params)
+            put_preserve(i, params)
+            if not result.isfinite().all().item():
+                bad_rows.add(i)
+                continue
+            state[i] = result.cpu()
+        else:
+            # We stop after the final transformer block, so every row must be
+            # advanced; the reference/error pass remains limited to five rows.
+            get_preserve(i, params)
+            state[i] = module.forward(state[i], params).cpu()
+            put_preserve(i, params)
+        ref = ref_states.get(i)
+        if ref is not None and have_linears and i not in bad_rows:
+            x = state[i].view(-1, state[i].shape[-1]).float()
+            y = ref.view(-1, ref.shape[-1]).float()
+            error += (torch_norm(x - y) / torch_norm(y)).item()
+            measured += 1
+            ref_states[i] = None
+    return error / max(measured, 1), measured
+
+
+def _set_new_tensors(stc, tensors) -> None:
+    """Set the in-memory reload layer on either a plain or variant STC.
+
+    VariantSafetensorsCollection intentionally leaves ``set_new_tensors``
+    unimplemented.  Dense keys resolve to its ``main`` collection, so the
+    converter's reload operation is applied to that collection only; the
+    routed-expert override remains the D2 collection.
+    """
+    target = getattr(stc, "main", stc)
+    target.set_new_tensors(tensors)
+
+
+def _inject_reload_keys(stc, tensors) -> list[str]:
+    """Make variant ``has_tensor_group`` see in-memory EXL3 keys.
+
+    The audited VariantSafetensorsCollection delegates ``get_tensor`` but its
+    ``has_tensor_group`` checks only ``tensor_file_map``.  Adding temporary
+    sentinels lets Linear.load choose its EXL3 branch; get_tensor still takes
+    the actual values from ``new_tensors``.  The sentinels are removed before
+    the collection is closed.
+    """
+    main = getattr(stc, "main", stc)
+    added = []
+    for key in tensors:
+        if key not in main.tensor_file_map:
+            main.tensor_file_map[key] = "__dense_bake_memory__"
+            added.append(key)
+    return added
+
+
+def _remove_reload_keys(stc, added: list[str]) -> None:
+    main = getattr(stc, "main", stc)
+    for key in added:
+        main.tensor_file_map.pop(key, None)
+
+
+def torch_norm(tensor):
+    # Kept as a tiny late-bound helper so importing this tool for header-only
+    # merge planning does not initialize torch/CUDA.
+    import torch
+    return torch.linalg.norm(tensor, "fro")
+
+
+def _checkpoint(runtime, work: Path, job: dict, state, original_ids, args_dict: dict):
+    ckpt = work / "ckpt"
+    ckpt.mkdir(parents=True, exist_ok=True)
+    runtime.save_tensor(state, "ckpt/state.safetensors", args_dict)
+    runtime.save_tensor(original_ids, "ckpt/original_input_ids.safetensors", args_dict)
+    (ckpt / "job.json").write_text(json.dumps(job, indent=2) + "\n", encoding="utf-8")
+
+
+def _read_job(work: Path) -> dict:
+    path = work / "ckpt" / "job.json"
+    if not path.is_file():
+        return {"next_module_idx": 0, "bad_rows": []}
+    try:
+        data = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError) as exc:
+        raise RuntimeError(f"invalid checkpoint {path}: {exc}") from exc
+    if not isinstance(data, dict) or not isinstance(data.get("next_module_idx"), int):
+        raise RuntimeError(f"checkpoint {path} has no integer next_module_idx")
+    return data
+
+
+def _prepare_real_state(runtime, experts: Path, args, model, job, work: Path):
+    import torch
+    args_dict = vars(args).copy()
+    args_dict["work_dir"] = str(work)
+    args_dict["apply_out_scales"] = True
+    args_dict["image_dump"] = False
+    args_dict["codebook"] = "mcg"
+    args_dict["verbose"] = args.verbose
+    quant_preserves = []
+    bad_rows = set(job.get("bad_rows") or [])
+    if job["next_module_idx"] > 0:
+        state = runtime.load_tensor("ckpt/state.safetensors", args_dict)
+        original_ids = runtime.load_tensor("ckpt/original_input_ids.safetensors", args_dict)
+        if not isinstance(state, list) or not isinstance(original_ids, list):
+            raise RuntimeError("checkpoint state/original_input_ids must both be tensor lists")
+        quant_preserves = [{} for _ in state]
+        return state, original_ids, quant_preserves, bad_rows, args_dict
+
+    tokenizer_config = runtime.Config.from_directory(str(experts))
+    tokenizer = runtime.Tokenizer.from_config(tokenizer_config)
+    original_ids = runtime.get_default_calibration(
+        {"cal_rows": args.cal_rows, "cal_cols": args.cal_cols}, tokenizer
+    )
+    state = list(original_ids)
+    quant_preserves = [{} for _ in state]
+    # The converter starts with token rows and advances them through embed and
+    # hc_expand before entering layers.N.  These modules are intentionally not
+    # quantized or written to qtensors.
+    for idx, module in enumerate(model.modules[: model.first_block_idx]):
+        log(f"LOAD_PREFIX module={module.key}")
+        _load_module(module, torch.device(f"cuda:{args.devices[0]}"))
+        for i in range(len(state)):
+            params = {"attn_mode": "flash_attn_nc", "input_ids": original_ids[i]}
+            state[i] = module.forward(module.prepare_for_device(state[i], params), params).cpu()
+        module.unload()
+    return state, original_ids, quant_preserves, bad_rows, args_dict
+
+
+def bake(args: argparse.Namespace) -> int:
+    src = Path(args.src).expanduser().resolve()
+    experts = Path(args.experts).expanduser().resolve()
+    work = Path(args.work).expanduser().resolve()
+    if not src.is_dir() or not experts.is_dir():
+        raise RuntimeError(f"--src and --experts must be directories: {src}, {experts}")
+    if args.bits < 1 or args.bits > 8 or any(
+        value is not None and (value < 1 or value > 8)
+        for value in (args.attn_bits, args.shared_bits)
+    ):
+        raise ValueError("bits must be in the range 1..8")
+    devices = _parse_devices(args.devices)
+    if not args.dry_run:
+        import torch
+        if not torch.cuda.is_available():
+            raise RuntimeError("real bake requires CUDA; use --dry-run for the offline checks")
+        count = torch.cuda.device_count()
+        if any(device >= count for device in devices):
+            raise RuntimeError(f"requested CUDA devices {devices}, but only {count} are visible")
+
+    runtime = _runtime_import(dry_run=args.dry_run)
+    config, model = _load_variant_model(src, experts, runtime)
+    plan = _module_plan(model, runtime.Linear)
+    log(
+        f"BAKE_PLAN layers={len(plan)} dense_linears={sum(len(t) for _, _, t in plan)} "
+        f"bits={args.bits:g} attn_bits={(args.attn_bits if args.attn_bits is not None else args.bits):g} "
+        f"shared_bits={(args.shared_bits if args.shared_bits is not None else args.bits):g} codebook=mcg"
+    )
+    if args.dry_run:
+        for _, module, targets in plan[:2]:
+            log(f"DRY_RUN_BAKE module={module.key} targets={len(targets)}")
+        log(f"DRY_RUN_BAKE module={plan[-1][1].key} targets={len(plan[-1][2])}")
+        if args.merge:
+            merge_plan = plan_merge(work, experts, args)
+            log(f"DRY_RUN_MERGE dropped={len(merge_plan['dropped'])} new={len(merge_plan['new_plan'])}")
+        return 0
+
+    work.mkdir(parents=True, exist_ok=True)
+    (work / "qtensors").mkdir(exist_ok=True)
+    (work / "debug").mkdir(exist_ok=True)
+    signature = {
+        "src": str(src), "experts": str(experts), "bits": args.bits,
+        "attn_bits": args.attn_bits, "shared_bits": args.shared_bits,
+        "cal_rows": args.cal_rows, "cal_cols": args.cal_cols, "codebook": "mcg",
+    }
+    signature_path = work / "args.json"
+    if args.resume:
+        if not signature_path.is_file():
+            raise RuntimeError(f"--resume requested but {signature_path} is missing")
+        if json.loads(signature_path.read_text(encoding="utf-8")) != signature:
+            raise RuntimeError("resume settings differ from work/args.json")
+    else:
+        qtensors_dir = work / "qtensors"
+        existing_qtensors = qtensors_dir.is_dir() and any(qtensors_dir.glob("layers.*.safetensors"))
+        if signature_path.exists() or (work / "ckpt" / "job.json").exists() or existing_qtensors:
+            raise RuntimeError("work directory already contains a bake; use --resume or choose a new --work path")
+        signature_path.write_text(json.dumps(signature, indent=2) + "\n", encoding="utf-8")
+
+    job = _read_job(work) if args.resume else {"next_module_idx": 0, "bad_rows": []}
+    state, original_ids, quant_preserves, bad_rows, args_dict = _prepare_real_state(
+        runtime, experts, args, model, job, work
+    )
+    job["cal_rows"] = args.cal_rows
+    job["cal_cols"] = args.cal_cols
+    job["codebook"] = "mcg"
+    job["bad_rows"] = sorted(bad_rows)
+    parallel = len(devices) > 1
+    replicas = [runtime.Model.from_config(config) for _ in devices[1:]] if parallel else []
+    for replica in replicas:
+        replica.calibration_all_experts = True
+    import torch
+
+    for module_idx, module, targets in plan:
+        if module_idx < job["next_module_idx"]:
+            continue
+        layer = _layer_number(module.key)
+        log(f"BAKE_LAYER start={module.key} targets={len(targets)}")
+        device = torch.device(f"cuda:{devices[0]}")
+        _load_module(module, device)
+        # Recalculate recursive linears after load; dense targets must be FP16
+        # here, while the routed expert variant is already EXL3.
+        targets = [m for m in module if isinstance(m, runtime.Linear) and is_dense_key(m.key)]
+        if len(targets) == 0 or any(getattr(m, "inner", None).__class__.__name__ != "LinearFP16" for m in targets):
+            raise RuntimeError(f"{module.key}: a dense target did not load as LinearFP16")
+        if any(m.qmap is None for m in targets):
+            raise RuntimeError(f"{module.key}: a dense target has no Hessian qmap")
+        keys = [m.key for m in targets]
+        if sorted(keys) != sorted(dense_keys_for_layer(layer, keys)):  # LNA-LAB: module order is not lexical
+            raise RuntimeError(f"{module.key}: dense target ordering/filter changed after load; got={keys} expected={dense_keys_for_layer(layer, keys)}")
+
+        capture_replicas = None
+        if parallel:
+            capture_replicas = runtime.load_parallel_calib_modules(
+                replicas, module_idx, devices, None
+            )
+        if capture_replicas is not None:
+            def get_preserve(i, params):
+                params.update(quant_preserves[i])
+                params["quant_preserve"] = quant_preserves[i]
+            def put_preserve(i, params):
+                quant_preserves[i] = params["quant_preserve"]
+            capture_h, ref_states = runtime.capture_module_parallel(
+                model, [module] + capture_replicas, devices, None, state,
+                original_ids, get_preserve, put_preserve, False, 0,
+                f"CAPTURE {module.key}", bad_rows,
+            )
+            for replica in capture_replicas:
+                replica.unload()
+        else:
+            capture_h, ref_states, get_preserve, put_preserve = _capture_serial(
+                model, module, state, original_ids, quant_preserves, bad_rows
+            )
+
+        strategy = {m.key: bits_for_key(m.key, args.bits, args.attn_bits, args.shared_bits) for m in targets}
+        for linear in targets:
+            linear.inner.swap_cpu()
+        if len(targets) >= len(devices) and all(bits <= 8 for bits in strategy.values()):
+            runtime.quantize_linears_parallel(
+                args_dict, targets, config, strategy, module_idx, devices, None, capture_h, state
+            )
+        else:
+            runtime.quantize_linears_single(
+                args_dict, targets, config, strategy, module_idx, devices, None, capture_h, state
+            )
+        qtensors = {}
+        for linear in targets:
+            qtensors.update(linear.get_tensors())
+        expected = {f"{m.key}.{suffix}" for m in targets for suffix in EXL3_SUFFIXES}
+        if set(qtensors) != expected:
+            raise RuntimeError(f"{module.key}: converter emitted unexpected tensors")
+        runtime.save_tensor(qtensors, f"qtensors/layers.{layer}.safetensors", args_dict)
+        module.unload()
+        reload_keys = _inject_reload_keys(config.stc, qtensors)
+        _set_new_tensors(config.stc, qtensors)
+        _load_module(module, device, source=qtensors, close=False)
+        advance_replicas = None
+        if parallel:
+            advance_replicas = runtime.load_parallel_calib_modules(
+                replicas, module_idx, devices, None, source=qtensors
+            )
+        _set_new_tensors(config.stc, None)
+        _remove_reload_keys(config.stc, reload_keys)
+        if advance_replicas is not None:
+            def get_preserve(i, params):
+                params.update(quant_preserves[i])
+                params["quant_preserve"] = quant_preserves[i]
+            def put_preserve(i, params):
+                quant_preserves[i] = params["quant_preserve"]
+            runtime.advance_state_parallel(
+                model, [module] + advance_replicas, devices, None, state,
+                original_ids, get_preserve, put_preserve, ref_states, True,
+                module_idx == plan[-1][0], f"ADVANCE {module.key}", bad_rows,
+            )
+            for replica in advance_replicas:
+                replica.unload()
+        else:
+            _advance_serial(model, module, state, original_ids, quant_preserves, ref_states, bad_rows, True)
+        module.unload()
+        job["next_module_idx"] = module_idx + 1
+        job["bad_rows"] = sorted(bad_rows)
+        job["completed_layers"] = sorted(set(job.get("completed_layers", [])) | {layer})
+        _checkpoint(runtime, work, job, state, original_ids, args_dict)
+        log(f"BAKE_LAYER done={module.key} checkpoint=after_layer_{layer}")
+    log(f"BAKE_COMPLETE work={work} layers={len(plan)}")
+    return 0
+
+
+def _read_work_tensors(work: Path) -> list[dict[str, Any]]:
+    qtensors = work / "qtensors"
+    if not qtensors.is_dir():
+        raise RuntimeError(f"missing work qtensors directory: {qtensors}")
+    result = []
+    seen = set()
+    for path in sorted(qtensors.glob("layers.*.safetensors"), key=lambda p: p.name):
+        match = _LAYER_FILE_RE.fullmatch(path.name)
+        if match is None:
+            continue
+        header_len, header = overlay.read_header(path)
+        for name, meta in header.items():
+            if name == "__metadata__":
+                continue
+            if not is_dense_key(name.rsplit(".", 1)[0]) or name.rsplit(".", 1)[1] not in EXL3_SUFFIXES:
+                raise RuntimeError(f"work file contains non-dense/non-EXL3 tensor: {path.name}:{name}")
+            base, suffix = name.rsplit(".", 1)
+            if _layer_number(base) != int(match.group("layer")):
+                raise RuntimeError(f"tensor {name} is in the wrong work layer file {path.name}")
+            if name in seen:
+                raise RuntimeError(f"duplicate work tensor: {name}")
+            expected_dtype = EXL3_DTYPES[suffix]
+            if meta.get("dtype") != expected_dtype:
+                raise RuntimeError(f"{name}: expected dtype {expected_dtype}, got {meta.get('dtype')}")
+            seen.add(name)
+            result.append({"target": name, "base": base, "suffix": suffix,
+                           "meta": meta, "path": path, "header_len": header_len})
+    if not result:
+        raise RuntimeError(f"no complete dense EXL3 tensors under {qtensors}")
+    grouped = {}
+    for item in result:
+        grouped.setdefault(item["base"], set()).add(item["suffix"])
+    incomplete = [base for base, suffixes in grouped.items() if suffixes != set(EXL3_SUFFIXES)]
+    if incomplete:
+        raise RuntimeError(f"work tensors have incomplete EXL3 groups, e.g. {incomplete[0]}")
+    for layer in sorted({_layer_number(base) for base in grouped}):
+        wo_a = {
+            int(base.rsplit(".", 1)[1])
+            for base in grouped
+            if _layer_number(base) == layer and _WO_A_RE.fullmatch(base.split(".", 2)[2])
+        }
+        if wo_a and wo_a != set(range(8)):
+            raise RuntimeError(f"work layer {layer} has incomplete wo_a slices: {sorted(wo_a)}")
+    return sorted(result, key=lambda item: item["target"])
+
+
+def _vllm_prefixes(layer: int, root: str, shared_prefix: str) -> dict[str, str]:
+    base = f"{root}.layers.{layer}"
+    return {
+        "attn.wq_a": f"{base}.attn.fused_wqa_wkv",
+        "attn.wkv": f"{base}.attn.fused_wqa_wkv",
+        "attn.wq_b": f"{base}.attn.wq_b",
+        "attn.wo_b": f"{base}.attn.wo_b",
+        "attn.compressor.wkv": f"{base}.attn.compressor.fused_wkv_wgate",
+        "attn.compressor.wgate": f"{base}.attn.compressor.fused_wkv_wgate",
+        "attn.indexer.compressor.wkv": f"{base}.attn.indexer.compressor.fused_wkv_wgate",
+        "attn.indexer.compressor.wgate": f"{base}.attn.indexer.compressor.fused_wkv_wgate",
+        "attn.indexer.wq_b": f"{base}.attn.indexer.wq_b",
+        "ffn.shared_experts.w1": f"{base}.{shared_prefix}.gate_up_proj",
+        "ffn.shared_experts.w3": f"{base}.{shared_prefix}.gate_up_proj",
+        "ffn.shared_experts.w2": f"{base}.{shared_prefix}.down_proj",
+    }
+
+
+def _config_with_vllm_block(base_config: dict, work_items: list[dict], args) -> dict:
+    config = copy.deepcopy(base_config)
+    q = config.setdefault("quantization_config", {})
+    layers = {}
+    for item in work_items:
+        base = item["base"]
+        tail = base.split(".", 2)[2]
+        if _WO_A_RE.fullmatch(tail):
+            # LNA-LAB: the plugin consumes one already-sliced wo_a per TP rank.
+            vllm = f"{args.vllm_root}.layers.{_layer_number(base)}.attn.wo_a"
+        else:
+            vllm = _vllm_prefixes(
+                _layer_number(base), args.vllm_root, args.vllm_shared_prefix
+            ).get(tail)
+        if vllm is None:
+            raise RuntimeError(f"no vLLM prefix mapping for {base}")
+        bits = args.attn_bits if dense_kind(base) == "attn" and args.attn_bits is not None else \
+            args.shared_bits if dense_kind(base) == "shared" and args.shared_bits is not None else args.bits
+        bits = _schema_bits(bits)
+        old = layers.setdefault(vllm, {"bits": bits})
+        if old["bits"] != bits:
+            raise RuntimeError(f"fused vLLM module has mixed bits: {vllm}")
+    # LNA-LAB: unmatched dense modules (notably indexer.weights_proj) stay BF16.
+    q.setdefault("non_routed_dtype_policy", "bf16_as_stored")
+    q["non_routed_exl3"] = {
+        "codebook": "mcg",
+        "layers": dict(sorted(layers.items())),
+    }
+    return config
+
+
+def _write_dense_shard(out: Path, items: list[dict]) -> dict[str, Any]:
+    destination = out / DENSE_SHARD
+    partial = out / (DENSE_SHARD + ".partial")
+    entries = [(item["target"], item["meta"]) for item in items]
+    with partial.open("wb") as dst:
+        header_len, _ = overlay.common.write_safetensors_header(
+            dst, entries, {"format": "pt", "source": "exllamav3-dense-bake", "codebook": "mcg"}
+        )
+        for item in items:
+            with item["path"].open("rb") as src:
+                overlay.common.copy_payload(src, dst, 8 + item["header_len"], item["meta"])
+    os.replace(partial, destination)
+    return {"name": DENSE_SHARD, "header_len": header_len, "entries": entries}
+
+
+def _link_base_files(experts: Path, out: Path, rewrite_files: set[str]) -> None:
+    excluded = rewrite_files | {CONFIG_NAME, INDEX_NAME, DENSE_SHARD, DENSE_SHARD + ".partial"}
+    for entry in sorted(experts.iterdir(), key=lambda p: p.name):
+        if entry.name in excluded or not (entry.is_file() or entry.is_symlink()):
+            continue
+        destination = out / entry.name
+        if destination.exists() or destination.is_symlink():
+            if destination.is_symlink() and destination.resolve() == entry.resolve():
+                continue
+            raise RuntimeError(f"output path already exists: {destination}")
+        destination.symlink_to(entry.resolve())
+
+
+def plan_merge(work: Path, experts: Path, args, *, require_complete: bool = True) -> dict[str, Any]:
+    work_items = _read_work_tensors(work)
+    source, source_map = overlay.source_headers(experts)
+    overlay.validate_source_index(experts, source_map)
+    dense_bases = sorted({item["base"] for item in work_items})
+    expected_bases = set()
+    for name in source:
+        if not name.endswith(".weight"):
+            continue
+        base = name[:-len(".weight")]
+        if base.startswith("layers.") and base.endswith(".attn.wo_a"):
+            layer = _layer_number(base)
+            expected_bases.update(f"layers.{layer}.attn.wo_a.slice.{i}" for i in range(8))
+        elif is_dense_key(base):
+            expected_bases.add(base)
+    missing_work = sorted(expected_bases - set(dense_bases))
+    extra_work = sorted(set(dense_bases) - expected_bases)
+    if (missing_work or extra_work) and require_complete:
+        raise RuntimeError(
+            "work directory is not a complete dense bake: "
+            f"missing={missing_work[:3]} extra={extra_work[:3]}"
+        )
+    if missing_work or extra_work:
+        log(
+            "WARNING dry-run merge is partial: "
+            f"missing={missing_work[:3]} extra={extra_work[:3]}"
+        )
+    dropped = set()  # LNA-LAB: the 8 wo_a slices replace the single source wo_a.weight
+    for base in dense_bases:
+        m = re.match(r"^(layers\.[0-9]+\.attn\.wo_a)\.slice\.[0-7]$", base)
+        dropped.add(f"{m.group(1)}.weight" if m else f"{base}.weight")
+    missing = sorted(name for name in dropped if name not in source)
+    if missing:
+        raise RuntimeError(f"dense work tensor has no BF16/fp8 source weight to replace: {missing[0]}")
+    by_file: dict[str, set[str]] = {}
+    for name in dropped:
+        by_file.setdefault(source[name][0].name, set()).add(name)
+    rewrite_files = set(by_file)
+    new_plan = [{"target": item["target"], "meta": item["meta"]} for item in work_items]
+    base_config_path = experts / CONFIG_NAME
+    try:
+        base_config = json.loads(base_config_path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError) as exc:
+        raise RuntimeError(f"cannot read {base_config_path}: {exc}") from exc
+    config = _config_with_vllm_block(base_config, work_items, args)
+    index = overlay.output_index(source, dropped, new_plan)
+    # layer_overlay's generic planner uses its historical shard name; T4 has
+    # a distinct dense shard so it cannot be confused with a layer overlay.
+    new_targets = {item["target"] for item in new_plan}  # LNA-LAB: only the dense tensors go to the dense shard;
+    index["weight_map"] = {                                # the base pack may itself carry a layer_overlay shard of that name
+        name: (DENSE_SHARD if (filename == overlay.NEW_SHARD and name in new_targets) else filename)
+        for name, filename in index["weight_map"].items()
+    }
+    return {
+        "source": source,
+        "source_map": source_map,
+        "items": work_items,
+        "dropped": dropped,
+        "rewrite_files": rewrite_files,
+        "new_plan": new_plan,
+        "config": config,
+        "index": index,
+    }
+
+
+def _verify_merge(experts: Path, out: Path, plan: dict[str, Any]) -> None:
+    index_path = out / INDEX_NAME
+    if json.loads(index_path.read_text(encoding="utf-8")) != plan["index"]:
+        raise RuntimeError("merge index differs from plan")
+    output_headers = {}
+    for filename in sorted(set(plan["index"]["weight_map"].values())):
+        path = out / filename
+        if not path.is_file():
+            raise RuntimeError(f"indexed output shard missing: {path}")
+        header_len, header = overlay.read_header(path)
+        expected_names = {
+            name for name, mapped in plan["index"]["weight_map"].items() if mapped == filename
+        }
+        actual_names = {name for name in header if name != "__metadata__"}
+        if actual_names != expected_names:
+            raise RuntimeError(
+                f"output shard {filename} has unexpected tensor set: "
+                f"missing={sorted(expected_names - actual_names)[:2]} "
+                f"extra={sorted(actual_names - expected_names)[:2]}"
+            )
+        for name, mapped in plan["index"]["weight_map"].items():
+            if mapped == filename:
+                if name not in header:
+                    raise RuntimeError(f"indexed tensor missing from {filename}: {name}")
+                output_headers[name] = (path, header_len, header)
+    if any(name in plan["dropped"] for name in output_headers):
+        raise RuntimeError("dropped source tensor remains in merged output")
+    for filename in plan["rewrite_files"]:
+        path = out / filename
+        if path.is_symlink():
+            raise RuntimeError(f"rewritten shard is still a symlink: {filename}")
+        _, header = overlay.read_header(path)
+        if any(name in plan["dropped"] for name in header):
+            raise RuntimeError(f"dropped tensor remains in rewritten shard: {filename}")
+    for name, (source_path, source_header_len, source_header) in plan["source"].items():
+        if name in plan["dropped"] or source_path.name not in plan["rewrite_files"]:
+            continue
+        output_path, output_header_len, output_header = output_headers[name]
+        if source_header[name].get("dtype") != output_header[name].get("dtype") or \
+                source_header[name].get("shape") != output_header[name].get("shape"):
+            raise RuntimeError(f"retained tensor metadata changed: {name}")
+        overlay.compare_payloads(source_path, output_path, source_header[name], output_header[name],
+                                 source_header_len, output_header_len)
+    for item in plan["items"]:
+        path, header_len, header = output_headers[item["target"]]
+        meta = header[item["target"]]
+        if meta.get("dtype") != item["meta"].get("dtype") or meta.get("shape") != item["meta"].get("shape"):
+            raise RuntimeError(f"new tensor metadata changed: {item['target']}")
+        if item["target"].endswith(".mcg"):
+            start, end = meta["data_offsets"]
+            with path.open("rb") as stream:
+                stream.seek(8 + header_len + start)
+                if struct.unpack("<i", stream.read(end - start))[0] != MCG_MARKER:
+                    raise RuntimeError(f"MCG marker mismatch: {item['target']}")
+    if json.loads((out / CONFIG_NAME).read_text(encoding="utf-8")) != plan["config"]:
+        raise RuntimeError("merge config differs from plan")
+    for entry in experts.iterdir():
+        if entry.name in plan["rewrite_files"] or entry.name in {CONFIG_NAME, INDEX_NAME, DENSE_SHARD}:
+            continue
+        if not (entry.is_file() or entry.is_symlink()):
+            continue
+        linked = out / entry.name
+        if not linked.is_symlink() or linked.resolve() != entry.resolve():
+            raise RuntimeError(f"unchanged source file is not linked unchanged: {entry.name}")
+    log(f"MERGE_VERIFY_OK indexed_tensors={len(output_headers)} dropped_absent=true retained_payloads_byte_identical=true")
+
+
+def merge(args: argparse.Namespace) -> int:
+    work = Path(args.work).expanduser().resolve()
+    # LNA-LAB: the merged config must describe the bits that were actually baked.
+    # Unless --bits was given explicitly, take bits/attn_bits/shared_bits from work/args.json.
+    if not getattr(args, "_bits_explicit", False):
+        recorded = work / "args.json"
+        if recorded.is_file():
+            rec = json.loads(recorded.read_text())
+            args.bits = rec.get("bits", args.bits)
+            args.attn_bits = rec.get("attn_bits", args.attn_bits)
+            args.shared_bits = rec.get("shared_bits", args.shared_bits)
+            log(f"MERGE_BITS from {recorded}: bits={args.bits:g} attn_bits={args.attn_bits} shared_bits={args.shared_bits}")
+    experts = Path(args.experts).expanduser().resolve()
+    out = Path(args.merge).expanduser().resolve()
+    overlay.safe_output_path(experts, out)
+    plan = plan_merge(work, experts, args, require_complete=not args.dry_run)
+    log(f"MERGE_PLAN rewrite_files={len(plan['rewrite_files'])} dropped={len(plan['dropped'])} new={len(plan['new_plan'])}")
+    if args.dry_run:
+        log(f"DRY_RUN_MERGE output={out} symlinks=all_unchanged_source_files")
+        return 0
+    if out.exists() and any(out.iterdir()):
+        raise RuntimeError(f"refusing to merge into non-empty directory: {out}")
+    out.mkdir(parents=True, exist_ok=True)
+    _link_base_files(experts, out, plan["rewrite_files"])
+    for filename in sorted(plan["rewrite_files"]):
+        overlay.rewrite_source_shard(experts / filename, out / filename, plan["dropped"])
+    _write_dense_shard(out, plan["items"])
+    overlay.write_output_metadata(out, plan["index"], plan["config"])
+    _verify_merge(experts, out, plan)
+    return 0
+
+
+def make_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--src", required=True, help="bf16/fp8 DeepSeek-V4 source checkpoint")
+    parser.add_argument("--experts", required=True, help="existing routed-expert EXL3 pack")
+    parser.add_argument("--work", required=True, help="resumable qtensors/checkpoint directory")
+    parser.add_argument("--merge", help="write merged overlay pack to this directory")
+    parser.add_argument("--bits", type=float, default=4, help="dense default EXL3 K (default: 4)")
+    parser.add_argument("--attn-bits", type=float, default=None)
+    parser.add_argument("--shared-bits", type=float, default=None)
+    parser.add_argument("--cal-rows", type=int, default=250)
+    parser.add_argument("--cal-cols", type=int, default=2048)
+    parser.add_argument("--devices", default="0,1", help="CUDA device IDs, e.g. 0,1")
+    parser.add_argument("--resume", action="store_true", help="resume completed module checkpoints")
+    parser.add_argument("--dry-run", action="store_true", help="offline layout/merge planning; no writes")
+    parser.add_argument("--vllm-root", default="language_model.model", help="vLLM model root prefix (LNA-LAB: vision class roots the text model under language_model.model)")
+    parser.add_argument("--vllm-shared-prefix", default="ffn.shared_experts",
+                        help="vLLM shared-expert child prefix under model.layers.N")
+    parser.add_argument("--verbose", action="store_true")
+    return parser
+
+
+def main(argv=None) -> int:
+    args = make_parser().parse_args(argv)
+    args._bits_explicit = any(a == "--bits" or a.startswith("--bits=") for a in (argv if argv is not None else sys.argv[1:]))
+    try:
+        if args.merge:
+            # A merge is independent of CUDA; --dry-run additionally performs
+            # the requested source/model layout check before header planning.
+            if args.dry_run:
+                dry_run_layout(Path(args.src).expanduser().resolve(), Path(args.experts).expanduser().resolve())
+            return merge(args)
+        import torch  # LNA-LAB: the converter runs under inference_mode; loaded tensors are inference tensors
+        with torch.inference_mode():
+            return bake(args)
+    except (RuntimeError, ValueError, OSError) as exc:
+        log(f"ERROR {exc}")
+        return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

lna-lab/tools/layer_overlay.py

@@ -0,0 +1,957 @@
+#!/usr/bin/env python3
+"""Build a selected-main-layer K2 EXL3 overlay.
+
+The source pack is never changed.  Unchanged source files are linked into the
+output, source shards containing the selected K3 routed experts are rewritten
+tensor-by-tensor, and the replacement K2 tensors are written to one new shard.
+Donor tensors are fetched with HTTP Range requests one tensor at a time; a
+local donor-shard directory can be supplied through ``LAYER_OVERLAY_LOCAL_DIR``.
+
+``--dry-run`` is deliberately offline: it scans the local source headers and
+donor index and plans the replacement using a K2 ABI exemplar already in the
+source pack.  A donor header cache, when supplied, adds header-level shape and
+dtype validation without network access.
+"""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+import os
+import re
+import struct
+import sys
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+from typing import BinaryIO, Iterable
+
+try:
+    from . import mtp_overlay as common
+except ImportError:  # Running as ``python tools/layer_overlay.py``.
+    import mtp_overlay as common
+
+
+CONFIG_NAME = "config.json"
+INDEX_NAME = "model.safetensors.index.json"
+NEW_SHARD = "model-layers-k2.safetensors"
+DEFAULT_DONOR_INDEX = "/run/media/tonoken3/DATA1/.tmp/k2v1-index.json"
+DEFAULT_HEADER_CACHE = "/run/media/tonoken3/DATA1/.tmp/layer-overlay-donor-headers.json"
+CHUNK_SIZE = common.CHUNK_SIZE
+RETRY_COUNT = common.RETRY_COUNT
+
+DTYPE_BYTES = common.DTYPE_BYTES
+EXL3_SUFFIXES = ("trellis", "suh", "svh", "mcg")
+EXL3_DTYPE = {"trellis": "I16", "suh": "F16", "svh": "F16", "mcg": "I32"}
+PROJECTION_MAP = {"gate_proj": "w1", "up_proj": "w3", "down_proj": "w2"}
+PROJECTION_ORDER = {"gate_proj": 0, "up_proj": 1, "down_proj": 2}
+
+SOURCE_EXPERT_RE = re.compile(
+    r"^layers\.(?P<layer>[0-9]+)\.ffn\.experts\."
+    r"(?P<expert>[0-9]+)\.(?P<projection>w[123])\."
+    r"(?P<suffix>trellis|suh|svh|mcg)$"
+)
+DONOR_EXPERT_RE = re.compile(
+    r"^model\.layers\.(?P<layer>[0-9]+)\.mlp\.experts\."
+    r"(?P<expert>[0-9]+)\.(?P<projection>gate_proj|up_proj|down_proj)\."
+    r"(?P<suffix>trellis|suh|svh|mcg)$"
+)
+
+# These helpers are intentionally re-exported for small offline tests and for
+# callers that used the corresponding mtp_overlay helpers.
+parse_header = common.parse_header
+validate_header = common.validate_header
+read_header = common.read_header
+tensor_nbytes = common.tensor_nbytes
+
+
+def log(message: str) -> None:
+    print(message, flush=True)
+
+
+def parse_layers(value: str) -> tuple[int, ...]:
+    try:
+        values = [int(part.strip()) for part in value.split(",") if part.strip()]
+    except ValueError as exc:
+        raise ValueError(f"invalid --layers value {value!r}") from exc
+    if not values or any(layer < 0 for layer in values):
+        raise ValueError(f"--layers must contain non-negative layer numbers: {value!r}")
+    if len(set(values)) != len(values):
+        raise ValueError(f"--layers contains duplicates: {value!r}")
+    return tuple(sorted(values))
+
+
+def source_layer(name: str) -> int | None:
+    match = SOURCE_EXPERT_RE.fullmatch(name)
+    return None if match is None else int(match.group("layer"))
+
+
+def donor_layer(name: str) -> int | None:
+    match = DONOR_EXPERT_RE.fullmatch(name)
+    return None if match is None else int(match.group("layer"))
+
+
+def map_donor_name(name: str) -> str:
+    """Map one donor main-layer expert name to the target pack ABI."""
+    match = DONOR_EXPERT_RE.fullmatch(name)
+    if match is None:
+        raise ValueError(f"not a donor main-layer EXL3 expert tensor: {name}")
+    return (
+        f"layers.{match.group('layer')}.ffn.experts.{int(match.group('expert'))}."
+        f"{PROJECTION_MAP[match.group('projection')]}.{match.group('suffix')}"
+    )
+
+
+# Short alias used by offline tests and useful to importers.
+map_name = map_donor_name
+
+
+def is_donor_expert_name(name: str) -> bool:
+    return DONOR_EXPERT_RE.fullmatch(name) is not None
+
+
+def is_source_expert_name(name: str) -> bool:
+    return SOURCE_EXPERT_RE.fullmatch(name) is not None
+
+
+def donor_base_url(donor: str) -> str:
+    return donor.rstrip("/") if donor.startswith("http") else f"https://huggingface.co/{donor}/resolve/main"
+
+
+def local_or_remote_bytes(url: str, byte_range: tuple[int, int] | None = None) -> bytes:
+    """Read a range from a local donor shard or a remote URL.
+
+    The real build calls this only for header ranges and selected tensor
+    ranges.  It never requests a complete donor shard.
+    """
+    local_dir = os.environ.get("LAYER_OVERLAY_LOCAL_DIR")
+    filename = url.split("?", 1)[0].rsplit("/", 1)[-1]
+    if local_dir:
+        local_path = Path(local_dir).expanduser() / filename
+        if local_path.is_file():
+            with local_path.open("rb") as stream:
+                if byte_range is None:
+                    return stream.read()
+                stream.seek(byte_range[0])
+                data = stream.read(byte_range[1] - byte_range[0] + 1)
+            expected = byte_range[1] - byte_range[0] + 1
+            if len(data) != expected:
+                raise RuntimeError(
+                    f"short local range read {local_path}: got {len(data)}, expected {expected}"
+                )
+            return data
+
+    for attempt in range(RETRY_COUNT):
+        try:
+            request = urllib.request.Request(url, headers={"User-Agent": "layer_overlay/1"})
+            if byte_range is not None:
+                request.add_header("Range", f"bytes={byte_range[0]}-{byte_range[1]}")
+            with urllib.request.urlopen(request, timeout=60) as response:
+                status = getattr(response, "status", response.getcode())
+                if byte_range is not None and status != 206:
+                    raise RuntimeError(f"expected HTTP 206 for Range request, got {status}")
+                data = response.read()
+            expected = None if byte_range is None else byte_range[1] - byte_range[0] + 1
+            if expected is not None and len(data) != expected:
+                raise RuntimeError(f"short range read: got {len(data)}, expected {expected}")
+            return data
+        except (OSError, TimeoutError, urllib.error.URLError, urllib.error.HTTPError, RuntimeError) as exc:
+            if attempt == RETRY_COUNT - 1:
+                raise RuntimeError(f"failed fetching {url} range={byte_range}: {exc}") from exc
+            delay = 2**attempt
+            log(f"RETRY attempt={attempt + 1}/{RETRY_COUNT} range={byte_range}: {exc}")
+            time.sleep(delay)
+    raise AssertionError("unreachable")
+
+
+def donor_index_from_file(path: Path) -> dict[str, str]:
+    try:
+        data = json.loads(path.read_text())
+    except (OSError, json.JSONDecodeError) as exc:
+        raise RuntimeError(f"cannot read donor index {path}: {exc}") from exc
+    weight_map = data.get("weight_map")
+    if not isinstance(weight_map, dict) or not all(
+        isinstance(name, str) and isinstance(filename, str)
+        for name, filename in weight_map.items()
+    ):
+        raise RuntimeError(f"donor index {path} has no valid weight_map")
+    return weight_map
+
+
+def source_headers(src: Path) -> tuple[dict[str, tuple[Path, int, dict]], dict[str, str]]:
+    """Scan every local safetensors file, including an existing overlay shard."""
+    candidates = sorted(
+        (path for path in src.iterdir() if path.name.endswith(".safetensors") and path.is_file()),
+        key=lambda path: path.name,
+    )
+    if not candidates:
+        raise RuntimeError(f"no safetensors shards found under {src}")
+    headers: dict[str, tuple[Path, int, dict]] = {}
+    weight_map: dict[str, str] = {}
+    for path in candidates:
+        header_len, header = read_header(path)
+        for name in header:
+            if name == "__metadata__":
+                continue
+            if name in weight_map:
+                raise RuntimeError(f"duplicate source tensor {name!r}")
+            headers[name] = (path, header_len, header)
+            weight_map[name] = path.name
+    log(f"SOURCE_HEADERS files={len(candidates)} tensors={len(headers)}")
+    return headers, weight_map
+
+
+def validate_source_index(src: Path, source_map: dict[str, str]) -> None:
+    """Cross-check the numbered source headers against an existing index."""
+    path = src / INDEX_NAME
+    if not path.is_file():
+        log("SOURCE_INDEX absent=allowed; numbered headers are authoritative")
+        return
+    try:
+        data = json.loads(path.read_text())
+        index_map = data["weight_map"]
+    except (OSError, json.JSONDecodeError, KeyError, TypeError) as exc:
+        raise RuntimeError(f"invalid source index {path}: {exc}") from exc
+    if not isinstance(index_map, dict):
+        raise RuntimeError(f"source index {path} has no object weight_map")
+    if set(index_map) != set(source_map):
+        raise RuntimeError(
+            f"source index tensor set differs from local headers: "
+            f"missing={sorted(set(source_map) - set(index_map))[:5]} "
+            f"extra={sorted(set(index_map) - set(source_map))[:5]}"
+        )
+    for name, filename in source_map.items():
+        if index_map.get(name) != filename:
+            raise RuntimeError(f"source index disagrees with header for {name}: {index_map.get(name)!r} != {filename!r}")
+    log(f"SOURCE_INDEX checked=local_headers tensors={len(source_map)} indexed_total={len(index_map)}")
+
+
+def _abi_meta(meta: dict) -> dict:
+    return {"dtype": meta.get("dtype"), "shape": list(meta.get("shape", []))}
+
+
+def validate_abi(abi: dict[str, dict[str, dict]], label: str) -> None:
+    for projection in ("w1", "w2", "w3"):
+        if projection not in abi:
+            raise RuntimeError(f"{label}: missing projection {projection}")
+        for suffix in EXL3_SUFFIXES:
+            meta = abi[projection].get(suffix)
+            if not isinstance(meta, dict) or meta.get("dtype") != EXL3_DTYPE[suffix]:
+                raise RuntimeError(f"{label}: unexpected {projection}.{suffix} metadata: {meta}")
+            shape = meta.get("shape")
+            if not isinstance(shape, list) or not all(isinstance(dim, int) and dim >= 0 for dim in shape):
+                raise RuntimeError(f"{label}: invalid {projection}.{suffix} shape: {shape}")
+        if abi[projection]["trellis"]["shape"][-1:] != [32]:
+            raise RuntimeError(f"{label}: {projection}.trellis is not K2: {abi[projection]['trellis']}")
+
+
+def find_source_k2_abi(
+    source: dict[str, tuple[Path, int, dict]],
+    selected_layers: Iterable[int],
+) -> tuple[dict[str, dict[str, dict]], int]:
+    """Use one unselected local main layer as the expected K2 ABI."""
+    selected = set(selected_layers)
+    available = sorted({layer for name in source if (layer := source_layer(name)) is not None})
+    for layer in available:
+        if layer in selected:
+            continue
+        abi: dict[str, dict[str, dict]] = {}
+        complete = True
+        for projection in ("w1", "w2", "w3"):
+            abi[projection] = {}
+            for suffix in EXL3_SUFFIXES:
+                name = f"layers.{layer}.ffn.experts.0.{projection}.{suffix}"
+                if name not in source:
+                    complete = False
+                    break
+                abi[projection][suffix] = _abi_meta(source[name][2][name])
+            if not complete:
+                break
+        if complete:
+            try:
+                validate_abi(abi, f"source layer {layer} K2 exemplar")
+            except RuntimeError:
+                continue
+            log(f"MAIN_K2_ABI exemplar_layer={layer} validated=true")
+            return abi, layer
+    raise RuntimeError("could not find an unselected K2 main-layer ABI exemplar in the source pack")
+
+
+def expected_names(layers: Iterable[int], *, donor: bool) -> set[str]:
+    names = set()
+    for layer in layers:
+        for expert in range(256):
+            for projection in ("gate_proj", "up_proj", "down_proj") if donor else ("w1", "w2", "w3"):
+                for suffix in EXL3_SUFFIXES:
+                    if donor:
+                        names.add(f"model.layers.{layer}.mlp.experts.{expert}.{projection}.{suffix}")
+                    else:
+                        names.add(f"layers.{layer}.ffn.experts.{expert}.{projection}.{suffix}")
+    return names
+
+
+def validate_source_selection(
+    source: dict[str, tuple[Path, int, dict]],
+    selected_layers: tuple[int, ...],
+    k2_abi: dict[str, dict[str, dict]],
+) -> tuple[set[str], dict[str, list[tuple[str, dict]]], int]:
+    """Validate selected source layers are complete K3 sets and plan drops."""
+    selected = set(selected_layers)
+    expected = expected_names(selected_layers, donor=False)
+    actual = {name for name in source if source_layer(name) in selected}
+    if actual != expected:
+        raise RuntimeError(
+            f"source selected expert set mismatch: actual={len(actual)} "
+            f"expected={len(expected)} missing={sorted(expected - actual)[:5]} "
+            f"extra={sorted(actual - expected)[:5]}"
+        )
+    for name in source:
+        raw_layer = re.match(r"^layers\.(?P<layer>[0-9]+)\.ffn\.experts\.", name)
+        if (
+            raw_layer is not None
+            and int(raw_layer.group("layer")) in selected
+            and not is_source_expert_name(name)
+        ):
+            raise RuntimeError(f"malformed selected source expert name: {name}")
+
+    dropped: set[str] = set()
+    by_file: dict[str, list[tuple[str, dict]]] = {}
+    for name in sorted(actual):
+        path, _, header = source[name]
+        meta = header[name]
+        match = SOURCE_EXPERT_RE.fullmatch(name)
+        assert match is not None
+        expert = int(match.group("expert"))
+        projection = match.group("projection")
+        suffix = match.group("suffix")
+        if expert >= 256:
+            raise RuntimeError(f"selected source expert id out of range: {name}")
+        if meta.get("dtype") != EXL3_DTYPE[suffix]:
+            raise RuntimeError(f"selected source dtype mismatch for {name}: {meta}")
+        shape = meta.get("shape")
+        expected_shape = list(k2_abi[projection][suffix]["shape"])
+        if suffix == "trellis":
+            if not isinstance(shape, list) or shape[:-1] != expected_shape[:-1] or shape[-1:] != [48]:
+                raise RuntimeError(f"selected source is not K3 for {name}: {meta}")
+        elif shape != expected_shape:
+            raise RuntimeError(f"selected source ABI mismatch for {name}: {meta}, expected {k2_abi[projection][suffix]}")
+        dropped.add(name)
+        by_file.setdefault(path.name, []).append((name, meta))
+
+    expected_per_layer = 256 * 3 * 4
+    for layer in selected_layers:
+        count = sum(1 for name in actual if source_layer(name) == layer)
+        if count != expected_per_layer:
+            raise RuntimeError(f"source layer {layer}: found {count} expert tensors, expected {expected_per_layer}")
+    for filename, items in sorted(by_file.items()):
+        # A shard may contain one or more selected layers; every complete layer
+        # contributes exactly 256 * 3 * 4 expert tensors.
+        layers_here = {source_layer(name) for name, _ in items}
+        expected_count = expected_per_layer * len(layers_here)
+        if len(items) != expected_count:
+            raise RuntimeError(f"{filename}: found {len(items)} selected tensors, expected {expected_count}")
+    drop_bytes = sum(tensor_nbytes(meta) for items in by_file.values() for _, meta in items)
+    log(
+        f"SOURCE_DROP tensors={len(dropped)} bytes={drop_bytes} "
+        f"files={sorted(by_file)} layers={list(selected_layers)}"
+    )
+    return dropped, by_file, drop_bytes
+
+
+def validate_donor_selection(weight_map: dict[str, str], layers: tuple[int, ...]) -> list[str]:
+    selected = set(layers)
+    candidates = [
+        name for name in weight_map
+        if name.startswith("model.layers.") and ".mlp.experts." in name
+    ]
+    malformed = sorted(name for name in candidates if not is_donor_expert_name(name))
+    if malformed:
+        raise RuntimeError(f"malformed donor main expert names (first 5): {malformed[:5]}")
+    selected_names = [name for name in candidates if donor_layer(name) in selected]
+    expected = expected_names(layers, donor=True)
+    actual = set(selected_names)
+    if actual != expected:
+        raise RuntimeError(
+            f"donor selected expert set mismatch: actual={len(actual)} expected={len(expected)} "
+            f"missing={sorted(expected - actual)[:5]} extra={sorted(actual - expected)[:5]}"
+        )
+    selected_names.sort(
+        key=lambda name: (
+            int(name.split(".")[2]),
+            int(name.split(".")[5]),
+            PROJECTION_ORDER[name.split(".")[6]],
+            EXL3_SUFFIXES.index(name.split(".")[7]),
+        )
+    )
+    log(
+        f"DONOR_INDEX selected_tensors={len(selected_names)} layers={list(layers)} "
+        f"files={sorted({weight_map[name] for name in selected_names})}"
+    )
+    return selected_names
+
+
+def donor_headers_from_remote(
+    donor: str,
+    weight_map: dict[str, str],
+    selected_names: Iterable[str],
+    cache_path: Path | None,
+    *,
+    write_cache: bool = True,
+    allow_network: bool = True,
+) -> dict[str, tuple[int, dict]]:
+    """Read one header per required donor shard, locally or with ranges."""
+    base = donor_base_url(donor)
+    files = sorted({weight_map[name] for name in selected_names})
+    cached: dict = {}
+    if cache_path is not None and cache_path.exists():
+        try:
+            cached = json.loads(cache_path.read_text()).get("headers", {})
+        except (OSError, json.JSONDecodeError) as exc:
+            raise RuntimeError(f"cannot read donor header cache {cache_path}: {exc}") from exc
+
+    result: dict[str, tuple[int, dict]] = {}
+    for filename in files:
+        item = cached.get(filename)
+        if isinstance(item, list) and len(item) == 2:
+            header_len, header = int(item[0]), item[1]
+            validate_header(header)
+            result[filename] = (header_len, header)
+            log(f"DONOR_HEADER cache file={filename} tensors={len(header) - 1}")
+            continue
+        if not allow_network:
+            raise RuntimeError(f"donor header cache has no entry for {filename}")
+        url = f"{base}/{filename}"
+        length_blob = local_or_remote_bytes(url, (0, 7))
+        if len(length_blob) != 8:
+            raise RuntimeError(f"donor header length read for {filename} was not 8 bytes")
+        header_len = struct.unpack("<Q", length_blob)[0]
+        header_blob = local_or_remote_bytes(url, (8, 8 + header_len - 1))
+        parsed_len, header = parse_header(length_blob + header_blob)
+        validate_header(header)
+        if parsed_len != header_len:
+            raise RuntimeError(f"donor header length changed for {filename}")
+        result[filename] = (header_len, header)
+        log(f"DONOR_HEADER remote_or_local file={filename} tensors={len(header) - 1}")
+
+    if cache_path is not None and write_cache:
+        cache_path.parent.mkdir(parents=True, exist_ok=True)
+        cache_path.write_text(
+            json.dumps({"headers": {name: [hlen, header] for name, (hlen, header) in result.items()}}, indent=2)
+        )
+        log(f"DONOR_HEADER_CACHE wrote={cache_path}")
+    return result
+
+
+def expected_output_meta(
+    donor_name: str,
+    k2_abi: dict[str, dict[str, dict]],
+) -> dict:
+    match = DONOR_EXPERT_RE.fullmatch(donor_name)
+    assert match is not None
+    projection = PROJECTION_MAP[match.group("projection")]
+    suffix = match.group("suffix")
+    if suffix == "mcg":
+        # Donor markers are scalar [], while the destination parameter is [1].
+        return {"dtype": "I32", "shape": [1]}
+    return copy.deepcopy(k2_abi[projection][suffix])
+
+
+def validate_and_plan_donor(
+    selected: list[str],
+    weight_map: dict[str, str],
+    donor_headers: dict[str, tuple[int, dict]] | None,
+    k2_abi: dict[str, dict[str, dict]],
+) -> list[dict]:
+    plan: list[dict] = []
+    seen_targets: set[str] = set()
+    for donor_name in selected:
+        target_name = map_donor_name(donor_name)
+        if target_name in seen_targets:
+            raise RuntimeError(f"donor name map collision at {target_name}")
+        seen_targets.add(target_name)
+        expected = expected_output_meta(donor_name, k2_abi)
+        source_file = weight_map[donor_name]
+        actual = copy.deepcopy(expected)
+        donor_meta = None
+        if donor_headers is not None:
+            if source_file not in donor_headers:
+                raise RuntimeError(f"missing donor header for {source_file}")
+            _, header = donor_headers[source_file]
+            if donor_name not in header:
+                raise RuntimeError(f"donor header lacks indexed tensor {donor_name}")
+            donor_meta = copy.deepcopy(header[donor_name])
+            validate_header({donor_name: donor_meta})
+            actual = _abi_meta(donor_meta)
+            if donor_name.endswith(".mcg") and actual.get("shape") == []:
+                actual["shape"] = [1]
+            if actual != expected:
+                raise RuntimeError(
+                    f"donor shape/dtype mismatch for {donor_name}: got {actual}, expected {expected}"
+                )
+        plan.append(
+            {
+                "donor": donor_name,
+                "target": target_name,
+                "file": source_file,
+                "meta": actual,
+                "expected": expected,
+                "donor_meta": donor_meta,
+            }
+        )
+    expected_count = len(selected)
+    if len(plan) != expected_count:
+        raise RuntimeError(f"planned donor tensor count is {len(plan)}, expected {expected_count}")
+    return plan
+
+
+def tensor_range(header_len: int, meta: dict) -> tuple[int, int]:
+    start, end = meta["data_offsets"]
+    return 8 + header_len + start, 8 + header_len + end - 1
+
+
+def build_new_header(plan: list[dict], donor: str, layers: tuple[int, ...]) -> tuple[list[tuple[str, dict]], int, dict]:
+    entries = [(item["target"], item["meta"]) for item in plan]
+    offset = 0
+    header: dict = {
+        "__metadata__": {
+            "format": "pt",
+            "source": donor,
+            "overlay": "main-layer-k2",
+            "layers": ",".join(str(layer) for layer in layers),
+        }
+    }
+    for name, meta in entries:
+        size = tensor_nbytes(meta)
+        header[name] = {
+            "dtype": meta["dtype"],
+            "shape": list(meta["shape"]),
+            "data_offsets": [offset, offset + size],
+        }
+        offset += size
+    return entries, offset, header
+
+
+def header_without_offsets(header: dict) -> dict:
+    return {
+        name: {"dtype": meta["dtype"], "shape": list(meta["shape"])}
+        for name, meta in header.items()
+        if name != "__metadata__"
+    }
+
+
+def write_new_shard(
+    out: Path,
+    plan: list[dict],
+    donor: str,
+    layers: tuple[int, ...],
+    donor_headers: dict[str, tuple[int, dict]],
+) -> int:
+    entries, payload_bytes, expected_header = build_new_header(plan, donor, layers)
+    destination = out / NEW_SHARD
+    partial = out / (NEW_SHARD + ".partial")
+    header_len = None
+    actual_header = None
+    resume_at = 0
+
+    if destination.exists():
+        existing_len, existing_header = read_header(destination)
+        if header_without_offsets(existing_header) != header_without_offsets(expected_header):
+            raise RuntimeError(f"existing {destination} has a different tensor plan")
+        if destination.stat().st_size != 8 + existing_len + payload_bytes:
+            raise RuntimeError(f"existing {destination} is incomplete or has trailing bytes")
+        log(f"NEW_SHARD existing_complete={destination}")
+        return payload_bytes
+
+    if partial.exists():
+        existing_len, existing_header = read_header(partial, check_data=False)
+        if header_without_offsets(existing_header) == header_without_offsets(expected_header):
+            data_start = 8 + existing_len
+            data_bytes = partial.stat().st_size - data_start
+            boundaries = {0} | {
+                meta["data_offsets"][1]
+                for name, meta in existing_header.items()
+                if name != "__metadata__"
+            }
+            if data_bytes in boundaries and 0 <= data_bytes <= payload_bytes:
+                header_len, actual_header, resume_at = existing_len, existing_header, data_bytes
+                log(f"NEW_SHARD resume={partial} completed_bytes={resume_at}")
+            else:
+                log(f"NEW_SHARD discard_invalid_partial={partial}")
+                partial.unlink()
+        else:
+            log(f"NEW_SHARD discard_stale_partial={partial}")
+            partial.unlink()
+
+    if header_len is None:
+        with partial.open("wb") as stream:
+            header_len, actual_header = common.write_safetensors_header(
+                stream,
+                entries,
+                {"format": "pt", "source": donor, "overlay": "main-layer-k2", "layers": ",".join(map(str, layers))},
+            )
+        resume_at = 0
+
+    assert header_len is not None and actual_header is not None
+    with partial.open("r+b") as stream:
+        data_start = 8 + header_len
+        stream.truncate(data_start + resume_at)
+        stream.seek(data_start + resume_at)
+        for index, item in enumerate(plan):
+            name = item["target"]
+            meta = actual_header[name]
+            start, end = meta["data_offsets"]
+            if end <= resume_at:
+                log(f"FETCH skip_existing tensor={index + 1}/{len(plan)} name={name}")
+                continue
+            if start != resume_at:
+                raise RuntimeError(f"partial new shard boundary mismatch before {name}")
+            donor_header_len, donor_header = donor_headers[item["file"]]
+            donor_meta = donor_header[item["donor"]]
+            absolute = tensor_range(donor_header_len, donor_meta)
+            log(f"FETCH tensor={index + 1}/{len(plan)} name={name} bytes={end - start}")
+            data = local_or_remote_bytes(f"{donor_base_url(donor)}/{item['file']}", absolute)
+            if len(data) != end - start:
+                raise RuntimeError(f"{name}: fetched {len(data)} bytes, expected {end - start}")
+            stream.write(data)
+            resume_at = end
+    if resume_at != payload_bytes:
+        raise RuntimeError(f"new shard ended at {resume_at} bytes, expected {payload_bytes}")
+    os.replace(partial, destination)
+    log(f"NEW_SHARD wrote={destination} payload_bytes={payload_bytes}")
+    return payload_bytes
+
+
+def rewrite_source_shard(source_path: Path, destination_path: Path, dropped: set[str]) -> int:
+    """Copy one source shard while streaming every retained tensor payload."""
+    header_len, header = read_header(source_path)
+    entries = [
+        (name, meta)
+        for name, meta in header.items()
+        if name != "__metadata__" and name not in dropped
+    ]
+    copied = sum(tensor_nbytes(meta) for _, meta in entries)
+    partial = destination_path.with_name(destination_path.name + ".partial")
+    with source_path.open("rb") as source_stream, partial.open("wb") as destination_stream:
+        common.write_safetensors_header(destination_stream, entries, header.get("__metadata__"))
+        source_data_start = 8 + header_len
+        for name, meta in entries:
+            common.copy_payload(source_stream, destination_stream, source_data_start, meta)
+    os.replace(partial, destination_path)
+    log(
+        f"REWRITE file={source_path.name} kept_tensors={len(entries)} copied_bytes={copied} "
+        f"dropped_tensors={sum(name in dropped for name in header if name != '__metadata__')}"
+    )
+    return copied
+
+
+def safe_output_path(src: Path, out: Path) -> None:
+    src_real = src.resolve()
+    out_real = out.resolve(strict=False)
+    if out_real == src_real or src_real in out_real.parents:
+        raise RuntimeError(f"refusing output inside source pack: {out}")
+
+
+def link_source_files(src: Path, out: Path, rewrite_files: set[str]) -> int:
+    excluded = rewrite_files | {CONFIG_NAME, INDEX_NAME, NEW_SHARD}
+    linked = 0
+    for entry in sorted(src.iterdir(), key=lambda path: path.name):
+        if entry.name in excluded or not (entry.is_file() or entry.is_symlink()):
+            continue
+        destination = out / entry.name
+        if os.path.lexists(destination):
+            if destination.is_symlink() and destination.resolve() == entry.resolve():
+                continue
+            raise RuntimeError(f"output path already exists and is not the source link: {destination}")
+        # Resolve source symlinks so the overlay remains usable independently
+        # of a symlink chain in the source overlay.
+        destination.symlink_to(entry.resolve())
+        linked += 1
+    log(f"LINKED source_files={linked}")
+    return linked
+
+
+def output_index(
+    source: dict[str, tuple[Path, int, dict]],
+    dropped: set[str],
+    new_plan: list[dict],
+) -> dict:
+    weight_map: dict[str, str] = {}
+    total_size = 0
+    for name, (path, _, header) in sorted(source.items()):
+        if name in dropped:
+            continue
+        weight_map[name] = path.name
+        total_size += tensor_nbytes(header[name])
+    for item in new_plan:
+        if item["target"] in weight_map:
+            raise RuntimeError(f"new tensor collides with retained source tensor: {item['target']}")
+        weight_map[item["target"]] = NEW_SHARD
+        total_size += tensor_nbytes(item["meta"])
+    return {"metadata": {"total_size": total_size}, "weight_map": dict(sorted(weight_map.items()))}
+
+
+def edited_config(src: Path, layers: tuple[int, ...]) -> dict:
+    config_path = src / CONFIG_NAME
+    try:
+        config = json.loads(config_path.read_text())
+    except (OSError, json.JSONDecodeError) as exc:
+        raise RuntimeError(f"cannot read {config_path}: {exc}") from exc
+    q = config.get("quantization_config")
+    if not isinstance(q, dict):
+        raise RuntimeError("source config has no quantization_config object")
+    if q.get("bits") != 2:
+        raise RuntimeError(f"source base quantization bits must be 2, got {q.get('bits')!r}")
+    layer_bits = q.get("layer_bits")
+    if not isinstance(layer_bits, dict):
+        raise RuntimeError("source quantization_config.layer_bits is not an object")
+    layer_bits = dict(layer_bits)
+    for layer in layers:
+        key = str(layer)
+        if layer_bits.get(key) != 3:
+            raise RuntimeError(f"source config layer_bits[{key!r}] must be 3 before replacement, got {layer_bits.get(key)!r}")
+        layer_bits.pop(key)
+    q["layer_bits"] = layer_bits
+    return config
+
+
+def write_output_metadata(out: Path, index: dict, config: dict) -> None:
+    (out / INDEX_NAME).write_text(json.dumps(index, indent=2) + "\n")
+    (out / CONFIG_NAME).write_text(json.dumps(config, indent=2) + "\n")
+    log(f"WROTE metadata files={INDEX_NAME},{CONFIG_NAME}")
+
+
+def compare_payloads(
+    source_path: Path,
+    output_path: Path,
+    source_meta: dict,
+    output_meta: dict,
+    source_header_len: int,
+    output_header_len: int,
+) -> None:
+    source_start, source_end = source_meta["data_offsets"]
+    output_start, output_end = output_meta["data_offsets"]
+    if source_end - source_start != output_end - output_start:
+        raise RuntimeError("retained tensor payload length changed")
+    with source_path.open("rb") as source_stream, output_path.open("rb") as output_stream:
+        source_stream.seek(8 + source_header_len + source_start)
+        output_stream.seek(8 + output_header_len + output_start)
+        remaining = source_end - source_start
+        while remaining:
+            size = min(CHUNK_SIZE, remaining)
+            source_data = source_stream.read(size)
+            output_data = output_stream.read(size)
+            if len(source_data) != size or source_data != output_data:
+                raise RuntimeError(f"retained tensor payload differs: {source_path.name}")
+            remaining -= size
+
+
+def verify_output(
+    src: Path,
+    out: Path,
+    source: dict[str, tuple[Path, int, dict]],
+    dropped: set[str],
+    plan: list[dict],
+    expected_index: dict,
+    expected_config: dict,
+    rewrite_files: set[str],
+) -> None:
+    index_path = out / INDEX_NAME
+    config_path = out / CONFIG_NAME
+    if not index_path.is_file() or not config_path.is_file():
+        raise RuntimeError("overlay index/config is missing")
+    actual_index = json.loads(index_path.read_text())
+    if actual_index != expected_index:
+        raise RuntimeError("overlay index differs from the planned index")
+
+    output_headers: dict[str, tuple[Path, int, dict]] = {}
+    for filename in sorted(set(actual_index["weight_map"].values())):
+        path = out / filename
+        if not path.is_file():
+            raise RuntimeError(f"indexed shard is missing: {path}")
+        header_len, header = read_header(path)
+        for name, mapped_file in actual_index["weight_map"].items():
+            if mapped_file == filename:
+                if name not in header:
+                    raise RuntimeError(f"indexed tensor is missing from {filename}: {name}")
+                output_headers[name] = (path, header_len, header)
+    if set(output_headers) != set(actual_index["weight_map"]):
+        raise RuntimeError("not every indexed tensor was opened")
+
+    for entry in src.iterdir():
+        if not (entry.is_file() or entry.is_symlink()):
+            continue
+        if entry.name in rewrite_files or entry.name in (CONFIG_NAME, INDEX_NAME, NEW_SHARD):
+            continue
+        linked = out / entry.name
+        if not linked.is_symlink() or linked.resolve() != entry.resolve():
+            raise RuntimeError(f"source file is not linked unchanged: {entry.name}")
+    for filename in rewrite_files:
+        if (out / filename).is_symlink():
+            raise RuntimeError(f"rewritten source shard is still a symlink: {filename}")
+        _, rewritten_header = read_header(out / filename)
+        leftover = [n for n in rewritten_header if n != "__metadata__" and n in dropped]
+        if leftover:
+            raise RuntimeError(f"rewritten shard {filename} still carries {len(leftover)} dropped tensors, e.g. {leftover[0]}")
+
+    for name, (source_path, source_header_len, source_header) in source.items():
+        if name in dropped:
+            # Replaced in place: the same name must now point at the new K2 shard.
+            where = actual_index["weight_map"].get(name)
+            if where is not None and where != NEW_SHARD:
+                raise RuntimeError(f"dropped source tensor remains indexed in {where}: {name}")
+            continue
+        if source_path.name not in rewrite_files:
+            continue
+        output_path, output_header_len, output_header = output_headers[name]
+        if output_header[name]["dtype"] != source_header[name]["dtype"] or output_header[name]["shape"] != source_header[name]["shape"]:
+            raise RuntimeError(f"retained tensor metadata changed: {name}")
+        compare_payloads(
+            source_path,
+            output_path,
+            source_header[name],
+            output_header[name],
+            source_header_len,
+            output_header_len,
+        )
+
+    for item in plan:
+        path, header_len, header = output_headers[item["target"]]
+        actual = header[item["target"]]
+        if actual["dtype"] != item["meta"]["dtype"] or actual["shape"] != item["meta"]["shape"]:
+            raise RuntimeError(f"new tensor metadata differs from donor plan: {item['target']}")
+        if item["target"].endswith(".mcg"):
+            with path.open("rb") as stream:
+                start, end = actual["data_offsets"]
+                stream.seek(8 + header_len + start)
+                payload = stream.read(end - start)
+            if len(payload) != 4 or struct.unpack("<i", payload)[0] != -877912083:
+                raise RuntimeError(f"MCG marker mismatch for {item['target']}")
+
+    actual_config = json.loads(config_path.read_text())
+    if actual_config != expected_config:
+        raise RuntimeError("overlay config differs from the planned config")
+    log(f"VERIFY_OK indexed_tensors={len(output_headers)} retained_payloads_byte_identical=true")
+
+
+def build(args: argparse.Namespace) -> int:
+    src = Path(args.src).expanduser().resolve()
+    out = Path(args.out).expanduser()
+    donor = args.donor
+    layers = parse_layers(args.layers)
+    if not src.is_dir():
+        raise RuntimeError(f"source directory does not exist: {src}")
+    safe_output_path(src, out)
+
+    source, source_map = source_headers(src)
+    validate_source_index(src, source_map)
+    k2_abi, exemplar_layer = find_source_k2_abi(source, layers)
+    dropped, dropped_by_file, dropped_bytes = validate_source_selection(source, layers, k2_abi)
+    rewrite_files = set(dropped_by_file)
+    expected_config = edited_config(src, layers)
+    log(f"CONFIG_PLAN remove_layer_bits={[str(layer) for layer in layers]} exemplar_layer={exemplar_layer}")
+
+    donor_index_path = Path(args.donor_index).expanduser() if args.donor_index else None
+    if donor_index_path is not None and donor_index_path.exists():
+        donor_map = donor_index_from_file(donor_index_path)
+        log(f"DONOR_INDEX read={donor_index_path}")
+    elif args.dry_run:
+        raise RuntimeError(f"--dry-run requires the local donor index; not found: {donor_index_path}")
+    else:
+        index_blob = local_or_remote_bytes(f"{donor_base_url(donor)}/{INDEX_NAME}")
+        try:
+            donor_map = json.loads(index_blob)["weight_map"]
+        except (json.JSONDecodeError, KeyError, TypeError) as exc:
+            raise RuntimeError(f"remote donor index is invalid: {exc}") from exc
+        log(f"DONOR_INDEX remote tensors={len(donor_map)}")
+    selected = validate_donor_selection(donor_map, layers)
+
+    cache_path = Path(args.donor_headers).expanduser() if args.donor_headers else None
+    donor_headers: dict[str, tuple[int, dict]] | None = None
+    if args.dry_run:
+        if cache_path is not None and cache_path.exists():
+            donor_headers = donor_headers_from_remote(
+                donor,
+                donor_map,
+                selected,
+                cache_path,
+                write_cache=False,
+                allow_network=False,
+            )
+        else:
+            log("DONOR_HEADERS offline=not-read; dry-run uses local K2 ABI")
+    else:
+        donor_headers = donor_headers_from_remote(donor, donor_map, selected, cache_path)
+    plan = validate_and_plan_donor(selected, donor_map, donor_headers, k2_abi)
+
+    fetched_bytes = sum(tensor_nbytes(item["meta"]) for item in plan)
+    rewritten_bytes = sum(
+        tensor_nbytes(header[name])
+        for name, (path, _, header) in source.items()
+        if name not in dropped and path.name in rewrite_files
+    )
+    expected_index = output_index(source, dropped, plan)
+    log(
+        f"PLAN layers={list(layers)} donor_tensors={len(plan)} bytes_to_fetch={fetched_bytes} "
+        f"({fetched_bytes / (1 << 30):.3f} GiB)"
+    )
+    log(
+        f"PLAN rewrite_files={sorted(rewrite_files)} bytes_to_rewrite={rewritten_bytes} "
+        f"bytes_removed={dropped_bytes} dropped_tensors={len(dropped)}"
+    )
+    log(
+        f"PLAN output symlinks=source_files_except_rewritten_config_index "
+        f"new_shard={NEW_SHARD} index={INDEX_NAME} config=remove_selected_layer_bits"
+    )
+    if args.dry_run:
+        log("DRY_RUN_OK network=false writes=false")
+        return 0
+
+    out.mkdir(parents=True, exist_ok=True)
+    link_source_files(src, out, rewrite_files)
+    if donor_headers is None:
+        raise RuntimeError("donor headers were not loaded for a real build")
+    write_new_shard(out, plan, donor, layers, donor_headers)
+    for filename in sorted(rewrite_files):
+        rewrite_source_shard(src / filename, out / filename, {n for n, _ in dropped_by_file[filename]})
+    write_output_metadata(out, expected_index, expected_config)
+    log(f"BUILD_OK out={out}")
+    if args.verify:
+        verify_output(src, out, source, dropped, plan, expected_index, expected_config, rewrite_files)
+    return 0
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--src", required=True, help="source pack directory")
+    parser.add_argument("--donor", required=True, help="Hugging Face repo id or base URL")
+    parser.add_argument("--out", required=True, help="new overlay directory")
+    parser.add_argument("--layers", required=True, help="comma-separated main layer numbers to replace")
+    parser.add_argument(
+        "--donor-index",
+        default=DEFAULT_DONOR_INDEX,
+        help=f"offline donor weight_map JSON (default: {DEFAULT_DONOR_INDEX})",
+    )
+    parser.add_argument(
+        "--donor-headers",
+        default=DEFAULT_HEADER_CACHE,
+        help=f"donor header cache (default: {DEFAULT_HEADER_CACHE})",
+    )
+    parser.add_argument("--dry-run", action="store_true", help="plan only; no network and no writes")
+    parser.add_argument("--verify", action="store_true", help="verify every indexed tensor after building")
+    args = parser.parse_args()
+    if args.dry_run and args.verify:
+        parser.error("--dry-run and --verify cannot be combined")
+    try:
+        return build(args)
+    except KeyboardInterrupt:
+        log("ABORT interrupted")
+        return 130
+    except Exception as exc:
+        log(f"ABORT {type(exc).__name__}: {exc}")
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

lna-lab/tools/mtp_overlay.py

@@ -0,0 +1,1032 @@
+#!/usr/bin/env python3
+"""Build an EXL3-routed-expert overlay for the three DSpark MTP layers.
+
+The source pack is never changed.  The output directory links every source file
+except ``config.json`` and source shards 46--48, rewrites those three shards with
+only their non-draft tensors, and adds ``model-mtp-exl3.safetensors`` containing
+the donor's EXL3 MTP experts.  Donor tensor payloads are fetched with HTTP Range
+requests one tensor at a time; whole donor shards are never downloaded.
+
+Offline planning is intentional.  ``--dry-run`` reads the source headers and the
+local donor weight map but never opens a network connection or writes an output.
+The default local donor index is the one named in orders/T2-mtp-overlay.md.
+"""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+import os
+import re
+import struct
+import sys
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+from typing import BinaryIO, Iterable
+
+
+SOURCE_REWRITE_FILES = tuple(
+    f"model-{i:05d}-of-00048.safetensors" for i in (46, 47, 48)
+)
+NEW_SHARD = "model-mtp-exl3.safetensors"
+INDEX_NAME = "model.safetensors.index.json"
+CONFIG_NAME = "config.json"
+DEFAULT_DONOR_INDEX = "/run/media/tonoken3/DATA1/.tmp/d2-index.json"
+DEFAULT_HEADER_CACHE = "/run/media/tonoken3/DATA1/.tmp/mtp-overlay-donor-headers.json"
+CHUNK_SIZE = 8 << 20
+RETRY_COUNT = 5
+
+DTYPE_BYTES = {
+    "BOOL": 1,
+    "U8": 1,
+    "I8": 1,
+    "F8_E4M3": 1,
+    "F8_E4M3FN": 1,
+    "F8_E5M2": 1,
+    "F8_E8M0": 1,
+    "U16": 2,
+    "I16": 2,
+    "F16": 2,
+    "BF16": 2,
+    "U32": 4,
+    "I32": 4,
+    "F32": 4,
+    "U64": 8,
+    "I64": 8,
+    "F64": 8,
+}
+
+DONOR_EXPERT_RE = re.compile(
+    r"^mtp\.(?P<layer>[0-2])\.mlp\.experts\."
+    r"(?P<expert>[0-9]+)\.(?P<projection>gate_proj|up_proj|down_proj)\."
+    r"(?P<suffix>trellis|suh|svh|mcg)$"
+)
+SOURCE_EXPERT_RE = re.compile(
+    r"^mtp\.(?P<layer>[0-2])\.ffn\.experts\."
+    r"(?P<expert>[0-9]+)\.(?P<projection>w[123])\."
+    r"(?P<suffix>weight|scale)$"
+)
+SOURCE_SHARD_RE = re.compile(r"^model-(?P<number>[0-9]+)-of-(?P<total>[0-9]+)\.safetensors$")
+
+PROJECTION_MAP = {"gate_proj": "w1", "up_proj": "w3", "down_proj": "w2"}
+PROJECTION_ORDER = {"gate_proj": 0, "up_proj": 1, "down_proj": 2}
+EXL3_SUFFIXES = ("trellis", "suh", "svh", "mcg")
+EXL3_DTYPE = {"trellis": "I16", "suh": "F16", "svh": "F16", "mcg": "I32"}
+
+
+def log(message: str) -> None:
+    print(message, flush=True)
+
+
+def tensor_nbytes(meta: dict) -> int:
+    """Return the payload size and reject malformed safetensors metadata."""
+    dtype = meta.get("dtype")
+    if dtype not in DTYPE_BYTES:
+        raise ValueError(f"unsupported safetensors dtype {dtype!r}")
+    shape = meta.get("shape")
+    if not isinstance(shape, list) or any(not isinstance(x, int) or x < 0 for x in shape):
+        raise ValueError(f"invalid safetensors shape {shape!r}")
+    size = DTYPE_BYTES[dtype]
+    for dim in shape:
+        size *= dim
+    return size
+
+
+def parse_header(blob: bytes) -> tuple[int, dict]:
+    """Parse a safetensors header from a blob containing at least its header."""
+    if len(blob) < 8:
+        raise ValueError("safetensors header is shorter than the 8-byte length")
+    header_len = struct.unpack("<Q", blob[:8])[0]
+    end = 8 + header_len
+    if end > len(blob):
+        raise ValueError(f"truncated safetensors header: need {end}, have {len(blob)}")
+    try:
+        header = json.loads(blob[8:end].decode("utf-8"))
+    except (UnicodeDecodeError, json.JSONDecodeError) as exc:
+        raise ValueError(f"invalid safetensors JSON header: {exc}") from exc
+    if not isinstance(header, dict):
+        raise ValueError("safetensors header JSON must be an object")
+    return header_len, header
+
+
+def validate_header(header: dict, data_bytes: int | None = None) -> None:
+    """Validate the structural parts needed by the range copier."""
+    for name, meta in header.items():
+        if name == "__metadata__":
+            if not isinstance(meta, dict):
+                raise ValueError("safetensors __metadata__ must be an object")
+            continue
+        if not isinstance(meta, dict):
+            raise ValueError(f"tensor {name!r} metadata is not an object")
+        offsets = meta.get("data_offsets")
+        if (
+            not isinstance(offsets, list)
+            or len(offsets) != 2
+            or any(not isinstance(x, int) for x in offsets)
+            or offsets[0] < 0
+            or offsets[1] < offsets[0]
+        ):
+            raise ValueError(f"invalid data_offsets for {name!r}: {offsets!r}")
+        expected = tensor_nbytes(meta)
+        actual = offsets[1] - offsets[0]
+        if expected != actual:
+            raise ValueError(
+                f"payload size mismatch for {name!r}: metadata={expected}, offsets={actual}"
+            )
+        if data_bytes is not None and offsets[1] > data_bytes:
+            raise ValueError(
+                f"data_offsets for {name!r} exceed data region: {offsets[1]} > {data_bytes}"
+            )
+
+
+def read_header(path: Path, *, check_data: bool = True) -> tuple[int, dict]:
+    """Read and validate only a local file's 8-byte length and JSON header."""
+    file_size = path.stat().st_size
+    with path.open("rb") as stream:
+        prefix = stream.read(8)
+        if len(prefix) != 8:
+            raise ValueError(f"{path}: missing 8-byte safetensors header length")
+        header_len = struct.unpack("<Q", prefix)[0]
+        header_blob = prefix + stream.read(header_len)
+    if len(header_blob) != 8 + header_len:
+        raise ValueError(f"{path}: truncated safetensors JSON header")
+    parsed_len, header = parse_header(header_blob)
+    if parsed_len != header_len:
+        raise AssertionError("header length changed during parsing")
+    validate_header(header, file_size - 8 - header_len if check_data else None)
+    return header_len, header
+
+
+def map_donor_name(name: str) -> str:
+    """Map one donor EXL3 expert name to the target pack's expert ABI."""
+    match = DONOR_EXPERT_RE.fullmatch(name)
+    if match is None:
+        raise ValueError(f"not a donor MTP EXL3 expert tensor: {name}")
+    projection = match.group("projection")
+    return (
+        f"mtp.{match.group('layer')}.ffn.experts.{int(match.group('expert'))}."
+        f"{PROJECTION_MAP[projection]}.{match.group('suffix')}"
+    )
+
+
+# Short alias used by the offline tests and useful to callers importing this tool.
+map_name = map_donor_name
+
+
+def is_donor_expert_name(name: str) -> bool:
+    return DONOR_EXPERT_RE.fullmatch(name) is not None
+
+
+def tensor_range(header_len: int, meta: dict) -> tuple[int, int]:
+    """Return the inclusive absolute byte range for one tensor."""
+    start, end = meta["data_offsets"]
+    return 8 + header_len + start, 8 + header_len + end - 1
+
+
+def source_headers(src: Path) -> tuple[dict[str, tuple[Path, int, dict]], dict[str, str]]:
+    """Scan all numbered source shards; the source intentionally has no index."""
+    candidates = []
+    for path in src.iterdir():
+        match = SOURCE_SHARD_RE.fullmatch(path.name)
+        if match and path.is_file():
+            candidates.append((int(match.group("number")), path))
+    if not candidates:
+        raise RuntimeError(f"no numbered safetensors shards found under {src}")
+
+    headers: dict[str, tuple[Path, int, dict]] = {}
+    weight_map: dict[str, str] = {}
+    for _, path in sorted(candidates):
+        header_len, header = read_header(path)
+        for name in header:
+            if name == "__metadata__":
+                continue
+            if name in weight_map:
+                raise RuntimeError(f"duplicate source tensor {name!r}")
+            headers[name] = (path, header_len, header)
+            weight_map[name] = path.name
+    log(f"SOURCE_HEADERS files={len(candidates)} tensors={len(headers)}")
+    return headers, weight_map
+
+
+def donor_index_from_file(path: Path) -> dict[str, str]:
+    try:
+        data = json.loads(path.read_text())
+    except (OSError, json.JSONDecodeError) as exc:
+        raise RuntimeError(f"cannot read donor index {path}: {exc}") from exc
+    weight_map = data.get("weight_map")
+    if not isinstance(weight_map, dict) or not all(
+        isinstance(name, str) and isinstance(filename, str)
+        for name, filename in weight_map.items()
+    ):
+        raise RuntimeError(f"donor index {path} has no valid weight_map")
+    return weight_map
+
+
+def http_bytes(url: str, byte_range: tuple[int, int] | None = None) -> bytes:
+    """Fetch one complete response, retrying transient/range failures.
+
+    Lna-Lab: if MTP_OVERLAY_LOCAL_DIR holds a file with the URL's basename (a
+    pre-downloaded donor shard), serve the bytes from it instead of the network.
+    """
+    local_dir = os.environ.get("MTP_OVERLAY_LOCAL_DIR")
+    if local_dir:
+        local_path = Path(local_dir) / url.rsplit("/", 1)[-1].split("?")[0]
+        if local_path.is_file():
+            with local_path.open("rb") as fh:
+                if byte_range is None:
+                    return fh.read()
+                fh.seek(byte_range[0])
+                data = fh.read(byte_range[1] - byte_range[0] + 1)
+            expected = byte_range[1] - byte_range[0] + 1
+            if len(data) != expected:
+                raise RuntimeError(f"short local range read {local_path}: got {len(data)}, expected {expected}")
+            return data
+    for attempt in range(RETRY_COUNT):
+        try:
+            request = urllib.request.Request(url, headers={"User-Agent": "mtp_overlay/1"})
+            if byte_range is not None:
+                request.add_header("Range", f"bytes={byte_range[0]}-{byte_range[1]}")
+            with urllib.request.urlopen(request, timeout=60) as response:
+                status = getattr(response, "status", response.getcode())
+                if byte_range is not None and status != 206:
+                    raise RuntimeError(f"expected HTTP 206 for Range request, got {status}")
+                data = response.read()
+            expected = None if byte_range is None else byte_range[1] - byte_range[0] + 1
+            if expected is not None and len(data) != expected:
+                raise RuntimeError(f"short range read: got {len(data)}, expected {expected}")
+            return data
+        except (OSError, TimeoutError, urllib.error.URLError, urllib.error.HTTPError, RuntimeError) as exc:
+            if attempt == RETRY_COUNT - 1:
+                raise RuntimeError(f"failed fetching {url} range={byte_range}: {exc}") from exc
+            delay = 2**attempt
+            log(f"RETRY attempt={attempt + 1}/{RETRY_COUNT} range={byte_range}: {exc}")
+            time.sleep(delay)
+    raise AssertionError("unreachable")
+
+
+def donor_headers_from_remote(
+    donor: str,
+    weight_map: dict[str, str],
+    selected_names: Iterable[str],
+    cache_path: Path | None,
+    *,
+    write_cache: bool = True,
+    allow_network: bool = True,
+) -> dict[str, tuple[int, dict]]:
+    """Read one remote safetensors header per selected donor shard."""
+    base = donor.rstrip("/") if donor.startswith("http") else f"https://huggingface.co/{donor}/resolve/main"
+    files = sorted({weight_map[name] for name in selected_names})
+    cached: dict = {}
+    if cache_path is not None and cache_path.exists():
+        try:
+            cached = json.loads(cache_path.read_text()).get("headers", {})
+        except (OSError, json.JSONDecodeError) as exc:
+            raise RuntimeError(f"cannot read donor header cache {cache_path}: {exc}") from exc
+
+    result: dict[str, tuple[int, dict]] = {}
+    for filename in files:
+        item = cached.get(filename)
+        if isinstance(item, list) and len(item) == 2:
+            header_len, header = int(item[0]), item[1]
+            validate_header(header)
+            result[filename] = (header_len, header)
+            log(f"DONOR_HEADER cache file={filename} tensors={len(header) - 1}")
+            continue
+        if not allow_network:
+            raise RuntimeError(f"donor header cache has no entry for {filename}")
+        url = f"{base}/{filename}"
+        length_blob = http_bytes(url, (0, 7))
+        if len(length_blob) != 8:
+            raise RuntimeError(f"donor header length read for {filename} was not 8 bytes")
+        header_len = struct.unpack("<Q", length_blob)[0]
+        header_blob = http_bytes(url, (8, 8 + header_len - 1))
+        parsed_len, header = parse_header(length_blob + header_blob)
+        validate_header(header)
+        if parsed_len != header_len:
+            raise RuntimeError(f"donor header length changed for {filename}")
+        result[filename] = (header_len, header)
+        log(f"DONOR_HEADER remote file={filename} tensors={len(header) - 1}")
+
+    if cache_path is not None and write_cache:
+        cache_path.parent.mkdir(parents=True, exist_ok=True)
+        cache_path.write_text(
+            json.dumps(
+                {"headers": {name: [hlen, header] for name, (hlen, header) in result.items()}},
+                indent=2,
+            )
+        )
+        log(f"DONOR_HEADER_CACHE wrote={cache_path}")
+    return result
+
+
+def expected_donor_names() -> set[str]:
+    return {
+        f"mtp.{layer}.mlp.experts.{expert}.{projection}.{suffix}"
+        for layer in range(3)
+        for expert in range(256)
+        for projection in PROJECTION_MAP
+        for suffix in EXL3_SUFFIXES
+    }
+
+
+def source_draft_shapes(
+    source: dict[str, tuple[Path, int, dict]],
+) -> dict[str, tuple[int, int]]:
+    """Validate the measured source FP8 draft weight shapes and return out/in dims."""
+    matches = [
+        (name, meta[2][name])
+        for name, meta in source.items()
+        if SOURCE_EXPERT_RE.fullmatch(name)
+    ]
+    expected = {
+        f"mtp.{layer}.ffn.experts.{expert}.{projection}.{suffix}"
+        for layer in range(3)
+        for expert in range(256)
+        for projection in ("w1", "w2", "w3")
+        for suffix in ("weight", "scale")
+    }
+    actual = {name for name, _ in matches}
+    if actual != expected:
+        missing = sorted(expected - actual)[:5]
+        extra = sorted(actual - expected)[:5]
+        raise RuntimeError(f"source draft expert set mismatch missing={missing} extra={extra}")
+
+    shapes: dict[str, tuple[int, int]] = {}
+    for projection in ("w1", "w2", "w3"):
+        name = f"mtp.0.ffn.experts.0.{projection}.weight"
+        meta = source[name][2][name]
+        if meta.get("dtype") != "I8" or len(meta.get("shape", [])) != 2:
+            raise RuntimeError(f"unexpected source draft metadata for {name}: {meta}")
+        out_dim, in_dim = meta["shape"]
+        if out_dim % 16 or in_dim % 16:
+            raise RuntimeError(f"source draft dimensions are not tile-aligned for {name}: {meta['shape']}")
+        shapes[projection] = (out_dim, in_dim)
+        scale_name = f"mtp.0.ffn.experts.0.{projection}.scale"
+        scale = source[scale_name][2][scale_name]
+        if scale.get("dtype") != "F8_E8M0":
+            raise RuntimeError(f"unexpected source draft scale dtype for {scale_name}: {scale}")
+
+    for name, (_, _, header) in source.items():
+        match = SOURCE_EXPERT_RE.fullmatch(name)
+        if match is None:
+            continue
+        meta = header[name]
+        projection = match.group("projection")
+        suffix = match.group("suffix")
+        if suffix == "weight":
+            if meta["dtype"] != "I8" or tuple(meta["shape"]) != shapes[projection]:
+                raise RuntimeError(f"inconsistent source draft weight {name}: {meta}")
+        elif meta["dtype"] != "F8_E8M0":
+            raise RuntimeError(f"inconsistent source draft scale {name}: {meta}")
+
+    log(
+        "SOURCE_DRAFT_SHAPES "
+        + " ".join(
+            f"{projection}.weight=I8{list(shapes[projection])}"
+            for projection in ("w1", "w2", "w3")
+        )
+        + " scale=F8_E8M0"
+    )
+    return shapes
+
+
+def main_k2_shapes(
+    source: dict[str, tuple[Path, int, dict]],
+) -> dict[str, dict[str, dict]]:
+    """Read the concrete K2 EXL3 ABI from the requested main-layer exemplar."""
+    result: dict[str, dict[str, dict]] = {}
+    for projection in ("w1", "w2", "w3"):
+        result[projection] = {}
+        for suffix in EXL3_SUFFIXES:
+            name = f"layers.5.ffn.experts.0.{projection}.{suffix}"
+            if name not in source:
+                raise RuntimeError(f"required main-layer K2 exemplar is missing: {name}")
+            result[projection][suffix] = copy.deepcopy(source[name][2][name])
+    for projection, parts in result.items():
+        trellis = parts["trellis"]
+        if trellis.get("dtype") != "I16" or trellis.get("shape", [])[-1:] != [32]:
+            raise RuntimeError(f"main-layer exemplar is not K2 EXL3 for {projection}: {trellis}")
+        for suffix in EXL3_SUFFIXES:
+            if parts[suffix].get("dtype") != EXL3_DTYPE[suffix]:
+                raise RuntimeError(f"main-layer exemplar dtype mismatch for {projection}.{suffix}")
+    log(
+        "MAIN_K2_ABI "
+        + " ".join(
+            f"{projection}="
+            + ",".join(
+                f"{suffix}:{result[projection][suffix]['dtype']}{result[projection][suffix]['shape']}"
+                for suffix in EXL3_SUFFIXES
+            )
+            for projection in ("w1", "w2", "w3")
+        )
+    )
+    return result
+
+
+def expected_mtp_meta(
+    projection: str,
+    suffix: str,
+    draft_shapes: dict[str, tuple[int, int]],
+    main_shapes: dict[str, dict[str, dict]],
+) -> dict:
+    """Build the offline expected MTP ABI from the concrete main K2 ABI.
+
+    The source FP8 MTP weights are TP-local (w1/w3 [2048, 2048], w2
+    [4096, 1024]), while the donor EXL3 tensors are full-width and match the
+    main-layer K2 tensors.  ``draft_shapes`` is still passed here so the
+    source dimensions remain part of the validated plan and cannot silently
+    disappear from the offline check.  EXL3 donor MCG markers are scalar
+    tensors (shape []) although the vLLM destination parameter is [1].
+    """
+    del draft_shapes
+    if suffix == "mcg":
+        # Lna-Lab: donor markers are scalar; the vLLM destination parameter is [1].
+        return {"dtype": EXL3_DTYPE[suffix], "shape": [1]}
+    reference = main_shapes[projection][suffix]
+    return {"dtype": reference["dtype"], "shape": list(reference["shape"])}
+
+
+def validate_donor_selection(
+    weight_map: dict[str, str],
+) -> list[str]:
+    names = [name for name in weight_map if name.startswith("mtp.") and ".experts." in name]
+    malformed = sorted(name for name in names if not is_donor_expert_name(name))
+    if malformed:
+        raise RuntimeError(f"malformed donor MTP expert names (first 5): {malformed[:5]}")
+    expected = expected_donor_names()
+    actual = set(names)
+    if actual != expected:
+        raise RuntimeError(
+            f"donor MTP expert set mismatch: selected={len(actual)} "
+            f"missing={sorted(expected - actual)[:5]} extra={sorted(actual - expected)[:5]}"
+        )
+    selected = sorted(
+        names,
+        key=lambda name: (
+            int(name.split(".")[1]),
+            int(name.split(".")[4]),
+            PROJECTION_ORDER[name.split(".")[5]],
+            EXL3_SUFFIXES.index(name.split(".")[6]),
+        ),
+    )
+    log(
+        f"DONOR_INDEX experts={len(selected)} non_expert_mtp={sum(name.startswith('mtp.') for name in weight_map) - len(selected)} "
+        f"files={sorted({weight_map[name] for name in selected})}"
+    )
+    return selected
+
+
+def validate_and_plan_donor(
+    selected: list[str],
+    weight_map: dict[str, str],
+    donor_headers: dict[str, tuple[int, dict]] | None,
+    draft_shapes: dict[str, tuple[int, int]],
+    main_shapes: dict[str, dict[str, dict]],
+) -> list[dict]:
+    plan = []
+    seen_targets: set[str] = set()
+    for donor_name in selected:
+        target_name = map_donor_name(donor_name)
+        if target_name in seen_targets:
+            raise RuntimeError(f"donor name map collision at {target_name}")
+        seen_targets.add(target_name)
+        projection = PROJECTION_MAP[donor_name.split(".")[5]]
+        suffix = donor_name.split(".")[6]
+        expected = expected_mtp_meta(projection, suffix, draft_shapes, main_shapes)
+        source_file = weight_map[donor_name]
+        actual = expected
+        if donor_headers is not None:
+            if source_file not in donor_headers:
+                raise RuntimeError(f"missing donor header for {source_file}")
+            _, header = donor_headers[source_file]
+            if donor_name not in header:
+                raise RuntimeError(f"donor header lacks indexed tensor {donor_name}")
+            actual = copy.deepcopy(header[donor_name])
+            if donor_name.endswith(".mcg") and actual.get("shape") == []:
+                actual["shape"] = [1]  # same 4 bytes; vLLM wants a (1,) parameter
+            # Dtype and shape must agree with the concrete main K2 EXL3 ABI.
+            # The source MTP FP8 tensors are TP-local; the donor EXL3 tensors
+            # are full-width, as the local donor shard header confirms.
+            if actual.get("dtype") != expected["dtype"] or actual.get("shape") != expected["shape"]:
+                raise RuntimeError(
+                    f"donor shape/dtype mismatch for {donor_name}: got "
+                    f"{actual.get('dtype')}{actual.get('shape')}, expected "
+                    f"{expected['dtype']}{expected['shape']} from local MTP dimensions"
+                )
+            validate_header({donor_name: actual})
+        plan.append(
+            {
+                "donor": donor_name,
+                "target": target_name,
+                "file": source_file,
+                "meta": actual,
+                "expected": expected,
+            }
+        )
+    if len(plan) != 9216:
+        raise RuntimeError(f"planned donor tensor count is {len(plan)}, expected 9216")
+    return plan
+
+
+def source_drop_plan(
+    source: dict[str, tuple[Path, int, dict]],
+) -> tuple[set[str], dict[str, list[tuple[str, dict]]], int, int]:
+    dropped: set[str] = set()
+    by_file: dict[str, list[tuple[str, dict]]] = {name: [] for name in SOURCE_REWRITE_FILES}
+    for name, (path, _, header) in source.items():
+        if name.startswith("mtp.") and ".ffn.experts." in name and SOURCE_EXPERT_RE.fullmatch(name) is None:
+            raise RuntimeError(f"malformed source MTP expert tensor name: {name}")
+        match = SOURCE_EXPERT_RE.fullmatch(name)
+        if match is None:
+            continue
+        if path.name not in by_file:
+            raise RuntimeError(f"draft expert tensor is outside source shards 46-48: {name} in {path.name}")
+        if int(match.group("expert")) >= 256:
+            raise RuntimeError(f"source draft expert id out of range: {name}")
+        dropped.add(name)
+        by_file[path.name].append((name, header[name]))
+    expected_per_file = 256 * 3 * 2
+    for filename, items in by_file.items():
+        if len(items) != expected_per_file:
+            raise RuntimeError(f"{filename}: found {len(items)} draft tensors, expected {expected_per_file}")
+    drop_bytes = sum(tensor_nbytes(meta) for items in by_file.values() for _, meta in items)
+    log(f"SOURCE_DROP tensors={len(dropped)} bytes={drop_bytes} files={list(SOURCE_REWRITE_FILES)}")
+    return dropped, by_file, len(dropped), drop_bytes
+
+
+def write_safetensors_header(
+    stream: BinaryIO,
+    entries: list[tuple[str, dict]],
+    metadata: dict | None = None,
+) -> tuple[int, dict]:
+    header: dict = {}
+    if metadata is not None:
+        header["__metadata__"] = metadata
+    offset = 0
+    for name, meta in entries:
+        size = tensor_nbytes(meta)
+        header[name] = {
+            "dtype": meta["dtype"],
+            "shape": list(meta["shape"]),
+            "data_offsets": [offset, offset + size],
+        }
+        offset += size
+    blob = json.dumps(header, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
+    blob += b" " * (-len(blob) % 8)
+    stream.write(struct.pack("<Q", len(blob)))
+    stream.write(blob)
+    return len(blob), header
+
+
+def copy_payload(
+    source: BinaryIO,
+    destination: BinaryIO,
+    source_data_start: int,
+    meta: dict,
+) -> None:
+    start, end = meta["data_offsets"]
+    source.seek(source_data_start + start)
+    remaining = end - start
+    while remaining:
+        block = source.read(min(CHUNK_SIZE, remaining))
+        if not block:
+            raise RuntimeError("short source tensor payload while rewriting shard")
+        destination.write(block)
+        remaining -= len(block)
+
+
+def rewrite_source_shard(
+    source_path: Path,
+    destination_path: Path,
+    dropped: set[str],
+) -> int:
+    header_len, header = read_header(source_path)
+    entries = [(name, meta) for name, meta in header.items() if name != "__metadata__" and name not in dropped]
+    copied = sum(tensor_nbytes(meta) for _, meta in entries)
+    partial = destination_path.with_name(destination_path.name + ".partial")
+    with source_path.open("rb") as source_stream, partial.open("wb") as destination_stream:
+        _, new_header = write_safetensors_header(
+            destination_stream,
+            entries,
+            header.get("__metadata__"),
+        )
+        del new_header
+        source_data_start = 8 + header_len
+        for name, meta in entries:
+            copy_payload(source_stream, destination_stream, source_data_start, meta)
+    os.replace(partial, destination_path)
+    log(
+        f"REWRITE file={source_path.name} kept_tensors={len(entries)} "
+        f"copied_bytes={copied} dropped_tensors={sum(name in dropped for name, _ in header.items())}"
+    )
+    return copied
+
+
+def safe_output_path(src: Path, out: Path) -> None:
+    src_real = src.resolve()
+    out_real = out.resolve(strict=False)
+    if out_real == src_real or src_real in out_real.parents:
+        raise RuntimeError(f"refusing output inside source pack: {out}")
+
+
+def link_source_files(src: Path, out: Path) -> int:
+    excluded = set(SOURCE_REWRITE_FILES) | {CONFIG_NAME, INDEX_NAME}
+    linked = 0
+    for entry in sorted(src.iterdir(), key=lambda p: p.name):
+        if entry.name in excluded or not (entry.is_file() or entry.is_symlink()):
+            continue
+        destination = out / entry.name
+        if os.path.lexists(destination):
+            if destination.is_symlink() and destination.resolve() == entry.resolve():
+                continue
+            raise RuntimeError(f"output path already exists and is not the source link: {destination}")
+        destination.symlink_to(entry.resolve())
+        linked += 1
+    log(f"LINKED source_files={linked}")
+    return linked
+
+
+def build_new_header(plan: list[dict], donor: str) -> tuple[list[tuple[str, dict]], int, dict]:
+    entries = [(item["target"], item["meta"]) for item in plan]
+    offset = 0
+    header: dict = {"__metadata__": {"format": "pt", "source": donor, "overlay": "mtp-exl3"}}
+    for name, meta in entries:
+        size = tensor_nbytes(meta)
+        header[name] = {
+            "dtype": meta["dtype"],
+            "shape": list(meta["shape"]),
+            "data_offsets": [offset, offset + size],
+        }
+        offset += size
+    return entries, offset, header
+
+
+def header_without_offsets(header: dict) -> dict:
+    return {
+        name: {"dtype": meta["dtype"], "shape": list(meta["shape"])}
+        for name, meta in header.items()
+        if name != "__metadata__"
+    }
+
+
+def write_new_shard(
+    out: Path,
+    plan: list[dict],
+    donor: str,
+    donor_headers: dict[str, tuple[int, dict]],
+) -> int:
+    entries, payload_bytes, expected_header = build_new_header(plan, donor)
+    destination = out / NEW_SHARD
+    partial = out / (NEW_SHARD + ".partial")
+    header_len = None
+    actual_header = None
+    resume_at = 0
+
+    if destination.exists():
+        existing_len, existing_header = read_header(destination)
+        if header_without_offsets(existing_header) != header_without_offsets(expected_header):
+            raise RuntimeError(f"existing {destination} has a different tensor plan")
+        if destination.stat().st_size != 8 + existing_len + payload_bytes:
+            raise RuntimeError(f"existing {destination} is incomplete or has trailing bytes")
+        log(f"NEW_SHARD existing_complete={destination}")
+        return payload_bytes
+
+    if partial.exists():
+        # A process can stop in the middle of the current tensor.  Read and
+        # validate the JSON header without requiring the partial data region to
+        # reach its final offset, then resume only at a completed boundary.
+        existing_len, existing_header = read_header(partial, check_data=False)
+        if header_without_offsets(existing_header) == header_without_offsets(expected_header):
+            data_start = 8 + existing_len
+            data_bytes = partial.stat().st_size - data_start
+            boundaries = {0} | {meta["data_offsets"][1] for name, meta in existing_header.items() if name != "__metadata__"}
+            if data_bytes in boundaries and 0 <= data_bytes <= payload_bytes:
+                header_len, actual_header, resume_at = existing_len, existing_header, data_bytes
+                log(f"NEW_SHARD resume={partial} completed_bytes={resume_at}")
+            else:
+                log(f"NEW_SHARD discard_invalid_partial={partial}")
+                partial.unlink()
+        else:
+            log(f"NEW_SHARD discard_stale_partial={partial}")
+            partial.unlink()
+
+    if header_len is None:
+        with partial.open("wb") as stream:
+            header_len, actual_header = write_safetensors_header(
+                stream, entries, {"format": "pt", "source": donor, "overlay": "mtp-exl3"}
+            )
+        resume_at = 0
+
+    assert header_len is not None and actual_header is not None
+    with partial.open("r+b") as stream:
+        data_start = 8 + header_len
+        stream.truncate(data_start + resume_at)
+        stream.seek(data_start + resume_at)
+        for index, item in enumerate(plan):
+            name = item["target"]
+            meta = actual_header[name]
+            start, end = meta["data_offsets"]
+            if end <= resume_at:
+                log(f"FETCH skip_existing tensor={index + 1}/{len(plan)} name={name}")
+                continue
+            if start != resume_at:
+                raise RuntimeError(f"partial new shard boundary mismatch before {name}")
+            remote_header_len, remote_header = donor_headers[item["file"]]
+            remote_meta = remote_header[item["donor"]]
+            absolute = tensor_range(remote_header_len, remote_meta)
+            base = donor.rstrip("/") if donor.startswith("http") else f"https://huggingface.co/{donor}/resolve/main"
+            log(f"FETCH tensor={index + 1}/{len(plan)} name={name} bytes={end - start}")
+            data = http_bytes(f"{base}/{item['file']}", absolute)
+            if len(data) != end - start:
+                raise RuntimeError(f"{name}: fetched {len(data)} bytes, expected {end - start}")
+            stream.write(data)
+            resume_at = end
+    if resume_at != payload_bytes:
+        raise RuntimeError(f"new shard ended at {resume_at} bytes, expected {payload_bytes}")
+    os.replace(partial, destination)
+    log(f"NEW_SHARD wrote={destination} payload_bytes={payload_bytes}")
+    return payload_bytes
+
+
+def output_index(
+    source: dict[str, tuple[Path, int, dict]],
+    dropped: set[str],
+    new_plan: list[dict],
+) -> dict:
+    weight_map: dict[str, str] = {}
+    total_size = 0
+    for name, (path, _, header) in sorted(source.items()):
+        if name in dropped:
+            continue
+        output_file = path.name if path.name not in SOURCE_REWRITE_FILES else path.name
+        weight_map[name] = output_file
+        total_size += tensor_nbytes(header[name])
+    for item in new_plan:
+        if item["target"] in weight_map:
+            raise RuntimeError(f"new tensor collides with retained source tensor: {item['target']}")
+        weight_map[item["target"]] = NEW_SHARD
+        total_size += tensor_nbytes(item["meta"])
+    return {"metadata": {"total_size": total_size}, "weight_map": dict(sorted(weight_map.items()))}
+
+
+def edited_config(src: Path) -> dict:
+    config_path = src / CONFIG_NAME
+    try:
+        config = json.loads(config_path.read_text())
+    except (OSError, json.JSONDecodeError) as exc:
+        raise RuntimeError(f"cannot read {config_path}: {exc}") from exc
+    q = config.get("quantization_config")
+    if not isinstance(q, dict):
+        raise RuntimeError("source config has no quantization_config object")
+    q.pop("mtp_experts", None)
+    q.pop("mtp_experts_start_layer", None)
+    layer_bits = q.get("layer_bits")
+    if not isinstance(layer_bits, dict):
+        raise RuntimeError("source quantization_config.layer_bits is not an object")
+    layer_bits = dict(layer_bits)
+    layer_bits.update({"43": 2, "44": 2, "45": 2})
+    q["layer_bits"] = layer_bits
+    return config
+
+
+def write_output_metadata(out: Path, index: dict, config: dict) -> None:
+    (out / INDEX_NAME).write_text(json.dumps(index, indent=2) + "\n")
+    (out / CONFIG_NAME).write_text(json.dumps(config, indent=2) + "\n")
+    log(f"WROTE metadata files={INDEX_NAME},{CONFIG_NAME}")
+
+
+def compare_payloads(source_path: Path, output_path: Path, source_meta: dict, output_meta: dict, source_header_len: int, output_header_len: int) -> None:
+    source_start, source_end = source_meta["data_offsets"]
+    output_start, output_end = output_meta["data_offsets"]
+    if source_end - source_start != output_end - output_start:
+        raise RuntimeError("retained tensor payload length changed")
+    with source_path.open("rb") as source_stream, output_path.open("rb") as output_stream:
+        source_stream.seek(8 + source_header_len + source_start)
+        output_stream.seek(8 + output_header_len + output_start)
+        remaining = source_end - source_start
+        while remaining:
+            size = min(CHUNK_SIZE, remaining)
+            a = source_stream.read(size)
+            b = output_stream.read(size)
+            if a != b:
+                raise RuntimeError(f"retained tensor payload differs: {source_path.name}")
+            remaining -= len(a)
+
+
+def verify_output(
+    src: Path,
+    out: Path,
+    source: dict[str, tuple[Path, int, dict]],
+    dropped: set[str],
+    plan: list[dict],
+    expected_index: dict,
+    source_config: dict,
+) -> None:
+    index_path = out / INDEX_NAME
+    config_path = out / CONFIG_NAME
+    if not index_path.is_file() or not config_path.is_file():
+        raise RuntimeError("overlay index/config is missing")
+    actual_index = json.loads(index_path.read_text())
+    if actual_index.get("weight_map") != expected_index.get("weight_map"):
+        raise RuntimeError("overlay index weight_map differs from the planned map")
+
+    # Every indexed tensor is opened through its file header and its declared
+    # shape/dtype/byte length is checked against the index.
+    output_headers: dict[str, tuple[Path, int, dict]] = {}
+    for filename in sorted(set(actual_index["weight_map"].values())):
+        path = out / filename
+        if not path.is_file():
+            raise RuntimeError(f"indexed shard is missing: {path}")
+        header_len, header = read_header(path)
+        for name, mapped_file in actual_index["weight_map"].items():
+            if mapped_file != filename:
+                continue
+            if name not in header:
+                raise RuntimeError(f"indexed tensor is missing from {filename}: {name}")
+            meta = header[name]
+            if tensor_nbytes(meta) != meta["data_offsets"][1] - meta["data_offsets"][0]:
+                raise RuntimeError(f"invalid indexed tensor payload: {name}")
+            output_headers[name] = (path, header_len, header)
+    if set(output_headers) != set(actual_index["weight_map"]):
+        raise RuntimeError("not every indexed tensor was opened")
+
+    # Confirm source links and byte identity of every retained tensor in the
+    # three rewritten shards.  Other source shards are links and need no copy.
+    for entry in src.iterdir():
+        if not (entry.is_file() or entry.is_symlink()) or entry.name in SOURCE_REWRITE_FILES or entry.name in (CONFIG_NAME, INDEX_NAME):
+            continue
+        linked = out / entry.name
+        if not linked.is_symlink() or linked.resolve() != entry.resolve():
+            raise RuntimeError(f"source file is not linked unchanged: {entry.name}")
+    for name, (source_path, source_header_len, source_header) in source.items():
+        if name in dropped:
+            if name in actual_index["weight_map"]:
+                raise RuntimeError(f"dropped source tensor remains indexed: {name}")
+            continue
+        if source_path.name not in SOURCE_REWRITE_FILES:
+            continue
+        output_path, output_header_len, output_header = output_headers[name]
+        if output_header[name]["dtype"] != source_header[name]["dtype"] or output_header[name]["shape"] != source_header[name]["shape"]:
+            raise RuntimeError(f"retained tensor metadata changed: {name}")
+        compare_payloads(
+            source_path,
+            output_path,
+            source_header[name],
+            output_header[name],
+            source_header_len,
+            output_header_len,
+        )
+
+    # New names, shapes, dtypes and MCG markers must match the donor plan.
+    for item in plan:
+        path, header_len, header = output_headers[item["target"]]
+        del path
+        actual = header[item["target"]]
+        if actual["dtype"] != item["meta"]["dtype"] or actual["shape"] != item["meta"]["shape"]:
+            raise RuntimeError(f"new tensor metadata differs from donor plan: {item['target']}")
+        if item["target"].endswith(".mcg"):
+            with (out / NEW_SHARD).open("rb") as stream:
+                start, end = actual["data_offsets"]
+                stream.seek(8 + header_len + start)
+                value = struct.unpack("<i", stream.read(end - start))[0]
+            if value != -877912083:
+                raise RuntimeError(f"MCG marker mismatch for {item['target']}: {value}")
+
+    config = json.loads(config_path.read_text())
+    q = config.get("quantization_config", {})
+    if "mtp_experts" in q or "mtp_experts_start_layer" in q:
+        raise RuntimeError("MTP source-format config keys were not removed")
+    if q.get("non_routed_quantization") != source_config["quantization_config"].get("non_routed_quantization"):
+        raise RuntimeError("non_routed_quantization changed")
+    if q.get("non_routed_dtype_policy") != source_config["quantization_config"].get("non_routed_dtype_policy"):
+        raise RuntimeError("non_routed_dtype_policy changed")
+    for key in ("43", "44", "45"):
+        if q.get("layer_bits", {}).get(key) != 2:
+            raise RuntimeError(f"layer_bits[{key}] is not explicitly 2")
+    log(f"VERIFY_OK indexed_tensors={len(output_headers)} retained_payloads_byte_identical=true")
+
+
+def build(args: argparse.Namespace) -> int:
+    src = Path(args.src).expanduser().resolve()
+    out = Path(args.out).expanduser()
+    donor = args.donor
+    if not src.is_dir():
+        raise RuntimeError(f"source directory does not exist: {src}")
+    safe_output_path(src, out)
+
+    source, source_map = source_headers(src)
+    dropped, by_file, dropped_count, dropped_bytes = source_drop_plan(source)
+    del by_file
+    draft_shapes = source_draft_shapes(source)
+    main_shapes = main_k2_shapes(source)
+
+    donor_index_path = Path(args.donor_index).expanduser() if args.donor_index else None
+    if donor_index_path is not None and donor_index_path.exists():
+        donor_map = donor_index_from_file(donor_index_path)
+        log(f"DONOR_INDEX read={donor_index_path}")
+    elif args.dry_run:
+        raise RuntimeError(
+            f"--dry-run requires the local donor index; not found: {donor_index_path}"
+        )
+    else:
+        base = donor.rstrip("/") if donor.startswith("http") else f"https://huggingface.co/{donor}/resolve/main"
+        donor_index_blob = http_bytes(f"{base}/model.safetensors.index.json")
+        try:
+            donor_map = json.loads(donor_index_blob)["weight_map"]
+        except (json.JSONDecodeError, KeyError, TypeError) as exc:
+            raise RuntimeError(f"remote donor index is invalid: {exc}") from exc
+        log(f"DONOR_INDEX remote tensors={len(donor_map)}")
+    selected = validate_donor_selection(donor_map)
+
+    cache_path = Path(args.donor_headers).expanduser() if args.donor_headers else None
+    donor_headers: dict[str, tuple[int, dict]] | None = None
+    if args.dry_run:
+        if cache_path is not None and cache_path.exists():
+            donor_headers = donor_headers_from_remote(
+                donor,
+                donor_map,
+                selected,
+                cache_path,
+                write_cache=False,
+                allow_network=False,
+            )
+        else:
+            log("DONOR_HEADERS offline=not-read; dry-run uses local MTP dimensions")
+    else:
+        donor_headers = donor_headers_from_remote(donor, donor_map, selected, cache_path)
+    plan = validate_and_plan_donor(selected, donor_map, donor_headers, draft_shapes, main_shapes)
+
+    fetched_bytes = sum(tensor_nbytes(item["meta"]) for item in plan)
+    rewritten_bytes = sum(
+        tensor_nbytes(meta)
+        for name, (_, _, header) in source.items()
+        if name not in dropped and Path(source_map[name]).name in SOURCE_REWRITE_FILES
+        for meta in [header[name]]
+    )
+    source_config = edited_config(src)
+    expected_index = output_index(source, dropped, plan)
+    log(
+        f"PLAN donor_tensors={len(plan)} bytes_to_fetch={fetched_bytes} "
+        f"({fetched_bytes / (1 << 30):.3f} GiB)"
+    )
+    log(
+        f"PLAN rewrite_files=3 bytes_to_rewrite={rewritten_bytes} "
+        f"bytes_removed={dropped_bytes} dropped_tensors={dropped_count}"
+    )
+    log(
+        f"PLAN output symlinks=source_files_except_46_48_and_config "
+        f"new_shard={NEW_SHARD} index={INDEX_NAME} config_edits=remove_mtp_source_keys+layer_bits_43_44_45=2"
+    )
+    if args.dry_run:
+        log("DRY_RUN_OK network=false writes=false")
+        return 0
+
+    out.mkdir(parents=True, exist_ok=True)
+    link_source_files(src, out)
+    if donor_headers is None:
+        raise RuntimeError("donor headers were not loaded for a real build")
+    write_new_shard(out, plan, donor, donor_headers)
+    for filename in SOURCE_REWRITE_FILES:
+        rewrite_source_shard(src / filename, out / filename, dropped)
+    write_output_metadata(out, expected_index, source_config)
+    log(f"BUILD_OK out={out}")
+    if args.verify:
+        verify_output(src, out, source, dropped, plan, expected_index, json.loads((src / CONFIG_NAME).read_text()))
+    return 0
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--src", required=True, help="source pack directory")
+    parser.add_argument("--donor", required=True, help="Hugging Face repo id or base URL")
+    parser.add_argument("--out", required=True, help="new overlay directory")
+    parser.add_argument(
+        "--donor-index",
+        default=DEFAULT_DONOR_INDEX,
+        help=f"offline donor weight_map JSON (default: {DEFAULT_DONOR_INDEX})",
+    )
+    parser.add_argument(
+        "--donor-headers",
+        default=DEFAULT_HEADER_CACHE,
+        help=f"remote donor header cache (default: {DEFAULT_HEADER_CACHE})",
+    )
+    parser.add_argument("--dry-run", action="store_true", help="plan only; no network and no writes")
+    parser.add_argument("--verify", action="store_true", help="verify every indexed tensor after building")
+    args = parser.parse_args()
+    if args.dry_run and args.verify:
+        parser.error("--dry-run and --verify cannot be combined")
+    try:
+        return build(args)
+    except KeyboardInterrupt:
+        log("ABORT interrupted")
+        return 130
+    except Exception as exc:
+        log(f"ABORT {type(exc).__name__}: {exc}")
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

model-00001-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:367c971dc3cd6a042a9bec1caff508e77eabfaef1df2de3a827397ef8bbc6af3
+size 1991876272

model-00002-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d654a0be1cc7be7222d79d1e3971ee64fa78867e01fbcae815032e71951de193
+size 1631836440

model-00003-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3443031c34ff01bf4ef0b23fb36d60cd6cbf45a4e01128c816a3389adf7ac46e
+size 1631836440

model-00004-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34687841798a15ce9b46052ff8770237d8e8343b08b3b8cd4e985de2913ac512
+size 1632383008

model-00005-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa3b869fe2e277e6f46d2618d4f5e8625d4ba359d2ebaa5fda303e5286c49440
+size 5529472

model-00006-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e97556b8989a374dec330c3c886dcd6e19a56001c7daf9e835cc306587a2e1f
+size 1626176072

model-00007-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dfecfd503aabac065be311b383150950786df03bbce177b881c0bd37b64b92a9
+size 1625892784

model-00008-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc8bf1d81af397ef98daf0fb1653ae08454236a6e5e57d65c425fd0082dd7f51
+size 1626176072

model-00009-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fad91e6c1eaa4ca3e48d4e3f2f98c69c86b06ad1acd9cf407dd46728556be008
+size 1625892784

model-00010-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c1e892b046ab791432bddf923111f705ea762638d52ce9d0499401b5c623d80
+size 1626176072

model-00011-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7d4ca138c51b71490a0d18daa61e38fe66d001f09e3365ce52146f5228091bf
+size 1625892784

model-00012-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3a4d98b8205cf0f531abba739bfd513266cb29f1420bddc6ddc5068b864133a
+size 1626179160

model-00013-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac720c288d0664776005ce618afb0f225916cf0a017549129913ddbe1f68c676
+size 1625895872

model-00014-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cac7ec09d5cedda548777f9a1cabea3427df3a12261e8fbc44780c88a9ddfbf8
+size 1626179160

model-00015-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5716a6ecfde594977d45ddc7120dec61367503d9b47714f9019973a2f8360ebc
+size 2431202576

model-00016-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ee66c56843d7343a08f5ead3acd6ac4f2eb26edb6d6067e015bc51a90428d84
+size 1626179160

model-00017-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b9fffdfcecaa4924223fd6861a3281e331f52610e4406077f19d854a6d02ee5
+size 1625895872

model-00018-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3e3be162b08f58e7ab5fcabeed8c9933c416a9be7798706f4b4a382d5cddaee
+size 1626179160

model-00019-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67283c44bf0daababc353a34bf14d3858cb71561ce0cd1189e2012e275964d05
+size 1625895872

model-00020-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ae488fb14a99df7c32f0d4547d6ec7db03a120c14b4251e309460ddb82b1dad
+size 1626179160

model-00021-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e080a0e3dc4607142b37a9b85ae3b1802fcd0c8b872815ea7c51a227e4e1ea3d
+size 1625895872

model-00022-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb567d322e29b3ce20ca332de7ac3d451c91836518bfdf6bbafb097e9d56ef3c
+size 1626179160

model-00023-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae4fe98e86714fcfdece132565b3240d88567815c3c51ab6246fa9789c937326
+size 5529488

model-00024-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8e884e9ad843f62dcd625763e4750b5aabc9a0822b4db786c722c4284389756
+size 2431485872

model-00025-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98804432b6124d2a0d7fe527e9a9e4c7d691d78ef8489f3c20e9d30a406af022
+size 1625895872

model-00026-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3e1295c53d8f9b711d57b10112929685927590327a257372288ba28185c0906
+size 1626179160

model-00027-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8bdc5ef60398f85f6ea7d5828915ad2fea0b0415adf8b1615e19cb423737632
+size 1625895872

model-00028-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d79711081aee89cb63a88758c625ae9998fc49d75da6e520a65dfb34531af68d
+size 1626179160