Document the MTP heads and vision support

README.md

@@ -21,16 +21,16 @@ tags:
 machine. Built and tested on an NVIDIA DGX Spark (GB10).
 
 The repo also includes Xiaomi's DFlash speculative-decoding drafter, fixed so it loads and
-quantized to 4 bpw.
+quantized to 4 bpw, and the model's own MTP heads and vision tower.
 
 | | |
 |---|---|
-| Weights | 83.45 GiB, 12 shards |
+| Weights | 85.28 GiB, 12 shards, including the MTP heads and the vision tower |
 | Bitrate | 2.27 bpw (excluding head), head 6 bpw |
 | Perplexity | 5.40, wikitext-2 test, 64 x 2048 tokens |
-| Decode, batch 1 | ~31 tok/s without drafter; 40–60 tok/s with the 4 bpw drafter (see [Speed](#speed)) |
-| Context | 262,144 tokens with the drafter on a DGX Spark |
-| Modalities | Text only (no vision, audio or MTP heads) |
+| Decode, batch 1 | ~31 tok/s without a drafter; 35–80 tok/s with the DFlash drafter (see [Speed](#speed)) |
+| Context | 262,144 tokens on a DGX Spark, with the drafter and vision loaded |
+| Modalities | Text and images (no audio) |
 
 Requires exllamav3 v1.5.2 or later; see [How to run](#how-to-run).
 
@@ -39,7 +39,7 @@ Requires exllamav3 v1.5.2 or later; see [How to run](#how-to-run).
 ```
 model-*.safetensors, model.safetensors.index.json   EXL3 weights
 quantization_config.json                            per-tensor storage record
-config.json, tokenizer files, chat_template.jinja
+config.json, tokenizer files, chat_template.jinja, preprocessor_config.json
 dflash/        drafter, 4 bpw EXL3 (use this one)
 dflash-bf16/   the same drafter, unquantized
 eval/          benchmark outputs
@@ -56,7 +56,8 @@ Converted with `convert.py -b 2.25 -hq -cr 250 -cc 2048`. The per-module result:
 | attention | 4.0 |
 | dense MLP (layer 0) | 3.0 |
 | `lm_head` | 6.0 |
-| embeddings, norms, router | BF16 |
+| MTP heads | 4.0 |
+| embeddings, norms, router, vision tower | BF16 |
 
 **Layer 47:** some of its experts produce intermediate values past the fp16 limit.
 This build scales `up_proj` down by 128 in that layer (`interm_div`) and restores the scale in
@@ -84,15 +85,19 @@ Per-task scores and run settings are in [`eval/bench/`](eval/bench/).
 
 ## Speed
 
-DGX Spark, batch 1, greedy, 512 generated tokens, median of 3 runs:
+exllamav3 v1.5.2 on a DGX Spark, batch 1, greedy, 512 generated tokens, median of 3 runs, in
+tok/s:
 
-| prompt | no drafter | 4 bpw drafter | speedup |
+| prompt | no drafter | DFlash drafter | MTP heads |
 |---|---|---|---|
-| coding | 31.4 tok/s | 48.5 tok/s | 1.54x |
-| prose | 31.4 tok/s | 40.2 tok/s | 1.28x |
-| reasoning | 31.4 tok/s | 59.5 tok/s | 1.89x |
+| coding | 31.5 | 49.5 | 40.7 |
+| prose | 31.3 | 35.4 | 31.8 |
+| reasoning | 31.2 | 61.1 | 45.1 |
+| code edit | 31.0 | 80.7 | 51.3 |
 
-The target model verifies every drafted token, so the drafter does not affect output quality.
+DFlash is the faster drafter on every prompt. The MTP heads draft 3 tokens per step; asking for
+6 was at most 5% faster. Prose gains the least and varies the most from prompt to prompt. The target model verifies every drafted token, so neither drafter
+affects output quality.
 
 Long context with the BF16 drafter, generating ~400 tokens of code against a large
 repository prompt:
@@ -152,6 +157,17 @@ draft_model:
   dynamic_draft: true
 ```
 
+To use the MTP heads instead of the DFlash drafter:
+
+```yaml
+draft_model:
+  draft_mode: mtp
+  dynamic_draft: true
+```
+
+For image input, add `vision: true` under `model:`. The vision tower adds about 1.3 GiB; with it
+and the DFlash drafter loaded at 262,144 context, a 250K-token prompt still left 15 GiB free.
+
 Thinking can be turned off per request with `"chat_template_kwargs": {"enable_thinking": false}`.
 Tool calls use the `qwen3_coder` format, which TabbyAPI detects automatically.