Upload README.md with huggingface_hub

README.md

@@ -18,11 +18,11 @@ Mixed-precision **INT4 (W4A16)** quantization of [`Qwen/Qwen3.8-27B`](https://hu
 built with [llm-compressor](https://github.com/vllm-project/llm-compressor) using
 **AWQ activation-aware scaling followed by GPTQ**.
 
-**21 GB, and measurably more faithful to the BF16 model than the official FP8 release**
+**22 GB, and measurably more faithful to the BF16 model than the official FP8 release**
 on high-confidence token agreement — at 8 GB less. Runs on anything Turing or newer
 (no FP8/FP4 hardware required).
 
-This is the size-optimised variant. A 23 GB sibling,
+This is the size-optimised variant. A 24 GB sibling,
 [`Qwen3.8-27B-INT4-AWQ-GPTQ`](https://huggingface.co/TelperionAI/Qwen3.8-27B-INT4-AWQ-GPTQ),
 keeps the GDN projections at 8-bit and is more accurate again (0.93% vs 1.15% confident).
 
@@ -56,12 +56,12 @@ Against the BF16 base on 142,727 tokens of self-distilled thinking-mode output p
 | checkpoint | size ↓ | top-1 ↑ | near-tie ↓ | moderate ↓ | confident ↓ | certain ↓ | divmed ↑ | tok/s ↑ |
 |---|---:|---:|---:|---:|---:|---:|---:|---:|
 | `Qwen/Qwen3.8-27B-FP8` *(8-bit ref)* | 29 GB | 96.15% | 22.70% | 3.48% | 1.45% | 0.08% | 47 | 8711 |
-| **this model (GDN 4-bit)** | 21 GB | **95.88%** | **25.26%** | **3.60%** | **1.15%** | 0.13% | **51** | 4716 |
-| our 23 GB sibling (GDN 8-bit) | 23 GB | 96.30% | 22.29% | 3.52% | 0.93% | 0.09% | 48 | 4617 |
+| **this model (GDN 4-bit)** | 22 GB | **95.88%** | **25.26%** | **3.60%** | **1.15%** | 0.13% | **51** | 4716 |
+| our 24 GB sibling (GDN 8-bit) | 24 GB | 96.30% | 22.29% | 3.52% | 0.93% | 0.09% | 48 | 4617 |
 | `cyankiwi/Qwen3.8-27B-AWQ-INT4` | **20 GB** | 94.35% | 33.74% | 5.59% | 1.35% | **0.12%** | 29 | 4787 |
 
-Bold marks the best value in each column **among the ~20–21 GB checkpoints**; the FP8 row and
-our own 23 GB sibling sit in different size classes and are shown for reference only.
+Bold marks the best value in each column **among the ~20–22 GB checkpoints**; the FP8 row and
+our own 24 GB sibling sit in different size classes and are shown for reference only.
 
 **Columns.** `top-1` is raw argmax agreement with BF16. The four bucket columns are
 *disagreement* rates, split by how confident the base model was at that position
@@ -76,7 +76,7 @@ damage. Do not rank quantizations of this model by perplexity.
 
 ### What the GDN precision costs
 
-Moving the GDN projections from 8-bit to 4-bit is what takes this build from 23 GB to 21 GB.
+Moving the GDN projections from 8-bit to 4-bit is what takes this build from 24 GB to 22 GB.
 It costs `confident` **0.93% → 1.15%** and `certain` **0.09% → 0.13%** — real but modest.
 Note the two are not a pure bit-width comparison: at 4-bit the GDN projections also move
 into GPTQ's error compensation, where at 8-bit they were plain round-to-nearest, so the
@@ -109,13 +109,32 @@ W4A16 needs only compute capability ≥ 7.5 (Turing), so this runs on 3090/4090/
 as well as Blackwell. Throughput above was measured on B300 and is not representative of
 Ampere, where Marlin INT4 is comparatively better tuned.
 
+## Speculative decoding (MTP)
+
+The model's **MTP (multi-token prediction) head is included**, in BF16, and works with
+vLLM's `mtp` speculative decoding:
+
+```python
+from vllm import LLM
+llm = LLM("TelperionAI/Qwen3.8-27B-INT4-AWQ-GPTQ-gdn4", tensor_parallel_size=2,
+          speculative_config={"method": "mtp", "num_speculative_tokens": 2})
+```
+
+`Qwen3_5ForConditionalGeneration` does not carry `mtp.*` in its state dict, so
+llm-compressor never sees it and it is absent from most community quantizations of this
+model — even though `config.json` declares `mtp_num_hidden_layers: 1`. It is grafted back
+in here from the base checkpoint and excluded from quantization (`re:.*mtp.*` in
+`quantization_config.ignore`; without that exclusion the quantization target regexes also
+match `mtp.layers.0.mlp.*` and vLLM fails to load). Draft quality drives acceptance rate,
+so it is kept at full precision rather than quantized.
+
+Acceptance rate has not been measured; the head is verified to load and generate.
+
 ## Limitations
 
-- **21 GB is still tight on a single 24 GB card** once KV cache is accounted for.
+- **22 GB is still tight on a single 24 GB card** once KV cache is accounted for.
   Quantizing `lm_head` would free a further ~1.3 GB but was not measured here.
 - **Throughput on Blackwell is ~2.3× below NVFP4.** This checkpoint trades speed for fidelity.
-- **The MTP head is not included** — `Qwen3_5ForConditionalGeneration` does not carry it in
-  its state dict, so it is dropped during quantization. MTP speculative decoding is unavailable.
 - **Single evaluation corpus.** All numbers come from one self-distilled corpus. The margins
   over FP8 and cyankiwi are statistically solid but have not been replicated on a second
   distribution, nor on downstream task benchmarks.