README.md

---
base_model:
- nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- nvfp4
- modelopt
- vllm
- speculative-decoding
- mtp
- dgx-spark
---

# Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP

NVFP4 quantisation of
[nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451](https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451),
built to run on a 128 GB DGX Spark, **with the MTP speculative-decoding head
intact**.

This is the same quantisation as
[PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4)
plus one 105 MB tensor that the other repo is missing. **If you want speculative
decoding, use this one.** If you do not, either works.

| | this repo (`-MTP`) | the plain `-NVFP4` repo |
|---|---|---|
| tensors | **2399** | 2398 |
| `mtp.fc.weight` | **present** | absent |
| runs `--speculative-config method=mtp` | **yes** | loads, then drafts garbage |
| size | 20.6 GB | 20.5 GB |

---

## Read this first: the missing tensor problem

The upstream bf16 repo was **re-uploaded on 2026-07-21** to add
`mtp.fc.weight` - the fusion projection the MTP head needs to combine its two
inputs. **A checkpoint pulled before that date has `mtp.layers.0.*` but no
`mtp.fc`, and this is a silent failure.**

vLLM disables strict weight-initialisation checking for quantized configs. The
missing tensor is allocated, nothing is loaded into it, and it keeps whatever
was in that memory. The MTP head then drafts from a **random projection**: no
exception, no warning, nothing in the log. Every drafted token is rejected, you
pay the drafting cost for nothing, and decode gets *slower*. The only symptom is
a number you were probably not measuring.

Check your own copy:

```python
import json, urllib.request
idx = json.load(urllib.request.urlopen(
    "https://huggingface.co/<repo>/resolve/main/model.safetensors.index.json"))
print("mtp.fc.weight:", "mtp.fc.weight" in idx["weight_map"])
```

`False` means MTP will not work for you, however healthy it looks.

## Serving it

Tested on vLLM `0.25.2.dev0` (arm64/CUDA 13 build), DGX Spark GB10, sm_121a.

**Plain, no speculative decoding:**

```bash
vllm serve PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP \
  --kv-cache-dtype fp8 --attention-backend flashinfer \
  --max-model-len 131072 --gpu-memory-utilization 0.72 --max-num-seqs 64 \
  --reasoning-parser qwen3 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml
```

**With MTP (recommended):** add

```bash
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
```

No separate draft model - vLLM rewrites this checkpoint's own config into a
`Qwen3_5MTP` draft and reads the head out of these weights. It is a reasoning
model, so pass `chat_template_kwargs: {"enable_thinking": false}` if you do not
want the budget spent inside `<think>`.

**With DFlash** (much faster on file-editing work, much slower on prose, and it
needs a patched draft because the published one does not load on vLLM at all):
see the recipe repo linked below.

---

## Is speculative decoding worth it? Measured, not guessed

540 measurements on this checkpoint: 3 configs x 3 workloads x 10 concurrency
levels x 6 shuffled rounds, all clean, standard deviation mostly under 1%.

### The plain-English version

An AI writes one word at a time, and each word means re-reading the whole model
from memory. That memory trip is the slow part. **Speculative decoding** adds a
small fast helper that guesses the next few words so the big model can check
them in one go. Right guesses are free. Wrong guesses are thrown away and cost
you time.

**So it is a bet on whether the next words are predictable.**

- **MTP** guesses **2 words** ahead and is usually right. Small bet, reliable.
- **DFlash** guesses **15 words** ahead at once. Astonishing when the text is
  predictable; wasteful when it is not.

Text is predictable when the model is mostly **copying something you just gave
it** - "here is my file, rename this variable" - which is what coding assistants
do all day. It is unpredictable when the model is **writing something new**.

How often each helper guessed right, on this model:

| | writing prose | writing code | editing a file |
|---|---|---|---|
| **MTP** | 47% | 76% | **100%** |
| **DFlash** | **6%** | 19% | 83% |

**DFlash guessing right 6% of the time on prose is the whole reason it is not a
magic speed-up button.**

![draft acceptance rate by workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c5.svg)

Rank the workloads by that hit rate and you have ranked them by outcome, for
both methods, at every concurrency level. It is not really about concurrency.

### MTP is a safe upgrade. Use it by default.

Aggregate tokens/sec vs no speculative decoding:

| workload | MTP vs no-spec |
|---|---|
| **code** | wins at **every** concurrency level tested, up to **+70%** |
| **edit** | wins at **every** level, up to **+104%** |
| prose | wins at 1-8, 24, 32 streams (up to +32%); loses at 12 (-11%), 16 (-10%), 64 (-7%) |

It costs about **10% of the KV cache pool** and nothing else. On code and editing
there is no concurrency at which it is the wrong choice. On free-form prose above
8 concurrent streams it is roughly a coin flip - that is where its acceptance is
weakest (47%).

**Writing code** - MTP (orange) leads base (blue) at every level:

![aggregate throughput, code workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c1-code.svg)

**Editing files** - DFlash (green) is in another league, MTP still well clear of base:

![aggregate throughput, edit workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c1-edit.svg)

**Free-form prose** - DFlash drops below *doing nothing* from 4 streams up:

![aggregate throughput, prose workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c1-prose.svg)

### DFlash is a specialist tool with a sharp edge

Single stream on editing work it is **+716%** over no speculation and **+311%**
over MTP. That is real. But:

- On **prose** it is beaten by *doing nothing* from 4 concurrent streams up.
- It costs **40%** of the KV pool (second model + 15 draft slots per stream).
- It caps at about **22 concurrent streams** on this hardware and queues the
  rest - median time to first token at 64 streams goes to **138 seconds**.

Ask for 64 streams and count how many actually run at once. Base and MTP track
the diagonal; DFlash flattens at 22, so its throughput lines beyond that point
are a capacity limit rather than a speed result:

![concurrency achieved vs requested](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c2c.svg)

**Rule of thumb: DFlash below its ~22-stream ceiling for edit-heavy agentic
coding; MTP for everything else and for crowds.**

Full charts, tables, the 540-cell dataset and the fix that makes DFlash load on
vLLM at all:

**[-> Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4_Dflash_DGX_recipe](https://github.com/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4_Dflash_DGX_recipe)**

### A warning about averages

Averaging prose/code/edit assumes editing is exactly a third of your traffic.
**72% of DFlash's average comes from the edit workload alone.** Drop editing and
average only prose and code, and MTP wins from 4 streams up. Measure your own
mix instead - acceptance rate tells you directly:

```bash
curl -s http://127.0.0.1:8000/metrics \
  | grep -E 'spec_decode_num_(accepted|draft)_tokens_total'
```

Near 0.06 is prose-like (do not use DFlash); near 0.83 is edit-like (do).

---

## How this was made

Source: `nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451` - bf16, 1199
tensors, 55,457,998,304 bytes, a `nuslerp` mergekit merge of two Qwen3.6-27B
Architect-Polaris Fable variants. Uploaded with nightmedia's permission.

Quantised with **NVIDIA ModelOpt 0.45.0**, `examples/llm_ptq/hf_ptq.py`:

```
--qformat nvfp4 --calib_size 512 --dataset cnn_dailymail
--batch_size 0 --inference_tensor_parallel 1
```

ModelOpt auto-excludes `linear_attn`, the vision tower, `mtp.*`, embeddings and
`lm_head` from quantisation - see `quant_summary.txt` in this repo for the
per-layer record, and `hf_quant_config.json` for the exclusion patterns.
Output: 2398 tensors, 20,487,991,904 bytes.

`mtp.fc.weight` ([5120, 10240] BF16, 104,857,600 bytes) was then taken from the
corrected upstream revision and added as `model-mtp-fc.safetensors`, with the
index updated - **2399 tensors, 20,592,849,504 bytes**. vLLM's own source
expects this tensor to be unquantized in NVFP4 checkpoints
(`qwen3_5_mtp.py`: *"mtp.fc is stored as BF16 in NVFP4 checkpoints ... Force
unquantized"*), so a spliced BF16 tensor is exactly what a fresh PTQ run would
have emitted.

The three VL processor configs (`preprocessor_config.json`,
`processor_config.json`, `video_preprocessor_config.json`) are included -
ModelOpt does not emit them and vLLM hard-fails a VL model at init without them.

## Limitations

- **Measured on one machine, one model.** DGX Spark GB10, 121.7 GiB unified
  memory, sm_121a, arm64, a community arm64/CUDA-13 vLLM build. No claim about
  other hardware, multi-node, or other quantizations.
- **Acceptance rates are specific to this checkpoint.** The MTP head was
  inherited through a four-generation mergekit merge and never retrained. A
  different model will have different acceptance, and therefore different
  crossover points.
- **No quality evaluation was run here.** NVFP4 is a lossy 4-bit quantisation.
  These numbers are throughput only; nothing in this card says the output is as
  good as the bf16 source.
- **Benchmarks ran with thinking disabled** and 512-token generations. Long
  reasoning traces were not measured.
- `min_p` and `logit_bias` do not work under speculative decoding (vLLM warns at
  startup).

## Credits

- **[nightmedia](https://huggingface.co/nightmedia)** - the bf16 merge this is
  quantised from, and its Architect-Polaris merge stages. All the model work is
  theirs; uploaded with their permission. That merge in turn credits
  Qwen/Qwen3.6-27B and its own upstreams.
- **[z-lab](https://huggingface.co/z-lab)** - the DFlash draft used in the
  comparison.
- **NVIDIA ModelOpt** - the PTQ pipeline.
- **vLLM** - the serving engine, including the `Qwen3_5MTP` support that makes
  the MTP head usable at all.