--- base_model: - nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451 library_name: transformers pipeline_tag: image-text-to-text tags: - nvfp4 - modelopt - vllm - speculative-decoding - mtp - dgx-spark --- # Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP NVFP4 quantisation of [nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451](https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451), built to run on a 128 GB DGX Spark, **with the MTP speculative-decoding head intact**. This is the same quantisation as [PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4) plus one 105 MB tensor that the other repo is missing. **If you want speculative decoding, use this one.** If you do not, either works. | | this repo (`-MTP`) | the plain `-NVFP4` repo | |---|---|---| | tensors | **2399** | 2398 | | `mtp.fc.weight` | **present** | absent | | runs `--speculative-config method=mtp` | **yes** | loads, then drafts garbage | | size | 20.6 GB | 20.5 GB | --- ## Read this first: the missing tensor problem The upstream bf16 repo was **re-uploaded on 2026-07-21** to add `mtp.fc.weight` - the fusion projection the MTP head needs to combine its two inputs. **A checkpoint pulled before that date has `mtp.layers.0.*` but no `mtp.fc`, and this is a silent failure.** vLLM disables strict weight-initialisation checking for quantized configs. The missing tensor is allocated, nothing is loaded into it, and it keeps whatever was in that memory. The MTP head then drafts from a **random projection**: no exception, no warning, nothing in the log. Every drafted token is rejected, you pay the drafting cost for nothing, and decode gets *slower*. The only symptom is a number you were probably not measuring. Check your own copy: ```python import json, urllib.request idx = json.load(urllib.request.urlopen( "https://huggingface.co//resolve/main/model.safetensors.index.json")) print("mtp.fc.weight:", "mtp.fc.weight" in idx["weight_map"]) ``` `False` means MTP will not work for you, however healthy it looks. ## Serving it Tested on vLLM `0.25.2.dev0` (arm64/CUDA 13 build), DGX Spark GB10, sm_121a. **Plain, no speculative decoding:** ```bash vllm serve PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP \ --kv-cache-dtype fp8 --attention-backend flashinfer \ --max-model-len 131072 --gpu-memory-utilization 0.72 --max-num-seqs 64 \ --reasoning-parser qwen3 --trust-remote-code \ --enable-auto-tool-choice --tool-call-parser qwen3_xml ``` **With MTP (recommended):** add ```bash --speculative-config '{"method":"mtp","num_speculative_tokens":2}' ``` No separate draft model - vLLM rewrites this checkpoint's own config into a `Qwen3_5MTP` draft and reads the head out of these weights. It is a reasoning model, so pass `chat_template_kwargs: {"enable_thinking": false}` if you do not want the budget spent inside ``. **With DFlash** (much faster on file-editing work, much slower on prose, and it needs a patched draft because the published one does not load on vLLM at all): see the recipe repo linked below. --- ## Is speculative decoding worth it? Measured, not guessed 540 measurements on this checkpoint: 3 configs x 3 workloads x 10 concurrency levels x 6 shuffled rounds, all clean, standard deviation mostly under 1%. ### The plain-English version An AI writes one word at a time, and each word means re-reading the whole model from memory. That memory trip is the slow part. **Speculative decoding** adds a small fast helper that guesses the next few words so the big model can check them in one go. Right guesses are free. Wrong guesses are thrown away and cost you time. **So it is a bet on whether the next words are predictable.** - **MTP** guesses **2 words** ahead and is usually right. Small bet, reliable. - **DFlash** guesses **15 words** ahead at once. Astonishing when the text is predictable; wasteful when it is not. Text is predictable when the model is mostly **copying something you just gave it** - "here is my file, rename this variable" - which is what coding assistants do all day. It is unpredictable when the model is **writing something new**. How often each helper guessed right, on this model: | | writing prose | writing code | editing a file | |---|---|---|---| | **MTP** | 47% | 76% | **100%** | | **DFlash** | **6%** | 19% | 83% | **DFlash guessing right 6% of the time on prose is the whole reason it is not a magic speed-up button.** ![draft acceptance rate by workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c5.svg) Rank the workloads by that hit rate and you have ranked them by outcome, for both methods, at every concurrency level. It is not really about concurrency. ### MTP is a safe upgrade. Use it by default. Aggregate tokens/sec vs no speculative decoding: | workload | MTP vs no-spec | |---|---| | **code** | wins at **every** concurrency level tested, up to **+70%** | | **edit** | wins at **every** level, up to **+104%** | | prose | wins at 1-8, 24, 32 streams (up to +32%); loses at 12 (-11%), 16 (-10%), 64 (-7%) | It costs about **10% of the KV cache pool** and nothing else. On code and editing there is no concurrency at which it is the wrong choice. On free-form prose above 8 concurrent streams it is roughly a coin flip - that is where its acceptance is weakest (47%). **Writing code** - MTP (orange) leads base (blue) at every level: ![aggregate throughput, code workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c1-code.svg) **Editing files** - DFlash (green) is in another league, MTP still well clear of base: ![aggregate throughput, edit workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c1-edit.svg) **Free-form prose** - DFlash drops below *doing nothing* from 4 streams up: ![aggregate throughput, prose workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c1-prose.svg) ### DFlash is a specialist tool with a sharp edge Single stream on editing work it is **+716%** over no speculation and **+311%** over MTP. That is real. But: - On **prose** it is beaten by *doing nothing* from 4 concurrent streams up. - It costs **40%** of the KV pool (second model + 15 draft slots per stream). - It caps at about **22 concurrent streams** on this hardware and queues the rest - median time to first token at 64 streams goes to **138 seconds**. Ask for 64 streams and count how many actually run at once. Base and MTP track the diagonal; DFlash flattens at 22, so its throughput lines beyond that point are a capacity limit rather than a speed result: ![concurrency achieved vs requested](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c2c.svg) **Rule of thumb: DFlash below its ~22-stream ceiling for edit-heavy agentic coding; MTP for everything else and for crowds.** Full charts, tables, the 540-cell dataset and the fix that makes DFlash load on vLLM at all: **[-> Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4_Dflash_DGX_recipe](https://github.com/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4_Dflash_DGX_recipe)** ### A warning about averages Averaging prose/code/edit assumes editing is exactly a third of your traffic. **72% of DFlash's average comes from the edit workload alone.** Drop editing and average only prose and code, and MTP wins from 4 streams up. Measure your own mix instead - acceptance rate tells you directly: ```bash curl -s http://127.0.0.1:8000/metrics \ | grep -E 'spec_decode_num_(accepted|draft)_tokens_total' ``` Near 0.06 is prose-like (do not use DFlash); near 0.83 is edit-like (do). --- ## How this was made Source: `nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451` - bf16, 1199 tensors, 55,457,998,304 bytes, a `nuslerp` mergekit merge of two Qwen3.6-27B Architect-Polaris Fable variants. Uploaded with nightmedia's permission. Quantised with **NVIDIA ModelOpt 0.45.0**, `examples/llm_ptq/hf_ptq.py`: ``` --qformat nvfp4 --calib_size 512 --dataset cnn_dailymail --batch_size 0 --inference_tensor_parallel 1 ``` ModelOpt auto-excludes `linear_attn`, the vision tower, `mtp.*`, embeddings and `lm_head` from quantisation - see `quant_summary.txt` in this repo for the per-layer record, and `hf_quant_config.json` for the exclusion patterns. Output: 2398 tensors, 20,487,991,904 bytes. `mtp.fc.weight` ([5120, 10240] BF16, 104,857,600 bytes) was then taken from the corrected upstream revision and added as `model-mtp-fc.safetensors`, with the index updated - **2399 tensors, 20,592,849,504 bytes**. vLLM's own source expects this tensor to be unquantized in NVFP4 checkpoints (`qwen3_5_mtp.py`: *"mtp.fc is stored as BF16 in NVFP4 checkpoints ... Force unquantized"*), so a spliced BF16 tensor is exactly what a fresh PTQ run would have emitted. The three VL processor configs (`preprocessor_config.json`, `processor_config.json`, `video_preprocessor_config.json`) are included - ModelOpt does not emit them and vLLM hard-fails a VL model at init without them. ## Limitations - **Measured on one machine, one model.** DGX Spark GB10, 121.7 GiB unified memory, sm_121a, arm64, a community arm64/CUDA-13 vLLM build. No claim about other hardware, multi-node, or other quantizations. - **Acceptance rates are specific to this checkpoint.** The MTP head was inherited through a four-generation mergekit merge and never retrained. A different model will have different acceptance, and therefore different crossover points. - **No quality evaluation was run here.** NVFP4 is a lossy 4-bit quantisation. These numbers are throughput only; nothing in this card says the output is as good as the bf16 source. - **Benchmarks ran with thinking disabled** and 512-token generations. Long reasoning traces were not measured. - `min_p` and `logit_bias` do not work under speculative decoding (vLLM warns at startup). ## Credits - **[nightmedia](https://huggingface.co/nightmedia)** - the bf16 merge this is quantised from, and its Architect-Polaris merge stages. All the model work is theirs; uploaded with their permission. That merge in turn credits Qwen/Qwen3.6-27B and its own upstreams. - **[z-lab](https://huggingface.co/z-lab)** - the DFlash draft used in the comparison. - **NVIDIA ModelOpt** - the PTQ pipeline. - **vLLM** - the serving engine, including the `Qwen3_5MTP` support that makes the MTP head usable at all.