Name the checkpoint correctly: Fable-B-F451-NVFP4 (F451 is part of the source merge identity)

README.md

@@ -1,256 +1,254 @@
----
-base_model:
-- nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451
-library_name: transformers
-pipeline_tag: image-text-to-text
-tags:
-- nvfp4
-- modelopt
-- vllm
-- speculative-decoding
-- mtp
-- dgx-spark
----
-
-# Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4-MTP
-
-NVFP4 quantisation of
-[nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451](https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451),
-built to run on a 128 GB DGX Spark, All of the actual model work here belongs to nightmedia, who built the base model this is quantized from. That model is itself a merge sitting on top of a longer chain of work by other people in the community: **with the MTP speculative-decoding head
-intact**.
-
-This is the same quantisation as
-[PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4)
-plus one 105 MB tensor that the other repo is missing. **If you want speculative
-decoding, use this one.** If you do not, either works.
-
-| | this repo (`-MTP`) | the plain `-NVFP4` repo |
-|---|---|---|
-| tensors | **2399** | 2398 |
-| `mtp.fc.weight` | **present** | absent |
-| runs `--speculative-config method=mtp` | **yes** | loads, then drafts garbage |
-| size | 20.6 GB | 20.5 GB |
-
-DFLASH Recipe can be found here 
-**[-> Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4_Dflash_DGX_recipe](https://github.com/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4_Dflash_DGX_recipe)**
----
-
-## Read this first: the missing tensor problem
-
-The upstream bf16 repo was **re-uploaded on 2026-07-21** to add
-`mtp.fc.weight` - the fusion projection the MTP head needs to combine its two
-inputs. **A checkpoint pulled before that date has `mtp.layers.0.*` but no
-`mtp.fc`, and this is a silent failure.**
-
-vLLM disables strict weight-initialisation checking for quantized configs. The
-missing tensor is allocated, nothing is loaded into it, and it keeps whatever
-was in that memory. The MTP head then drafts from a **random projection**: no
-exception, no warning, nothing in the log. Every drafted token is rejected, you
-pay the drafting cost for nothing, and decode gets *slower*. The only symptom is
-a number you were probably not measuring.
-
-Check your own copy:
-
-```python
-import json, urllib.request
-idx = json.load(urllib.request.urlopen(
-    "https://huggingface.co/<repo>/resolve/main/model.safetensors.index.json"))
-print("mtp.fc.weight:", "mtp.fc.weight" in idx["weight_map"])
-```
-
-`False` means MTP will not work for you, however healthy it looks.
-
-## Serving it
-
-Tested on vLLM `0.25.2.dev0` (arm64/CUDA 13 build), DGX Spark GB10, sm_121a.
-
-**Plain, no speculative decoding:**
-
-```bash
-vllm serve PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4-MTP \
-  --kv-cache-dtype fp8 --attention-backend flashinfer \
-  --max-model-len 131072 --gpu-memory-utilization 0.72 --max-num-seqs 64 \
-  --reasoning-parser qwen3 --trust-remote-code \
-  --enable-auto-tool-choice --tool-call-parser qwen3_xml
-```
-
-**With MTP (recommended):** add
-
-```bash
-  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
-```
-
-No separate draft model - vLLM rewrites this checkpoint's own config into a
-`Qwen3_5MTP` draft and reads the head out of these weights. It is a reasoning
-model, so pass `chat_template_kwargs: {"enable_thinking": false}` if you do not
-want the budget spent inside `<think>`.
-
-**With DFlash** (much faster on file-editing work, much slower on prose, and it
-needs a patched draft because the published one does not load on vLLM at all):
-see the recipe repo linked below.
-
----
-
-## Is speculative decoding worth it? Measured, not guessed
-
-540 measurements on this checkpoint: 3 configs x 3 workloads x 10 concurrency
-levels x 6 shuffled rounds, all clean, standard deviation mostly under 1%.
-
-### The plain-English version
-
-An AI writes one word at a time, and each word means re-reading the whole model
-from memory. That memory trip is the slow part. **Speculative decoding** adds a
-small fast helper that guesses the next few words so the big model can check
-them in one go. Right guesses are free. Wrong guesses are thrown away and cost
-you time.
-
-**So it is a bet on whether the next words are predictable.**
-
-- **MTP** guesses **2 words** ahead and is usually right. Small bet, reliable.
-- **DFlash** guesses **15 words** ahead at once. Astonishing when the text is
-  predictable; wasteful when it is not.
-
-Text is predictable when the model is mostly **copying something you just gave
-it** - "here is my file, rename this variable" - which is what coding assistants
-do all day. It is unpredictable when the model is **writing something new**.
-
-How often each helper guessed right, on this model:
-
-| | writing prose | writing code | editing a file |
-|---|---|---|---|
-| **MTP** | 47% | 76% | **100%** |
-| **DFlash** | **6%** | 19% | 83% |
-
-**DFlash guessing right 6% of the time on prose is the whole reason it is not a
-magic speed-up button.**
-
-![draft acceptance rate by workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4-MTP/resolve/main/charts/c5.svg)
-
-Rank the workloads by that hit rate and you have ranked them by outcome, for
-both methods, at every concurrency level. It is not really about concurrency.
-
-### MTP is a safe upgrade. Use it by default.
-
-Aggregate tokens/sec vs no speculative decoding:
-
-| workload | MTP vs no-spec |
-|---|---|
-| **code** | wins at **every** concurrency level tested, up to **+70%** |
-| **edit** | wins at **every** level, up to **+104%** |
-| prose | wins at 1-8, 24, 32 streams (up to +32%); loses at 12 (-11%), 16 (-10%), 64 (-7%) |
-
-It costs about **10% of the KV cache pool** and nothing else. On code and editing
-there is no concurrency at which it is the wrong choice. On free-form prose above
-8 concurrent streams it is roughly a coin flip - that is where its acceptance is
-weakest (47%).
-
-**Writing code** - MTP (orange) leads base (blue) at every level:
-
-![aggregate throughput, code workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4-MTP/resolve/main/charts/c1-code.svg)
-
-**Editing files** - DFlash (green) is in another league, MTP still well clear of base:
-
-![aggregate throughput, edit workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4-MTP/resolve/main/charts/c1-edit.svg)
-
-**Free-form prose** - DFlash drops below *doing nothing* from 4 streams up:
-
-![aggregate throughput, prose workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4-MTP/resolve/main/charts/c1-prose.svg)
-
-### DFlash is a specialist tool with a sharp edge
-
-Single stream on editing work it is **+716%** over no speculation and **+311%**
-over MTP. That is real. But:
-
-- On **prose** it is beaten by *doing nothing* from 4 concurrent streams up.
-- It costs **40%** of the KV pool (second model + 15 draft slots per stream).
-- It caps at about **22 concurrent streams** on this hardware and queues the
-  rest - median time to first token at 64 streams goes to **138 seconds**.
-
-Ask for 64 streams and count how many actually run at once. Base and MTP track
-the diagonal; DFlash flattens at 22, so its throughput lines beyond that point
-are a capacity limit rather than a speed result:
-
-![concurrency achieved vs requested](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4-MTP/resolve/main/charts/c2c.svg)
-
-**Rule of thumb: DFlash below its ~22-stream ceiling for edit-heavy agentic
-coding; MTP for everything else and for crowds.**
-
-Full charts, tables, the 540-cell dataset and the fix that makes DFlash load on
-vLLM at all:
-
-**[-> Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4_Dflash_DGX_recipe](https://github.com/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4_Dflash_DGX_recipe)**
-
-### A warning about averages
-
-Averaging prose/code/edit assumes editing is exactly a third of your traffic.
-**72% of DFlash's average comes from the edit workload alone.** Drop editing and
-average only prose and code, and MTP wins from 4 streams up. Measure your own
-mix instead - acceptance rate tells you directly:
-
-```bash
-curl -s http://127.0.0.1:8000/metrics \
-  | grep -E 'spec_decode_num_(accepted|draft)_tokens_total'
-```
-
-Near 0.06 is prose-like (do not use DFlash); near 0.83 is edit-like (do).
-
----
-
-## How this was made
-
-Source: `nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451` - bf16, 1199
-tensors, 55,457,998,304 bytes, a `nuslerp` mergekit merge of two Qwen3.6-27B
-Architect-Polaris Fable variants. Uploaded with nightmedia's permission.
-
-Quantised with **NVIDIA ModelOpt 0.45.0**, `examples/llm_ptq/hf_ptq.py`:
-
-```
---qformat nvfp4 --calib_size 512 --dataset cnn_dailymail
---batch_size 0 --inference_tensor_parallel 1
-```
-
-ModelOpt auto-excludes `linear_attn`, the vision tower, `mtp.*`, embeddings and
-`lm_head` from quantisation - see `quant_summary.txt` in this repo for the
-per-layer record, and `hf_quant_config.json` for the exclusion patterns.
-Output: 2398 tensors, 20,487,991,904 bytes.
-
-`mtp.fc.weight` ([5120, 10240] BF16, 104,857,600 bytes) was then taken from the
-corrected upstream revision and added as `model-mtp-fc.safetensors`, with the
-index updated - **2399 tensors, 20,592,849,504 bytes**. vLLM's own source
-expects this tensor to be unquantized in NVFP4 checkpoints
-(`qwen3_5_mtp.py`: *"mtp.fc is stored as BF16 in NVFP4 checkpoints ... Force
-unquantized"*), so a spliced BF16 tensor is exactly what a fresh PTQ run would
-have emitted.
-
-The three VL processor configs (`preprocessor_config.json`,
-`processor_config.json`, `video_preprocessor_config.json`) are included -
-ModelOpt does not emit them and vLLM hard-fails a VL model at init without them.
-
-## Limitations
-
-- **Measured on one machine, one model.** DGX Spark GB10, 121.7 GiB unified
-  memory, sm_121a, arm64, a community arm64/CUDA-13 vLLM build. No claim about
-  other hardware, multi-node, or other quantizations.
-- **Acceptance rates are specific to this checkpoint.** The MTP head was
-  inherited through a four-generation mergekit merge and never retrained. A
-  different model will have different acceptance, and therefore different
-  crossover points.
-- **No quality evaluation was run here.** NVFP4 is a lossy 4-bit quantisation.
-  These numbers are throughput only; nothing in this card says the output is as
-  good as the bf16 source.
-- **Benchmarks ran with thinking disabled** and 512-token generations. Long
-  reasoning traces were not measured.
-- `min_p` and `logit_bias` do not work under speculative decoding (vLLM warns at
-  startup).
-
-## Credits
-
-- **[nightmedia](https://huggingface.co/nightmedia)** - the bf16 merge this is
-  quantised from, and its Architect-Polaris merge stages. All the model work is
-  theirs; uploaded with their permission. That merge in turn credits
-  Qwen/Qwen3.6-27B and its own upstreams.
-- **[z-lab](https://huggingface.co/z-lab)** - the DFlash draft used in the
-  comparison.
-- **NVIDIA ModelOpt** - the PTQ pipeline.
-- **vLLM** - the serving engine, including the `Qwen3_5MTP` support that makes
-  the MTP head usable at all.
+---
+base_model:
+- nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451
+library_name: transformers
+pipeline_tag: image-text-to-text
+tags:
+- nvfp4
+- modelopt
+- vllm
+- speculative-decoding
+- mtp
+- dgx-spark
+---
+
+# Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP
+
+NVFP4 quantisation of
+[nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451](https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451),
+built to run on a 128 GB DGX Spark, **with the MTP speculative-decoding head
+intact**.
+
+This is the same quantisation as
+[PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4)
+plus one 105 MB tensor that the other repo is missing. **If you want speculative
+decoding, use this one.** If you do not, either works.
+
+| | this repo (`-MTP`) | the plain `-NVFP4` repo |
+|---|---|---|
+| tensors | **2399** | 2398 |
+| `mtp.fc.weight` | **present** | absent |
+| runs `--speculative-config method=mtp` | **yes** | loads, then drafts garbage |
+| size | 20.6 GB | 20.5 GB |
+
+---
+
+## Read this first: the missing tensor problem
+
+The upstream bf16 repo was **re-uploaded on 2026-07-21** to add
+`mtp.fc.weight` - the fusion projection the MTP head needs to combine its two
+inputs. **A checkpoint pulled before that date has `mtp.layers.0.*` but no
+`mtp.fc`, and this is a silent failure.**
+
+vLLM disables strict weight-initialisation checking for quantized configs. The
+missing tensor is allocated, nothing is loaded into it, and it keeps whatever
+was in that memory. The MTP head then drafts from a **random projection**: no
+exception, no warning, nothing in the log. Every drafted token is rejected, you
+pay the drafting cost for nothing, and decode gets *slower*. The only symptom is
+a number you were probably not measuring.
+
+Check your own copy:
+
+```python
+import json, urllib.request
+idx = json.load(urllib.request.urlopen(
+    "https://huggingface.co/<repo>/resolve/main/model.safetensors.index.json"))
+print("mtp.fc.weight:", "mtp.fc.weight" in idx["weight_map"])
+```
+
+`False` means MTP will not work for you, however healthy it looks.
+
+## Serving it
+
+Tested on vLLM `0.25.2.dev0` (arm64/CUDA 13 build), DGX Spark GB10, sm_121a.
+
+**Plain, no speculative decoding:**
+
+```bash
+vllm serve PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP \
+  --kv-cache-dtype fp8 --attention-backend flashinfer \
+  --max-model-len 131072 --gpu-memory-utilization 0.72 --max-num-seqs 64 \
+  --reasoning-parser qwen3 --trust-remote-code \
+  --enable-auto-tool-choice --tool-call-parser qwen3_xml
+```
+
+**With MTP (recommended):** add
+
+```bash
+  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
+```
+
+No separate draft model - vLLM rewrites this checkpoint's own config into a
+`Qwen3_5MTP` draft and reads the head out of these weights. It is a reasoning
+model, so pass `chat_template_kwargs: {"enable_thinking": false}` if you do not
+want the budget spent inside `<think>`.
+
+**With DFlash** (much faster on file-editing work, much slower on prose, and it
+needs a patched draft because the published one does not load on vLLM at all):
+see the recipe repo linked below.
+
+---
+
+## Is speculative decoding worth it? Measured, not guessed
+
+540 measurements on this checkpoint: 3 configs x 3 workloads x 10 concurrency
+levels x 6 shuffled rounds, all clean, standard deviation mostly under 1%.
+
+### The plain-English version
+
+An AI writes one word at a time, and each word means re-reading the whole model
+from memory. That memory trip is the slow part. **Speculative decoding** adds a
+small fast helper that guesses the next few words so the big model can check
+them in one go. Right guesses are free. Wrong guesses are thrown away and cost
+you time.
+
+**So it is a bet on whether the next words are predictable.**
+
+- **MTP** guesses **2 words** ahead and is usually right. Small bet, reliable.
+- **DFlash** guesses **15 words** ahead at once. Astonishing when the text is
+  predictable; wasteful when it is not.
+
+Text is predictable when the model is mostly **copying something you just gave
+it** - "here is my file, rename this variable" - which is what coding assistants
+do all day. It is unpredictable when the model is **writing something new**.
+
+How often each helper guessed right, on this model:
+
+| | writing prose | writing code | editing a file |
+|---|---|---|---|
+| **MTP** | 47% | 76% | **100%** |
+| **DFlash** | **6%** | 19% | 83% |
+
+**DFlash guessing right 6% of the time on prose is the whole reason it is not a
+magic speed-up button.**
+
+![draft acceptance rate by workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c5.svg)
+
+Rank the workloads by that hit rate and you have ranked them by outcome, for
+both methods, at every concurrency level. It is not really about concurrency.
+
+### MTP is a safe upgrade. Use it by default.
+
+Aggregate tokens/sec vs no speculative decoding:
+
+| workload | MTP vs no-spec |
+|---|---|
+| **code** | wins at **every** concurrency level tested, up to **+70%** |
+| **edit** | wins at **every** level, up to **+104%** |
+| prose | wins at 1-8, 24, 32 streams (up to +32%); loses at 12 (-11%), 16 (-10%), 64 (-7%) |
+
+It costs about **10% of the KV cache pool** and nothing else. On code and editing
+there is no concurrency at which it is the wrong choice. On free-form prose above
+8 concurrent streams it is roughly a coin flip - that is where its acceptance is
+weakest (47%).
+
+**Writing code** - MTP (orange) leads base (blue) at every level:
+
+![aggregate throughput, code workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c1-code.svg)
+
+**Editing files** - DFlash (green) is in another league, MTP still well clear of base:
+
+![aggregate throughput, edit workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c1-edit.svg)
+
+**Free-form prose** - DFlash drops below *doing nothing* from 4 streams up:
+
+![aggregate throughput, prose workload](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c1-prose.svg)
+
+### DFlash is a specialist tool with a sharp edge
+
+Single stream on editing work it is **+716%** over no speculation and **+311%**
+over MTP. That is real. But:
+
+- On **prose** it is beaten by *doing nothing* from 4 concurrent streams up.
+- It costs **40%** of the KV pool (second model + 15 draft slots per stream).
+- It caps at about **22 concurrent streams** on this hardware and queues the
+  rest - median time to first token at 64 streams goes to **138 seconds**.
+
+Ask for 64 streams and count how many actually run at once. Base and MTP track
+the diagonal; DFlash flattens at 22, so its throughput lines beyond that point
+are a capacity limit rather than a speed result:
+
+![concurrency achieved vs requested](https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP/resolve/main/charts/c2c.svg)
+
+**Rule of thumb: DFlash below its ~22-stream ceiling for edit-heavy agentic
+coding; MTP for everything else and for crowds.**
+
+Full charts, tables, the 540-cell dataset and the fix that makes DFlash load on
+vLLM at all:
+
+**[-> Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4_Dflash_DGX_recipe](https://github.com/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4_Dflash_DGX_recipe)**
+
+### A warning about averages
+
+Averaging prose/code/edit assumes editing is exactly a third of your traffic.
+**72% of DFlash's average comes from the edit workload alone.** Drop editing and
+average only prose and code, and MTP wins from 4 streams up. Measure your own
+mix instead - acceptance rate tells you directly:
+
+```bash
+curl -s http://127.0.0.1:8000/metrics \
+  | grep -E 'spec_decode_num_(accepted|draft)_tokens_total'
+```
+
+Near 0.06 is prose-like (do not use DFlash); near 0.83 is edit-like (do).
+
+---
+
+## How this was made
+
+Source: `nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451` - bf16, 1199
+tensors, 55,457,998,304 bytes, a `nuslerp` mergekit merge of two Qwen3.6-27B
+Architect-Polaris Fable variants. Uploaded with nightmedia's permission.
+
+Quantised with **NVIDIA ModelOpt 0.45.0**, `examples/llm_ptq/hf_ptq.py`:
+
+```
+--qformat nvfp4 --calib_size 512 --dataset cnn_dailymail
+--batch_size 0 --inference_tensor_parallel 1
+```
+
+ModelOpt auto-excludes `linear_attn`, the vision tower, `mtp.*`, embeddings and
+`lm_head` from quantisation - see `quant_summary.txt` in this repo for the
+per-layer record, and `hf_quant_config.json` for the exclusion patterns.
+Output: 2398 tensors, 20,487,991,904 bytes.
+
+`mtp.fc.weight` ([5120, 10240] BF16, 104,857,600 bytes) was then taken from the
+corrected upstream revision and added as `model-mtp-fc.safetensors`, with the
+index updated - **2399 tensors, 20,592,849,504 bytes**. vLLM's own source
+expects this tensor to be unquantized in NVFP4 checkpoints
+(`qwen3_5_mtp.py`: *"mtp.fc is stored as BF16 in NVFP4 checkpoints ... Force
+unquantized"*), so a spliced BF16 tensor is exactly what a fresh PTQ run would
+have emitted.
+
+The three VL processor configs (`preprocessor_config.json`,
+`processor_config.json`, `video_preprocessor_config.json`) are included -
+ModelOpt does not emit them and vLLM hard-fails a VL model at init without them.
+
+## Limitations
+
+- **Measured on one machine, one model.** DGX Spark GB10, 121.7 GiB unified
+  memory, sm_121a, arm64, a community arm64/CUDA-13 vLLM build. No claim about
+  other hardware, multi-node, or other quantizations.
+- **Acceptance rates are specific to this checkpoint.** The MTP head was
+  inherited through a four-generation mergekit merge and never retrained. A
+  different model will have different acceptance, and therefore different
+  crossover points.
+- **No quality evaluation was run here.** NVFP4 is a lossy 4-bit quantisation.
+  These numbers are throughput only; nothing in this card says the output is as
+  good as the bf16 source.
+- **Benchmarks ran with thinking disabled** and 512-token generations. Long
+  reasoning traces were not measured.
+- `min_p` and `logit_bias` do not work under speculative decoding (vLLM warns at
+  startup).
+
+## Credits
+
+- **[nightmedia](https://huggingface.co/nightmedia)** - the bf16 merge this is
+  quantised from, and its Architect-Polaris merge stages. All the model work is
+  theirs; uploaded with their permission. That merge in turn credits
+  Qwen/Qwen3.6-27B and its own upstreams.
+- **[z-lab](https://huggingface.co/z-lab)** - the DFlash draft used in the
+  comparison.
+- **NVIDIA ModelOpt** - the PTQ pipeline.
+- **vLLM** - the serving engine, including the `Qwen3_5MTP` support that makes
+  the MTP head usable at all.