Upload README.md with huggingface_hub

README.md

@@ -20,7 +20,7 @@ NVFP4 (W4A4, group 16) quantization of [**AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCEN
 
 **55.6 GB → 20.58 GB.** Fits on two 16 GB cards with real KV headroom.
 
-- The MTP draft head is preserved in bf16 and wired into the index, but **⚠️ do not enable speculative decoding on this model — see below.**
+- **MTP draft head preserved in bf16** and wired into the index — speculative decoding works (see the note on reasoning effort below).
 - bf16 kept for: `lm_head`, the vision tower, the DeltaNet `conv1d`, and the MTP head. Everything else is NVFP4 W4A4.
 - Built for **SM120 / Blackwell with vanilla vLLM v0.22.0** — compressed-tensors is auto-detected, no `--quantization` flag.
 - Quantized GPU-resident (full bf16 dispatched across 7× RTX PRO 2000 Blackwell via `device_map=auto`): **122 seconds end to end.**
@@ -42,23 +42,23 @@ Sanity-checked after quantization: Japanese self-introduction, arithmetic/primal
 ```bash
 vllm serve sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4 \
   --trust-remote-code --tensor-parallel-size 4 \
-  --max-model-len 32768 --kv-cache-dtype fp8 --reasoning-parser qwen3
+  --max-model-len 32768 --kv-cache-dtype fp8 --reasoning-parser qwen3 \
+  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
 ```
 
-**Do not add `--speculative-config` with this model.** See the first gotcha below.
-
 On boards without P2P add `NCCL_P2P_DISABLE=1` and `--disable-custom-all-reduce`.
 
 ## ⚠️ Gotchas
 
-- **Run this model WITHOUT speculative decoding.** With `--speculative-config '{"method":"mtp",...}'` enabled, the reasoning phase can run away: on identical prompts the `<think>` block grows several times longer and sometimes degenerates into repeating one line until the token budget is exhausted, so no answer is emitted. Measured on the same prompt at temperature 0 — MTP off: 2,516 chars of thinking then a 3,684-char answer; MTP on: 18,350 chars of thinking and no answer. Divergence at temperature 0 means the draft path is changing the output, not just accelerating it.
+- **Use `reasoning_effort: "medium"` for anything long-form.** At the default (highest) effort this model can spend its entire token budget inside `<think>` — the reasoning block grows past 18,000 characters, sometimes degenerating into repeating a single line, and no answer is ever emitted. At medium effort the same prompts finish cleanly every time: I ran a 9-case gate (French and English long-form, temperature 0 and 0.7) in three configurations — NVFP4 with MTP `num_speculative_tokens: 6`, NVFP4 without MTP, and the original bf16 checkpoint with MTP — and all 27 runs passed with thinking held to 1–3k characters.
 
-  **This is a problem introduced by my quantization, not by the fine-tune.** I served the original bf16 checkpoint at TP=8 with MTP `num_speculative_tokens: 6` and it behaved perfectly across the same prompts (no runaway, no repetition), so the bf16 MTP head is sound — it simply does not sit well on a W4A4 body for this fine-tune. The base Qwen3.8-27B quantized with the identical recipe does not show the problem either, so it is specific to this combination. I'm investigating a fix; until then, serve without `--speculative-config` and the model is well-behaved.
+  ```json
+  "chat_template_kwargs": {"reasoning_effort": "medium"}
+  ```
 
-  Reported by a user running it on Jetson with vLLM 0.27.1 — thank you.
+  **Speculative decoding is fine** — an earlier version of this card told you to disable it, which was wrong. MTP at n=6 passes the gate as long as the effort is medium. My apologies for the misdirection, and thanks to the user on Jetson / vLLM 0.27.1 who reported the looping.
 - **The 15 `mtp.*` modules are listed in `quantization_config.ignore` — do not remove them.** If vLLM treats the bf16 MTP head as NVFP4 the draft breaks silently: 0% acceptance, slower than no MTP at all.
 - W4A16 (NVFP4A16) does **not** serve on this architecture in vLLM 0.22 (`gptq_marlin_repack: size_n=24 not divisible by tile_n_size=64`). W4A4 only.
-- Reasoning model: give it `max_tokens ≥ 4096`, or use `chat_template_kwargs: {"reasoning_effort": "low"}` for long-form generation — at the default effort it can spend the entire budget thinking.
 - Long single-file code generation drops a closing paren roughly 1–2 times in 14 regardless of sampling temperature (measured with a JS parser on the base Qwen3.8-27B). Put a syntax check in the loop rather than tuning temperature.
 
 ## Recipe