pottokao/Huihui-Qwen3.6-27B-abliterated-NVFP4-DFlash-MXFP8

Huihui-Qwen3.6-27B-abliterated · NVFP4 + DFlash MXFP8 draft · 128k on 2×16GB

Qwen3.6-27B (abliterated), NVFP4-quantized, with an MXFP8-quantized DFlash speculative-decoding draft — running 128k context at ~128 tok/s average (up to 183 tok/s) on two RTX 5070 Ti 16GB cards, one of them on an M.2 riser.

To our knowledge this is the first vLLM-loadable (modelopt format) quantized DFlash draft for Qwen3.6-27B on HF (searched 2026-08-13: existing DFlash quants are GGUF / MLX / OpenVINO only, including the abliterated+DFlash combos).

TL;DR

Hardware	2× RTX 5070 Ti 16GB, TP2 — GPU1 sits on an M.2 → PCIe 4.0 x4 riser (¼ bandwidth, NCCL P2P disabled)
Context	131,072 tokens (KV pool 131,378, FP8 KV)
Single-stream decode	163–183 tok/s — median 163–168 on short code prompts (10-run), 183.5 sustained on Spec-Bench math_reasoning (n=25, 20 warmups) — single-RTX-5090 territory from a cheap asymmetric dual-card box
Spec-Bench avg	128.2 tok/s across 8 categories, +31% vs the same model with its native MTP head (K=3)
Acceptance length	3.6–5.7 @ K=8 (per-position decay is a gentle slope, not a cliff)

What's in the box

├── model-0000{1..3}-of-00003.safetensors   # 27B backbone, NVFP4 (FP8 attn/KV), MTP tensors physically removed
├── config.json / hf_quant_config.json      # modelopt mixed quant config
└── draft/
    ├── model.safetensors                   # DFlash draft, MXFP8 (self_attn excluded), 1.9 GB
    └── ...                                 # upstream z-lab card kept for provenance

Provenance

Piece	Source	What we did
Backbone	huihui-ai/Huihui-Qwen3.6-27B-abliterated (orthogonal ablation / uncensored)	Self-quantized to NVFP4 with NVIDIA TensorRT Model Optimizer, following NVIDIA's own recipe for Qwen3.6-27B (nvfp4_fp8_mha: W4A16 NVFP4 weights, FP8 attention, FP8-cast KV). Verified layer-by-layer identical structure vs NVIDIA's official quant. Then physically stripped the 15 native mtp.* tensors (this bundle uses DFlash instead — vLLM would otherwise pick the MTP weights up from the shards).
Draft	z-lab/Qwen3.6-27B-DFlash (BF16, 3.2 GB)	Self-quantized to MXFP8 (group 32, *self_attn* excluded): 3.22 → 1.90 GB. Acceptance is unharmed (pos0 ≈ 80–90%, see below). MXFP8 over NVFP4 on purpose — draft precision buys acceptance; we didn't need the extra VRAM.
Engine	vllm/vllm-openai:nightly	Pinned digest below — nightly moves, the digest doesn't.

vllm/vllm-openai@sha256:a671d5fcda70fe9ac6f245f9780821de459fb4ee22c018fd07a0f10a55279bf9

Run it (the exact 128k command)

docker run -d --name hh27dflash --gpus all \
  -e NCCL_P2P_DISABLE=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -v /path/to/this-repo:/model \
  -v /path/to/this-repo/draft:/draft \
  -p 8002:8000 \
  vllm/vllm-openai@sha256:a671d5fcda70fe9ac6f245f9780821de459fb4ee22c018fd07a0f10a55279bf9 \
    /model --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \
    --trust-remote-code --served-model-name hh27 \
    --kv-cache-dtype fp8 --attention-backend TRITON_ATTN \
    --mamba-cache-dtype bfloat16 --mamba-ssm-cache-dtype bfloat16 \
    --gpu-memory-utilization 0.96 --kv-cache-memory 2700000000 \
    --language-model-only --max-model-len 131072 \
    --max-num-seqs 3 --max-num-batched-tokens 1024 --enable-chunked-prefill \
    --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8,"attention_backend":"TRITON_ATTN","max_model_len":32768}' \
    --default-chat-template-kwargs '{"enable_thinking": false}' \
    --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

The five flags that make 128k possible on 2×16GB (each one paid for in blood)

This model's KV cost is brutal: 16 full-attention layers × head_dim 256 = 16,384 B/token/GPU — twice a normal 27B. 128k does not fit naively.

1. --language-model-only — skips loading the vision tower, frees ~0.5 GiB/GPU. Language weights are bit-identical; this is the flag that closes the 128k gap for free.
2. --mamba-cache-dtype bfloat16 + --mamba-ssm-cache-dtype bfloat16 — default fp32 mamba state is pure waste; bf16 grows the KV pool by ~22% with byte-identical outputs in our tests.
3. --kv-cache-memory 2700000000 — 2.85e9 boots fine and then OOMs the engine on the first heavy prefill (34 MB activation ask vs 18 MB free). Leave ~150–250 MB headroom; the pool is exactly 131,378 tokens at 2.7e9.
4. --attention-backend TRITON_ATTN and "attention_backend":"TRITON_ATTN" inside --speculative-config — the drafter does not inherit the CLI flag, and sm_120 FLASH_ATTN doesn't support FP8 KV. Miss either one and it dies.
5. "max_model_len":32768 inside --speculative-config — the draft's context can be set independently; leave it unset and it inflates to 128k, eating KV for nothing.

Also load-bearing: --max-num-seqs 3 (4 OOMs, 8 hits NCCL errors at this watermark), --max-num-batched-tokens 1024 (halves prefill activation spikes), K=8 (K=15 measured slower — deep draft tails fall under the ~12% break-even acceptance).

Benchmarks

All numbers: this exact config, vllm bench serve, Spec-Bench protocol, --ignore-eos, 20 warmups, concurrency 1, thinking off. Same machine, same day, same question sets as the MTP baseline.

DFlash (this repo, K=8) vs native MTP head (K=3)

The same abliterated NVFP4 backbone can run its native single-layer MTP head instead (K=3, FLASHINFER backend, 128k) — full baseline config at the bottom of this section. Head-to-head:

Category	n	MTP tok/s	DFlash tok/s	Δ	accept_len MTP → DFlash
math_reasoning	25	117.6	183.5	+56%	3.44 → 5.69
coding (Spec-Bench)	25	117.8	167.4	+42%	3.25 → 4.61
translation	25	116.1	151.6	+31%	3.21 → 4.28
qa	25	109.9	134.2	+22%	2.96 → 3.57
rag	25	89.6	111.9	+25%	3.30 → 4.51
summarization	25	83.9	100.1	+19%	3.01 → 3.67
agent (held-out)	25	77.9	95.5	+23%	3.33 → 4.60
code (held-out, long prompts)	157	69.6	81.3	+17%	3.10 → 4.17
mean (8 categories)	332	97.8	128.2	+31.1%

332 questions per side (664 fixed-length generations total), all with 20 warmup requests per category — not a handful of cherry-picked prompts.

Median ITL is ~25–26 ms for both — per-step cost is identical, the entire win is tokens-per-step.

Note on acceptance rate: don't compare it across different K — the denominator is K, so deeper drafts dilute the average by construction. Compare acceptance length and tok/s.

Per-position acceptance (DFlash, K=8, coding)

pos:    0     1     2     3     4     5     6     7
acc:  88.7  75.5  57.6  48.1  38.7  31.1  27.4  18.9   (%)

A gentle slope — unlike recursive MTP heads, which cliff after position 2. That flat tail is why DFlash affords K=8 while MTP saturates at K=3.

Rough single-stream feel (10 runs, short code prompt, 512 tok, warmed)

min 149.8 | median 163.1 | mean 161.2 | max 179.5 tok/s

On a box where one GPU talks through an M.2 riser at PCIe 4.0 x4. This is single-5090-class decode from two lame 16GB cards.

MTP baseline config (for reproducibility)

The MTP numbers above were produced with: native single-layer MTP head, {"method":"mtp","num_speculative_tokens":3}, FLASHINFER backend (VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE=134217728), FP8 KV, --kv-cache-memory 3000000000, --language-model-only, 131,072 context, seqs 8, batched 2048, same nightly image. Full script in the companion repo.

Credits

• huihui-ai — the abliterated base
• z-lab — DFlash (project page), the draft this quant derives from
• NVIDIA TensorRT Model Optimizer — quant recipe
• vLLM — engine

Uncensored model: you are responsible for what you generate with it.