Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-W4A4

NVFP4 W4A4 quantization (4-bit weights and 4-bit activations) of huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated — an uncensored / abliterated Qwen3.6-35B-A3B hybrid MoE (~3B active, GDN linear-attention + full-attention interleave). Self-quantized with llm-compressor (NVFP4 scheme) in compressed-tensors nvfp4-pack-quantized format.

The point of this checkpoint: on a single NVIDIA DGX Spark (GB10, SM121) it is servable in vLLM and decodes faster than the FP8 build — which most NVFP4 MoE quants on consumer/SM121 Blackwell are not (they typically fall back to a dequant slow path or refuse to compile the block-scaled MMA). With the right toolchain it lands at 66.9 tok/s vs 52.0 tok/s for FP8 (+29%) while using 16 GB less weight memory.

Lineage: BF16 abliterated base (huihui-ai) → NVFP4 W4A4 (this repo). Vision tower preserved and the model.language_model.visual.* → visual.* loader-prefix fix already applied (see note below). MTP heads preserved but off by default (see below).

Quick stats

Metric	Value
Base	huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated (BF16)
Format	compressed-tensors nvfp4-pack-quantized
Scheme	NVFP4 — W4A4 (weights FP4 + activations FP4), group size 16, FP8 e4m3fn scales
KV cache scheme	fp8
Disk size	~22 GB
Kept higher precision	lm_head, visual.*, mlp.gate (router), shared_expert_gate
MTP weights	Preserved (model_mtp.safetensors) — keep OFF, see note
Native context	256K (max_position_embeddings 262144), no YaRN
Modality	Text + Image (vision tower preserved, loader-prefix fixed)

Performance on DGX Spark (GB10, vLLM)

Paired, same-harness, single-stream decode, KV-cache fp8, warm:

Configuration	Weight mem	tok/s	vs FP8
FP8 dynamic	~38 GB	52.0	1.00×
This NVFP4 W4A4 (cudagraph on)	~22 GB	66.9	1.29×
This NVFP4 W4A4, --enforce-eager	~22 GB	~23	0.44×

The whole speedup lives in CUDA graphs. With --enforce-eager this model is only ~23 tok/s. On a hybrid MoE, eager-mode launch overhead from the many small per-expert kernels dominates wall-clock; capturing CUDA graphs removes it and recovers the full 66.9 tok/s. Do not pass --enforce-eager.

Full write-up (method, sweep, traces): DGX Spark Part 33 — NVFP4 W4A4 MoE + CUDA graphs.

Serving (vLLM, GB10 / SM121)

Toolchain requirement. You need a vLLM build where the SM121 NVFP4 block-scaled MMA actually compiles: cutlass-dsl 4.5.x + flashinfer 0.6.11+. Do not use any sm121-b12x style mod that downgrades cutlass-dsl to 4.4.2 — that breaks the compiled NVFP4 path and silently forces enforce-eager behaviour (back to ~23 tok/s).

vllm serve coolthor/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-W4A4 \
  --served-model-name qwen36-abliterated \
  --moe-backend flashinfer_cutlass \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.50 \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --trust-remote-code

Gotchas

• Never set --enforce-eager — it drops you from 66.9 → ~23 tok/s (kills the CUDA-graph win).
• --max-num-batched-tokens >= 2096 is required — the hybrid Mamba/GDN cache block alignment needs it; smaller values fail to allocate the conv/state cache.
• The flashinfer_cutlass NVFP4 MoE backend wants W4A4 (this model). A W4A16 weight-only NVFP4 export is rejected by this backend — the activations must be FP4 too, which is why this checkpoint is W4A4.
• MTP heads are preserved but should stay OFF. Swept num_speculative_tokens n=0–4; the no-spec baseline wins. On this hybrid at GB10 memory bandwidth, NVFP4 weight loading and MTP speculation are substitutes competing for the same bandwidth — stacking them does not help. Just don't pass a --speculative-config.

Vision loader-prefix fix (carried forward)

Qwen3_5MoeForConditionalGeneration saves vision weights under model.language_model.visual.*, but vLLM's loader looks for visual.*, so all vision tensors silently skip-load and image input degenerates into !!!!! loops (text-only unaffected). This checkpoint already has the prefix stripped in the affected shard + model.safetensors.index.json, so vision works out of the box.

Quantization recipe

# recipe.yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: ['re:.*lm_head', 're:visual.*', 're:model.visual.*', 're:.*mlp.gate$', 're:.*shared_expert_gate$']
      scheme: NVFP4
      bypass_divisibility_checks: false

from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_PATH = "huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated"
SAVE_DIR = "Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-W4A4"

model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    MODEL_PATH, dtype="auto", low_cpu_mem_usage=True,
)
processor = AutoProcessor.from_pretrained(MODEL_PATH)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",  # W4A4: 4-bit weights + 4-bit activations
    ignore=[
        "re:.*lm_head",
        "re:visual.*",
        "re:model.visual.*",
        "re:.*mlp.gate$",
        "re:.*shared_expert_gate$",
    ],
)

oneshot(model=model, recipe=recipe)
model.save_pretrained(SAVE_DIR, max_shard_size="2GB", safe_serialization=True)
processor.save_pretrained(SAVE_DIR)
save_mtp_tensors_to_checkpoint(source_model=MODEL_PATH, dest_dir=SAVE_DIR)

License

Apache-2.0, inherited from huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated and the upstream Qwen/Qwen3.6-35B-A3B (Apache-2.0). Redistribution permitted.

---

☕ If this saved you GPU hours, you can buy me a coffee.