Qwen3.5-122B-A10B-abliterix-NVFP4

NVFP4 (W4A4) quantization of wangzhang/Qwen3.5-122B-A10B-abliterix — the 122B/A10B Abliterix-abliterated (uncensored) Qwen3.5 MoE — packaged for vLLM serving on NVIDIA Blackwell hardware (DGX Spark GB10 / SM121).

Summary

Base model	wangzhang/Qwen3.5-122B-A10B-abliterix (BF16)
Quantization	NVFP4 W4A4 — group-wise (group_size=16), tensor-group strategy, symmetric
Format	compressed-tensors / nvfp4-pack-quantized (vLLM-native, packed 2 fp4/byte)
Weight layout (per Linear)	.weight_packed (uint8, packed) + .weight_scale (fp8 e4m3fn, per-group) + .weight_global_scale (float32) + .input_global_scale (float32)
Activation quant	dynamic local, group_size=16, fp8 e4m3fn scale
Skipped modules	lm_head, *.mlp.gate, *.mlp.shared_expert_gate, *.embed_tokens, all *.linear_attn.* (Mamba/GDN), all visual.* (text-only model) — kept BF16
NVFP4 layers	37,056
BF16 layers preserved	471 (norms, biases, A_log, dt_bias, embed_tokens, lm_head, gates, linear_attn)
Shards	15 × ~5.4 GB safetensors
Total size on disk	70.4 GB (~32% of BF16 baseline)
Tested vLLM image	ghcr.io/bjk110/vllm-spark:v022-d568
Runtime stack	NGC pytorch:26.04-py3 base • PyTorch 2.12.0a0 • CUDA 13.0 • vLLM v0.21.0 + PR #35568 cherry-pick • FlashInfer v0.6.11.post3 • NCCL 2.30.4 • Triton 3.7.0 • TensorRT 5.8.1
Topology	2× DGX Spark GB10, TP=2 over 200 Gbps RoCE

Why this quantization

The Abliterix-trained uncensored behavior (0.5% refusal rate, KL divergence 0.0115 vs the Qwen3.5-122B-A10B baseline) is preserved while dropping weight memory from BF16 (230 GB) to NVFP4 (70 GB) — fitting comfortably on two DGX Spark nodes (2 × 119 GiB unified memory) with headroom for KV-cache, with the same compressed-tensors runtime path as the FP8 sibling.

Compared to the FP8 W8A8 variant (bjk110/Qwen3.5-122B-A10B-abliterix-FP8, 116 GB), this NVFP4 build cuts another ~40% off-disk and unlocks the FlashInfer-CUTLASS NVFP4 MoE backend for higher throughput on Blackwell.

Quantization method

Direct safetensors-level conversion via convert_bf16_to_nvfp4.py, mirroring the layout of RedHatAI/Qwen3.5-122B-A10B-NVFP4 (built with llm-compressor) — but applied directly to the abliterix BF16 checkpoint, because llm-compressor 0.10 pins transformers <=4.57.6 while Qwen3.5MoeForCausalLM is only available in transformers >=5.5. The direct script needs only torch + safetensors.

For each 2D Linear weight W (shape [out, in]) not in the ignore list:

1. Identify the fused group the Linear belongs to (if any):
   • Attention QKV: q_proj, k_proj, v_proj of the same self_attn → 3-member group
   • Expert gate-up: experts.N.{gate_proj, up_proj} → 2-member group
   • Shared-expert gate-up: shared_expert.{gate_proj, up_proj} → 2-member group
   • Everything else (o_proj, down_proj) is independent.
2. Compute the per-tensor weight_global_scale (float32). For a fused group, all members share one global scale computed from max(abs(W)) across the entire group; vLLM merges these Linears at runtime (fused QKV / gate_up matmul) and requires a single coherent scale per merged matrix — independent per-projection scales produce garbage outputs (repeated tokens) even though the model boots.
3. Reshape W into groups of 16 along the input dim. Compute per-group fp8 e4m3fn weight_scale (one scale per 16-element group) so that each group fits the NVFP4 [-6, 6] range after dequant.
4. Quantize each group element to NVFP4 (e2m1), pack two values per uint8 → weight_packed.
5. The matching input_global_scale (float32, per-tensor) is stored for the runtime to compute on-the-fly NVFP4 activation quantization.

Activations are NVFP4 W4A4 — quantized at inference time by vLLM using the stored input_global_scale with dynamic local group_size=16 (fp8 e4m3fn scale).

config.json quantization block (excerpt)

{
  "quantization_config": {
    "config_groups": {
      "group_0": {
        "format": "nvfp4-pack-quantized",
        "targets": ["Linear"],
        "weights": {
          "num_bits": 4, "type": "float", "strategy": "tensor_group",
          "group_size": 16, "symmetric": true, "dynamic": false,
          "observer": "memoryless_minmax",
          "scale_dtype": "torch.float8_e4m3fn"
        },
        "input_activations": {
          "num_bits": 4, "type": "float", "strategy": "tensor_group",
          "group_size": 16, "symmetric": true, "dynamic": "local",
          "observer": "static_minmax",
          "scale_dtype": "torch.float8_e4m3fn"
        },
        "output_activations": null
      }
    },
    "format": "nvfp4-pack-quantized",
    "ignore": [
      "re:.*lm_head",
      "re:visual.*",
      "re:model.visual.*",
      "re:.*mlp.gate$",
      "re:.*embed_tokens$",
      "re:.*shared_expert_gate$",
      "re:.*linear_attn.*"
    ],
    "quant_method": "compressed-tensors"
  }
}

Serving with vLLM

# Tested with the v022-d568 image — see DGX Spark notes below.
vllm serve bjk110/Qwen3.5-122B-A10B-abliterix-NVFP4 \
    --tensor-parallel-size 2 \
    --distributed-executor-backend ray \
    --quantization compressed-tensors \
    --moe-backend flashinfer_cutlass \
    --kv-cache-dtype fp8 \
    --max-model-len 32768 \
    --max-num-seqs 4 \
    --gpu-memory-utilization 0.88 \
    --enable-chunked-prefill \
    --reasoning-parser qwen3

--moe-backend flashinfer_cutlass selects the FlashInfer CUTLASS NVFP4 grouped-GEMM kernel for the MoE expert path — required to get the full NVFP4 speedup on Blackwell.

A single Blackwell-class GPU is not enough to hold the 70 GB weights + KV cache + activation buffers, so --tensor-parallel-size 2 (or higher) is mandatory.

DGX Spark (GB10, SM121) notes

NVIDIA DGX Spark uses SM121, which on stock vLLM v0.21.0 was excluded from the Marlin/CUTLASS NVFP4 codepaths (the gates were SM120-only). vLLM PR #35568 (commit 06d020bb6) widens those gates to the SM12x family — covering both the FP8 W8A8 and the NVFP4 dispatch paths.

With that fix applied, the boot log reports the NVFP4 kernels are bound to the SM121 GPUs and the FlashInfer-CUTLASS MoE path is selected, confirming the full Blackwell NVFP4 pipeline is active.

Runtime stack (image v022-d568)

The image is the cumulative top of the v022 forward-stack build chain, rooted in NGC nvcr.io/nvidia/pytorch:26.04-py3 (CUDA 13.0, PyTorch 2.12.0a0). Each layer corresponds to one published image tag:

Stack layer	Component / version	Image tag
Base	NGC pytorch:26.04-py3 (CUDA 13.0, PyTorch 2.12.0a0)	v022-ngc2604
Inference	vLLM v0.21.0	v022-vllm021
FP4/FP8 attention & MoE kernels	FlashInfer v0.6.11.post3	v022-fi0611
Triton	3.7.0	v022-trt37
TensorRT runtime	5.8.1	v022-tx581
Collective comm	NCCL 2.30.4	v022-nccl234
SM121 enablement	vLLM PR #35568 cherry-pick (SM120 → SM12x gates)	v022-d568 ← this

Building on NGC 26.04 (vs. the older 26.03 base used by v021) gives the SM121 GPU the matching CUDA 13.0 driver/runtime split that the Blackwell NVFP4 kernels expect, and is required for FlashInfer v0.6.11.post3 (which assumes CUDA 13 headers).

Lineage

Stage	Repo / Tag
BF16 baseline	Qwen/Qwen3.5-122B-A10B
BF16 reference NVFP4 (text+vision)	RedHatAI/Qwen3.5-122B-A10B-NVFP4
Abliterix abliteration (BF16)	wangzhang/Qwen3.5-122B-A10B-abliterix
FP8 W8A8 abliterix	bjk110/Qwen3.5-122B-A10B-abliterix-FP8
NVFP4 W4A4 (this repo)	bjk110/Qwen3.5-122B-A10B-abliterix-NVFP4

Citation

@software{abliterix,
  author = {Wu, Wangzhang},
  title  = {Abliterix: Automated LLM Abliteration},
  year   = {2026},
  url    = {https://github.com/wuwangzhang1216/abliterix}
}

Acknowledgements

• Wu Wangzhang for the Abliterix framework and the BF16 abliterated checkpoint.
• Qwen team for the Qwen3.5-122B-A10B base model.
• Red Hat AI for the Qwen3.5-122B-A10B-NVFP4 reference layout.
• vLLM compressed-tensors + FlashInfer-CUTLASS NVFP4 MoE integration teams.
• DGX Spark SM121 enablement: vLLM PR #35568 by Blake Ledden (Second Nature Computing) + contributors.