Jianqiao1/Qwen3.6-27B-NVFP4-MTP-GGUF

Notice:

Model quality improved significantly after adopting the MoQ strategy; however, because the MoQ strategy offers very few interchangeable Q4_K options, the NVFP4 model loses almost all its performance advantages under this approach. Consequently, unless new technology emerges to enhance NVFP4's quantization performance, we will likely remain on the v3 version for the foreseeable future. If model quality is your priority, I recommend my MoQ series, as these models offer superior quality:

Jianqiao1/Qwen3.6-27B-MTP-MoQ-GGUF

Qwen3.6-27B-NVFP4-MTP-GGUF

This is a GGUF quantization of Qwen3.6-27B-MTP using a custom NVFP4 quantizer and a MoQ-derived mixed tensor policy. The model was quantized with my customized llama.cpp build, but the output GGUF uses standard tensor types and is compatible with mainline llama.cpp builds that support NVFP4.

This implementation incorporates ideas from michaelw9999's NVFP4 quantizer component in advanced-gguf-quantizer, uses the unsloth imatrix file for Qwen3.6-27B, and applies my rich+CJSO adaptive NVFP4 scale search with RSF-lite.

The tensors are stored using standard GGUF tensor types such as NVFP4, IQ4_XS, Q5_K, Q8_0, BF16, and F32. The model keeps the MTP-capable Qwen3.6 architecture metadata, including qwen35.nextn_predict_layers = 1.

Update History

• v3

  • Switches the mixed tensor policy to the projected MoQ-4.8 layer strategy. This layer strategy originates from the Qwen 3.5 9B model released by w-ahmad; we analyzed his strategy and adapted it for the 27B model. It performs exceptionally well, completely surpassing our original mixed-layer strategy—and, in fact, outperforming Unsloth's Dynamic 2.0 layer strategy as well.
  • Keeps MoQ-4.8's sensitive tensor choices, including BF16 guards and MTP large tensor Q8_0 protection.
  • Replaces the MoQ-4.8 Q4_K tensors with NVFP4; other MoQ-selected tensor types such as IQ4_XS, Q5_K, BF16, Q8_0, and F32 remain unchanged.
  • This version prioritizes the MoQ-4.8 quality/tail profile while retaining standard NVFP4 runtime compatibility.
  • While MoQ's layering strategy is excellent, it does not always benefit NVFP4; the current bpw-to-quality ratio of NVFP4 struggles to compete with IQ4_XS and Q4_K, resulting in fewer NVFP4-replaceable tensors within MoQ and, consequently, a very low speedup ratio for NVFP4 compared to the original MoQ model.

• v2

  • Introduced the stronger mixed NVFP4 layer policy built from the local NVFP4_M policy plus an XL-v2 style high-precision override policy.
  • Used the same adaptive NVFP4 quantizer stack: rich+CJSO scale search, RSF-lite, imatrix-aware two-objective selection.
  • Improved quality over the earlier NVFP4_M policy while keeping the model smaller than Q4_K_M.

Quantization Overview

This model uses an experimental NVFP4 weight-side quantization pipeline:

• NVFP4 adaptive scale selection

  • NVFP4 stores 16-value subblocks with E2M1 values and UE4M3 scales.
  • For each 16-value subblock, the quantizer searches multiple scale candidates instead of using only the standard amax / 6 scale.

• Rich scale search

  • Uses the top1 max-absolute value in each subblock as the anchor.
  • Candidate slots: 6, 5, 4, 3, 2, 1.5, 1.
  • For each slot, the quantizer searches the UE4M3 scale code with radius +/-1.

• CJSO local scale anchors

  • In addition to the rich slot candidates, the quantizer computes local least-squares scale anchors for each 16-value subblock.
  • It starts from the standard M=6 NVFP4 assignment, then estimates ordinary and imatrix-weighted scale anchors using the fixed FP4 assignment.
  • The final default search combines rich candidates with CJSO candidates using radius 2, then deduplicates UE4M3 scale codes before scoring.

• RSF-lite

  • A per-tensor scale multiplier is selected from: 0.875, 0.9375, 1.0, 1.0625, 1.125.
  • The selected multiplier is baked into each subblock's NVFP4 scale code.
  • No runtime side tensor or format change is required.

• Imatrix-aware candidate selection

  • The model was quantized with an imatrix.
  • Adaptive NVFP4 candidate selection uses a two-objective score that balances ordinary reconstruction MSE and imatrix-weighted MSE.

• MoQ-4.8 mixed tensor policy

  • The v3 model uses a projected MoQ-4.8 tensor strategy from w-ahmad/Qwen3.5-9B-GGUF-MoQ-MTP.
  • MoQ-4.8's Q4_K tensors are replaced by NVFP4.
  • MoQ-selected IQ4_XS, Q5_K, BF16, Q8_0, and F32 tensors are kept as-is.
  • MTP large tensors are kept at Q8_0.

Usage

Use standard llama.cpp; please use a version that supports NVFP4, such as b8297 or later.

./llama-cli \
  -m Qwen3.6-27B-MTP-NVFP4_MoQ_4.8.gguf \
  -ngl 999 \
  -fa 1 \
  -p "Write a concise technical explanation of transformer inference." \
  -n 256 \
  -st --simple-io

To enable MTP speculative decoding:

./llama-cli \
  -m Qwen3.6-27B-MTP-NVFP4_MoQ_4.8.gguf \
  -ngl 999 \
  -fa 1 \
  -p "Write a concise technical explanation of transformer inference." \
  -n 256 \
  -st --simple-io \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-default

Quantization Command

The v3 model was produced with a tensor policy file generated from the projected MoQ-4.8 strategy:

./llama-quantize \
  --imatrix imatrix_unsloth.gguf_file \
  --tensor-type-file qwen36_27b_mtp_nvfp4_moq48_mtp_q8_0_tensor_types.txt \
  --nvfp4-scale-mode adaptive \
  Qwen3.6-27B-MTP-BF16.gguf \
  Qwen3.6-27B-MTP-NVFP4_MoQ_4.8.gguf \
  NVFP4 \
  16

Default NVFP4 adaptive settings in the custom build:

• Search algorithm: rich_cjso
• CJSO radius: 2
• Rich slots: 6,5,4,3,2,1.5,1
• Rich scale-code radius: 1
• RSF-lite: enabled
• Imatrix selector: two-objective ordinary/weighted MSE balance

Evaluation

Evaluation was run on WikiText-2 raw test text with context length 512, using BF16 logits as the KLD base.

Model	Size BPW	Size GiB	Mean PPL(Q)	Mean PPL(base)	Corr	Mean KLD	Max KLD	99.9% KLD	RMS delta-p	Same top-p
NVFP4 v1	4.88	15.5319	7.061288 +/- 0.047106	6.902375 +/- 0.045392	98.71%	0.051408 +/- 0.000910	26.440189	3.930263	6.168 +/- 0.059%	91.188 +/- 0.074%
NVFP4 v2	4.98	15.8567	7.023239 +/- 0.046574	6.902375 +/- 0.045392	98.92%	0.043386 +/- 0.000789	23.118464	3.708230	5.651 +/- 0.057%	91.883 +/- 0.071%
NVFP4 v3	4.73	15.0410	7.019970 +/- 0.046483	6.902375 +/- 0.045392	99.28%	0.027350 +/- 0.000661	25.868000	2.546991	4.472 +/- 0.059%	93.673 +/- 0.063%
Michaelw9999 NVFP4	4.79	15.2367	7.117026 +/- 0.047435	6.902375 +/- 0.045392	98.61%	0.058210 +/- 0.000925	24.220201	4.471214	6.670 +/- 0.061%	90.549 +/- 0.076%
Michaelw9999 RSF NVFP4 v4	4.79	15.27	7.030348 +/- 0.046636	6.900856 +/- 0.045374	98.91%	0.044590 +/- 0.000832	24.703529	3.726478	5.774 +/- 0.060%	91.924 +/- 0.071%
Unsloth Q4_K_M	5.01	15.9319	6.936849 +/- 0.045687	6.902375 +/- 0.045392	99.39%	0.022354 +/- 0.000642	21.246073	2.224004	4.044 +/- 0.060%	94.329 +/- 0.060%

Throughput on RTX 5090, llama.cpp CUDA 13.0 build:

Model	pp512	tg128	pg32768,256	MTP p512 prefill	MTP gen128	MTP p32768 prefill	MTP gen256
NVFP4 v1	3743.22	71.14	2182.78	1489.10	112.80	2679.90	95.60
NVFP4 v2	3414.24	66.32	2017.06	1349.10	107.90	2550.50	105.20
NVFP4 v3	2236.78	67.19	1759.29	1284.30	160.60	2254.20	114.50
Michaelw9999 NVFP4	2816.75	72.61	2308.00	n/a	n/a	n/a	n/a
Michaelw9999 RSF NVFP4 v4	5174.31 +/- 1.30	76.61 +/- 0.18	2751.93 +/- 23.73	n/a	n/a	n/a	n/a
Unsloth Q4_K_M	2217.93	65.52	1755.85	1265.20	94.80	2171.10	82.00

Notes

• Thanks to michaelw9999 for the public NVFP4 quantizer work.
• Thanks to the unsloth team for the Qwen3.6 imatrix file.
• Thanks to the w-ahmad's MoQ tensor policy.