BroLaurens/gemma-4-31B-it-qat-NVFP4-GGUF

Gemma 4 31B IT — NVFP4-MSE Blackwell GGUF

A family of three GGUF files of Gemma 4 31B IT, converted from melcheikh/gemma-4-31B-it-qat-NVFP4-mse-Blackwell — a QAT checkpoint where every transformer weight is natively NVFP4 (NVIDIA's FP4 format, MSE per-block scaling + FP8 activation scale sweep via ModelOpt 0.44.0).

• ORIG — the source-preserving conversion: native NVFP4 on all 410 projection tensors + BF16 token embedding (the one tensor excluded from QAT). Highest fidelity.
• Q8_0-embd — same NVFP4 backbone, token embedding quantized to Q8_0. The balanced choice.
• Q4_K-embd — same NVFP4 backbone, token embedding quantized to Q4_K. Smallest.

The NVFP4 backbone is byte-identical across all three files (410/410 per-tensor SHA-256). The files differ only in token_embd.weight — which, since Gemma 4 ties embeddings to the LM head, is also output.weight. No other tensor varies.

Vision works. Gemma 4 31B is a native VLM (images and video). No mmproj is shipped here — pair with any Gemma 4 31B mmproj, e.g. mmproj-BF16.gguf from unsloth/gemma-4-31B-it-qat-GGUF (the vision tower is untouched by QAT in both models). MTP speculative decoding works — the model has been tested with unsloth's mtp-gemma-4-31B-it-Q4_0.gguf draft head.

The three files

File	Size	token_embd (= tied LM head)	Backbone
gemma-4-31B-it-qat-NVFP4-ORIG.gguf	19.31 GB	BF16	NVFP4 (410 tensors, byte-identical)
gemma-4-31B-it-qat-NVFP4-Q8_0-embd.gguf	17.99 GB	Q8_0	NVFP4 (410 tensors, byte-identical)
gemma-4-31B-it-qat-NVFP4-Q4_K-embd.gguf	17.29 GB	Q4_K	NVFP4 (410 tensors, byte-identical)

Tensor layout (1,653 tensors per file)

GGML type	Tensors	Size	Component
NVFP4	410	16.47 GB	all 60 layers × (attn_q/k/v/output + ffn_gate/up/down)
BF16 / Q8_0 / Q4_K	1	2.82 / 1.50 / 0.79 GB	token_embd (= tied LM head)
F32	1,242	0.01 GB	norms, q/k-norms, layer output scales, NVFP4 block scales, rope freqs

Lineage

1. Google — gemma-4-31B-it-qat-q4_0-unquantized (Gemma terms): the QAT-trained q4_0 checkpoint.
2. melcheikh — gemma-4-31B-it-qat-NVFP4-mse-Blackwell: NVIDIA ModelOpt 0.44.0 NVFP4 QAT with MSE per-block weight scaling (--qformat nvfp4_mse) plus an FP8 activation scale sweep, calibrated on cnn_dailymail (512 samples). lm_head and the vision tower excluded from quantization. This is the checkpoint converted here — the NVFP4 tensors are preserved natively (GGML type 40), never dequantized or re-quantized.
3. This repo — the GGUF conversion and the embedding tier ladder.

How this was made

# 1. Source-preserving conversion (everything already NVFP4 in the checkpoint)
python3 convert_hf_to_gguf.py <checkpoint> --outfile gemma-4-31B-it-qat-NVFP4-ORIG.gguf --outtype auto

# 2. Tiers: llama-quantize with per-tensor overrides pinning the NVFP4 backbone (verbatim copy),
#    only the token embedding is re-quantized.
#    overrides-nvfp4-backbone.txt (regex, matches all 410 projections):
#      blk\.[0-9]+\.attn_(q|k|v|output)\.weight=nvfp4
#      blk\.[0-9]+\.ffn_(gate|up|down)\.weight=nvfp4
llama-quantize --tensor-type-file overrides-nvfp4-backbone.txt \
    --token-embedding-type Q8_0 \
    gemma-4-31B-it-qat-NVFP4-ORIG.gguf \
    gemma-4-31B-it-qat-NVFP4-Q8_0-embd.gguf Q4_K
# (repeat with --token-embedding-type Q4_K for the Q4_K-embd tier)

The chat template is Google's current (updated) Gemma 4 template. general.quantization_version = 2, native context length 262,144.

Verified: 410/410 NVFP4 tensors SHA-256 byte-identical across all three files.

Serving

Requires a Blackwell GPU (sm_100/sm_120) for native FP4 — llama.cpp with BLACKWELL_NATIVE_FP4 support. Non-Blackwell devices will not get the point of this format.

llama-server \
    -m gemma-4-31B-it-qat-NVFP4-Q4_K-embd.gguf \
    --mmproj mmproj-BF16.gguf \
    -md mtp-gemma-4-31B-it-Q4_0.gguf \        # optional MTP draft (unsloth)
    --spec-type draft-mtp --spec-draft-n-max 3 \
    --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 --draft-p-min 0.0 \
    -c 81920 --temp 0 -ngl 999 --parallel 1 -fa on -t 8 \
    -b 512 -ub 512 -ctk q8_0 -ctv q8_0 \
    --jinja

Measured on a single RTX PRO 4000 Blackwell SFF (24 GiB), SPEED-Bench throughput_16k / high_entropy, osl 1024, temperature 0, identical server flags (MTP draft n-max 2, -c 81920, q8_0 KV cache, text-only):

Model file	Prompt processing t/s (16k prompt)
gemma-4-31B-it-qat-UD-Q4_K_XL.gguf (unsloth, Q4_K_XL)	662.6
gemma-4-31B-it-qat-NVFP4-Q8_0-embd.gguf	958.9
gemma-4-31B-it-qat-NVFP4-Q4_K-embd.gguf	943.9

On native FP4 hardware the NVFP4 backbone processes 16k prompts ~45% faster than the Q4_K_XL quantization of the same model.

Honest trade-off notes

• The backbone is QAT-trained NVFP4 — this is not a PTQ round trip; the 4-bit weights were trained by the checkpoint authors, and this conversion preserves them exactly.
• The embedding tiers are PTQ on one tensor. token_embd (and therefore the tied LM head) is the only tensor excluded from QAT in the source. Quantizing it to Q8_0/Q4_K is a small, local fidelity loss vs ORIG — Q8_0 is closer to BF16 than Q4_K is. If output quality matters more than ~2 GB, use ORIG; if footprint matters, Q4_K-embd costs 2.0 GB vs ORIG.
• Vision — the vision tower was untouched by QAT; any Gemma 4 31B mmproj works (unsloth's mmproj-BF16.gguf is the same unquantized tower).

Attribution & license

This is a derivative work — no training or fine-tuning happened here. Credit belongs to:

1. Google for the base model and QAT training.
2. NVIDIA for ModelOpt and the NVFP4-MSE method.
3. melcheikh for the NVFP4-MSE Blackwell checkpoint converted here.

Gemma 4 models are released by Google under the Apache 2.0 license — see the Gemma 4 license. These files are derivatives of Apache-2.0 artifacts and are distributed under the same Apache 2.0 terms.

SHA-256

34195d8261c853a9baaa89f59787abcae36c12455c5b5809c16083395e4524a4  gemma-4-31B-it-qat-NVFP4-ORIG.gguf
af8617be45d407f185e7a78252903f9d02917b71adab6657625c4714334b9e0f  gemma-4-31B-it-qat-NVFP4-Q8_0-embd.gguf
60bca34fa5e5c9281e06a265c5b90ed9f522c3ce2c5a837f651a4690de032510  gemma-4-31B-it-qat-NVFP4-Q4_K-embd.gguf