LibertAIDAI

/

Qwen3.6-27B-NVFP4-GGUF

Like 15

Follow

LibertAI 60

Why it's 15 GB

It's not the FFN-NVFP4 win that does most of the work — NVFP4 is ~4.5 bits/elem packed (4-bit weight + FP8 block scale
per 16 elems), versus Q4_K_M at ~4.84 effective bits/elem. That's only ~0.3 bits/weight cheaper. The bigger savings
come from not boosting embed/lm_head/etc. to higher precision, which most "Q4" GGUFs (unsloth UD, etc.) routinely do.

The actual tensor inventory of Qwen3.6-27B-NVFP4-Q4_K_M.gguf:

  ┌───────────┬───────┬──────────────────────────────────────────────────────────────────────────┐
  │ # tensors │ dtype │                                   what                                   │
  ├───────────┼───────┼──────────────────────────────────────────────────────────────────────────┤
  │ 192       │ NVFP4 │ every ffn_gate / ffn_up / ffn_down weight (64 layers × 3)                │
  ├───────────┼───────┼──────────────────────────────────────────────────────────────────────────┤
  │ 33        │ Q6_K  │ output.weight + the biggest projections                                  │
  ├───────────┼───────┼──────────────────────────────────────────────────────────────────────────┤
  │ 273       │ Q4_K  │ token_embd, attn_gate, attn q/k/v on the 16 full-attn layers, ssm_out, … │
  ├───────────┼───────┼──────────────────────────────────────────────────────────────────────────┤
  │ 737       │ F32   │ norms, biases, NVFP4 per-tensor input scales                             │
  └───────────┴───────┴──────────────────────────────────────────────────────────────────────────┘

MTP and vision are already gone. The base GGUF was built without the MTP head; the vision tower is in the separate
mmproj-Qwen3.6-27B-F16.gguf (889 MB) and you only need it for image input. So for your "text-only on a 4090" goal you
can ignore the mmproj — that's already what you want.

How to reproduce / shrink further

1. Source: mmangkad/Qwen3.6-27B-NVFP4 — NVIDIA ModelOpt v0.42 activation-aware NVFP4 calibration, repacked. You cannot
   add NVFP4 to more tensors without re-running ModelOpt — NVFP4 needs activation calibration, llama.cpp will not
   synthesize it from BF16. (MXFP4 can be assigned at quantize-time, no calibration needed.)
2. llama.cpp at master (≥ #22196 for Blackwell NVFP4 MMA, plus #20505/#20506/#22611 for the Qwen3.5/3.6 convert path).
3. python convert_hf_to_gguf.py → BF16 GGUF with the 192 FFN tensors already typed as NVFP4 (the convert script
   picks up hf_quant_config.json).
4. llama-quantize --tensor-type '=' input.gguf out.gguf Q4_K_M to retype anything you want. To push lower
   than Q4_K_M:
   - --tensor-type 'attn_q.weight=MXFP4' --tensor-type 'attn_k.weight=MXFP4' --tensor-type 'attn_v.weight=MXFP4'
   --tensor-type 'attn_output.weight=MXFP4'
   - Or flip the SSM in_proj (attn_qkv.weight in this arch) — that's the 48× Q6_K block that dominates the non-FFN
   footprint.
5. For 4090 specifically: NVFP4 and MXFP4 both fall back to the dp4a / MMQ paths on Ada (sm_89) — no hardware
   tensor-core speedup there. The size shrinks, but it won't go faster than well-tuned Q4_K_M. Your real constraint on a
   24 GB 4090 won't be weights, it'll be KV cache (64 layers × 4 KV heads × 256 dim is heavy at long context). Use -ctk
   q4_0 -ctv q4_0 and you'll fit way more context than weight quant tweaks will buy you.