---
license: apache-2.0
license_link: https://ai.google.dev/gemma/docs/gemma_4_license
base_model: WaveCut/gemma-4-26B-A4B-it-heretic
base_model_relation: quantized
library_name: vllm
pipeline_tag: image-text-to-text
tags:
  - gemma4
  - heretic
  - vllm
  - llm-compressor
  - compressed-tensors
  - w4a16
  - int4
  - rtn
  - g64
  - moe
  - multimodal
---

# Gemma 4 26B A4B Heretic W4A16 G64 RTN for vLLM

`compressed-tensors` W4A16 quantization of
[`WaveCut/gemma-4-26B-A4B-it-heretic`](https://huggingface.co/WaveCut/gemma-4-26B-A4B-it-heretic)
for vLLM on a single 24 GB RTX 3090.

## Quantization

| Setting | Value |
| --- | --- |
| Method | RTN / `memoryless_minmax` |
| Weights | signed INT4, symmetric |
| Activations | BF16 at runtime |
| Group size | 64 |
| Format | `compressed-tensors`, `pack-quantized` |
| Source architecture | `Gemma4ForConditionalGeneration` |
| Source Heretic commit | `e6d09d5c434a5e91326ce9d779786a91e7aaae35` |
| Original quantization stack | `llm-compressor` main, Transformers 5.8.0 |

Vision modules, routers, embeddings, and `lm_head` remain unquantized. Group
size 64 is required because Gemma 4 projection widths such as 2112 and 704 are
not divisible by 128. The exact recipe is stored in `recipe.yaml`.

## Source refresh

On 2026-07-16, Google updated the Gemma 4 chat template but did not update the
model weights. The initial and current Google revisions have identical LFS
SHA256 values for both BF16 weight shards. This repository therefore keeps the
existing verified W4A16 tensors and refreshes the chat template, tokenizer
metadata, and processor metadata from Google revision
`01e5b3ee840d3a9e0b0b493c593e85398a30ef75`.

## vLLM

The production profile uses:

```bash
vllm serve WaveCut/gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm \
  --revision 51b3bbe6b47f12a1841e3934ce24e767bb8dec17 \
  --served-model-name "Gemma 4 26B Heretic" \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 40192 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.95 \
  --generation-config vllm \
  --no-enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4
```

The checkpoint was accepted for production on AI Farm with vLLM 0.25.1,
Torch 2.11.0+cu130, Transformers 5.8.0, and compressed-tensors 0.17.0.
On an RTX 3090, both vLLM 0.23.0 and 0.25.1 passed 13/13 text and vision case
groups plus streaming checks. The matrix included exact needle recovery at
39,186 text tokens and 38,490 multimodal tokens, including concurrency 2.
Both runtimes also passed 24/24 production-shaped Russian quality probes;
median probe latency improved from 0.6385 s to 0.6125 s on vLLM 0.25.1.

The full environment, timing table, harness, and raw JSONL artifacts are in
the [AI Farm vLLM report](https://github.com/iamwavecut/aifarm-vllm-openai/blob/af3a2915246a73585c175355af387c23e7360238/results/20260716-vllm-023-vs-0251.md).
The historical RTX 3090 quantization benchmark and harness remain available
under `bench/` and `scripts/`.

## Limitations

- RTN is a weight-only quantization; compare output quality against the BF16
  Heretic checkpoint for quality-sensitive workloads.
- The 40,192-token, two-request profile is a tight fit on 24 GB and leaves
  little free VRAM.
- This is a third-party post-processed and quantized checkpoint. It is not
  affiliated with or endorsed by Google.