| --- |
| license: apache-2.0 |
| license_link: https://ai.google.dev/gemma/docs/gemma_4_license |
| base_model: WaveCut/gemma-4-26B-A4B-it-heretic |
| base_model_relation: quantized |
| library_name: vllm |
| pipeline_tag: image-text-to-text |
| tags: |
| - gemma4 |
| - heretic |
| - vllm |
| - llm-compressor |
| - compressed-tensors |
| - w4a16 |
| - int4 |
| - rtn |
| - g64 |
| - moe |
| - multimodal |
| --- |
| |
| # Gemma 4 26B A4B Heretic W4A16 G64 RTN for vLLM |
|
|
| `compressed-tensors` W4A16 quantization of |
| [`WaveCut/gemma-4-26B-A4B-it-heretic`](https://huggingface.co/WaveCut/gemma-4-26B-A4B-it-heretic) |
| for vLLM on a single 24 GB RTX 3090. |
|
|
| ## Quantization |
|
|
| | Setting | Value | |
| | --- | --- | |
| | Method | RTN / `memoryless_minmax` | |
| | Weights | signed INT4, symmetric | |
| | Activations | BF16 at runtime | |
| | Group size | 64 | |
| | Format | `compressed-tensors`, `pack-quantized` | |
| | Source architecture | `Gemma4ForConditionalGeneration` | |
| | Source Heretic commit | `e6d09d5c434a5e91326ce9d779786a91e7aaae35` | |
| | Original quantization stack | `llm-compressor` main, Transformers 5.8.0 | |
|
|
| Vision modules, routers, embeddings, and `lm_head` remain unquantized. Group |
| size 64 is required because Gemma 4 projection widths such as 2112 and 704 are |
| not divisible by 128. The exact recipe is stored in `recipe.yaml`. |
|
|
| ## Source refresh |
|
|
| On 2026-07-16, Google updated the Gemma 4 chat template but did not update the |
| model weights. The initial and current Google revisions have identical LFS |
| SHA256 values for both BF16 weight shards. This repository therefore keeps the |
| existing verified W4A16 tensors and refreshes the chat template, tokenizer |
| metadata, and processor metadata from Google revision |
| `01e5b3ee840d3a9e0b0b493c593e85398a30ef75`. |
|
|
| ## vLLM |
|
|
| The production profile uses: |
|
|
| ```bash |
| vllm serve WaveCut/gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm \ |
| --revision 51b3bbe6b47f12a1841e3934ce24e767bb8dec17 \ |
| --served-model-name "Gemma 4 26B Heretic" \ |
| --trust-remote-code \ |
| --dtype bfloat16 \ |
| --max-model-len 40192 \ |
| --max-num-seqs 2 \ |
| --max-num-batched-tokens 8192 \ |
| --gpu-memory-utilization 0.95 \ |
| --generation-config vllm \ |
| --no-enable-prefix-caching \ |
| --enable-auto-tool-choice \ |
| --tool-call-parser gemma4 \ |
| --reasoning-parser gemma4 |
| ``` |
|
|
| The checkpoint was accepted for production on AI Farm with vLLM 0.25.1, |
| Torch 2.11.0+cu130, Transformers 5.8.0, and compressed-tensors 0.17.0. |
| On an RTX 3090, both vLLM 0.23.0 and 0.25.1 passed 13/13 text and vision case |
| groups plus streaming checks. The matrix included exact needle recovery at |
| 39,186 text tokens and 38,490 multimodal tokens, including concurrency 2. |
| Both runtimes also passed 24/24 production-shaped Russian quality probes; |
| median probe latency improved from 0.6385 s to 0.6125 s on vLLM 0.25.1. |
|
|
| The full environment, timing table, harness, and raw JSONL artifacts are in |
| the [AI Farm vLLM report](https://github.com/iamwavecut/aifarm-vllm-openai/blob/af3a2915246a73585c175355af387c23e7360238/results/20260716-vllm-023-vs-0251.md). |
| The historical RTX 3090 quantization benchmark and harness remain available |
| under `bench/` and `scripts/`. |
|
|
| ## Limitations |
|
|
| - RTN is a weight-only quantization; compare output quality against the BF16 |
| Heretic checkpoint for quality-sensitive workloads. |
| - The 40,192-token, two-request profile is a tight fit on 24 GB and leaves |
| little free VRAM. |
| - This is a third-party post-processed and quantized checkpoint. It is not |
| affiliated with or endorsed by Google. |
|
|
