Gemma-4-E4B-it-GLQ-6bpw

GLQ (6.0 bpw uniform) quantization of google/gemma-4-E4B-it.

Property	Value
Average bpw	6.0
Quantization scheme	uniform
File size	7.06 GB (2.11× smaller than bf16)
gsm8k limit=50 chat+thinking strict-match	92% (bf16 baseline: 86%)
Recommended max_new_tokens for thinking mode	16K

License

License: Apache 2.0 — see https://ai.google.dev/gemma/docs/gemma_4_license

Original model: https://huggingface.co/google/gemma-4-E4B-it

Derivative quantization work; the base Gemma 4 model and the GLQ tooling (https://github.com/cnygaard/glq) are both Apache 2.0. Quantizing the weights does not change the license.

Method

This is a Golay-Leech Quantization (GLQ) of the original bf16 weights:

• E8 lattice shell codebook (65536 entries, 8-D blocks)
• Randomized Hadamard Transform (RHT) for input/output rotation
• LDLQ (block LDL decomposition) feedback during encoding
• N-stage residual quantization for ≥3 bpw layers
• Mixed-precision allocation via Hessian-trace-derived sensitivity proxy (where applicable)

See the GLQ repo for details: https://github.com/cnygaard/glq

Usage (HuggingFace transformers)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import glq, glq.hf_integration  # registers the GLQ quantization config

model_id = "xv0y5ncu/Gemma-4-E4B-it-GLQ-6bpw"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="cuda", dtype=torch.bfloat16,
)

# For thinking-mode (math reasoning, multi-step problems):
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "What is 12 * 17?"}],
    tokenize=False, add_generation_prompt=True,
    enable_thinking=True,
)
ids = tok(prompt, return_tensors="pt").input_ids.cuda()
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=16384, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=False))

Installation

pip install glq "transformers>=5.13.1,<5.15"

Pin transformers below 5.15 — 5.15.0 moved gemma-4 to a per-layer config, so config.head_dim raises and vLLM dies before loading a single weight. Verified on 5.14.1 with vLLM 0.27.1. Not specific to GLQ: stock bf16 gemma-4 fails the same way.

The custom CUDA kernel JIT-compiles on first use (~30s). Requires torch>=2.0, transformers>=5.0, CUDA 12.x.

Note on max_new_tokens

Quantized variants use the model's chat template with enable_thinking=True and may require a larger thinking budget than bf16 to converge to the final answer. Empirically:

Variant	Recommended max_new_tokens for thinking
8bpw, 7bpw	2048
4bpw, 5bpw mix, 6bpw	8192-16384

Insufficient budget will cause the model to truncate mid-thought and the final answer may be missing. This is a generation-config concern, not a quality concern: when given enough budget, all variants converge to within stderr of bf16 quality on gsm8k.

lm-eval workaround for thinking mode

The lm-evaluation-harness does not currently expose enable_thinking=True through apply_chat_template. Until upstream support lands, monkey-patch the tokenizer before evaluating:

def patch_thinking(tokenizer):
    orig = tokenizer.apply_chat_template
    def patched(*args, **kwargs):
        kwargs.setdefault("enable_thinking", True)
        return orig(*args, **kwargs)
    tokenizer.apply_chat_template = patched

---

🔗 GLQ on GitHub: https://github.com/cnygaard/glq — if you like it, a ⭐ is appreciated.