--- tags: - abliterated - uncensored - heretic - ega - biprojection - vllm - gemma4 - gemma4-26b-A4B-it - qat library_name: transformers license: apache-2.0 license_link: https://ai.google.dev/gemma/docs/gemma_4_license pipeline_tag: image-text-to-text base_model: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized provider: xdavxd name: xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega description: Refusal-ablated (biprojection + EGA) variant of gemma-4-26B-A4B-it-qat-q4_0-unquantized. tool_calling_supported: true required_cli_args: - '--reasoning-parser gemma4' - '--enable-prefix-caching' default-chat-template-kwargs: '{"enable_thinking": true}' chat_template_file_name: chat_template.jinja tool_call_parser: gemma4 validated_tasks: - tool-calling tasks: - text-to-text - text-generation - tool-calling --- # gemma-4-26B-A4B-it-qat-heretic-ega ## Model Overview - **Model Architecture:** Gemma 4 - **Input:** Text / Image - **Output:** Text - **Model Modifications:** - **Refusal ablation:** Norm-preserving biprojection (o_proj + shared mlp.down_proj) + Expert-Granular Abliteration (all MoE expert down_proj slices) - **Precision:** BF16 (unchanged) - **Release Date:** 2026-09-06 - **Version:** 1.0 - **Ablated by:** xdavxd - **Base Model:** [google/gemma-4-26B-A4B-it-qat-q4_0-unquantized](https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-unquantized) (revision `f1e06dc`) - **Original Model:** [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) This model is a refusal-ablated version of [google/gemma-4-26B-A4B-it-qat-q4_0-unquantized](https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-unquantized). It was evaluated on several tasks to assess its quality in comparison to the original model. ### Model Modifications This model was obtained by applying [TrevorJS's](https://github.com/TrevorS/gemma-4-abliteration) norm-preserving biprojected abliteration plus Expert-Granular Abliteration (EGA) to the QAT (quantization-aware-trained, Q4_0) checkpoint of Gemma 4 26B A4B, using [heretic](https://github.com/p-e-w/heretic) for model loading, residual extraction and evaluation. Per-layer refusal directions are computed from winsorized (99.5th percentile) residual activations on 400 harmful + 400 harmless prompts, orthogonalized against the harmless mean, and projected out of `attn.o_proj` and the shared `mlp.down_proj` in all 30 layers with row-norm preservation (60 dense tensors). The same projection is then applied to each of the 128 experts' `down_proj` slices per layer (3,840 expert slices). Scale 1.0, expert scale 1.0. Attention q/k/v, gate/up projections, router, embeddings, vision tower and output head are untouched. Tensor manifest is identical to the base (1,013 tensors, same names). This is the "o_proj + down_proj" method family (capability-preserving per the E4B abliteration report), extended into the routed experts. Weights are BF16; this is not a quantized model. ## Deployment ### Use with vLLM This model can be deployed using [vLLM](https://docs.vllm.ai/en/latest/). For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the [Gemma 4 vLLM usage guide](https://recipes.vllm.ai/Google/gemma-4-26B-A4B-it). 1. Start the vLLM server: ``` vllm serve xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 ``` To enable thinking/reasoning and tool calling: ``` vllm serve xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --default-chat-template-kwargs '{"enable_thinking": true}' \ --limit-mm-per-prompt '{"image": 4}' \ --async-scheduling \ --speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant","num_speculative_tokens":3}' ``` > **MTP:** use the QAT-specific drafter `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant`, not the vanilla one. On reasoning workloads it accepts ~78% of drafts vs ~53% for the vanilla drafter, a 2.2× vs 1.6× speedup. Lossless in distribution. > > **Tip:** For text-only workloads, pass `--language-model-only` to skip vision encoder memory allocation and free up GPU memory for a longer context window. 2. Send requests to the server: ```python from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://:8001/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) model = "xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega" messages = [ {"role": "user", "content": "Explain quantum mechanics clearly and concisely."}, ] outputs = client.chat.completions.create( model=model, messages=messages, ) generated_text = outputs.choices[0].message.content print(generated_text) ``` ## Creation This model was created with [TrevorJS's gemma-4-abliteration scripts](https://github.com/TrevorS/gemma-4-abliteration) on top of [heretic](https://github.com/p-e-w/heretic) git HEAD (1.4 plugin API, via a small compatibility shim), as presented in the code snippet below.
Stack: transformers 5.16.1, torch 2.13.0+cu130, heretic git HEAD, DGX Spark GB10 (128 GB). ```shell pip install 'heretic-llm @ git+https://github.com/p-e-w/heretic' sentencepiece protobuf git clone https://github.com/TrevorS/gemma-4-abliteration && cd gemma-4-abliteration hf download google/gemma-4-26B-A4B-it-qat-q4_0-unquantized --local-dir ~/models/base/gemma-4-26B-qat HF_DATASETS_CACHE=/tmp/hf_datasets_cache python scripts/ega.py \ --model ~/models/base/gemma-4-26B-qat \ --strip-topic-markers --skip-prefix --batch-size 4 \ --save ~/models/abl/gemma-4-26B-qat-ega # heretic's save drops processor files; restore tokenizer/processor from base cp -L ~/models/base/gemma-4-26B-qat/{tokenizer*,*processor*.json,chat_template*,generation_config.json} ~/models/abl/gemma-4-26B-qat-ega/ ``` Settings passed to heretic: `offload_outputs_to_cpu=False`, `device_map={"": 0}`, `orthogonalize_direction=True`, `winsorization_quantile=0.995`. Refusal markers: heretic defaults minus the seven topic words (`violat`, `prohibit`, `illegal`, `harmful`, `inappropriate`, `unethical`, `ethical boundaries`) and minus `disclaimer` (added to heretic in June 2026; Gemma 4 prepends "Disclaimer:" to compliant answers, which inflates keyword refusal counts ~25×).
## Evaluation This model was evaluated on IFEval, GSM8K Platinum, MATH-500, GPQA Diamond, WikiText-2, TruthfulQA-MC2, a 300-problem GSM8K termination split, and a needle-in-haystack long-context test using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), served with [vLLM](https://docs.vllm.ai/en/latest/) (OpenAI-compatible API). ### Performance | Metric | This model | Original model ([google/gemma-4-26B-A4B-it-qat-q4_0-unquantized](https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-unquantized)) | | :----- | :--------: | :---------------------------: | | **KL divergence** (first-token, 100 harmless_alpaca) | 0.0789 | 0 *(by definition)* | | **Refusals** (mlabonne/harmful_behaviors, 100 prompts, keyword) | 4/100 | 100/100 | | **Refusals** (686-prompt cross-dataset audit, keyword) | 11/686 | - | | **Refusals** (686-prompt audit, manually audited) | ~1–3/686 | - | Cross-dataset audit: JailbreakBench 3/100, tulu-harmbench 3/320, NousResearch/RefusalDataset 1/166, mlabonne 4/100. All 11 keyword flags are long responses that answer the prompt; most trip on an "I am an AI, not a doctor/attorney" preamble or on a marker string occurring inside generated content. For reference, [TrevorJS/gemma-4-26B-A4B-it-uncensored](https://huggingface.co/TrevorJS/gemma-4-26B-A4B-it-uncensored) (same method, vanilla base) scores KL 0.090 and 3/686 on the identical harness. ### Accuracy Three columns: the original model, Google's QAT release, and this checkpoint. All measured on the same hardware (NVIDIA GB10), same vLLM build, same server config, same seed, with the QAT-specific MTP drafter on every server. The difference between the first two is the cost of QAT; between the last two is the cost of abliteration. Recovery is this model divided by the QAT base (abliteration cost only). Protocol follows [RedHatAI](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4)'s: 0-shot, temperature 1.0, top-p 0.95, top-k 64, `max_gen_toks=32000`, seed 1234, 1 repetition. The vanilla and QAT columns are complete for round 1 (IFEval, wikitext, TruthfulQA, GSM8K split) and show the two bases are equivalent on every chat-templated task; they differ only on raw-text perplexity, where vanilla scores 24% worse for reasons not yet explained. Round 2 (Platinum, MATH-500, GPQA, no-think) was run against QAT as the reference base and not repeated on vanilla. I ran these to confirm that abliterating the QAT base didn't cost capability. It's a sanity check with single seeds, not a statistically rigorous comparison. The result: abliteration costs about 5 points on TruthfulQA and 2–4 points on MATH-500, and is within noise everywhere else. Termination and long-context retrieval are unaffected. The NVFP4 quant of this checkpoint is reported on its own model card. One 26B-specific observation: MATH-500 scores higher *without* thinking on all three models (QAT 86.8 vs 83.8, EGA 83.2 vs 81.8). The MoE's reasoning traces cost it a few points on this benchmark. **With thinking**
Category Benchmark google/gemma-4-26B-A4B-it google/gemma-4-26B-A4B-it-qat-q4_0-unquantized xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega Recovery
Instruction Following IFEval (0-shot, prompt-level strict) 94.82 94.64 92.98 98.2%
IFEval (0-shot, inst-level strict) 96.40 96.16 95.32 99.1%
Reasoning GSM8K Platinum (0-shot, flexible-extract) - 95.70 96.03 100.3%
GSM8K Platinum (0-shot, strict-match) - 95.70 96.03 100.3%
MATH-500 (0-shot, math_verify) - 83.80 81.80 97.6%
GPQA Diamond (0-shot, CoT) - 78.79 78.28 99.4%
**Without thinking**
Category Benchmark google/gemma-4-26B-A4B-it google/gemma-4-26B-A4B-it-qat-q4_0-unquantized xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega Recovery
Instruction Following IFEval (0-shot, prompt-level strict) - 88.17 87.80 99.6%
IFEval (0-shot, inst-level strict) - 91.73 91.73 100.0%
Reasoning GSM8K Platinum (0-shot, flexible-extract) - 94.79 95.37 100.6%
GSM8K Platinum (0-shot, strict-match) - 94.87 95.29 100.4%
MATH-500 (0-shot, math_verify) - 86.80 83.20 95.9%
**Perplexity, truthfulness, and reasoning stability**
Category Benchmark google/gemma-4-26B-A4B-it google/gemma-4-26B-A4B-it-qat-q4_0-unquantized xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega Recovery
Perplexity WikiText-2 (byte perplexity, lower is better) 4.1616 3.3544 3.3800 +0.8%
Truthfulness TruthfulQA-MC2 (0-shot) 63.80 63.37 58.10 91.7%
Reasoning stability GSM8K (0-shot, temp 0, headline) 91.7 90.7 89.0 98.1%
GSM8K (0-shot, temp 0, empty rate) 3.0 5.7 6.0 -
GSM8K (0-shot, temp 0, answered-only) 94.5 96.1 94.7 98.5%
Long context Needle-in-haystack (4k–64k, 5 depths each) - 25/25 25/25 100%
### Reproduction The results were obtained using the following commands:
Single seed (1234) per benchmark. All three models served on NVIDIA GB10 with the same vLLM build and identical server flags apart from the model path. **vLLM server:** `ghcr.io/timothystewart6/vllm-gb10:latest` (`v0.28.1.dev0+g2cf0a6915.d20260828`, transformers 5.16.1). The QAT-specific MTP drafter is lossless under rejection sampling and only affects throughput. ```shell docker run --rm -it \ --gpus all --ipc=host --network host \ -v ~/models/abl/gemma-4-26B-qat-ega:/models/qat-ega:ro \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e VLLM_USE_V2_MODEL_RUNNER=1 \ ghcr.io/timothystewart6/vllm-gb10:latest \ vllm serve /models/qat-ega \ --host 0.0.0.0 --port 8001 \ --served-model-name qat-ega \ --max-model-len 65536 \ --gpu-memory-utilization 0.75 \ --max-num-seqs 32 \ --max-num-batched-tokens 8192 \ --language-model-only \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --async-scheduling \ --default-chat-template-kwargs '{"enable_thinking": true}' \ --speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant","num_speculative_tokens":3}' ``` > To reproduce the without-thinking results, remove `--default-chat-template-kwargs '{"enable_thinking": true}'`. > > **Deviations from RedHatAI's protocol:** `--max-model-len 65536` rather than 32768 — with `max_gen_toks=32000`, a 32768 ceiling leaves 768 tokens for the prompt and MATH-500 has longer problems, which vLLM rejects with HTTP 400. `timeout=3600` rather than 1200 — BF16 on GB10 needs ~31 minutes to exhaust a 32000-token budget. `until=[]` on all tasks — the default stop sequences truncate thinking traces mid-reasoning. Model's shipped chat template rather than `examples/tool_chat_template_gemma4.jinja`. `max_retries=6`. #### GSM8K Platinum (lm-eval, 0-shot) ```shell lm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 --apply_chat_template \ --output_path results/qat-ega_gsm8k_platinum \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]" ``` #### IFEval (lm-eval, 0-shot) ```shell lm_eval --model local-chat-completions \ --tasks ifeval \ --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 --apply_chat_template \ --output_path results/qat-ega_ifeval \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]" ``` #### MATH-500 (lm-eval `minerva_math500`, 0-shot) Requires `pip install 'lm-eval[math]'` and `antlr4-python3-runtime==4.11`. Same 500-problem subset as lighteval's `math_500`, scored with sympy-based answer equivalence (`math_verify`). The `exact_match` filter reports 0 on thinking-mode output and is not the reported number. ```shell lm_eval --model local-chat-completions \ --tasks minerva_math500 \ --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 --apply_chat_template \ --output_path results/qat-ega_math500 \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]" ``` #### GPQA Diamond (lm-eval `gpqa_diamond_cot_zeroshot`, 0-shot) Dataset is gated; requires an HF token that has accepted the terms for `Idavidrein/gpqa`. `flexible-extract` is the reported number. ```shell lm_eval --model local-chat-completions \ --tasks gpqa_diamond_cot_zeroshot \ --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 --apply_chat_template \ --output_path results/qat-ega_gpqa \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]" ``` #### WikiText-2 and TruthfulQA-MC2 (lm-eval, loglikelihood) ```shell lm_eval --model local-completions \ --model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \ --tasks wikitext --num_fewshot 0 --batch_size 1 lm_eval --model local-completions \ --model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \ --tasks truthfulqa_mc2 --num_fewshot 0 --apply_chat_template ``` #### GSM8K three-number split (lm-eval, 0-shot, temperature 0, 300-problem subset) ```shell lm_eval --model local-chat-completions \ --tasks gsm8k \ --model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=16,max_retries=3,timeout=3600" \ --num_fewshot 0 --limit 300 --apply_chat_template \ --gen_kwargs "max_gen_toks=8192,until=[]" \ --output_path results/qat-ega_gsm8k_300 --log_samples ``` Rescored from the samples file: a response is *empty* if it contains no content (thinking never terminated); *headline* is correct / total; *answered-only* is correct / (total − empty). Extraction takes the last number in the response after stripping markdown and thousands separators, since lm-eval's flexible-extract filter returns `[invalid]` on this model's bolded answer formatting. #### Needle-in-a-haystack Custom script: single needle (a random vault code) buried in WikiText-2 filler at depths 0/0.25/0.5/0.75/1.0 for context lengths 4k/8k/16k/32k/64k, thinking off, temperature 0, exact-match on the code. 25 requests per model.