Instructions to use xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega")
model = AutoModelForMultimodalLM.from_pretrained("xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega

SGLang

How to use xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega with Docker Model Runner:
```
docker model run hf.co/xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

gemma-4-26B-A4B-it-qat-heretic-ega

Model Overview

Model Architecture: Gemma 4
- Input: Text / Image
- Output: Text
Model Modifications:
- Refusal ablation: Norm-preserving biprojection (o_proj + shared mlp.down_proj) + Expert-Granular Abliteration (all MoE expert down_proj slices)
- Precision: BF16 (unchanged)
Release Date: 2026-09-06
Version: 1.0
Ablated by: xdavxd
Base Model: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized (revision f1e06dc)
Original Model: google/gemma-4-26B-A4B-it

This model is a refusal-ablated version of google/gemma-4-26B-A4B-it-qat-q4_0-unquantized. It was evaluated on several tasks to assess its quality in comparison to the original model.

Model Modifications

This model was obtained by applying TrevorJS's norm-preserving biprojected abliteration plus Expert-Granular Abliteration (EGA) to the QAT (quantization-aware-trained, Q4_0) checkpoint of Gemma 4 26B A4B, using heretic for model loading, residual extraction and evaluation.

Per-layer refusal directions are computed from winsorized (99.5th percentile) residual activations on 400 harmful + 400 harmless prompts, orthogonalized against the harmless mean, and projected out of attn.o_proj and the shared mlp.down_proj in all 30 layers with row-norm preservation (60 dense tensors). The same projection is then applied to each of the 128 experts' down_proj slices per layer (3,840 expert slices). Scale 1.0, expert scale 1.0. Attention q/k/v, gate/up projections, router, embeddings, vision tower and output head are untouched. Tensor manifest is identical to the base (1,013 tensors, same names).

This is the "o_proj + down_proj" method family (capability-preserving per the E4B abliteration report), extended into the routed experts. Weights are BF16; this is not a quantized model.

Deployment

Use with vLLM

This model can be deployed using vLLM. For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the Gemma 4 vLLM usage guide.

Start the vLLM server:

vllm serve xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

To enable thinking/reasoning and tool calling:

vllm serve xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --limit-mm-per-prompt '{"image": 4}' \
  --async-scheduling \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant","num_speculative_tokens":3}'

MTP: use the QAT-specific drafter google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant, not the vanilla one. On reasoning workloads it accepts ~78% of drafts vs ~53% for the vanilla drafter, a 2.2× vs 1.6× speedup. Lossless in distribution.

Tip: For text-only workloads, pass --language-model-only to skip vision encoder memory allocation and free up GPU memory for a longer context window.

Send requests to the server:

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8001/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

model = "xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega"

messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
    model=model,
    messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)

Creation

This model was created with TrevorJS's gemma-4-abliteration scripts on top of heretic git HEAD (1.4 plugin API, via a small compatibility shim), as presented in the code snippet below.

Stack: transformers 5.16.1, torch 2.13.0+cu130, heretic git HEAD, DGX Spark GB10 (128 GB).

pip install 'heretic-llm @ git+https://github.com/p-e-w/heretic' sentencepiece protobuf
git clone https://github.com/TrevorS/gemma-4-abliteration && cd gemma-4-abliteration

hf download google/gemma-4-26B-A4B-it-qat-q4_0-unquantized --local-dir ~/models/base/gemma-4-26B-qat

HF_DATASETS_CACHE=/tmp/hf_datasets_cache python scripts/ega.py \
  --model ~/models/base/gemma-4-26B-qat \
  --strip-topic-markers --skip-prefix --batch-size 4 \
  --save ~/models/abl/gemma-4-26B-qat-ega

# heretic's save drops processor files; restore tokenizer/processor from base
cp -L ~/models/base/gemma-4-26B-qat/{tokenizer*,*processor*.json,chat_template*,generation_config.json} ~/models/abl/gemma-4-26B-qat-ega/

Settings passed to heretic: offload_outputs_to_cpu=False, device_map={"": 0}, orthogonalize_direction=True, winsorization_quantile=0.995. Refusal markers: heretic defaults minus the seven topic words (violat, prohibit, illegal, harmful, inappropriate, unethical, ethical boundaries) and minus disclaimer (added to heretic in June 2026; Gemma 4 prepends "Disclaimer:" to compliant answers, which inflates keyword refusal counts ~25×).

Evaluation

This model was evaluated on IFEval, GSM8K Platinum, MATH-500, GPQA Diamond, WikiText-2, TruthfulQA-MC2, a 300-problem GSM8K termination split, and a needle-in-haystack long-context test using lm-evaluation-harness, served with vLLM (OpenAI-compatible API).

Performance

Metric	This model	Original model (google/gemma-4-26B-A4B-it-qat-q4_0-unquantized)
KL divergence (first-token, 100 harmless_alpaca)	0.0789	0 (by definition)
Refusals (mlabonne/harmful_behaviors, 100 prompts, keyword)	4/100	100/100
Refusals (686-prompt cross-dataset audit, keyword)	11/686	-
Refusals (686-prompt audit, manually audited)	~1–3/686	-

Cross-dataset audit: JailbreakBench 3/100, tulu-harmbench 3/320, NousResearch/RefusalDataset 1/166, mlabonne 4/100. All 11 keyword flags are long responses that answer the prompt; most trip on an "I am an AI, not a doctor/attorney" preamble or on a marker string occurring inside generated content. For reference, TrevorJS/gemma-4-26B-A4B-it-uncensored (same method, vanilla base) scores KL 0.090 and 3/686 on the identical harness.

Accuracy

Three columns: the original model, Google's QAT release, and this checkpoint. All measured on the same hardware (NVIDIA GB10), same vLLM build, same server config, same seed, with the QAT-specific MTP drafter on every server. The difference between the first two is the cost of QAT; between the last two is the cost of abliteration. Recovery is this model divided by the QAT base (abliteration cost only). Protocol follows RedHatAI's: 0-shot, temperature 1.0, top-p 0.95, top-k 64, max_gen_toks=32000, seed 1234, 1 repetition.

The vanilla and QAT columns are complete for round 1 (IFEval, wikitext, TruthfulQA, GSM8K split) and show the two bases are equivalent on every chat-templated task; they differ only on raw-text perplexity, where vanilla scores 24% worse for reasons not yet explained. Round 2 (Platinum, MATH-500, GPQA, no-think) was run against QAT as the reference base and not repeated on vanilla.

I ran these to confirm that abliterating the QAT base didn't cost capability. It's a sanity check with single seeds, not a statistically rigorous comparison. The result: abliteration costs about 5 points on TruthfulQA and 2–4 points on MATH-500, and is within noise everywhere else. Termination and long-context retrieval are unaffected. The NVFP4 quant of this checkpoint is reported on its own model card.

One 26B-specific observation: MATH-500 scores higher without thinking on all three models (QAT 86.8 vs 83.8, EGA 83.2 vs 81.8). The MoE's reasoning traces cost it a few points on this benchmark.

With thinking

Category	Benchmark	google/gemma-4-26B-A4B-it	google/gemma-4-26B-A4B-it-qat-q4_0-unquantized	xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega	Recovery
Instruction Following	IFEval (0-shot, prompt-level strict)	94.82	94.64	92.98	98.2%
Instruction Following	IFEval (0-shot, inst-level strict)	96.40	96.16	95.32	99.1%
Reasoning	GSM8K Platinum (0-shot, flexible-extract)	-	95.70	96.03	100.3%
	GSM8K Platinum (0-shot, strict-match)	-	95.70	96.03	100.3%
	MATH-500 (0-shot, math_verify)	-	83.80	81.80	97.6%
	GPQA Diamond (0-shot, CoT)	-	78.79	78.28	99.4%

Without thinking

Category	Benchmark	google/gemma-4-26B-A4B-it	google/gemma-4-26B-A4B-it-qat-q4_0-unquantized	xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega	Recovery
Instruction Following	IFEval (0-shot, prompt-level strict)	-	88.17	87.80	99.6%
Instruction Following	IFEval (0-shot, inst-level strict)	-	91.73	91.73	100.0%
Reasoning	GSM8K Platinum (0-shot, flexible-extract)	-	94.79	95.37	100.6%
	GSM8K Platinum (0-shot, strict-match)	-	94.87	95.29	100.4%
	MATH-500 (0-shot, math_verify)	-	86.80	83.20	95.9%

Perplexity, truthfulness, and reasoning stability

Category	Benchmark	google/gemma-4-26B-A4B-it	google/gemma-4-26B-A4B-it-qat-q4_0-unquantized	xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega	Recovery
Perplexity	WikiText-2 (byte perplexity, lower is better)	4.1616	3.3544	3.3800	+0.8%
Truthfulness	TruthfulQA-MC2 (0-shot)	63.80	63.37	58.10	91.7%
Reasoning stability	GSM8K (0-shot, temp 0, headline)	91.7	90.7	89.0	98.1%
	GSM8K (0-shot, temp 0, empty rate)	3.0	5.7	6.0	-
	GSM8K (0-shot, temp 0, answered-only)	94.5	96.1	94.7	98.5%
Long context	Needle-in-haystack (4k–64k, 5 depths each)	-	25/25	25/25	100%

Reproduction

The results were obtained using the following commands:

Single seed (1234) per benchmark. All three models served on NVIDIA GB10 with the same vLLM build and identical server flags apart from the model path.

vLLM server:

ghcr.io/timothystewart6/vllm-gb10:latest (v0.28.1.dev0+g2cf0a6915.d20260828, transformers 5.16.1). The QAT-specific MTP drafter is lossless under rejection sampling and only affects throughput.

docker run --rm -it \
  --gpus all --ipc=host --network host \
  -v ~/models/abl/gemma-4-26B-qat-ega:/models/qat-ega:ro \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  ghcr.io/timothystewart6/vllm-gb10:latest \
  vllm serve /models/qat-ega \
  --host 0.0.0.0 --port 8001 \
  --served-model-name qat-ega \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.75 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 8192 \
  --language-model-only \
  --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --async-scheduling \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant","num_speculative_tokens":3}'

To reproduce the without-thinking results, remove --default-chat-template-kwargs '{"enable_thinking": true}'.

Deviations from RedHatAI's protocol: --max-model-len 65536 rather than 32768 — with max_gen_toks=32000, a 32768 ceiling leaves 768 tokens for the prompt and MATH-500 has longer problems, which vLLM rejects with HTTP 400. timeout=3600 rather than 1200 — BF16 on GB10 needs ~31 minutes to exhaust a 32000-token budget. until=[] on all tasks — the default stop sequences truncate thinking traces mid-reasoning. Model's shipped chat template rather than examples/tool_chat_template_gemma4.jinja. max_retries=6.

GSM8K Platinum (lm-eval, 0-shot)

lm_eval --model local-chat-completions \
  --tasks gsm8k_platinum_cot_llama \
  --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 --apply_chat_template \
  --output_path results/qat-ega_gsm8k_platinum \
  --seed 1234 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"

IFEval (lm-eval, 0-shot)

lm_eval --model local-chat-completions \
  --tasks ifeval \
  --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 --apply_chat_template \
  --output_path results/qat-ega_ifeval \
  --seed 1234 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"

MATH-500 (lm-eval `minerva_math500`, 0-shot)

Requires pip install 'lm-eval[math]' and antlr4-python3-runtime==4.11. Same 500-problem subset as lighteval's math_500, scored with sympy-based answer equivalence (math_verify). The exact_match filter reports 0 on thinking-mode output and is not the reported number.

lm_eval --model local-chat-completions \
  --tasks minerva_math500 \
  --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 --apply_chat_template \
  --output_path results/qat-ega_math500 \
  --seed 1234 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"

GPQA Diamond (lm-eval `gpqa_diamond_cot_zeroshot`, 0-shot)

Dataset is gated; requires an HF token that has accepted the terms for Idavidrein/gpqa. flexible-extract is the reported number.

lm_eval --model local-chat-completions \
  --tasks gpqa_diamond_cot_zeroshot \
  --model_args "model=qat-ega,max_length=65536,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=32,max_retries=6,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 --apply_chat_template \
  --output_path results/qat-ega_gpqa \
  --seed 1234 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234,until=[]"

WikiText-2 and TruthfulQA-MC2 (lm-eval, loglikelihood)

lm_eval --model local-completions \
  --model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \
  --tasks wikitext --num_fewshot 0 --batch_size 1

lm_eval --model local-completions \
  --model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/completions,tokenizer=/path/to/checkpoint,num_concurrent=8,max_retries=3,tokenized_requests=True" \
  --tasks truthfulqa_mc2 --num_fewshot 0 --apply_chat_template

GSM8K three-number split (lm-eval, 0-shot, temperature 0, 300-problem subset)

lm_eval --model local-chat-completions \
  --tasks gsm8k \
  --model_args "model=qat-ega,base_url=http://0.0.0.0:8001/v1/chat/completions,num_concurrent=16,max_retries=3,timeout=3600" \
  --num_fewshot 0 --limit 300 --apply_chat_template \
  --gen_kwargs "max_gen_toks=8192,until=[]" \
  --output_path results/qat-ega_gsm8k_300 --log_samples

Rescored from the samples file: a response is empty if it contains no content (thinking never terminated); headline is correct / total; answered-only is correct / (total − empty). Extraction takes the last number in the response after stripping markdown and thousands separators, since lm-eval's flexible-extract filter returns [invalid] on this model's bolded answer formatting.

Needle-in-a-haystack

Custom script: single needle (a random vault code) buried in WikiText-2 filler at depths 0/0.25/0.5/0.75/1.0 for context lengths 4k/8k/16k/32k/64k, thinking off, temperature 0, exact-match on the code. 25 requests per model.

Downloads last month: 78

Safetensors

Model size

26B params

Tensor type

BF16

Model tree for xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega

Base model

google/gemma-4-26B-A4B

Finetuned

google/gemma-4-26B-A4B-it

Finetuned

google/gemma-4-26B-A4B-it-qat-q4_0-unquantized

Finetuned

(16)

this model

Quantizations

1 model

gemma-4-26B-A4B-it-qat-heretic-ega

Model Overview

Model Modifications

Deployment

Use with vLLM

Creation

Evaluation

Performance

Accuracy

Reproduction

GSM8K Platinum (lm-eval, 0-shot)

IFEval (lm-eval, 0-shot)

MATH-500 (lm-eval minerva_math500, 0-shot)

GPQA Diamond (lm-eval gpqa_diamond_cot_zeroshot, 0-shot)

WikiText-2 and TruthfulQA-MC2 (lm-eval, loglikelihood)

GSM8K three-number split (lm-eval, 0-shot, temperature 0, 300-problem subset)

Needle-in-a-haystack

Model tree for xdavxd/gemma-4-26B-A4B-it-qat-heretic-ega

MATH-500 (lm-eval `minerva_math500`, 0-shot)

GPQA Diamond (lm-eval `gpqa_diamond_cot_zeroshot`, 0-shot)