---
license: apache-2.0
license_link: https://ai.google.dev/gemma/docs/gemma_4_license
pipeline_tag: text-generation
library_name: transformers
tags:
- heretic
- uncensored
- abliteration
- diffusion
- moe
base_model: google/diffusiongemma-26B-A4B-it
---

> **Note:** All credit to edwixx/diffusiongemma-26B-A4B-it-HERETIC-Uncensored, this repo only includes the processor_config.json

# diffusiongemma-26B-A4B-it-HERETIC-Uncensored

**26.6B params · 4B active · MoE · Diffusion · Apache-2.0** · ![Downloads](https://img.shields.io/badge/downloads-1.3k-blue)

This is the first abliteration of [DiffusionGemma 26B A4B](https://huggingface.co/google/diffusiongemma-26B-A4B-it), produced using [heretic](https://github.com/p-e-w/heretic) with custom patches to support its block-diffusion architecture and MoE expert layers.

DiffusionGemma is not a standard autoregressive transformer, so this required significant engineering work that hasn't been done before for this model class.

## Usage

Load using the `DiffusionGemmaForBlockDiffusion` class directly, not `AutoModelForCausalLM`:

```python
import torch
from transformers import AutoProcessor
from transformers.models.diffusion_gemma import DiffusionGemmaForBlockDiffusion

model_id = "FredyRivera-dev/diffusiongemma-26B-A4B-it-HERETIC-Uncensored"

processor = AutoProcessor.from_pretrained(model_id)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
).to("cuda")

messages = [
    {"role": "user", "content": "Hola, como estas?"}
]

inputs = processor.apply_chat_template(messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt").to("cuda")


out = model.generate(**inputs, max_new_tokens=200)

response = processor.decode(out.sequences[0], skip_special_tokens=True)
print(response)
```

### Multimodal Input

```python
import torch
from transformers import AutoProcessor
from transformers.models.diffusion_gemma import DiffusionGemmaForBlockDiffusion
from PIL import Image

model_id = "FredyRivera-dev/diffusiongemma-26B-A4B-it-HERETIC-Uncensored"

processor = AutoProcessor.from_pretrained(model_id)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
).to("cuda")

image = Image.open("web.png")

messages = [
    {"role": "user", "content": 
        [
            {"type": "text", "text": "Create the HTML code for the web page shown in the following image."},
            {"type": "image", "image": image}
        ]
    }
]

inputs = processor.apply_chat_template(messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt").to("cuda")


out = model.generate(**inputs, max_new_tokens=12288)

response = processor.decode(out.sequences[0], skip_special_tokens=True)
print(response)
```

## Results

| | |
|---|---|
| Base model | google/diffusiongemma-26B-A4B-it |
| Method | Heretic (directional ablation + LoRA + Optuna TPE) |
| Trials run | 200 |
| Best trial | #89 |
| Refusals | 13/100 (down from 100/100) |
| KL Divergence | 0.4909 |

## What had to be patched

Heretic assumes standard autoregressive models. DiffusionGemma needed several custom changes:

**Expert-Granular Abliteration (EGA):** The MoE `experts.down_proj` is a batched parameter `[128, 2816, 704]`, not a regular linear layer. Heretic skips it by default. We iterate over all 128 expert slices per layer and apply norm-preserving biprojected ablation to each one. Without this, refusals barely moved. Credit to [TrevorS](https://huggingface.co/TrevorS) for the original EGA idea on Gemma 4.

**Weight tying fix:** The encoder and decoder share the exact same weight tensors (confirmed via `data_ptr`). PEFT only wraps the encoder side, so the decoder wouldn't see the LoRA delta during generation. Fixed with a context manager that temporarily merges the LoRA into the shared base weights before each generation call.

**Task type:** DiffusionGemma's generation mixin doesn't implement `prepare_inputs_for_generation`, which the default `CAUSAL_LM` PEFT task type requires. Switched to `FEATURE_EXTRACTION`.

**Hidden states:** DiffusionGemma's `generate()` doesn't support `output_hidden_states`. Switched to forward hooks on encoder layers to capture per-layer activations for the refusal direction PCA.

**Output handling:** The model returns `DiffusionGemmaGenerationOutput` with a `.sequences` attribute, not a raw tensor like standard models. Patched all heretic output handling.

## Notes

This is a research release. The model will attempt to answer prompts it previously refused. 13/100 sensitive prompts still trigger refusals in testing.

## Known issue: "own" tokens

Some token positions in generated outputs show the word **"own"** where other content was expected.

Example output:
> "It pulls the **own** with **own** hands" (should be "It pulls the tide with gentle hands")

Initially assumed to be a diffusion denoising fallback, but [kabachuha pointed out](https://github.com/p-e-w/heretic/pull/378#issuecomment-4698062118) this artifact shows up in base autoregressive Gemma4 models too when weights are pushed hard (e.g. overfit LoRA training). Likely a Gemma4-level placeholder token, not specific to the diffusion architecture. A small LoRA fine-tune on clean data would probably reduce it. PRs welcome.

## Citation

```bibtex
@misc{edwixx-diffusiongemma-26B-A4B-it-HERETIC-Uncensored,
  author = {Anurag Kanade},
  title = {diffusiongemma-26B-A4B-it-HERETIC-Uncensored},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub},
  howpublished = {\url{https://huggingface.co/edwixx/diffusiongemma-26B-A4B-it-HERETIC-Uncensored}}
}
```