README.md

---
license: apache-2.0
language:
- en
base_model:
- Altworld/Astrea-R8-Chat-9B
tags:
- gguf
- llama.cpp
- qwen3.5
- multimodal
- vision
- heretic
- abliteration
- directional-ablation
- uncensored
- quantized
---
# Astrea-R8-Chat-9B-Heretic-T32-GGUF

GGUF conversions of `Altworld/Astrea-R8-Chat-9B`, modified using the Heretic automated directional-ablation process.

This release is based on **Heretic** and includes several quantization levels for use with `llama.cpp`, LM Studio, KoboldCpp, and other GGUF-compatible runtimes.

The model retains its original multimodal architecture. Vision use requires the separate matching `mmproj` file included in this repository.

## Model details

| Property           | Value                                  |
| ------------------ | -------------------------------------- |
| Base model         | `Altworld/Astrea-R8-Chat-9B`           |
| Parameter class    | Approximately 9B                       |
| Modification       | Heretic automated directional ablation |
| Selected candidate | KL divergence: 0.0316, Refusals: 8/100 |
| Format             | GGUF                                   |
| Text runtime       | llama.cpp-compatible                   |
| Vision support     | Yes, with matching `mmproj`            |
| Context metadata   | 262,144 tokens                         |
| MTP included       | No                                     |
| Primary precision  | BF16                                   |
| Quantizations      | Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q4_K_S     |

Practical context length depends on available memory, runtime configuration, and workload. A smaller context such as 8,192 or 16,384 tokens is a reasonable starting point for local use.

## What was changed?

The source model was processed using `p-e-w/heretic`, which applies optimized directional ablation—sometimes called abliteration—to reduce learned refusal behavior.

Heretic searches for ablation parameters that reduce refusals while minimizing divergence from the original model. This is not a conventional supervised fine-tune, LoRA, merge, or prompt modification.

This release uses the output selected as **KL divergence: 0.0316, Refusals: 8/100**.

The language-model weights were modified. The vision encoder and multimodal projector were not intentionally ablated.

## Important MTP note

The source configuration declared an additional Multi-Token Prediction or NextN layer, but the saved checkpoint did not contain the associated MTP tensors.

Including that declaration caused affected llama.cpp builds to expect a nonexistent tensor:

```text
blk.32.attn_norm.weight
```

The language-model GGUFs in this repository were therefore exported with:

```text
--no-mtp
```

This excludes the unavailable MTP head. It does not remove normal text generation, chat, reasoning, or vision capabilities.

## Available files

| File                                             | Intended use                                   |
| ------------------------------------------------ | ---------------------------------------------- |
| `Astrea-R8-Chat-9B-Heretic-T32-BF16.gguf`        | Archival master and maximum weight fidelity    |
| `Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf`        | Very high quality; near-BF16 practical version |
| `Astrea-R8-Chat-9B-Heretic-T32-Q6_K.gguf`        | High-quality local inference                   |
| `Astrea-R8-Chat-9B-Heretic-T32-Q5_K_M.gguf`      | Strong quality-to-size balance                 |
| `Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf`      | Recommended compact version                    |
| `Astrea-R8-Chat-9B-Heretic-T32-Q4_K_S.gguf`      | Smaller mainstream quant                       |
| `mmproj-Astrea-R8-Chat-9B-Heretic-T32-BF16.gguf` | Maximum-quality vision encoder/projector       |
| `mmproj-Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf` | Recommended vision encoder/projector           |

Actual filenames may differ slightly depending on the final files uploaded to this repository.

## Quantization recommendations

### Best overall compact version

```text
Q4_K_M language model
Q8_0 mmproj
```

### Better text quality

```text
Q5_K_M or Q6_K language model
Q8_0 mmproj
```

### Maximum practical quality

```text
Q8_0 language model
Q8_0 or BF16 mmproj
```

### Archival/reference version

```text
BF16 language model
BF16 mmproj
```

The `mmproj` is much smaller than the language model and directly affects visual understanding. Q8_0 or BF16 is recommended for the projector rather than aggressively quantizing it.


## Text-only usage with llama.cpp

```bash
llama-cli \
  -m Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  -ngl 999 \
  -c 8192 \
  -n 512 \
  -cnv
```

For a one-shot prompt:

```bash
llama-cli \
  -m Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  -ngl 999 \
  -c 8192 \
  -n 512 \
  -p "Explain why the sky appears blue."
```

On Windows CMD, place the command on one line or use `^` instead of `\` for line continuation.

## Vision usage with llama.cpp

Vision requires both the language-model GGUF and the matching `mmproj`:

```bash
llama-cli \
  -m Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  --mmproj mmproj-Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf \
  --image example.jpg \
  -ngl 999 \
  -c 8192 \
  -n 512 \
  -p "Describe this image in detail."
```

You may also use the dedicated multimodal test utility:

```bash
llama-mtmd-cli \
  -m Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  --mmproj mmproj-Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf \
  --image example.jpg \
  -ngl 999 \
  -c 8192 \
  -p "Describe this image."
```

## Server usage

Text and vision can be exposed through llama.cpp’s OpenAI-compatible server:

```bash
llama-server \
  -m Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  --mmproj mmproj-Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf \
  -ngl 999 \
  -c 8192 \
  --host 0.0.0.0 \
  --port 8080
```

Omit `--mmproj` when only text generation is needed.

Binding to `0.0.0.0` exposes the server to the local network. Use appropriate firewall rules and authentication controls before exposing it beyond a trusted network.

## Chat template

The model’s embedded chat template was preserved during conversion.

GGUF-compatible applications should normally detect and use the embedded template automatically. Avoid manually selecting an unrelated Qwen template unless your runtime fails to detect it.

## Evaluation

This release has not yet been subjected to a comprehensive independent benchmark suite.

The selected Heretic candidate was chosen from the automated optimization process, but that should not be treated as evidence of performance on general reasoning, coding, multilingual, multimodal, or safety benchmarks.

Suggested evaluations include:

* Comparison against the unmodified source model
* KL-divergence or perplexity comparison
* Instruction-following tests
* Refusal-rate testing
* General reasoning and coding tasks
* Vision-question answering
* OCR and detailed image-description tests
* Long-context retrieval
* Quantization comparisons across Q4_K_S through Q8_0

Community benchmark results and reproducible test reports are welcome.

## Intended use

This model is intended for:

* Local experimentation
* Research into directional ablation and refusal behavior
* General conversational use
* Creative writing
* Coding and technical assistance
* Multimodal image understanding
* Comparing quantization formats
* Evaluating alignment and behavioral changes

## Limitations

Directional ablation can change behavior in ways that are not fully captured by refusal-rate or divergence measurements.

Possible limitations include:

* Reduced safety refusals
* Harmful, offensive, misleading, or inappropriate output
* Incorrect factual claims
* Overconfidence
* Degraded behavior on prompts unlike those used during optimization
* Differences between quantization levels
* Reduced visual accuracy when using a quantized projector
* Inconsistent tool-call or structured-output behavior
* High memory use at very long context lengths

The absence of a refusal does not mean a response is correct, safe, legal, or appropriate.

Users are responsible for evaluating the model for their own application and implementing any safeguards required for that application.

## Safety notice

This is a deliberately less-restricted model.

It may comply with requests that the upstream model would decline, and it may generate content that is inaccurate, offensive, unsafe, or unsuitable for deployment without additional controls.

Do not rely on the model as the sole source of medical, legal, financial, security, or other high-stakes advice.

## Reproducibility

The language model was converted from the selected Heretic output using llama.cpp.

Representative BF16 conversion:

```bash
python convert_hf_to_gguf.py PATH_TO_HERETIC_OUTPUT \
  --no-mtp \
  --outfile Astrea-R8-Chat-9B-Heretic-T32-BF16-noMTP.gguf \
  --outtype bf16
```

Representative direct Q8_0 conversion:

```bash
python convert_hf_to_gguf.py PATH_TO_HERETIC_OUTPUT \
  --no-mtp \
  --outfile Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf \
  --outtype q8_0
```

Representative Q4_K_M quantization from BF16:

```bash
llama-quantize \
  Astrea-R8-Chat-9B-Heretic-T32-BF16-noMTP.gguf \
  Astrea-R8-Chat-9B-Heretic-T32-Q4_K_M.gguf \
  Q4_K_M
```

Representative Q8_0 mmproj conversion:

```bash
python convert_hf_to_gguf.py PATH_TO_HERETIC_OUTPUT \
  --mmproj \
  --outfile Astrea-R8-Chat-9B-Heretic-T32-Q8_0.gguf \
  --outtype q8_0
```

The converter automatically adds the `mmproj-` filename prefix when exporting a multimodal projector.

All lower-precision language-model quants should be generated independently from the BF16 master rather than from another quantized file.

## Credits

* Original model: `Altworld/Astrea-R8-Chat-9B`
* Underlying architecture: Qwen3.5
* Directional-ablation tooling: `p-e-w/heretic`
* GGUF conversion and runtime: `ggml-org/llama.cpp`

This repository distributes a modified and quantized derivative. It does not claim authorship of the original model, architecture, training data, or the Heretic and llama.cpp tooling.

## License

This repository is licensed under the Apache License 2.0, following the
license of the upstream model.

This is a modified derivative of:

- Altworld/Astrea-R8-Chat-9B
- Qwen/Qwen3.5-9B

The modifications, GGUF conversion, and quantization performed for this
repository do not imply endorsement by the original model authors.

See the included LICENSE file for the complete license terms.

## Disclaimer

This model is provided without warranties. The uploader is not responsible for outputs produced by the model or for downstream uses by third parties.