---
base_model: FedorFesarov/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic
base_model_relation: quantized
license: apache-2.0
language:
- en
- zh
library_name: gguf
pipeline_tag: text-generation
tags:
- gguf
- llama.cpp
- quantized
- abliteration
- heretic
- uncensored
- nanbeige
- looped-transformer
---

# Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic — GGUF

GGUF quantizations of [**FedorFesarov/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic**](https://huggingface.co/FedorFesarov/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic) — an abliterated build of [AnkitAI/Parable-Nanbeige4.2-3B-Claude-Fable-5](https://huggingface.co/AnkitAI/Parable-Nanbeige4.2-3B-Claude-Fable-5), itself a Claude Fable 5 fine-tune of [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B).

Every quant below was **measured, not assumed** — perplexity on `wikitext-2` and throughput on an RTX 5070 Ti.

---

## Which file should I download?

**Short answer: `IQ4_XS-imat` for the best all-round tradeoff, `Q4_K_M-imat` for maximum safety margin.**

| Priority | Pick | Size |
|---|---|---|
| Best overall balance | **IQ4_XS-imat** | 2.21 GB |
| Highest fidelity at reasonable size | **Q4_K_M-imat** | 2.40 GB |
| Smallest usable | **IQ3_M-imat** | 1.94 GB |
| Near-lossless, size no object | **Q6_K** | 3.19 GB |

---

## Measured results

`F16` baseline perplexity: **17.772**. Lower is better; "Δ" is degradation versus F16.

### With importance matrix (recommended)

The imatrix was computed on the full `wikitext-2` **train** split (4933 chunks, ctx 512) using the F16 weights. Calibration and evaluation use disjoint splits. `pp512` / `tg128` are prompt-processing and generation throughput (tokens/s) on an RTX 5070 Ti.

| File | Size | PPL | Δ vs F16 | pp512 t/s | tg128 t/s |
|---|---|---|---|---|---|
| `...-Q4_K_M-imat.gguf` | 2.40 GB | 18.125 | +2.0% | 7123 | 151 |
| `...-Q4_K_S-imat.gguf` | 2.33 GB | 18.248 | +2.7% | 7325 | 155 |
| **`...-IQ4_XS-imat.gguf`** | **2.21 GB** | **18.746** | **+5.5%** | **7875** | 162 |
| `...-Q3_K_L-imat.gguf` | 2.15 GB | 18.972 | +6.8% | 6262 | 148 |
| `...-IQ3_M-imat.gguf` | 1.94 GB | 19.227 | +8.2% | 7223 | 171 |
| `...-Q3_K_M-imat.gguf` | 2.02 GB | 19.293 | +8.6% | 6354 | 155 |
| `...-Q3_K_S-imat.gguf` | 1.86 GB | 21.253 | +19.6% | 5996 | 156 |
| `...-Q2_K-imat.gguf` | 1.64 GB | 25.359 | +42.7% | 5548 | 178 |

### Without importance matrix

| File | Size | PPL | Δ vs F16 |
|---|---|---|---|
| `...-Q6_K.gguf` | 3.19 GB | 17.710 | −0.3% |
| `...-Q8_0.gguf` | 4.13 GB | 17.900 | +0.7% |
| `...-Q4_K_S.gguf` | 2.33 GB | 17.978 | +1.2% |
| `...-Q5_K_M.gguf` | 2.78 GB | 18.281 | +2.9% |
| `...-Q4_K_M.gguf` | 2.40 GB | 18.298 | +3.0% |
| `...-Q3_K_L.gguf` | 2.15 GB | 20.434 | +15.0% |
| `...-Q3_K_M.gguf` | 2.02 GB | 20.535 | +15.5% |
| `...-Q3_K_S.gguf` | 1.86 GB | 25.490 | +43.4% |
| `...-Q2_K.gguf` | 1.64 GB | 42.863 | +141% |
| `...-F16.gguf` | 7.77 GB | 17.772 | baseline |

### Reading these numbers

- **The imatrix matters enormously below 4 bits.** Q2_K goes from +141% to +42.7%; Q3_K_M from +15.5% to +8.6%. At 4 bits and above the difference is negligible, which is why the high-precision files are provided without one.
- **Q2_K is not recommended at any setting.** Even with the imatrix it loses 42% — that is a qualitatively different model, not a slightly duller one. It is published for completeness and for extremely memory-constrained experiments only.
- **Above Q4_K_S the quants are statistically indistinguishable** from each other (differences smaller than the ±0.15 measurement error). Choose by size, not by decimal places.
- **IQ beats K-quants on GPU here.** IQ4_XS is smaller *and* faster than Q3_K_L while scoring better. Note this benchmark is CUDA — on pure CPU inference IQ types are typically slower to dequantize, so K-quants may win on CPU-only setups.

---

## Format integrity

The full-precision model was checked (greedy decoding) for damage to output structure versus the original fine-tune. Abliteration left all three intact:

| Check | Original | Abliterated (bf16) |
|---|---|---|
| `<think>` block opens and closes correctly | 10/10 | 10/10 |
| Valid parseable JSON on request | 10/10 | 10/10 |
| Well-formed `<tool_call>` given tools | 10/10 | 10/10 |

> These checks were run on the **bf16** weights. Aggressive quantization can degrade format adherence independently of abliteration — a Q2_K model (+42% perplexity) may drop JSON quotes or malform tool-calls even though the bf16 source does not. If you need reliable structured output, prefer **Q4_K_S or above**.

---

## Usage

```bash
llama-cli -m Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-IQ4_XS-imat.gguf \
  -ngl 99 \
  -c 8192 \
  -sys "You are a helpful assistant." \
  -p "Explain how HTTPS certificate validation works."
```

### Three flags you actually need

**`-c` — set it explicitly. This is not optional.**
This model has a 262144-token context and 44 KV-cache layers. Left at default, llama.cpp will try to allocate roughly **45 GB** of KV cache, spill it to system RAM, and inference will crawl. Budget approximately **176 MB of cache per 1024 tokens of context**. On a 16 GB card, 32k context is comfortable.

**`-ngl 99`** — offload all layers to GPU.

**`-sys "..."`** — override the default system prompt, which is in Chinese (`你是南北阁…`) and inherited from the base model. Without it, English prompts may get Chinese answers.

### Optional: shrink the KV cache

```bash
-ctk q8_0 -ctv q8_0
```
Roughly halves cache memory at minimal quality cost — worth it on this architecture given how large the cache is.

### Tool calling

The model emits an XML-style `<tool_call>` block. Front-ends that support llama.cpp's tool/grammar features (llama-server with `--jinja`, LM Studio, Open WebUI) can parse it directly using the bundled chat template.

### Reasoning block

The model emits a `<think>` block before answering. Compatible front-ends (llama.cpp server, LM Studio, Open WebUI) will collapse it. To skip reasoning for latency-sensitive extraction work, use a chat template with an empty think block.

---

## Compatibility

Requires a llama.cpp build with **Nanbeige looped-transformer support** — the runtime must honour the `num_loops = 2` metadata and execute 44 layers, not 22. Verified working on **b10227** and **b10236**.

If output is incoherent, your build is too old. Mobile apps bundling older llama.cpp (PocketPal, ChatterUI) may be affected; check that generation is coherent before assuming a quant is broken.

An `imatrix.dat` (the importance matrix used for the `-imat` quants) is included for reproducibility — you can use it to roll your own quants from the F16 file.

---

## Provenance

| Step | Source |
|---|---|
| Pretrained | [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B) |
| Fine-tuned | [AnkitAI/Parable-Nanbeige4.2-3B-Claude-Fable-5](https://huggingface.co/AnkitAI/Parable-Nanbeige4.2-3B-Claude-Fable-5) |
| Abliterated | [FedorFesarov/…-heretic](https://huggingface.co/FedorFesarov/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic) with [Heretic](https://github.com/p-e-w/heretic) |
| Quantized | this repository, llama.cpp b10236 |

**Note:** refusal behaviour in these weights is intentionally suppressed. Deploy accordingly.