FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic — GGUF

GGUF quantizations of FedorFesarov/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic — an abliterated build of AnkitAI/Parable-Nanbeige4.2-3B-Claude-Fable-5, itself a Claude Fable 5 fine-tune of Nanbeige/Nanbeige4.2-3B.

Every quant below was measured, not assumed — perplexity on wikitext-2 and throughput on an RTX 5070 Ti.

---

Which file should I download?

Short answer: IQ4_XS-imat for the best all-round tradeoff, Q4_K_M-imat for maximum safety margin.

Priority	Pick	Size
Best overall balance	IQ4_XS-imat	2.21 GB
Highest fidelity at reasonable size	Q4_K_M-imat	2.40 GB
Smallest usable	IQ3_M-imat	1.94 GB
Near-lossless, size no object	Q6_K	3.19 GB

---

Measured results

F16 baseline perplexity: 17.772. Lower is better; "Δ" is degradation versus F16.

With importance matrix (recommended)

The imatrix was computed on the full wikitext-2 train split (4933 chunks, ctx 512) using the F16 weights. Calibration and evaluation use disjoint splits. pp512 / tg128 are prompt-processing and generation throughput (tokens/s) on an RTX 5070 Ti.

File	Size	PPL	Δ vs F16	pp512 t/s	tg128 t/s
...-Q4_K_M-imat.gguf	2.40 GB	18.125	+2.0%	7123	151
...-Q4_K_S-imat.gguf	2.33 GB	18.248	+2.7%	7325	155
...-IQ4_XS-imat.gguf	2.21 GB	18.746	+5.5%	7875	162
...-Q3_K_L-imat.gguf	2.15 GB	18.972	+6.8%	6262	148
...-IQ3_M-imat.gguf	1.94 GB	19.227	+8.2%	7223	171
...-Q3_K_M-imat.gguf	2.02 GB	19.293	+8.6%	6354	155
...-Q3_K_S-imat.gguf	1.86 GB	21.253	+19.6%	5996	156
...-Q2_K-imat.gguf	1.64 GB	25.359	+42.7%	5548	178

Without importance matrix

File	Size	PPL	Δ vs F16
...-Q6_K.gguf	3.19 GB	17.710	−0.3%
...-Q8_0.gguf	4.13 GB	17.900	+0.7%
...-Q4_K_S.gguf	2.33 GB	17.978	+1.2%
...-Q5_K_M.gguf	2.78 GB	18.281	+2.9%
...-Q4_K_M.gguf	2.40 GB	18.298	+3.0%
...-Q3_K_L.gguf	2.15 GB	20.434	+15.0%
...-Q3_K_M.gguf	2.02 GB	20.535	+15.5%
...-Q3_K_S.gguf	1.86 GB	25.490	+43.4%
...-Q2_K.gguf	1.64 GB	42.863	+141%
...-F16.gguf	7.77 GB	17.772	baseline

Reading these numbers

• The imatrix matters enormously below 4 bits. Q2_K goes from +141% to +42.7%; Q3_K_M from +15.5% to +8.6%. At 4 bits and above the difference is negligible, which is why the high-precision files are provided without one.
• Q2_K is not recommended at any setting. Even with the imatrix it loses 42% — that is a qualitatively different model, not a slightly duller one. It is published for completeness and for extremely memory-constrained experiments only.
• Above Q4_K_S the quants are statistically indistinguishable from each other (differences smaller than the ±0.15 measurement error). Choose by size, not by decimal places.
• IQ beats K-quants on GPU here. IQ4_XS is smaller and faster than Q3_K_L while scoring better. Note this benchmark is CUDA — on pure CPU inference IQ types are typically slower to dequantize, so K-quants may win on CPU-only setups.

---

Format integrity

The full-precision model was checked (greedy decoding) for damage to output structure versus the original fine-tune. Abliteration left all three intact:

Check	Original	Abliterated (bf16)
<think> block opens and closes correctly	10/10	10/10
Valid parseable JSON on request	10/10	10/10
Well-formed <tool_call> given tools	10/10	10/10

These checks were run on the bf16 weights. Aggressive quantization can degrade format adherence independently of abliteration — a Q2_K model (+42% perplexity) may drop JSON quotes or malform tool-calls even though the bf16 source does not. If you need reliable structured output, prefer Q4_K_S or above.

---

Usage

llama-cli -m Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-IQ4_XS-imat.gguf \
  -ngl 99 \
  -c 8192 \
  -sys "You are a helpful assistant." \
  -p "Explain how HTTPS certificate validation works."

Three flags you actually need

-c — set it explicitly. This is not optional. This model has a 262144-token context and 44 KV-cache layers. Left at default, llama.cpp will try to allocate roughly 45 GB of KV cache, spill it to system RAM, and inference will crawl. Budget approximately 176 MB of cache per 1024 tokens of context. On a 16 GB card, 32k context is comfortable.

-ngl 99 — offload all layers to GPU.

-sys "..." — override the default system prompt, which is in Chinese (你是南北阁…) and inherited from the base model. Without it, English prompts may get Chinese answers.

Optional: shrink the KV cache

-ctk q8_0 -ctv q8_0

Roughly halves cache memory at minimal quality cost — worth it on this architecture given how large the cache is.

Tool calling

The model emits an XML-style <tool_call> block. Front-ends that support llama.cpp's tool/grammar features (llama-server with --jinja, LM Studio, Open WebUI) can parse it directly using the bundled chat template.

Reasoning block

The model emits a <think> block before answering. Compatible front-ends (llama.cpp server, LM Studio, Open WebUI) will collapse it. To skip reasoning for latency-sensitive extraction work, use a chat template with an empty think block.

---

Compatibility

Requires a llama.cpp build with Nanbeige looped-transformer support — the runtime must honour the num_loops = 2 metadata and execute 44 layers, not 22. Verified working on b10227 and b10236.

If output is incoherent, your build is too old. Mobile apps bundling older llama.cpp (PocketPal, ChatterUI) may be affected; check that generation is coherent before assuming a quant is broken.

An imatrix.dat (the importance matrix used for the -imat quants) is included for reproducibility — you can use it to roll your own quants from the F16 file.

---

Provenance

Step	Source
Pretrained	Nanbeige/Nanbeige4.2-3B
Fine-tuned	AnkitAI/Parable-Nanbeige4.2-3B-Claude-Fable-5
Abliterated	FedorFesarov/…-heretic with Heretic
Quantized	this repository, llama.cpp b10236

Note: refusal behaviour in these weights is intentionally suppressed. Deploy accordingly.