Instructions to use FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

Use Docker

docker model run hf.co/FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

Ollama
How to use FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF with Ollama:
```
ollama run hf.co/FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M
```
Unsloth Desktop

How to use FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF with Docker Model Runner:
```
docker model run hf.co/FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M
```

Lemonade

How to use FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "FesarovLab/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

FedorFesarov commited on 28 days ago

Commit

50f104c

verified ·

1 Parent(s): 176aa14

Upload folder using huggingface_hub

Browse files

Files changed (21) hide show

.gitattributes +19 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-F16.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-IQ3_M-imat.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-IQ4_XS-imat.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q2_K-imat.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q2_K.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_L-imat.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_L.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_M-imat.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_M.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_S-imat.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_S.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_M-imat.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_M.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_S-imat.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_S.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q5_K_M.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q6_K.gguf +3 -0
Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q8_0.gguf +3 -0
README.md +134 -0
imatrix.dat +3 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,22 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+imatrix.dat filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-F16.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-IQ3_M-imat.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-IQ4_XS-imat.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q2_K-imat.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_L-imat.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_M-imat.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_S-imat.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_M-imat.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_S-imat.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-F16.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:259fe67612e037a95226afad12825318828370f268b790d722501a8a97a99900
+size 8343845440

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-IQ3_M-imat.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:396eadb437641acba72dee59216f9a3e37ce7a8a50f9088206506298f6826572
+size 2081819488

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-IQ4_XS-imat.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:186a39a64707ee802c4a49506c4e278a53b16bbfa31dcebba5dc1a0baacfdefc
+size 2378003296

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q2_K-imat.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6e121f6d5236a52ec11f47b1678d2e0458ea21091fc8b19ab179fe8e4086181e
+size 1756095328

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b42df8f66b805426c12a1f96bae9f68184b34be4228de75d0e014bf05c3c7260
+size 1756095040

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_L-imat.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c4c24cd50909b679ec4e8a8f93383531ec211b021eccfb8dcc6c1818a529411a
+size 2307783520

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4aef5bed7b66ef1497423560e8b1114f97bd400d925639b379f8408d92e97c74
+size 2307783232

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_M-imat.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b3af7e90dc046cf569d986b6590008915849db19b5f99066866773210e069c9c
+size 2165439328

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3c4daa2d753daff3210e05d48604f49808f19625e07f19ee2da3f7bd5b650b1e
+size 2165439040

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_S-imat.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:808a978e9f162915a4bb255c3824b99afcb1200ce635b85a481bf009782c7aa5
+size 1995545440

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:add1dd77c708dd8d3695a0d059ad7faf01787eb6e8dfa5192ddbabe7d8e2ab29
+size 1995545152

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_M-imat.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3234f4adb1e35d103d05c0b088cb9f2771cc362e12025ae7ee20a0abbdbf1b48
+size 2574807904

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3a1f9abc052366e03a1433dabcccbd38ba16940e42be39a5bef98c2103a89d80
+size 2574807616

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_S-imat.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b706dd83123c7b69dba3c20bb5415a6113d58e99f6eb095a47ae5dc28b18e95a
+size 2503758688

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8ed3f56724a93e4646f5fcd3d2ee7a2b14c1be2ba212fee05df9e7d4183204f5
+size 2503758400

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5832e5676cd08145398ef530e83160e2540db1f72181b4642cab0a8b28ab5188
+size 2986996288

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c6e6b090a89ad9be01d5585f526720c62a23648c4cc275204ce02c03654220c3
+size 3424946752

Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-Q8_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:18942fb13e026d94528ce3cdad4eb3b5f5935258639ef3e45f1bdd7fcc905e93
+size 4434786880

README.md ADDED Viewed

	@@ -0,0 +1,134 @@

+---
+base_model: FedorFesarov/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic
+base_model_relation: quantized
+license: apache-2.0
+language:
+- en
+- zh
+library_name: gguf
+pipeline_tag: text-generation
+tags:
+- gguf
+- llama.cpp
+- quantized
+- abliteration
+- heretic
+- uncensored
+- nanbeige
+- looped-transformer
+---
+# Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic — GGUF
+GGUF quantizations of [**FedorFesarov/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic**](https://huggingface.co/FedorFesarov/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic) — an abliterated build of [AnkitAI/Parable-Nanbeige4.2-3B-Claude-Fable-5](https://huggingface.co/AnkitAI/Parable-Nanbeige4.2-3B-Claude-Fable-5), itself a Claude Fable 5 fine-tune of [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B).
+Every quant below was **measured, not assumed** — perplexity on `wikitext-2` and throughput on an RTX 5070 Ti.
+---
+## Which file should I download?
+**Short answer: `IQ4_XS-imat` if you want the best all-round tradeoff, `Q4_K_M-imat` if you want maximum safety margin.**
+| Priority | Pick | Size |
+|---|---|---|
+| Best overall balance | **IQ4_XS-imat** | 2.21 GB |
+| Highest fidelity at reasonable size | **Q4_K_M-imat** | 2.40 GB |
+| Smallest usable | **IQ3_M-imat** | 1.94 GB |
+| Near-lossless, size no object | **Q6_K** | 3.19 GB |
+---
+## Measured results
+`F16` baseline perplexity: **17.772**. Lower is better; "Δ" is degradation versus F16.
+### With importance matrix (recommended)
+The imatrix was computed on the full `wikitext-2` **train** split (4933 chunks, ctx 512) using the F16 weights. Calibration and evaluation use disjoint splits.
+| File | Size | PPL | Δ vs F16 | pp512 t/s | tg128 t/s |
+|---|---|---|---|---|---|
+| `...-Q4_K_M-imat.gguf` | 2.40 GB | 18.125 | +2.0% | 7123 | 151 |
+| `...-Q4_K_S-imat.gguf` | 2.33 GB | 18.248 | +2.7% | 7325 | 155 |
+| **`...-IQ4_XS-imat.gguf`** | **2.21 GB** | **18.746** | **+5.5%** | **7875** | 162 |
+| `...-Q3_K_L-imat.gguf` | 2.15 GB | 18.972 | +6.8% | 6262 | 148 |
+| `...-IQ3_M-imat.gguf` | 1.94 GB | 19.227 | +8.2% | 7223 | 171 |
+| `...-Q3_K_M-imat.gguf` | 2.02 GB | 19.293 | +8.6% | 6354 | 155 |
+| `...-Q3_K_S-imat.gguf` | 1.86 GB | 21.253 | +19.6% | 5996 | 156 |
+| `...-Q2_K-imat.gguf` | 1.64 GB | 25.359 | +42.7% | 5548 | 178 |
+### Without importance matrix
+| File | Size | PPL | Δ vs F16 |
+|---|---|---|---|
+| `...-Q6_K.gguf` | 3.19 GB | 17.710 | −0.3% |
+| `...-Q8_0.gguf` | 4.13 GB | 17.900 | +0.7% |
+| `...-Q4_K_S.gguf` | 2.33 GB | 17.978 | +1.2% |
+| `...-Q5_K_M.gguf` | 2.78 GB | 18.281 | +2.9% |
+| `...-Q4_K_M.gguf` | 2.40 GB | 18.298 | +3.0% |
+| `...-Q3_K_L.gguf` | 2.15 GB | 20.434 | +15.0% |
+| `...-Q3_K_M.gguf` | 2.02 GB | 20.535 | +15.5% |
+| `...-Q3_K_S.gguf` | 1.86 GB | 25.490 | +43.4% |
+| `...-Q2_K.gguf` | 1.64 GB | 42.863 | +141% |
+| `...-F16.gguf` | 7.77 GB | 17.772 | baseline |
+### Reading these numbers
+- **The imatrix matters enormously below 4 bits.** Q2_K goes from +141% to +42.7%; Q3_K_M from +15.5% to +8.6%. At 4 bits and above the difference is negligible, which is why the high-precision files are provided without one.
+- **Q2_K is not recommended at any setting.** Even with the imatrix it loses 42% — that is a qualitatively different model, not a slightly duller one. It is published for completeness and for extremely memory-constrained experiments only.
+- **Above Q4_K_S the quants are statistically indistinguishable** from each other (differences smaller than the ±0.15 measurement error). Choose by size, not by decimal places.
+- **IQ beats K-quants on GPU here.** IQ4_XS is smaller *and* faster than Q3_K_L while scoring better. Note this benchmark is CUDA — on pure CPU inference IQ types are typically slower to dequantize, so K-quants may win on CPU-only setups.
+---
+## Usage
+```bash
+llama-cli -m Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-IQ4_XS-imat.gguf \
+  -ngl 99 \
+  -c 8192 \
+  -sys "You are a helpful assistant." \
+  -p "Explain how HTTPS certificate validation works."
+```
+### Three flags you actually need
+**`-c` — set it explicitly. This is not optional.**
+This model has a 262144-token context and 44 KV-cache layers. Left at default, llama.cpp will try to allocate roughly **45 GB** of KV cache, spill it to system RAM, and inference will crawl. Budget approximately **176 MB of cache per 1024 tokens of context**. On a 16 GB card, 32k context is comfortable.
+**`-ngl 99`** — offload all layers to GPU.
+**`-sys "..."`** — override the default system prompt, which is in Chinese (`你是南北阁…`) and inherited from the base model. Without it, English prompts may get Chinese answers.
+### Optional: shrink the KV cache
+```bash
+-ctk q8_0 -ctv q8_0
+```
+Roughly halves cache memory at minimal quality cost — worth it on this architecture given how large the cache is.
+### Reasoning block
+The model emits a `<think>` block before answering. Compatible front-ends (llama.cpp server, LM Studio, Open WebUI) will collapse it. To skip reasoning for latency-sensitive extraction work, use a chat template with an empty think block.
+---
+## Compatibility
+Requires a llama.cpp build with **Nanbeige looped-transformer support** — the runtime must honour the `num_loops = 2` metadata and execute 44 layers, not 22. Verified working on **b10227** and **b10236**.
+If output is incoherent, your build is too old. Mobile apps bundling older llama.cpp (PocketPal, ChatterUI) may be affected; check that generation is coherent before assuming a quant is broken.
+---
+## Provenance
+| Step | Source |
+|---|---|
+| Pretrained | [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B) |
+| Fine-tuned | [AnkitAI/Parable-Nanbeige4.2-3B-Claude-Fable-5](https://huggingface.co/AnkitAI/Parable-Nanbeige4.2-3B-Claude-Fable-5) |
+| Abliterated | [FedorFesarov/…-heretic](https://huggingface.co/FedorFesarov/Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic) with [Heretic](https://github.com/p-e-w/heretic) |
+| Quantized | this repository, llama.cpp b10236 |
+**Note:** refusal behaviour in these weights is intentionally suppressed. Deploy accordingly.

imatrix.dat ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bb3922274d496b4833ea8de73a7dc7d315efff5fd96b94df0f911916bdd33f3b
+size 2862304