Instructions to use Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF
# Run inference directly in the terminal:
llama cli -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF
# Run inference directly in the terminal:
llama cli -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF
# Run inference directly in the terminal:
./llama-cli -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

Use Docker

docker model run hf.co/Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

LM Studio
Jan

vLLM

How to use Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

Ollama
How to use Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF with Ollama:
```
ollama run hf.co/Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF
```
Unsloth Desktop

How to use Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF with Docker Model Runner:
```
docker model run hf.co/Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF
```

Lemonade

How to use Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

Run and chat with the model

lemonade run user.IBM-granite-4.1-8b-heretic-PMRA-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Asystemoffields/IBM-granite-4.1-8b-heretic-PMRA-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Granite 4.1 8B Heretic · PMRA mixed-precision GGUF

A ~3.6 GB GGUF of IBM's Granite 4.1 8B (decensored) at the IQ3_XS size budget that scores ~0.42 nats lower NLL than a plain IQ3_XS quant on a held-out test split — not just on the calibration set. A single standard GGUF, ready for llama.cpp or Ollama.

The model

Granite 4.1 8B is an 8-billion-parameter open model from IBM, part of the enterprise-focused Granite family. It's a dense decoder-only transformer (GraniteForCausalLM): 40 layers, 4096 hidden, grouped-query attention (32 query / 8 KV heads), a 100k-token vocabulary, and a long 131,072-token (128K) context window — built for document-scale, tool-using, and RAG workloads. Multilingual at the base; calibrated and measured here on English.

This build sits on heretic-org/IBM-granite-4.1-8b-heretic — a decensored Granite 4.1 8B produced with Heretic v1.2.0 (the Self-Organizing-Maps abliteration method), which suppresses refusal directions while preserving the model's general behavior. Credit for the upstream checkpoint goes to IBM, and for the decensored release to heretic-org.

⚠️ Uncensored. The base has had its refusal behavior abliterated.

Why this build (PMRA)

A normal GGUF quant uses one format for nearly every tensor, paying the same bit-rate everywhere. Production Mixed-Rate Allocation (PMRA) measures each tensor group's contribution to quality and spends bits where they buy the most: starting from a low-bit IQ2_M floor, it promotes the groups that matter to stronger formats under a fixed byte budget, producing one standard GGUF the size of IQ3_XS that is more faithful to the original weights.

Crucially, the selection here was frozen on calibration data and then re-scored on a held-out Wikitext-2 test split — so the gain is a generalization result, not calibration overfit.

Headline (held-out Wikitext-2 test, lower NLL is better):

	NLL	size
this PMRA build	4.539	3.597 GB
plain `IQ3_XS` (same budget)	4.960	3.598 GB

→ −0.42 NLL at the same footprint, public-eval decision GO.

Quick start

llama-cli -m granite4_1_8b_heretic_pmra_layer_family_iq3xs_budget.gguf \
  -p "Write a short hello from PMRA." -n 80 --ctx-size 2048

Needs a recent llama.cpp build (or Ollama) with Granite 4.1 support. ~3.6 GB on disk; runs on CPU.

Footprint

file: granite4_1_8b_heretic_pmra_layer_family_iq3xs_budget.gguf
size: 3,600,448,224 bytes (≈ 3.60 GB) · payload 3,596,877,824 bytes
file bpw: 3.437 · payload bpw: 3.434
SHA-256: 29d3d2b33583127789ee26b0b5e1d7204cb5330af2c265bef6b42c7a4a4a291a
tensor reload mismatches: 0

Benchmarks

Calibration: Wikitext-2-raw train (12 prompts). Selector eval: Wikitext-2-raw validation (128 prompts). Held-out eval: Wikitext-2-raw test (512 prompts). Lower NLL is better.

Held-out Wikitext-2 test (the result that matters):

Variant	NLL	Payload bpw	Payload bytes
fp16 reference	`2.742451`	`16.000000`	`17,583,185,920`
`IQ2_M`	`5.150425`	`2.710999`	`2,979,250,176`
`IQ3_XS` (target / control)	`4.960251`	`3.274283`	`3,598,270,464`
`Q2_K`	`4.754195`	`2.979089`	`3,273,867,264`
`Q3_K_S`	`4.933018`	`3.423967`	`3,762,765,824`
`IQ4_XS`	`4.672932`	`4.184315`	`4,598,349,824`
PMRA knapsack	`4.539084`	`3.273016`	`3,596,877,824`
same-budget random	`4.939853`	`3.273016`	`3,596,877,824`

Selector validation split (Wikitext-2 validation): PMRA knapsack 4.469497 vs IQ3_XS 4.845994 — consistent with the held-out result.

public eval decision: GO
vs IQ3_XS: −0.421167 NLL, −1,392,640 bytes
vs Q3_K_S: −0.393934 NLL, −165,888,000 bytes
vs same-budget random allocation: −0.400769 NLL

How it was built

base: heretic-org/IBM-granite-4.1-8b-heretic (upstream-listed original: ibm-granite/granite-4.1-8b)
GGUF sources: mradermacher/IBM-granite-4.1-8b-heretic-i1-GGUF
tensor profile granite · group mode layer_family · selector c2_calib_knapsack_mixed
low source IQ2_M → target/control IQ3_XS; promotion menu Q2_K_S, Q2_K, Q3_K_S, Q3_K_M, IQ4_XS

Source mix

Source	Tensors	Payload bytes
`IQ2_M`	`110`	`585,269,248`
`Q2_K_S`	`40`	`516,259,840`
`Q2_K`	`56`	`359,530,496`
`Q3_K_S`	`62`	`1,035,780,096`
`Q3_K_M`	`46`	`423,198,720`
`IQ4_XS`	`48`	`676,839,424`

Files

granite4_1_8b_heretic_pmra_layer_family_iq3xs_budget.gguf — the model
artifact_report.json / .md, selector_result.json / .md
public_eval_wikitext_test_result.json / .md — the held-out evaluation
GRANITE4_1_8B_HERETIC_PMRA.md — release card

Attribution & license

Derived from, with thanks to:

heretic-org/IBM-granite-4.1-8b-heretic and heretic-org, credited for the Heretic decensored release
ibm-granite/granite-4.1-8b — the original base model (IBM)
GGUF quantizations from mradermacher/IBM-granite-4.1-8b-heretic-i1-GGUF
llama.cpp GGUF tooling

Released under apache-2.0. Preserve upstream model, Heretic release, license, and quantization attribution when redistributing derived artifacts.

Method + reproduction: https://github.com/asystemoffields/PMRA