Instructions to use pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M

Use Docker

docker model run hf.co/pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M

LM Studio
Jan
Ollama
How to use pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M with Ollama:
```
ollama run hf.co/pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M
```
Unsloth Desktop

How to use pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M with Docker Model Runner:
```
docker model run hf.co/pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M
```

Lemonade

How to use pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-31B-it-qat-heretic-Q4_K_M-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "pegasus912/gemma-4-31B-it-qat-heretic-Q4_K_M:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

q4_0

by Theory-of-mind - opened Jun 7

Discussion

Theory-of-mind

Jun 7

Try making Q4_0 - it should have better quality, since the model is trained specifically for 4-bit noise. Q4_K_XL from unsloth is actually Q4_0.

pegasus912

Owner Jun 7

That was my first thought and I did but I didn't notice any quality differences in my (albeit limited) testing. I have a Q4_0 quant of this locally that I've been testing to compare.

Theory-of-mind

Jun 7

If there's no difference, could you post Q4_0 specifically? It's smaller than Q4_K_M.

pegasus912

Owner Jun 7

Sure, it will be up in a little while.

pegasus912

Owner Jun 7

It's up now: https://huggingface.co/pegasus912/gemma-4-31B-it-qat-heretic-Q4_0.gguf

IrisColt

Jun 7

Did you notice less degradation in this model when compared to any Q4_foo quant of the non-QAT heretic model? My guess is that heretic annihilated QAT and we are back to square one, sigh...

pegasus912

Owner Jun 7

I haven't noticed, but I haven't done any coding yet on the QAT heretic versions. I did check to make sure they were still uncensored and did some limited creative writing tests with no issues.

Theory-of-mind

Jun 7

The only thing that comes to mind is token_embd.weight in Q6_K (instead of the required Q4_0). This could be the cause of the quality degradation. You can try requantizing, and if that doesn't help, the problem is definitely abliteration. Unfortunately, my PC can only quantize up to 12B.

pegasus912

Owner Jun 7

I've been researching and I'm working on just using the same layout as Unsloth for these since there's some concerns about BF16 QAT not converting correctly to 4_0 via llama.cpp. Hopefully I'll have these versions up a little later today.

Theory-of-mind

Jun 7

Thanks, I'm looking forward to it. I noticed a significant difference between the regular (non-heretic) 26B-A4B in Q6_K and QAT4. Now I'm really interested in comparing the regular heretic and QAT.

pegasus912

Owner Jun 7

The MOE is up now if you want to try it out. I've only done some basic testing so far and it seems good. I'm not sure how heretic and QAT mix, so this is all new. https://huggingface.co/pegasus912/gemma-4-26b-a4b-it-qat-heretic-ud-q4-k-xl

Theory-of-mind

Jun 7

I have a regular 26B in QAT4 and a Heretic from llmfan46 in Q6_K. I'll try to compare them with the QAT-heretic.

pegasus912

Owner Jun 7

I'm finding the Q4_0 to be better than UD_4k_XL so far in coding.

Theory-of-mind

Jun 7

I briefly tested the 26B-A4B ud-q4-k-xl on general and 'harmful' questions. Regarding general questions, it's dumber than a regular (non-heretic) QAT – probably on par with a regular Q6_K. It's still good for its size – I work in a non-Latin language, and abliteration/quantization artifacts appear faster there than in EN. On the other hand, the model does answer 'harmful' questions, but with disclaimers (which is not the case with llmfan's Q6_K heretic). I haven't tried coder3101's regular 26B-A4B heretic, so I can't say whether the issue is QAT or abliteration.
Overall, the model seems suitable as an uncen – it doesn't improve on the quality of a regular heretic Q6_K, but it takes up significantly less RAM and is faster. For a complete picture, it would be nice to compare it with the regular heretic in Q4_K_S by coder3101 to draw conclusions about whether QAT provides a gain in quality at a similar size.

pegasus912

Owner Jun 7

It's a little flaky, I think quantized QAT models need some extra research. Looking at this thread: https://old.reddit.com/r/LocalLLaMA/comments/1tzib7d/qat_variant_of_gemma4_26b_a4b_is_not_working_well/

IrisColt

Jun 9

Thanks, I'm looking forward to it. I noticed a significant difference between the regular (non-heretic) 26B-A4B in Q6_K and QAT4. Now I'm really interested in comparing the regular heretic and QAT.

Pretty please?

Theory-of-mind

Jun 9

Pretty please?

I'm still working on it, but everything points to Q6_K being of higher quality than QAT4. These are regular models without abliteration (to eliminate its influence). I asked them to write code for an SVG file with a chess game – both models made mistakes, but QAT4 had more. I'm also testing word diversity (checking what synonyms the model can produce), and QAT4 is losing here too.
QAT4 is probably better than the static Q4_K_M (without iMatrix). Overall logic is well preserved, but knowledge suffers (which is predictable: 21 GB holds more information than 13 GB). I think it's a great choice for its size (13-16 Gb), but if you need better quality, it's better to look at the Q6_K.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment