gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF

Instructions to use maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

Use Docker

docker model run hf.co/maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

Ollama
How to use maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF with Ollama:
```
ollama run hf.co/maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M
```
Unsloth Desktop

How to use maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF with Docker Model Runner:
```
docker model run hf.co/maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M
```

Lemonade

How to use maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "maxkru92/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF / README.md

maxkru92

Duplicate from yuxinlu1/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF

2c54356 3 months ago

preview code

Raw

History Blame Contribute Delete

6.45 kB

metadata

license: gemma
base_model: google/gemma-4-12B-it
library_name: gguf
pipeline_tag: text-generation
tags:
  - gemma4
  - reasoning
  - thinking
  - gguf
  - llama.cpp
  - local-llm

✨ Gemma4-12B-Reasoning-Distill (GGUF) ✨

🐣 Tiny footprint, big brain — local AI for everyone

No matter your GPU. No matter your RAM. If you've got ~4.5 GB of VRAM or unified memory free, you can run your own private, offline AI right now. 🚀 Tuned on Opus 4.6, 4.7 & 4.8 reasoning data, it delivers a major leap in reasoning power — whether you're asking questions or writing code. 🧠💻 All local, all yours, no API, no cloud.

⚡ NEW — the MTP version is here! Free speed 🎉

As of June 7, 2026, mainline llama.cpp just merged Gemma 4 MTP support — so the MTP draft model is now live in the MTP/ folder. Drop it next to any quant and generation gets noticeably faster with identical output (speculative decoding is lossless) — just add a couple of flags. 👉 See ⚡ Speed it up with MTP below. 💚

📦 Pick your size (GGUF quants)

Quant	Size	Vibe
🟢 Q2_K	4.5 GB	tiniest — runs almost anywhere
🔵 Q4_K_M	6.87 GB	the sweet spot 👌 (recommended)
🟣 Q6_K	9.11 GB	near-lossless
⚪ Q8_0	11.8 GB	basically full quality
(f16)	22.2 GB	full precision (overkill for most)

🧮 "Will it fit?" — context length cheat-sheet

Rough estimates 🤓 (assumes q8_0 KV cache + ~1.5 GB overhead; use q4_0 KV cache for ≈2× more context!). Max context is 131K. "—" = won't fit, pick a smaller quant. ✂️

Your VRAM / unified mem	🟢 Q2_K (4.5G)	🔵 Q4_K_M (6.87G)	🟣 Q6_K (9.11G)	⚪ Q8_0 (11.8G)
8 GB	~16K ctx	tight (~2–4K)	—	—
12 GB	~48K	~30K	~12K	—
16 GB	~80K	~64K	~44K	~22K
24 GB	131K (max) 🎉	~128K	~110K	~88K
32 GB	131K	131K	131K	131K

💡 Apple Silicon / integrated GPUs with unified memory count too — same numbers, just slower than a dGPU. 💡 Low on room? Drop a quant or switch KV cache to q4_0 and your context roughly doubles.

⚡ Speed it up with MTP (free & lossless) 🏎️

New as of June 7, 2026! Gemma 4's Multi-Token Prediction drafter lets the model guess a few tokens ahead and verify them in one shot — so you get more tokens/sec with byte-for-byte identical output. Pure speed, zero quality cost. 🪄

1. Grab the tiny draft from the MTP/ folder:

Draft file	Size	Use it for
⚪ `gemma-4-12B-it-MTP-Q8_0.gguf`	0.44 GB	recommended — tiny + full speed
`…-F16.gguf` / `…-BF16.gguf`	0.82 GB	full-precision draft (overkill)

💡 The draft is tiny — keep it Q8 or higher (over-quantizing a draft just lowers its hit rate). It pairs with any quant of the main model.

2. You need a fresh llama.cpp build — June 7 2026 (b9553) or newer. MTP was just merged, so older builds can't load the draft (unknown architecture: 'gemma4-assistant').

3. Run it exactly like below, just +3 flags (--model-draft, --spec-type, --n-gpu-layers-draft):

@echo off
cd /d C:\llama.cpp
llama-server.exe ^
  -m C:\models\gemma4-opus48-Q4_K_M.gguf ^
  --model-draft C:\models\MTP\gemma-4-12B-it-MTP-Q8_0.gguf ^
  --spec-type draft-mtp --spec-draft-n-max 4 ^
  --ctx-size 16384 --n-gpu-layers 99 --n-gpu-layers-draft 99 ^
  --no-mmap -fa on ^
  --temp 1.0 --top-p 0.95 --top-k 64 ^
  --host 0.0.0.0 --port 18080
pause

Measured on a single RTX 5090 (Q4_K_M main + Q8 draft): ~1.3× faster at greedy and ~1.2× at the default thinking sampling — free, with no change to output. 🎈

🔧 Heads-up: this is the stock Gemma drafter (trained on base Gemma 4), so on this fine-tune the hit rate — and thus the speedup — is a little lower than on vanilla Gemma 4. A re-aligned draft could push it higher (maybe a future update). Either way: free speed, no downside. 💚

🚀 How to run it (super easy)

Option A — llama.cpp (recommended) 🦙

Grab a quant above (e.g. …-Q4_K_M.gguf) and llama-server from llama.cpp.

⚠️ Needs a recent llama.cpp (this is the gemma4_unified architecture — older builds won't load it).
Run a server (Windows .bat shown — tweak --port, --ctx-size to taste):

@echo off
cd /d C:\llama.cpp
llama-server.exe ^
  -m C:\models\gemma4-opus48-Q4_K_M.gguf ^
  --ctx-size 16384 ^
  --n-gpu-layers 99 ^
  --no-mmap ^
  -fa on ^
  --cache-type-k q8_0 --cache-type-v q8_0 ^
  --temp 1.0 --top-p 0.95 --top-k 64 ^
  --host 0.0.0.0 --port 18080
pause

Open http://localhost:18080 and chat. 🎉 (Tip: bump --ctx-size per the table; use q4_0 KV for more.)

Option B — one-click apps 🖱️

Works in LM Studio, Jan, Ollama, etc. — just import the GGUF, pick your quant, go. 🐾

🧠 Thinking mode

This model thinks in Gemma's native thought channel. Keep enable_thinking=true (the default chat template handles it). Recommended sampling: temp 1.0, top_p 0.95, top_k 64.

⚠️ Good to know

Reduced refusals: the training data omits safety hedging, so this refuses less than the base model. It is not safety-aligned — add your own guardrails for production. Use responsibly. 🙏
Reasoning is stylistic synthetic CoT — great for structure, but double-check facts/numbers.
English-centric.

📚 Data & License

Base model: google/gemma-4-12B-it. Subject to the Gemma Terms of Use (derivatives must comply).
Training data: built on the public, Apache-2.0 dataset angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k, augmented with additional Opus 4.8-generated reasoning samples I curated and mixed in.
Personal/hobby project — shared as-is, no warranty. Have fun! 🐾✨