README.md

---
license: gemma
base_model: google/gemma-4-12B-it
library_name: gguf
pipeline_tag: text-generation
tags: [gemma4, reasoning, thinking, gguf, llama.cpp, local-llm]
---

# ✨ Gemma4-12B-Reasoning-Distill (GGUF) ✨
### 🐣 Tiny footprint, big brain — local AI for *everyone*

> **No matter your GPU. No matter your RAM.** If you've got **~4.5 GB** of VRAM *or* unified memory free,
> you can run your own private, offline AI right now. 🚀
> Tuned on **Opus 4.6, 4.7 & 4.8** reasoning data, it delivers a **major leap in reasoning power** —
> whether you're asking questions or writing code. 🧠💻 All local, all yours, no API, no cloud.

### ⚡ NEW — the **MTP version is here!** Free speed 🎉
**As of June 7, 2026**, mainline llama.cpp *just* merged Gemma 4 MTP support — so the **MTP draft model** is now
live in the [`MTP/`](https://huggingface.co/yuxinlu1/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF/tree/main/MTP) folder.
Drop it next to any quant and generation gets **noticeably faster with identical output** (speculative decoding is
lossless) — just add a couple of flags. 👉 See **⚡ Speed it up with MTP** below. 💚

---

## 📦 Pick your size (GGUF quants)

| Quant | Size | Vibe |
|------|------|------|
| 🟢 **Q2_K** | **4.5 GB** | tiniest — runs almost anywhere |
| 🔵 **Q4_K_M** | **6.87 GB** | the sweet spot 👌 (recommended) |
| 🟣 **Q6_K** | **9.11 GB** | near-lossless |
| ⚪ **Q8_0** | **11.8 GB** | basically full quality |
| (f16) | 22.2 GB | full precision (overkill for most) |

---

## 🧮 "Will it fit?" — context length cheat-sheet

Rough estimates 🤓 (assumes `q8_0` KV cache + ~1.5 GB overhead; **use `q4_0` KV cache for ≈2× more context!**).
Max context is **131K**. "—" = won't fit, pick a smaller quant. ✂️

| Your VRAM / unified mem | 🟢 Q2_K (4.5G) | 🔵 Q4_K_M (6.87G) | 🟣 Q6_K (9.11G) | ⚪ Q8_0 (11.8G) |
|---|---|---|---|---|
| **8 GB**  | ~16K ctx | tight (~2–4K) | — | — |
| **12 GB** | ~48K | ~30K | ~12K | — |
| **16 GB** | ~80K | ~64K | ~44K | ~22K |
| **24 GB** | 131K (max) 🎉 | ~128K | ~110K | ~88K |
| **32 GB** | 131K | 131K | 131K | 131K |

> 💡 Apple Silicon / integrated GPUs with **unified memory** count too — same numbers, just slower than a dGPU.
> 💡 Low on room? Drop a quant or switch KV cache to `q4_0` and your context roughly doubles.

---

## ⚡ Speed it up with MTP (free & lossless) 🏎️

**New as of June 7, 2026!** Gemma 4's *Multi-Token Prediction* drafter lets the model **guess a few tokens ahead** and
verify them in one shot — so you get more tokens/sec with **byte-for-byte identical output**. Pure speed, zero quality
cost. 🪄

**1.** Grab the tiny draft from the [`MTP/`](https://huggingface.co/yuxinlu1/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF/tree/main/MTP) folder:

| Draft file | Size | Use it for |
|---|---|---|
| ⚪ **`gemma-4-12B-it-MTP-Q8_0.gguf`** | **0.44 GB** | **recommended** — tiny + full speed |
| `…-F16.gguf` / `…-BF16.gguf` | 0.82 GB | full-precision draft (overkill) |

> 💡 The draft is **tiny** — keep it **Q8 or higher** (over-quantizing a draft just lowers its hit rate). It pairs with **any** quant of the main model.

**2.** You need a **fresh llama.cpp build — June 7 2026 (`b9553`) or newer**. MTP was *just* merged, so older builds
can't load the draft (`unknown architecture: 'gemma4-assistant'`).

**3.** Run it exactly like below, just **+3 flags** (`--model-draft`, `--spec-type`, `--n-gpu-layers-draft`):

```bat
@echo off
cd /d C:\llama.cpp
llama-server.exe ^
  -m C:\models\gemma4-opus48-Q4_K_M.gguf ^
  --model-draft C:\models\MTP\gemma-4-12B-it-MTP-Q8_0.gguf ^
  --spec-type draft-mtp --spec-draft-n-max 4 ^
  --ctx-size 16384 --n-gpu-layers 99 --n-gpu-layers-draft 99 ^
  --no-mmap -fa on ^
  --temp 1.0 --top-p 0.95 --top-k 64 ^
  --host 0.0.0.0 --port 18080
pause
```

**Measured on a single RTX 5090** (Q4_K_M main + Q8 draft): **~1.3× faster** at greedy and **~1.2×** at the default
thinking sampling — free, with no change to output. 🎈

> 🔧 *Heads-up:* this is the **stock** Gemma drafter (trained on base Gemma 4), so on this fine-tune the hit rate —
> and thus the speedup — is a little lower than on vanilla Gemma 4. A re-aligned draft could push it higher (maybe a
> future update). Either way: free speed, no downside. 💚

---

## 🚀 How to run it (super easy)

### Option A — llama.cpp (recommended) 🦙
1. Grab a quant above (e.g. `…-Q4_K_M.gguf`) and `llama-server` from [llama.cpp](https://github.com/ggml-org/llama.cpp).
   > ⚠️ Needs a **recent llama.cpp** (this is the `gemma4_unified` architecture — older builds won't load it).
2. Run a server (Windows `.bat` shown — tweak `--port`, `--ctx-size` to taste):

```bat
@echo off
cd /d C:\llama.cpp
llama-server.exe ^
  -m C:\models\gemma4-opus48-Q4_K_M.gguf ^
  --ctx-size 16384 ^
  --n-gpu-layers 99 ^
  --no-mmap ^
  -fa on ^
  --cache-type-k q8_0 --cache-type-v q8_0 ^
  --temp 1.0 --top-p 0.95 --top-k 64 ^
  --host 0.0.0.0 --port 18080
pause
```
3. Open `http://localhost:18080` and chat. 🎉  (Tip: bump `--ctx-size` per the table; use `q4_0` KV for more.)

### Option B — one-click apps 🖱️
Works in **LM Studio**, **Jan**, **Ollama**, etc. — just import the GGUF, pick your quant, go. 🐾

### 🧠 Thinking mode
This model thinks in Gemma's native thought channel. Keep **`enable_thinking=true`** (the default chat template
handles it). Recommended sampling: `temp 1.0, top_p 0.95, top_k 64`.

---

## ⚠️ Good to know
- **Reduced refusals:** the training data omits safety hedging, so this refuses less than the base model.
  It is **not** safety-aligned — add your own guardrails for production. Use responsibly. 🙏
- Reasoning is *stylistic* synthetic CoT — great for structure, but double-check facts/numbers.
- English-centric.

---

## 📚 Data & License
- **Base model:** [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it). Subject to the
  **[Gemma Terms of Use](https://ai.google.dev/gemma/terms)** (derivatives must comply).
- **Training data:** built on the public, **Apache-2.0** dataset
  [`angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k`](https://huggingface.co/datasets/angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k),
  **augmented with additional Opus 4.8-generated reasoning samples** I curated and mixed in.
- Personal/hobby project — shared as-is, no warranty. Have fun! 🐾✨