README.md

---
license: gemma
base_model: google/gemma-4-12B-it
library_name: gguf
pipeline_tag: text-generation
tags: [gemma4, coding, code, reasoning, thinking, gguf, llama.cpp, local-llm]
---

# 💻 Gemma4-12B-Coder (GGUF) — Composer 2.5 × Fable 5 ✨
### 🐣 Tiny footprint, big brain — a local **coding** model for *everyone*

> **No matter your GPU. No matter your RAM.** If you've got **~4.5 GB** of VRAM *or* unified memory free,
> you can run your own private, offline coding assistant right now. 🚀
> This is the **v1 / code edition** — distilled from **real chain-of-thought** so it *thinks through* a problem
> before writing the solution. 🧠💻 All local, all yours, no API, no cloud.

### 🎯 What it is
A focused fine-tune of Gemma 4 12B on **verifiable Python coding** data — every training example's reasoning leads to
code that **actually passed its tests**. The result reasons in the open (edge cases, complexity, approach) and then
emits a clean, runnable solution. 💚

---

## 📚 Training data (the interesting part 🍳)

This is a **distillation** of two complementary chain-of-thought sources, both over verifiable Python coding tasks
(algorithmic / function-level problems that come with deterministic tests):

- **🥇 Main set — Composer 2.5 *real* CoT.** Genuine, model-authored reasoning traces. The teacher solved each problem,
  its code was **run against the task's tests, and only the passing solutions were kept**. So the reasoning you're
  learning from leads to code that *actually works*.
- **🥈 Aux set — Fable 5 (released today! 🎉).** A clever twist: we took the problems where **Composer 2.5 got it wrong**
  and handed them to **Fable 5** to *redo* — re-deriving a fresh, self-consistent chain-of-thought and a correct
  solution, again **gated on passing the tests**. This recovers the hard cases the main teacher missed. These traces
  are **synthetic** (rationalized CoT), and are tagged separately so the two sources stay distinguishable.

The recipe: real CoT for the bulk of solid coverage, plus synthetic "second-attempt" CoT to patch the failures —
both verified by execution before anything entered training. ✅

---

## 🗺️ Roadmap — v2 (if there's interest! 💚)

This is **v1**. If the likes / downloads add up, I'll ship a **v2** that **pushes for the benchmarks** 🏁.

### 📢 Update on v2 & the Fable 5 situation *(2026-06-14)*

Quick heads-up for everyone waiting on v2:

**Fable 5 access has been pulled.** The Fable 5 CoT data I managed to save beforehand is honestly a pretty
small set — not enough on its own to act as the *primary* signal for v2 without risking **overfitting**. So the
plan is shifting:

- **v2 will lean more heavily on Composer 2.5 verifiable CoT** as the backbone (the main, execution-verified
  source), and use the limited Fable 5 data carefully as a supplement rather than the core.
- **If Fable 5 access doesn't come back within ~a week**, I'm considering bringing in **GLM-5.2** as an
  additional teacher. I just went through the benchmarks: per **BridgeMind's eval posted on X**, GLM-5.2 actually
  edges out Fable 5 on both the **BS** and **reasoning** leaderboards. I haven't tested it hands-on myself yet —
  my gut says it'll land *slightly below* Fable 5 in practice, but likely **very close**.

Bottom line: v2 is still coming. I'd just rather take a little longer and ship something that **generalizes**
than rush out an overfit model. Thanks for the patience and support 💚

⭐ **Like & download if you'd like to see v2** — that's the signal I'm watching!

---

## 📦 Pick your size (GGUF quants)

| Quant | Size | Vibe |
|------|------|------|
| 🟢 **Q2_K** | **4.5 GB** | tiniest — runs almost anywhere |
| 🔵 **Q4_K_M** | **6.87 GB** | the sweet spot 👌 (recommended) |
| 🟣 **Q6_K** | **9.11 GB** | near-lossless |
| ⚪ **Q8_0** | **11.8 GB** | basically full quality |

---

## 🧮 "Will it fit?" — context length cheat-sheet

Rough estimates 🤓 (assumes `q8_0` KV cache + ~1.5 GB overhead; **use `q4_0` KV cache for ≈2× more context!**).
Max context is **131K**. "—" = won't fit, pick a smaller quant. ✂️

| Your VRAM / unified mem | 🟢 Q2_K (4.5G) | 🔵 Q4_K_M (6.87G) | 🟣 Q6_K (9.11G) | ⚪ Q8_0 (11.8G) |
|---|---|---|---|---|
| **8 GB**  | ~16K ctx | tight (~2–4K) | — | — |
| **12 GB** | ~48K | ~30K | ~12K | — |
| **16 GB** | ~80K | ~64K | ~44K | ~22K |
| **24 GB** | 131K (max) 🎉 | ~128K | ~110K | ~88K |
| **32 GB** | 131K | 131K | 131K | 131K |

> 💡 Apple Silicon / integrated GPUs with **unified memory** count too — same numbers, just slower than a dGPU.
> 💡 Low on room? Drop a quant or switch KV cache to `q4_0` and your context roughly doubles.

---

## 🚀 How to run it (super easy)

### Option A — llama.cpp (recommended) 🦙
1. Grab a quant above (e.g. `…-Q4_K_M.gguf`) and `llama-server` from [llama.cpp](https://github.com/ggml-org/llama.cpp).
   > ⚠️ Needs a **recent llama.cpp** (this is the `gemma4_unified` architecture — older builds won't load it).
2. Run a server (Windows `.bat` shown — tweak `--port`, `--ctx-size` to taste):

```bat
@echo off
cd /d C:\llama.cpp
llama-server.exe ^
  -m C:\models\gemma4-coding-Q4_K_M.gguf ^
  --ctx-size 16384 ^
  --n-gpu-layers 99 ^
  --no-mmap ^
  -fa on ^
  --cache-type-k q8_0 --cache-type-v q8_0 ^
  --temp 1.0 --top-p 0.95 --top-k 64 ^
  --host 0.0.0.0 --port 18080
pause
```
3. Open `http://localhost:18080` and chat. 🎉  (Tip: bump `--ctx-size` per the table; use `q4_0` KV for more.)

### Option B — one-click apps 🖱️
Works in **LM Studio**, **Jan**, **Ollama**, etc. — just import the GGUF, pick your quant, go. 🐾

### 🧠 Thinking mode
This model thinks in Gemma's native thought channel before answering — exactly how it was trained. Keep
**`enable_thinking=true`** (the default chat template handles it). Recommended sampling: `temp 1.0, top_p 0.95, top_k 64`.
For coding you can also go greedy (`temp 0`) for more deterministic solutions.

---

## ⚠️ Good to know
- **Reduced refusals:** the training data is task-focused with no safety hedging, so this refuses less than the base
  model. It is **not** safety-aligned — add your own guardrails for production. Use responsibly. 🙏
- Specialized for **Python / algorithmic** coding. Reasoning quality is strongest in that domain; general-knowledge
  facts/numbers should still be double-checked.
- English-centric.

---

## 📚 Base & License
- **Base model:** [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it). Subject to the
  **[Gemma Terms of Use](https://ai.google.dev/gemma/terms)** (derivatives must comply).
- Personal/hobby project — shared as-is, no warranty. Have fun, and happy hacking! 🐾✨