---
license: apache-2.0
base_model: ai9stars/G9v3-3B
pipeline_tag: text-generation
tags:
- g9v3
- gguf
- llama-cpp
- cpu
- tool-calling
- long-context
language:
- en
- zh
---

# G9v3-3B Q4_K_M GGUF — 6-core CPU tier

A CPU-tier-validated Q4_K_M cut of **[ai9stars/G9v3-3B](https://huggingface.co/ai9stars/G9v3-3B)**
(dense ~3B, LlamaForCausalLM, 131K context, think/no-think modes, XML tool calling),
packaged for llama.cpp on mid-range machines. The GGUF is quantized by
**[mradermacher/G9v3-3B-GGUF](https://huggingface.co/mradermacher/G9v3-3B-GGUF)**;
this repo adds the serving template and **measured performance numbers** for the
6-core tier. Companion repo: [Itopoly/G9v3-3B-Q3_K_M-GGUF](https://huggingface.co/Itopoly/G9v3-3B-Q3_K_M-GGUF)
(the 4-core tier cut).

## What's in this repo

| File | Size | What it is |
|---|---|---|
| `G9v3-3B.Q4_K_M.gguf` | 1.8 GB | the quantized model (4.9 BPW) |
| `g9v3_chat_template_low.jinja` | 12 KB | chat template — **required for tool calling** (see below) |

## Measured performance (6 threads, x86-64 AVX2)

Measured on an 8-vCPU EPYC @ 2.0 GHz run at 6 threads, with llama.cpp `llama-server`:

- **Decode: ~25 tok/s at short context** (~15 tok/s at 4K ctx, ~10 tok/s at 8K ctx —
  CPU decode degrades ~5 ms/token per 1K of context)
- **Prefill: ~73 tok/s** — a 5.5K-token prompt costs ~75–90 s *once*; the prefix cache
  makes repeat requests near-instant
- **A/B vs Q3_K_M at identical settings:** Q3 ≈ 26 tok/s, Q4 ≈ 25 tok/s — on a 3B,
  quant level barely moves decode; the bandwidth difference is too small. Q4 wins on
  quality (visibly fewer 3-bit artifacts, coherent reasoning, correct "Paris" sanity
  answer). **Threads matter more than quant on a 3B.**
- `-fa OFF` recommended — flash-attention is *slower* on CPU

RAM need: ≥10 GB free for 32K context (KV cache is 52 KiB/token on this model);
131K context fits a 12 GB box.

## Run with llama.cpp

```bash
# get the files
huggingface-cli download Itopoly/G9v3-3B-Q4_K_M-GGUF G9v3-3B.Q4_K_M.gguf --local-dir .
huggingface-cli download Itopoly/G9v3-3B-Q4_K_M-GGUF g9v3_chat_template_low.jinja --local-dir .

# serve (OpenAI-compatible on /v1/chat/completions, model name: g9v3-3b-q4_k_m)
llama-server -m G9v3-3B.Q4_K_M.gguf --alias g9v3-3b-q4_k_m \
  -t 6 -tb 8 -c 32768 -fa off --port 8000 --host 0.0.0.0 \
  --chat-template-file g9v3_chat_template_low.jinja
```

Reasoning text arrives in `message.reasoning`, the answer in `content`.

## Why the chat template matters

Don't serve this GGUF without `g9v3_chat_template_low.jinja`: the stock template drops
tool-result messages, and the model re-calls the same tool forever (an observed loop
bug). The bundled template normalizes legacy tool history so multi-turn tool use works.

## Notes

- No-think mode: `temperature=0.7, top_p=0.95`; think mode: `temperature=0.9, top_p=0.95`.
- Prefill is the bottleneck on CPU — keep prompts ≤ 1–2K tokens, or accept a one-time
  ~75–90 s first hit per distinct large prefix (cached afterwards).
- No idle eviction: the model and prompt cache stay resident, so second requests on the
  same prefix are near-instant.
- Tool calling works (verified single-turn + both tool-history formats), but tool-call
  quality on a 3B is tier-limited — the 39B family models are the tool-heavy choice.

## Credits

- Original model: [ai9stars/G9v3-3B](https://huggingface.co/ai9stars/G9v3-3B) (Apache-2.0)
- GGUF quantization: [mradermacher/G9v3-3B-GGUF](https://huggingface.co/mradermacher/G9v3-3B-GGUF)
- CPU-tier validation, template, and packaging: Itopoly
  (see also [our Q3_K_M 4-core cut](https://huggingface.co/Itopoly/G9v3-3B-Q3_K_M-GGUF))