---
license: apache-2.0
base_model: ai9stars/G9v3-3B
pipeline_tag: text-generation
tags:
- g9v3
- gguf
- llama-cpp
- cpu
- tool-calling
- long-context
language:
- en
- zh
---

# G9v3-3B Q3_K_M GGUF — 4-core CPU tier

A CPU-tier-validated Q3_K_M cut of **[ai9stars/G9v3-3B](https://huggingface.co/ai9stars/G9v3-3B)**
(dense ~3B, LlamaForCausalLM, 131K context, think/no-think modes, XML tool calling),
packaged for llama.cpp on low-core machines. The GGUF itself is quantized by
**[mradermacher/G9v3-3B-GGUF](https://huggingface.co/mradermacher/G9v3-3B-GGUF)**;
this repo adds the serving template and **measured performance numbers** for the
4-core tier.

## What's in this repo

| File | Size | What it is |
|---|---|---|
| `g9v3-3B.Q3_K_M.gguf` | 1.5 GB | the quantized model |
| `g9v3_chat_template_low.jinja` | 12 KB | chat template — **required for tool calling** (see below) |

## Measured performance (4 threads, x86-64 AVX2)

Measured on an 8-vCPU EPYC @ 2.0 GHz run at 4 threads to simulate a 4-core box,
with llama.cpp `llama-server`:

- **Decode: ~25 tok/s** (think-mode ~14 tok/s) — comfortably above the ~10 tok/s
  interactive floor for a reasoning model
- **Prompt processing: ~55 tok/s** — a 5.5K-token prompt costs ~100 s *once*; the
  prefix cache makes repeat requests ~0.6 s
- Sanity check: reasoning is separated from the answer, tool calls emit structured
  `tool_calls`, and legacy OpenAI `function_call` history is normalized by the template

RAM need: ≥6 GB. Context memory: KV cache is 52 KiB/token on this model — 8192 ctx
fits an 8 GB box, 32768 ctx fits 16 GB.

## Run with llama.cpp

```bash
# get the files
huggingface-cli download itopoly/G9v3-3B-Q3_K_M-GGUF g9v3-3B.Q3_K_M.gguf --local-dir .
huggingface-cli download itopoly/G9v3-3B-Q3_K_M-GGUF g9v3_chat_template_low.jinja --local-dir .

# serve (OpenAI-compatible on /v1/chat/completions, model name: g9v3-3b)
llama-server -m g9v3-3B.Q3_K_M.gguf --alias g9v3-3b \
  -t 4 -tb 4 -c 8192 --port 8000 --host 0.0.0.0 \
  --chat-template-file g9v3_chat_template_low.jinja
```

Reasoning text arrives in `message.reasoning`, the answer in `content`.

## Why the chat template matters

Don't serve this GGUF without `g9v3_chat_template_low.jinja`: the stock template drops
tool-result messages, and the model re-calls the same tool forever (an observed loop
bug). The bundled template normalizes legacy tool history so multi-turn tool use works.

## Notes

- No-think mode: `temperature=0.7, top_p=0.95`; think mode: `temperature=1.0, top_p=0.95`.
- Prefill is the bottleneck on CPU — keep prompts ≤ 1–2K tokens, or accept a one-time
  ~2 min first hit per distinct large prefix (cached afterwards).
- Tool calling works (verified single-turn + both tool-history formats), but tool-call
  quality on a 3B is tier-limited; the 39B family models are the tool-heavy choice.

## Credits

- Original model: [ai9stars/G9v3-3B](https://huggingface.co/ai9stars/G9v3-3B) (Apache-2.0)
- GGUF quantization: [mradermacher/G9v3-3B-GGUF](https://huggingface.co/mradermacher/G9v3-3B-GGUF)
- CPU-tier validation, template, and packaging: itopoly