README.md

---
license: apache-2.0
base_model: Agnes-AI/Agnes-3.0-Flash
pipeline_tag: image-text-to-text
language:
- en
- zh
tags:
- gguf
- agnes
- multimodal
---

# Agnes-3.0-Flash GGUF

**Follow [@procrastiness on Twitter/X](https://twitter.com/procrastiness) for more model releases and updates.**

GGUF quantizations of the original [Agnes-AI/Agnes-3.0-Flash](https://huggingface.co/Agnes-AI/Agnes-3.0-Flash), released by 0xKitkat. **This is the normal model, with no abliteration or fine-tuning.** Approximately 33.1B parameters; all 72 decoder blocks and the original vision tower are retained.

The original Apache-2.0 license is preserved. Downloads and local use require no hosted API subscription; you supply the hardware and electricity. The source revision is `8f0c484c363cdda8384195be4a5f7730f3915bde`.

Available validated quants: Q4_K_M, Q5_K_M, Q6_K, Q8_0. All planned quants are published.

<!-- GGUF-USAGE-BEGIN -->
## Recommended starting setup

Start with **Q4_K_M**, a **4,096-token context**, **one concurrent request**, and **thinking disabled**. This is the configuration closest to the functional checks reported below. Use the embedded chat template; do not select a generic ChatML or Llama template manually.

For normal chat, use the upstream sampling defaults: **temperature 1.0, top-p 0.95, top-k 20**. For repeatable troubleshooting or checking exact answers, use **temperature 0**. Start with **512 output tokens** for short answers or **2,048** for longer responses, keeping prompt + image tokens + output within the configured context.

### Choose a quant and hardware

Download **one text GGUF**. Add the **F16 vision projector** only if you want image input. The projector is shared by all quants in this release.

| Quant | When to choose it |
|---|---|
| **Q4_K_M** | Recommended first download; smallest planned file and the easiest fit on consumer hardware. |
| **Q5_K_M** | More weight precision if you have additional memory; compare the measured results below. |
| **Q6_K** | Higher precision with a larger memory footprint. |
| **Q8_0** | Highest precision in this quant set; intended for larger-memory machines. |

The validation table lists exact sizes for published files. **A quant being listed here does not mean its upload has finished**; check the available-files list and the Files tab.

| Hardware | Suggested starting point |
|---|---|
| Single 8–16 GB GPU + 32 GB or more system RAM | Q4_K_M with CPU offloading. Use automatic fitting initially; lower GPU layers if needed. Generation will be slower than a full GPU fit. |
| Single 24 GB GPU + 32 GB or more system RAM | Try Q4_K_M at 4K context. Leave room for buffers, display usage, and the projector; reduce GPU layers if allocation fails. |
| Two 12 GB GPUs + about 32 GB system RAM | Q4_K_M, layer split `1,1`, 4K context. This release was exercised on two RTX 2060 12 GB GPUs. |
| 48 GB or more GPU memory | Consider Q6_K or Q8_0 at 4K first, then increase context after checking memory use. |
| CPU only, or Apple Silicon unified memory | Start with Q4_K_M and preferably 32 GB or more available system/unified memory. CPU-only and Metal performance were not benchmarked for this release. |

These are starting recommendations, not exact minimums. Weight-file size excludes runtime buffers, recurrent state, attention cache, images, and other applications. A 16 GB machine is a poor fit for this approximately 33B model. CPU-only operation benefits from 48–64 GB RAM when other applications are running.

## 1. Install llama.cpp and the download tool

Use a recent **llama.cpp** build with Qwen3.5 text and vision support. The release was tested with commit `56381e407c0ccfb3a6f71e668a27a901001d22ce`. Older bundled runtimes can reject the model architecture or load it incorrectly.

### Windows

Install Python 3.10+ if needed, then open PowerShell:

```powershell
winget install llama.cpp
python -m pip install --upgrade huggingface_hub requests
llama-server --version
```

Open a new terminal after installing if commands are not found. For NVIDIA acceleration, use a compatible CUDA build from the [official llama.cpp releases](https://github.com/ggml-org/llama.cpp/releases), extract the complete archive, and keep its DLLs alongside `llama-server.exe`. From that folder, replace `llama-server` in the commands below with `.\llama-server.exe`. Confirm the startup log detects your GPU; installing a package alone does not establish which backend it uses.

### macOS

With [Homebrew](https://brew.sh/) and Python installed:

```bash
brew install llama.cpp
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade huggingface_hub requests
llama-server --version
```

### Ubuntu / Debian: reproducible NVIDIA build

Install a compatible NVIDIA driver and CUDA Toolkit first; `nvidia-smi` and `nvcc --version` should work. Then:

```bash
sudo apt-get update
sudo apt-get install -y git cmake build-essential libcurl4-openssl-dev python3-venv
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 56381e407c0ccfb3a6f71e668a27a901001d22ce
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4 --target llama-server
export PATH="$PWD/build/bin:$PATH"
cd ..
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade huggingface_hub requests
```

For a CPU-only build, configure with `-DGGML_CUDA=OFF` and start the server with `-ngl 0`. Other backends and installation options are covered in the [llama.cpp installation guide](https://github.com/ggml-org/llama.cpp/blob/56381e407c0ccfb3a6f71e668a27a901001d22ce/docs/install.md) and [build guide](https://github.com/ggml-org/llama.cpp/blob/56381e407c0ccfb3a6f71e668a27a901001d22ce/docs/build.md).

## 2. Download the model

Run these in the directory where you want the `agnes` folder. The public download does not require a paid account or hosted inference subscription.

```bash
hf download 0xKitkat/Agnes-3.0-Flash-GGUF Agnes-3.0-Flash-Q4_K_M.gguf --local-dir agnes
```

For image input, also download:

```bash
hf download 0xKitkat/Agnes-3.0-Flash-GGUF mmproj-Agnes-3.0-Flash-F16.gguf --local-dir agnes
```

For another published quant, replace `Q4_K_M` in the filename with `Q5_K_M`, `Q6_K`, or `Q8_0`. Rerun the same download command after an interruption. Allow disk space for the chosen file plus the approximately 0.93 GB projector if used.

SHA-256 checksums are recorded in [release-manifest.json](./release-manifest.json). To calculate a local checksum:

```powershell
Get-FileHash agnes/Agnes-3.0-Flash-Q4_K_M.gguf -Algorithm SHA256
```

On Linux use `sha256sum agnes/Agnes-3.0-Flash-Q4_K_M.gguf`; on macOS use `shasum -a 256` with the same path.

## 3. Start the local server

### Text-only: single GPU or automatic fitting

```bash
llama-server -m agnes/Agnes-3.0-Flash-Q4_K_M.gguf --alias agnes -c 4096 --parallel 1 --fit on --jinja --flash-attn on --batch-size 256 --ubatch-size 128 --threads 6 --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8080
```

Open **http://127.0.0.1:8080** for the built-in chat UI. Keep the terminal running. The API is at **http://127.0.0.1:8080/v1**; its model name is **`agnes`**. This local server does not need an API key. If a client requires a nonempty key field, use `local`.

The command lets llama.cpp choose GPU offloading with `--fit on`. If it runs out of GPU memory, specify a smaller layer count, for example `-ngl 20`, and adjust from there. Use `-ngl 0` for CPU-only inference. Six CPU threads is a starting value used in our checks; tune it for your CPU.

### Text and images: tested dual-GPU layout

```bash
llama-server -m agnes/Agnes-3.0-Flash-Q4_K_M.gguf --mmproj agnes/mmproj-Agnes-3.0-Flash-F16.gguf --alias agnes -c 4096 --parallel 1 -ngl 99 --split-mode layer --tensor-split 1,1 --jinja --flash-attn on --batch-size 256 --ubatch-size 128 --threads 6 --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8080
```

`-ngl 99` requests all eligible layers on the GPUs; it is not a claim that the model has 99 layers. `--tensor-split 1,1` distributes layers between two GPUs. On one GPU, use the first command and add `--mmproj agnes/mmproj-Agnes-3.0-Flash-F16.gguf`. For larger quants, lower the GPU layer count or let automatic fitting choose it.

### Thinking and context settings

The API examples below explicitly disable thinking, matching release testing. To make this the server default for the web UI too, set `LLAMA_ARG_CHAT_TEMPLATE_KWARGS` **before** starting the server:

```powershell
# PowerShell
$env:LLAMA_ARG_CHAT_TEMPLATE_KWARGS = '{"enable_thinking":false}'
```

```bash
# Bash / zsh
export LLAMA_ARG_CHAT_TEMPLATE_KWARGS='{"enable_thinking":false}'
```

For experimental thinking-on use, send `"chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "xhigh"}` in the request. The source template accepts `low`, `medium`, and `xhigh` effort (the actual pinned template uses `xhigh`, even though the upstream card describes `high`). Allocate more output tokens and context for reasoning. This release's measured results do **not** validate reasoning-on quality, tool calling, video, or the advertised 262,144-token context. Start at 4K; try 8K only after confirming adequate memory and prompt-plus-output space.

## 4. Use the API from Python

Save as `chat_agnes.py`, then run `python chat_agnes.py` while the server is running:

```python
import unicodedata
import requests

prompt = unicodedata.normalize("NFC", "Explain how a rainbow forms in three sentences.")
response = requests.post(
    "http://127.0.0.1:8080/v1/chat/completions",
    json={
        "model": "agnes",
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 1.0,
        "top_p": 0.95,
        "top_k": 20,
        "max_tokens": 512,
        "chat_template_kwargs": {"enable_thinking": False},
        "stream": False,
    },
    timeout=1800,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"]["content"])
```

For deterministic checks, change `temperature` to `0` and add `"seed": 20260912`. Sampling defaults above come from the [original model](https://huggingface.co/Agnes-AI/Agnes-3.0-Flash); they are a starting point, not a new tuning benchmark.

### Ask about a local image

Start the server with the projector. Save this as `image_agnes.py`, place an `image.jpg` alongside it, and run `python image_agnes.py`:

```python
import base64
from pathlib import Path
import requests

encoded = base64.b64encode(Path("image.jpg").read_bytes()).decode("ascii")
response = requests.post(
    "http://127.0.0.1:8080/v1/chat/completions",
    json={
        "model": "agnes",
        "messages": [{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the main objects in this image."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{encoded}"}},
            ],
        }],
        "temperature": 0,
        "max_tokens": 256,
        "chat_template_kwargs": {"enable_thinking": False},
        "stream": False,
    },
    timeout=1800,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"]["content"])
```

For PNG input, change the file path and use `data:image/png;base64,`. Begin with one modest-size image; image tokens also consume context. The API format follows [llama.cpp's multimodal server documentation](https://github.com/ggml-org/llama.cpp/blob/56381e407c0ccfb3a6f71e668a27a901001d22ce/docs/multimodal.md).

## Troubleshooting

| Symptom | What to change |
|---|---|
| Unknown architecture, missing tensor, or unsupported operation | Update llama.cpp to the tested revision or a newer compatible build. This is a Qwen3.5 GGUF graph conversion, so older runtimes may not support it. |
| GPU out of memory | Use Q4_K_M; reduce `-ngl`, context, or `--ubatch-size` to `64`; keep `--parallel 1`. Add the projector only when needed. |
| Very slow generation | Check the startup log for actual GPU offloading. Q5/Q6/Q8 may spill heavily to CPU on 24 GB total VRAM. Try Q4 and close memory-heavy applications. |
| Thought tags or unexpectedly long reasoning | Set `chat_template_kwargs.enable_thinking` to `false`, or set the server environment variable shown above. |
| Output stops early | Check the response's `finish_reason`. Raise `max_tokens` if it is `length`, while ensuring sufficient context remains. |
| Image request fails | Check that the matching projector is loaded and that the data URL's MIME type matches the image file. |
| Unicode tokenization differs from Transformers | NFC-normalize text before sending it. Stock llama.cpp does not reproduce the upstream tokenizer's NFC normalization automatically. |

Other applications must bundle a sufficiently recent compatible backend. LM Studio and Ollama were not part of this release's validation; the commands above use llama.cpp directly.
<!-- GGUF-USAGE-END -->

## Validation

| Quant | File GiB | 24 deterministic tasks | Native top-10 token overlap | 3 synthetic image checks | WikiText PPL |
|---|---:|---:|---:|---:|---:|
| Q4_K_M | 18.40 | 95.8% | 93.1% | 100.0% | 7.160 |
| Q5_K_M | 21.42 | 95.8% | 95.6% | 100.0% | 7.113 |
| Q6_K | 24.62 | 95.8% | 97.5% | 100.0% | 7.082 |
| Q8_0 | 31.89 | 95.8% | 98.8% | 100.0% | 7.093 |

These are small functional tests, not comprehensive capability benchmarks. Native-reference checks compare next-token distributions on 16 prompts with a layer-streamed upstream calculation using FP32 computation and FP16 residual storage. All generation checks use thinking disabled. Per-quant reports and SHA-256 hashes are included. Long-context limits, video, reasoning-on quality, and tool calling are not exhaustively tested here.

WikiText perplexity uses eight 512-token chunks from a pinned WikiText-2 test corpus. Each normal quant must stay within 5% of the unmodified Q4 baseline. This is a subset loss check, not a full-corpus benchmark.

Q4_K_M, Q5_K_M and Q6_K use a 32-chunk importance matrix computed from the unmodified Q8_0 model. Q8_0 is quantized directly from the original BF16 weights. The projector is F16 and is shared by all quants.

## Architecture mapping

Agnes uses the same attention computation as the supported Qwen3.5 GGUF graph. Its additional parallel SwiGLU branch is preserved exactly by concatenating both branches' gate/up matrices and concatenating their down matrices on the input dimension. This is an algebraic graph conversion, not a trained modification. MTP speculative decoding weights are omitted from GGUF. The vision tower maps to the Qwen3.5 vision projector graph. Structural, rotary, tokenizer, and FP32 next-token equivalence tests are included.

The upstream tokenizer applies NFC Unicode normalization; stock llama.cpp's Qwen3.5 tokenizer does not. For exact agreement on decomposed Unicode, normalize text with `unicodedata.normalize("NFC", text)` before rendering the chat prompt. Already-normalized English and Chinese text is unaffected.


Original model and implementation: Agnes AI. Build code was generated with AI assistance and tested as reported. See the original license and upstream model card for attribution.