Publish remotely verified release and evaluations

README.md

@@ -1,9 +1,261 @@
 ---
 license: apache-2.0
+base_model: Agnes-AI/Agnes-3.0-Flash
+pipeline_tag: image-text-to-text
+language:
+- en
+- zh
+tags:
+- gguf
+- agnes
+- multimodal
 ---
 
-# Validated release upload in progress
-
-Local validation has passed. Files are uploading; the release is not complete yet. The final model card and evaluation tables will appear after remote hash verification.
+# Agnes-3.0-Flash GGUF
 
 **Follow [@procrastiness on Twitter/X](https://twitter.com/procrastiness) for more model releases and updates.**
+
+GGUF quantizations of the original [Agnes-AI/Agnes-3.0-Flash](https://huggingface.co/Agnes-AI/Agnes-3.0-Flash), released by 0xKitkat. **This is the normal model, with no abliteration or fine-tuning.** Approximately 33.1B parameters; all 72 decoder blocks and the original vision tower are retained.
+
+The original Apache-2.0 license is preserved. Downloads and local use require no hosted API subscription; you supply the hardware and electricity. The source revision is `8f0c484c363cdda8384195be4a5f7730f3915bde`.
+
+Available validated quants: Q4_K_M, Q5_K_M. Additional quants are still being built and validated.
+
+<!-- GGUF-USAGE-BEGIN -->
+## Recommended starting setup
+
+Start with **Q4_K_M**, a **4,096-token context**, **one concurrent request**, and **thinking disabled**. This is the configuration closest to the functional checks reported below. Use the embedded chat template; do not select a generic ChatML or Llama template manually.
+
+For normal chat, use the upstream sampling defaults: **temperature 1.0, top-p 0.95, top-k 20**. For repeatable troubleshooting or checking exact answers, use **temperature 0**. Start with **512 output tokens** for short answers or **2,048** for longer responses, keeping prompt + image tokens + output within the configured context.
+
+### Choose a quant and hardware
+
+Download **one text GGUF**. Add the **F16 vision projector** only if you want image input. The projector is shared by all quants in this release.
+
+| Quant | When to choose it |
+|---|---|
+| **Q4_K_M** | Recommended first download; smallest planned file and the easiest fit on consumer hardware. |
+| **Q5_K_M** | More weight precision if you have additional memory; compare the measured results below. |
+| **Q6_K** | Higher precision with a larger memory footprint. |
+| **Q8_0** | Highest precision in this quant set; intended for larger-memory machines. |
+
+The validation table lists exact sizes for published files. **A quant being listed here does not mean its upload has finished**; check the available-files list and the Files tab.
+
+| Hardware | Suggested starting point |
+|---|---|
+| Single 8–16 GB GPU + 32 GB or more system RAM | Q4_K_M with CPU offloading. Use automatic fitting initially; lower GPU layers if needed. Generation will be slower than a full GPU fit. |
+| Single 24 GB GPU + 32 GB or more system RAM | Try Q4_K_M at 4K context. Leave room for buffers, display usage, and the projector; reduce GPU layers if allocation fails. |
+| Two 12 GB GPUs + about 32 GB system RAM | Q4_K_M, layer split `1,1`, 4K context. This release was exercised on two RTX 2060 12 GB GPUs. |
+| 48 GB or more GPU memory | Consider Q6_K or Q8_0 at 4K first, then increase context after checking memory use. |
+| CPU only, or Apple Silicon unified memory | Start with Q4_K_M and preferably 32 GB or more available system/unified memory. CPU-only and Metal performance were not benchmarked for this release. |
+
+These are starting recommendations, not exact minimums. Weight-file size excludes runtime buffers, recurrent state, attention cache, images, and other applications. A 16 GB machine is a poor fit for this approximately 33B model. CPU-only operation benefits from 48–64 GB RAM when other applications are running.
+
+## 1. Install llama.cpp and the download tool
+
+Use a recent **llama.cpp** build with Qwen3.5 text and vision support. The release was tested with commit `56381e407c0ccfb3a6f71e668a27a901001d22ce`. Older bundled runtimes can reject the model architecture or load it incorrectly.
+
+### Windows
+
+Install Python 3.10+ if needed, then open PowerShell:
+
+```powershell
+winget install llama.cpp
+python -m pip install --upgrade huggingface_hub requests
+llama-server --version
+```
+
+Open a new terminal after installing if commands are not found. For NVIDIA acceleration, use a compatible CUDA build from the [official llama.cpp releases](https://github.com/ggml-org/llama.cpp/releases), extract the complete archive, and keep its DLLs alongside `llama-server.exe`. From that folder, replace `llama-server` in the commands below with `.\llama-server.exe`. Confirm the startup log detects your GPU; installing a package alone does not establish which backend it uses.
+
+### macOS
+
+With [Homebrew](https://brew.sh/) and Python installed:
+
+```bash
+brew install llama.cpp
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade huggingface_hub requests
+llama-server --version
+```
+
+### Ubuntu / Debian: reproducible NVIDIA build
+
+Install a compatible NVIDIA driver and CUDA Toolkit first; `nvidia-smi` and `nvcc --version` should work. Then:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y git cmake build-essential libcurl4-openssl-dev python3-venv
+git clone https://github.com/ggml-org/llama.cpp
+cd llama.cpp
+git checkout 56381e407c0ccfb3a6f71e668a27a901001d22ce
+cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
+cmake --build build --config Release -j 4 --target llama-server
+export PATH="$PWD/build/bin:$PATH"
+cd ..
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade huggingface_hub requests
+```
+
+For a CPU-only build, configure with `-DGGML_CUDA=OFF` and start the server with `-ngl 0`. Other backends and installation options are covered in the [llama.cpp installation guide](https://github.com/ggml-org/llama.cpp/blob/56381e407c0ccfb3a6f71e668a27a901001d22ce/docs/install.md) and [build guide](https://github.com/ggml-org/llama.cpp/blob/56381e407c0ccfb3a6f71e668a27a901001d22ce/docs/build.md).
+
+## 2. Download the model
+
+Run these in the directory where you want the `agnes` folder. The public download does not require a paid account or hosted inference subscription.
+
+```bash
+hf download 0xKitkat/Agnes-3.0-Flash-GGUF Agnes-3.0-Flash-Q4_K_M.gguf --local-dir agnes
+```
+
+For image input, also download:
+
+```bash
+hf download 0xKitkat/Agnes-3.0-Flash-GGUF mmproj-Agnes-3.0-Flash-F16.gguf --local-dir agnes
+```
+
+For another published quant, replace `Q4_K_M` in the filename with `Q5_K_M`, `Q6_K`, or `Q8_0`. Rerun the same download command after an interruption. Allow disk space for the chosen file plus the approximately 0.93 GB projector if used.
+
+SHA-256 checksums are recorded in [release-manifest.json](./release-manifest.json). To calculate a local checksum:
+
+```powershell
+Get-FileHash agnes/Agnes-3.0-Flash-Q4_K_M.gguf -Algorithm SHA256
+```
+
+On Linux use `sha256sum agnes/Agnes-3.0-Flash-Q4_K_M.gguf`; on macOS use `shasum -a 256` with the same path.
+
+## 3. Start the local server
+
+### Text-only: single GPU or automatic fitting
+
+```bash
+llama-server -m agnes/Agnes-3.0-Flash-Q4_K_M.gguf --alias agnes -c 4096 --parallel 1 --fit on --jinja --flash-attn on --batch-size 256 --ubatch-size 128 --threads 6 --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8080
+```
+
+Open **http://127.0.0.1:8080** for the built-in chat UI. Keep the terminal running. The API is at **http://127.0.0.1:8080/v1**; its model name is **`agnes`**. This local server does not need an API key. If a client requires a nonempty key field, use `local`.
+
+The command lets llama.cpp choose GPU offloading with `--fit on`. If it runs out of GPU memory, specify a smaller layer count, for example `-ngl 20`, and adjust from there. Use `-ngl 0` for CPU-only inference. Six CPU threads is a starting value used in our checks; tune it for your CPU.
+
+### Text and images: tested dual-GPU layout
+
+```bash
+llama-server -m agnes/Agnes-3.0-Flash-Q4_K_M.gguf --mmproj agnes/mmproj-Agnes-3.0-Flash-F16.gguf --alias agnes -c 4096 --parallel 1 -ngl 99 --split-mode layer --tensor-split 1,1 --jinja --flash-attn on --batch-size 256 --ubatch-size 128 --threads 6 --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8080
+```
+
+`-ngl 99` requests all eligible layers on the GPUs; it is not a claim that the model has 99 layers. `--tensor-split 1,1` distributes layers between two GPUs. On one GPU, use the first command and add `--mmproj agnes/mmproj-Agnes-3.0-Flash-F16.gguf`. For larger quants, lower the GPU layer count or let automatic fitting choose it.
+
+### Thinking and context settings
+
+The API examples below explicitly disable thinking, matching release testing. To make this the server default for the web UI too, set `LLAMA_ARG_CHAT_TEMPLATE_KWARGS` **before** starting the server:
+
+```powershell
+# PowerShell
+$env:LLAMA_ARG_CHAT_TEMPLATE_KWARGS = '{"enable_thinking":false}'
+```
+
+```bash
+# Bash / zsh
+export LLAMA_ARG_CHAT_TEMPLATE_KWARGS='{"enable_thinking":false}'
+```
+
+For experimental thinking-on use, send `"chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "xhigh"}` in the request. The source template accepts `low`, `medium`, and `xhigh` effort (the actual pinned template uses `xhigh`, even though the upstream card describes `high`). Allocate more output tokens and context for reasoning. This release's measured results do **not** validate reasoning-on quality, tool calling, video, or the advertised 262,144-token context. Start at 4K; try 8K only after confirming adequate memory and prompt-plus-output space.
+
+## 4. Use the API from Python
+
+Save as `chat_agnes.py`, then run `python chat_agnes.py` while the server is running:
+
+```python
+import unicodedata
+import requests
+
+prompt = unicodedata.normalize("NFC", "Explain how a rainbow forms in three sentences.")
+response = requests.post(
+    "http://127.0.0.1:8080/v1/chat/completions",
+    json={
+        "model": "agnes",
+        "messages": [{"role": "user", "content": prompt}],
+        "temperature": 1.0,
+        "top_p": 0.95,
+        "top_k": 20,
+        "max_tokens": 512,
+        "chat_template_kwargs": {"enable_thinking": False},
+        "stream": False,
+    },
+    timeout=1800,
+)
+response.raise_for_status()
+print(response.json()["choices"][0]["message"]["content"])
+```
+
+For deterministic checks, change `temperature` to `0` and add `"seed": 20260912`. Sampling defaults above come from the [original model](https://huggingface.co/Agnes-AI/Agnes-3.0-Flash); they are a starting point, not a new tuning benchmark.
+
+### Ask about a local image
+
+Start the server with the projector. Save this as `image_agnes.py`, place an `image.jpg` alongside it, and run `python image_agnes.py`:
+
+```python
+import base64
+from pathlib import Path
+import requests
+
+encoded = base64.b64encode(Path("image.jpg").read_bytes()).decode("ascii")
+response = requests.post(
+    "http://127.0.0.1:8080/v1/chat/completions",
+    json={
+        "model": "agnes",
+        "messages": [{
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "Describe the main objects in this image."},
+                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{encoded}"}},
+            ],
+        }],
+        "temperature": 0,
+        "max_tokens": 256,
+        "chat_template_kwargs": {"enable_thinking": False},
+        "stream": False,
+    },
+    timeout=1800,
+)
+response.raise_for_status()
+print(response.json()["choices"][0]["message"]["content"])
+```
+
+For PNG input, change the file path and use `data:image/png;base64,`. Begin with one modest-size image; image tokens also consume context. The API format follows [llama.cpp's multimodal server documentation](https://github.com/ggml-org/llama.cpp/blob/56381e407c0ccfb3a6f71e668a27a901001d22ce/docs/multimodal.md).
+
+## Troubleshooting
+
+| Symptom | What to change |
+|---|---|
+| Unknown architecture, missing tensor, or unsupported operation | Update llama.cpp to the tested revision or a newer compatible build. This is a Qwen3.5 GGUF graph conversion, so older runtimes may not support it. |
+| GPU out of memory | Use Q4_K_M; reduce `-ngl`, context, or `--ubatch-size` to `64`; keep `--parallel 1`. Add the projector only when needed. |
+| Very slow generation | Check the startup log for actual GPU offloading. Q5/Q6/Q8 may spill heavily to CPU on 24 GB total VRAM. Try Q4 and close memory-heavy applications. |
+| Thought tags or unexpectedly long reasoning | Set `chat_template_kwargs.enable_thinking` to `false`, or set the server environment variable shown above. |
+| Output stops early | Check the response's `finish_reason`. Raise `max_tokens` if it is `length`, while ensuring sufficient context remains. |
+| Image request fails | Check that the matching projector is loaded and that the data URL's MIME type matches the image file. |
+| Unicode tokenization differs from Transformers | NFC-normalize text before sending it. Stock llama.cpp does not reproduce the upstream tokenizer's NFC normalization automatically. |
+
+Other applications must bundle a sufficiently recent compatible backend. LM Studio and Ollama were not part of this release's validation; the commands above use llama.cpp directly.
+<!-- GGUF-USAGE-END -->
+
+## Validation
+
+| Quant | File GiB | 24 deterministic tasks | Native top-10 token overlap | 3 synthetic image checks | WikiText PPL |
+|---|---:|---:|---:|---:|---:|
+| Q4_K_M | 18.40 | 95.8% | 93.1% | 100.0% | 7.160 |
+| Q5_K_M | 21.42 | 95.8% | 95.6% | 100.0% | 7.113 |
+
+These are small functional tests, not comprehensive capability benchmarks. Native-reference checks compare next-token distributions on 16 prompts with a layer-streamed upstream calculation using FP32 computation and FP16 residual storage. All generation checks use thinking disabled. Per-quant reports and SHA-256 hashes are included. Long-context limits, video, reasoning-on quality, and tool calling are not exhaustively tested here.
+
+WikiText perplexity uses eight 512-token chunks from a pinned WikiText-2 test corpus. Each normal quant must stay within 5% of the unmodified Q4 baseline. This is a subset loss check, not a full-corpus benchmark.
+
+Q4_K_M, Q5_K_M and Q6_K use a 32-chunk importance matrix computed from the unmodified Q8_0 model. Q8_0 is quantized directly from the original BF16 weights. The projector is F16 and is shared by all quants.
+
+## Architecture mapping
+
+Agnes uses the same attention computation as the supported Qwen3.5 GGUF graph. Its additional parallel SwiGLU branch is preserved exactly by concatenating both branches' gate/up matrices and concatenating their down matrices on the input dimension. This is an algebraic graph conversion, not a trained modification. MTP speculative decoding weights are omitted from GGUF. The vision tower maps to the Qwen3.5 vision projector graph. Structural, rotary, tokenizer, and FP32 next-token equivalence tests are included.
+
+The upstream tokenizer applies NFC Unicode normalization; stock llama.cpp's Qwen3.5 tokenizer does not. For exact agreement on decomposed Unicode, normalize text with `unicodedata.normalize("NFC", text)` before rendering the chat prompt. Already-normalized English and Chinese text is unaffected.
+
+
+Original model and implementation: Agnes AI. Build code was generated with AI assistance and tested as reported. See the original license and upstream model card for attribution.