README.md

---
language:
- en
- zh
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- agnes-ai
- uncensored
- reasoning
- multimodal
- fp8
- w8a8
- vllm
- sglang
- long-context
- hybrid-attention
---


# Agnes 3.0 Flash FP8 (W8A8, calibrated)

Full FP8 quantization of Agnes-3.0-Flash Preview (33B parameters, multimodal).
Weights and activations are FP8 e4m3 with static per-tensor scales, calibrated
with LLM Compressor. This gives you both the model and context at FP8. Text, 
image, and video inputs are preserved, and the chat template is upstream's.

The Sglang patch from the original model checkpoint is included here as well.

*Note that this model seems to be naturally uncensored. I ran Heretic's assessment
and it gave 0/300 refusals. So no need to abliterate this model. And with that,
my usual warning here:*

> **Uncensored model:** the language checkpoint has undergone abliteration 
to reduce refusal behavior. Treat outputs as untrusted, apply application-level 
safeguards, and do not assume the model will decline harmful requests.

> **User responsibility:** this model is provided without warranty. The 
creators, uploaders, and maintainers are not responsible or liable for what 
others generate, publish, deploy, or otherwise do with this abliterated model. 
Users must operate it responsibly, apply appropriate safeguards, comply with 
applicable law, and respect third-party rights. This model is for research 
purposes only and is not intended for production use.


## Quantization recipe

| Field | Value |
|---|---|
| Format | `compressed-tensors`, `float-quantized` |
| Weights | FP8 (e4m3), per-tensor symmetric, static |
| Activations | FP8 (e4m3), per-tensor symmetric, static, calibrated |
| Targets | all `Linear` modules except the protected list below |
| Toolkit | llm-compressor 0.13.0, compressed-tensors 0.18.0 |
| Calibration | `HuggingFaceH4/ultrachat_200k`, split `train_sft`, revision `8049631c405ae6576f93f445c6b8166f76f5505a` |
| Calibration size | 512 samples, max sequence length 2048, batch 1, seed 42 |

Kept at original precision: `lm_head`, `embed_tokens`, the full `model.visual`
tower, the recurrent-state layers of delta attention (`conv1d`, `in_proj_a`,
`in_proj_b`), and every MTP weight (shipped unquantized in
`model-mtp.safetensors`).

`agnes_quantization_manifest.json` records the full build, including the
SHA-256 of the calibration prompt ids used, so the run is reproducible piece
for piece. `recipe.yaml` is the machine-readable form of the table above.

## Why the quantization runs on a fused checkpoint

The upstream FFN carries two branches per layer: a main branch (width 17,408)
and a parallel branch (width 2,048). This checkpoint folds them into a single
set of projections per layer (`intermediate_size: 19456`,
`parallel_ffn_intermediate_size: 0`).

Fusing in bf16 is concatenation and it is exact: gate/up join along the output
dimension, down along the input dimension. Static per-tensor FP8 scales cannot
be merged after the fact, because each branch carries its own scale and a
single fused matrix has room for exactly one. Quantizing the fused layout
means every serving tensor is quantized once, from the values the model will
actually run, and calibration observes the same matrix geometry as inference.

## Numerics

The fused layout measures 5.9e-4 full-vocabulary KL against the two-branch
layout at identical weights. The residual comes from reduction order in the
fused projections, and it sits below the 6.5e-4 that independent inference
stacks produce from the same unquantized checkpoint. Perplexity moves 17.05 to
17.07 across the fusion.

The FP8 conversion itself has no formal before/after eval on this card. Run
your own benchmarks before trusting it for a workload.

## Files

| Group | Contents |
|---|---|
| Weights | `model-00001-of-00002.safetensors`, `model-00002-of-00002.safetensors`, `model-mtp.safetensors`, `model.safetensors.index.json` |
| Config | `config.json`, `generation_config.json`, custom `*_agnes.py` modeling and processor code |
| Tokenizer | `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja` |
| Provenance | `recipe.yaml`, `agnes_quantization_manifest.json` |
| Extras | `sglang_patch/`, `serve.sh` |

The checkpoint is standard `compressed-tensors float-quantized`, so any engine
that reads that format can load it.

## Upstream and license

Derived from the `Agnes-AI/Agnes-3.0-Flash` Preview checkpoint at revision
`891ce4f9ffb89b22888aa7fcc2bb2f3618867684`, Apache 2.0, unchanged. This is a
community quantization and is not affiliated with Agnes AI. The open-weight
Preview differs from the production/API model, and the upstream benchmark
figures below describe the Preview checkpoint before quantization.

---

**The original upstream model card follows, unchanged.**


<p align="center">
  <a href="https://agnes-ai.com/"><img src="https://img.shields.io/badge/Agnes_AI-Website-3248AF" alt="Agnes AI website"></a>
  <a href="#quickstart"><img src="https://img.shields.io/badge/Agnes--3.0--Flash_Preview-Open_Weights-3248AF" alt="Open weights"></a>
  <img src="https://img.shields.io/badge/License-Apache_2.0-111827" alt="Apache 2.0">
</p>



# Agnes-3.0-Flash Preview


## Model version clarification
This repository contains an earlier open-weight **Preview checkpoint** of Agnes 3.0 Flash. It is distinct from the newer **production/API checkpoint** listed on [Artificial Analysis](https://artificialanalysis.ai/models/agnes-3-0-flash).
The Preview release has **33B parameters** and a context window of **262,144 tokens**. The production/API model uses a different checkpoint and configuration, with a **1M-token context window**. Its benchmark results should not be attributed to the Preview weights released here.
This repository was initially published as `Agnes-3.0-Flash` without the `Preview` suffix. The model card now explicitly identifies this release as **Agnes-3.0-Flash Preview** to clarify the distinction between the open-weight release and the production/API model.
The specifications and Agnes benchmark results below refer to the **Preview checkpoint**.
Hello! 👋 Today we are introducing **Agnes-3.0-Flash Preview**, an **open-weights multimodal preview model** built for people who want flagship-class reasoning without flagship-class hardware.
Highlights:
- **Competitive across core capabilities.** Agnes-3.0-Flash Preview posts competitive results across reasoning, coding, and instruction-following evaluations.
- **Built for demanding work.** A **262 144-token context window**, adjustable reasoning effort, tool calling, and **text, image and video** understanding.


### Benchmarks
> **Benchmark scope:** The Agnes results in the chart and table below belong to the **Agnes-3.0-Flash Preview open-weight checkpoint** released in this repository. They are not results for the production/API Agnes 3.0 Flash model listed on Artificial Analysis.
<p align="center">
  <img style="width:100%;max-width:1100px" src="assets/benchmark-preview.png" alt="Agnes-3.0-Flash Preview benchmark reference results">
</p>


The Agnes-3.0-Flash Preview scores in the chart correspond to the open-weight checkpoint released in this repository.
Reference results across contemporary models are shown below. The figures were compiled from different sources, harnesses, and model snapshots and do not constitute a controlled head-to-head comparison.
<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;width:100%;margin:0 auto;padding:16px 0;overflow-x:auto">
<table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:11px;min-width:1180px">
<thead><tr>
<th style="width:15%;padding:9px 6px;text-align:left;border-bottom:2px solid #3248AF;color:#3248AF;font-size:11px">Benchmark</th>
<th style="width:10%;padding:9px 5px;text-align:center;font-weight:700;border-bottom:2px solid #3248AF;color:#3248AF;background:rgba(50,72,175,.09);font-size:10px;line-height:1.3">Agnes-3.0-Flash Preview</th>
<th style="width:8.33%;padding:9px 5px;text-align:center;border-bottom:2px solid #3248AF;font-size:10px;line-height:1.3">Qwen3.6-35B-A3B<br><span style="opacity:.65">35B / 3B active</span></th>
<th style="width:8.33%;padding:9px 5px;text-align:center;border-bottom:2px solid #3248AF;font-size:10px;line-height:1.3">Kimi K2.5<br><span style="opacity:.65">1T / 32B active</span></th>
<th style="width:8.33%;padding:9px 5px;text-align:center;border-bottom:2px solid #3248AF;font-size:10px;line-height:1.3">Muse Glimmer<br><span style="opacity:.65">30B</span></th>
<th style="width:8.33%;padding:9px 5px;text-align:center;border-bottom:2px solid #3248AF;font-size:10px;line-height:1.3">Qwen3.5<br><span style="opacity:.65">27B</span></th>
<th style="width:8.33%;padding:9px 5px;text-align:center;border-bottom:2px solid #3248AF;font-size:10px;line-height:1.3">DeepSeek V4 Flash 0731<br><span style="opacity:.65">284B / 13B active</span></th>
<th style="width:8.33%;padding:9px 5px;text-align:center;border-bottom:2px solid #3248AF;font-size:10px;line-height:1.3">Qwen3.8<br><span style="opacity:.65">27B</span></th>
<th style="width:8.33%;padding:9px 5px;text-align:center;border-bottom:2px solid #3248AF;font-size:10px;line-height:1.3">Gemini 3.5 Flash<br><span style="opacity:.65">undisclosed</span></th>
<th style="width:8.33%;padding:9px 5px;text-align:center;border-bottom:2px solid #3248AF;font-size:10px;line-height:1.3">Qwen3.8 Flash Next<br><span style="opacity:.65">125B / 6B active</span></th>
<th style="width:8.33%;padding:9px 5px;text-align:center;border-bottom:2px solid #3248AF;font-size:10px;line-height:1.3">MiniMax M3<br><span style="opacity:.65">428B / 23B active</span></th>
</tr></thead><tbody>
<tr><td style="padding:7px 6px">IFBench</td><td style="padding:7px 5px;text-align:center;font-weight:700;color:#3248AF;background:rgba(50,72,175,.05)">74.20</td><td style="padding:7px 5px;text-align:center">64.4</td><td style="padding:7px 5px;text-align:center">43.7</td><td style="padding:7px 5px;text-align:center">77.0</td><td style="padding:7px 5px;text-align:center">75.6</td><td style="padding:7px 5px;text-align:center">75.8</td><td style="padding:7px 5px;text-align:center">79.5</td><td style="padding:7px 5px;text-align:center">76.3</td><td style="padding:7px 5px;text-align:center">81.3</td><td style="padding:7px 5px;text-align:center">82.9</td></tr>
<tr><td style="padding:7px 6px">SciCode</td><td style="padding:7px 5px;text-align:center;font-weight:700;color:#3248AF;background:rgba(50,72,175,.05)">38.08</td><td style="padding:7px 5px;text-align:center">35.8</td><td style="padding:7px 5px;text-align:center">39.6</td><td style="padding:7px 5px;text-align:center">43.6</td><td style="padding:7px 5px;text-align:center">39.5</td><td style="padding:7px 5px;text-align:center">50.3</td><td style="padding:7px 5px;text-align:center">46.6</td><td style="padding:7px 5px;text-align:center">53.1</td><td style="padding:7px 5px;text-align:center">50.6</td><td style="padding:7px 5px;text-align:center">45.4</td></tr>
<tr><td style="padding:7px 6px">GPQA Diamond</td><td style="padding:7px 5px;text-align:center;font-weight:700;color:#3248AF;background:rgba(50,72,175,.05)">85.05</td><td style="padding:7px 5px;text-align:center">84.1</td><td style="padding:7px 5px;text-align:center">78.9</td><td style="padding:7px 5px;text-align:center">83.5</td><td style="padding:7px 5px;text-align:center">85.8</td><td style="padding:7px 5px;text-align:center">90.8</td><td style="padding:7px 5px;text-align:center">90.5</td><td style="padding:7px 5px;text-align:center">92.2</td><td style="padding:7px 5px;text-align:center">92.3</td><td style="padding:7px 5px;text-align:center">92.9</td></tr>
<tr><td style="padding:7px 6px">AA-LCR</td><td style="padding:7px 5px;text-align:center;font-weight:700;color:#3248AF;background:rgba(50,72,175,.05)">68.33</td><td style="padding:7px 5px;text-align:center">66.7</td><td style="padding:7px 5px;text-align:center">59.0</td><td style="padding:7px 5px;text-align:center">80.0</td><td style="padding:7px 5px;text-align:center">72.3</td><td style="padding:7px 5px;text-align:center">79.7</td><td style="padding:7px 5px;text-align:center">82.0</td><td style="padding:7px 5px;text-align:center">81.0</td><td style="padding:7px 5px;text-align:center">79.7</td><td style="padding:7px 5px;text-align:center">74.0</td></tr>
<tr><td style="padding:7px 6px">AA-Omniscience Accuracy</td><td style="padding:7px 5px;text-align:center;font-weight:700;color:#3248AF;background:rgba(50,72,175,.05)">23.00</td><td style="padding:7px 5px;text-align:center">18.8</td><td style="padding:7px 5px;text-align:center">22.9</td><td style="padding:7px 5px;text-align:center">27.0</td><td style="padding:7px 5px;text-align:center">20.7</td><td style="padding:7px 5px;text-align:center">40.4</td><td style="padding:7px 5px;text-align:center">18.4</td><td style="padding:7px 5px;text-align:center">51.4</td><td style="padding:7px 5px;text-align:center">24.5</td><td style="padding:7px 5px;text-align:center">16.7</td></tr>
</tbody></table></div>

<p style="font-size:11px;opacity:.72">
Higher is better for every row. Header parameter figures mix total and active counts, and harnesses and snapshot dates differ across sources, so treat cross-column comparisons as reference values rather than a controlled head-to-head evaluation.
</p>



## Architecture
Agnes-3.0-Flash Preview is a hybrid-attention decoder: three of every four layers run a gated delta rule (recurrent, with per-layer state independent of sequence length), and the fourth runs standard global attention. Only 18 of the 72 layers therefore hold a KV cache that grows with context.
| | |
|---|---|
| Context length | **262 144** tokens |
| Decoder layers | 72 = 54 delta-rule recurrent + 18 global attention, alternating 3 : 1 |
| Hidden size | 5120 |
| Global attention | 24 query heads / 4 KV heads (6 : 1 GQA), head dim 256; RMS-norm on q and k, sigmoid-gated output |
| Delta-rule layers | 16 key heads / 48 value heads, head dim 128; causal conv (kernel 4) in front, gated RMS-norm; recurrent state in fp32 |
| Feed-forward | SwiGLU, intermediate size 17408; plus a parallel SwiGLU 2048 branch in every layer |
| Positions | 3-axis rotary (text / height / width), interleaved mrope sections 11 : 11 : 10, base 1e7, applied to the first 25 % of each head dim (64 dims) |
| Vocabulary | 248 320 |
| Vision tower | 27 layers, hidden 1152, patch 16, 2 × 2 spatial merge, projected to 5120 |

## Quickstart
<div style="border-left:4px solid #3248AF;background:rgba(50,72,175,.08);border-radius:6px;padding:12px 16px;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;font-size:14px;line-height:1.6">
<div style="font-weight:700;color:#3248AF;margin-bottom:6px">REMOTE CODE REQUIRED</div>

<p style="margin:0"><b>Agnes-3.0-Flash Preview</b> ships its own model implementation. Always load it with <code>trust_remote_code=True</code>.</p>

</div>



### Requirements
```bash
pip install "transformers>=5.12" torch torchvision accelerate
```
Tested on transformers 5.12.1. Image and video inputs go through the bundled processor, which needs `torchvision`.

### Transformers
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
path = "Agnes-AI/Agnes-3.0-Flash"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(
    path, dtype="bfloat16", device_map="auto", trust_remote_code=True
)
msgs = [{"role": "user", "content": "请用三句话解释什么是人工智能。"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
```

### Images and video
Image and video inputs go through the bundled processor (also remote code):
```python
from transformers import AutoProcessor
proc = AutoProcessor.from_pretrained(path, trust_remote_code=True)
msgs = [{"role": "user", "content": [{"type": "image", "image": "photo.jpg"},
                                     {"type": "text", "text": "描述这张图。"}]}]
inputs = proc.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
                                  return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(proc.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
```

### Reasoning effort
The chat template exposes three reasoning levels — `high` (default), `medium`, `low` — plus a thinking-off switch:
```python
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt",
                              reasoning_effort="medium")   # or enable_thinking=False
```

### Tool calling
The chat template renders tool definitions for you. The model emits calls as `<tool_call><function=…><parameter=…>`, and you feed results back as a `tool` role message:
```python
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Look up current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string", "description": "City name"}},
            "required": ["city"],
        },
    },
}]
msgs = [{"role": "user", "content": "What's the weather in Beijing right now?"}]
ids = tok.apply_chat_template(msgs, tools=tools, add_generation_prompt=True,
                              return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256)
reply = tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)

# <tool_call>

# <function=get_weather>

# <parameter=city>

# Beijing

# </parameter>

# </function>

# </tool_call>

# run the tool, append the result, generate the final answer
msgs += [{"role": "assistant", "content": reply},
         {"role": "tool", "content": "Clear, 26°C, light northeasterly wind"}]
```
Over the OpenAI API pass `tools=` the same way. The server returns the text above verbatim by default; to get structured `tool_calls`, configure sglang with a tool-call parser matching this format (likewise a reasoning parser, if you want the thinking span in `reasoning_content`).

### SGLang
`serve.sh` starts a server from a stock public image, overlaying three files onto the image's `sglang` package and nothing else. See `sglang_patch/README.md`.
```bash
docker run --gpus all --shm-size 64g -p 30001:8080 \
    -v /path/to/agnes-3.0-flash:/model \
    lmsysorg/sglang:nightly-dev-20260908-20ca564b \
    bash /agnes-3.0-flash/serve.sh --served-model-name Agnes-3.0-Flash
```
`serve.sh` forwards extra command-line arguments to sglang, which is how `--served-model-name` takes effect; `--tp 2` works the same way. The server listens on port 8080 inside the container:
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:30001/v1")
response = client.chat.completions.create(
    model="Agnes-3.0-Flash",
    messages=[{"role": "user", "content": "Design a fault-tolerant event processing architecture."}],
    temperature=1.0,
    max_tokens=2000,
)
print(response.choices[0].message.content)
```
Pass `stream=True` for streaming; `tools=` and `reasoning_effort=` are accepted the same way.

## Hardware Requirements
| Resource | Recommendation |
|---|---|
| GPUs | 1 × NVIDIA H200 141 GB or NVIDIA H100 80 GB (or equivalent) at bf16 |
| Tensor parallel | `--tp 1`; `--tp 2` for maximum context and concurrency |
| Weights on disk | Approximately 66 GB for the bf16 checkpoint |
| Host memory | 128 GB or more recommended |

Actual context length and concurrency depend on KV-cache allocation, runtime overhead, and tensor-parallel configuration; validate the target workload on the intended hardware.

## Recommended Inference Settings
| Setting | Recommended |
|---|---|
| `temperature` | 1.0 |
| `top_p` | 0.95 |
| `top_k` | 20 |
| `reasoning_effort` | `high` for hard reasoning, `low` for latency-sensitive traffic |
| `max_tokens` | 2000 or higher |

These are the checkpoint's own `generation_config.json` defaults.

## Model Capabilities
| Capability | Support |
|---|---|
| Advanced reasoning | Yes, with `high` / `medium` / `low` effort levels |
| Coding and debugging | Yes |
| Long-context analysis | 262 144 tokens |
| Image understanding | Yes |
| Video understanding | Yes |
| Tool calling | Yes (`<tool_call>` / `<tool_response>`) |
| Streaming | Yes |
| OpenAI-compatible APIs | Chat Completions via sglang |

## License
Released under the [Apache License 2.0](LICENSE).

## Citation
```bibtex
@misc{agnes30flash2026,
  title        = {Agnes-3.0-Flash Preview},
  author       = {{Agnes AI}},
  year         = {2026},
  month        = sep,
  howpublished = {Open-weights preview checkpoint},
  url          = {https://agnes-ai.com/}
}
```