Eleri-1.5B

Eleri is a 1.5B-parameter model fine-tuned to do one job: given an AI agent's payment and the mandate it was authorized under, say whether the payment matches — and if not, exactly which of 15 specific failure modes it hit (overspend, wrong vendor, expired mandate, a lookalike wallet address, and more).

It's a LoRA adapter on top of Qwen/Qwen2.5-1.5B-Instruct, trained via QLoRA (against the 4-bit unsloth/qwen2.5-1.5b-instruct-unsloth-bnb-4bit quantization for training efficiency) and evaluated at full precision. It's small enough to run cheaply and fast enough to audit a real transaction stream, which was the actual design goal — not squeezing out marginal accuracy on a leaderboard, but being cheap enough to run on every single payment an agent makes.

Results

Evaluated on EleriBench's locked, 1,500-example test set, against two frontier few-shot baselines on the identical prompt and output schema:

model	verdict acc	anomaly macro-F1	ECE (10-bin)	cost / 1k audits
eleri-1.5b	0.972	0.989	0.050	$0.04–0.62 *
claude-haiku-4-5 (few-shot)	0.808	0.684	0.128	$5.26
gpt-4o-mini (few-shot)	0.682	0.392	0.240	$0.60

* $0.62/1k is the training-time evaluation harness's naive per-call GPU-time estimate; $0.04–0.08/1k is measured cost in a production serving setup (batched, constrained decoding via vLLM). Full per-anomaly breakdown and methodology in the EleriBench repo.

How to use

vLLM (recommended — what production serving actually uses)

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest
from vllm.sampling_params import StructuredOutputsParams

llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct", enable_lora=True, max_lora_rank=16)
lora_request = LoRARequest("eleri", 1, "<path to this adapter>")

# See EleriBench's schemas/python/prompt.py and output_schema.py for the
# exact system prompt and JSON schema this model was trained against —
# both matter for getting these results; a different prompt/schema will
# give different (likely worse) results.
outputs = llm.generate(
    prompts,
    SamplingParams(structured_outputs=StructuredOutputsParams(json=ELERI_VERDICT_JSON_SCHEMA)),
    lora_request=lora_request,
)

Transformers + PEFT

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "theakomolafe/eleri-1.5b")
tokenizer = AutoTokenizer.from_pretrained("theakomolafe/eleri-1.5b")

Without the exact system prompt and constrained-JSON output schema from EleriBench, you'll get free-text generations, not the structured verdicts this model was tuned to produce — grab schemas/python/prompt.py and schemas/python/output_schema.py from that repo before using this adapter for anything.

Training data

30,000 synthetic, label-first-generated examples (TransactionRecord → EleriVerdict pairs) across 16 spend categories, 15 anomaly types, and 3 verdicts, covering x402, AP2, ACP, and Stripe-shaped payment records. Full generation methodology, the label-quality bugs found and fixed during this model's own development, and the exact test-set hash are documented in EleriBench's datagen/.

Intended use and limitations

Built for one narrow task: verifying whether a specific payment matches a specific stated authorization. It is not a general chat model, not a fraud model for arbitrary transactions without a mandate to check against, and its outputs should inform a review process, not autonomously approve, block, or move money — that's the boundary Sawa (the product this model was built for) enforces in practice: Eleri audits after the fact and flags what looks off; it never has the ability to act on a payment itself.

The 30,000-example training set is synthetic. It's realistic by construction (protocol-accurate field shapes, a curated vendor/purpose pool per category, label-first generation so every ground truth is exact rather than inferred) but it is not real transaction data, and real-world distribution shift is untested beyond EleriBench's own held-out split.

License

Apache 2.0, same as the base model (Qwen2.5-1.5B-Instruct).

Framework versions

• PEFT 0.20.0