---
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: peft
pipeline_tag: text-classification
language:
  - en
tags:
  - decision-model
  - system-one
  - lora
  - calibrated
  - classification
  - routing
  - jev
  - jev-like
  - typesafe
---

# Certus v0007 — a Jev-like "System One" decision model

Try it: **[ait-hf/certus-jev-like-playground](https://huggingface.co/spaces/ait-hf/certus-jev-like-playground)** (free, no account).

Certus is a self-hosted decision model in the spirit of TypeSafe's Jev / System One models. It does **not generate
text**: it reads a `state` (any text or JSON) plus typed questions and returns **calibrated probability
distributions** over answer sets the caller defines — a choice among named options, a score on an ordered rubric,
or a yes/no. Every answer is the soft-max over the logits of the option letters at the assistant turn, so the output
is always one of your options and the cost is one forward pass, no decoding.

## What is in this repo

```
registry.json                 version registry (lineage, tasks, calibration temperatures, metrics)
v0007/                        trunk: LoRA r=16 adapter on Qwen2.5-1.5B-Instruct, 61 public tasks, 1 epoch
v0007-<domain>/               domain adapters (LoRA, parents = [v0007]); applied unmerged on top of the merged trunk
family_v0007/family.json      domain -> adapter map, routing threshold, max stack
family_v0007/heads/           router: one logistic head per adapter on the trunk's last hidden state + normalisation
serve/                        the API server + playground page (`python app.py`, see below)
```

| version | domain | trained on | LoRA r | val acc (mean over its tasks) |
|---|---|---|---:|---:|
| `v0007` | trunk | ag_news, banking77, boolq, clinc150, commonsense_qa, copa, dbpedia, emotion, facts, fits, imdb, jailbreak, massive_intent, mnli, mrpc, openbookqa, paws, qnli, read, rte, sciq, sst2, swag, tweet_emoji, tweet_hate, tweet_irony, tweet_offensive, tweet_sentiment, yahoo, yelp, anli, winogrande, hellaswag, race, scitail, qqp, stsb, toxic, stance_abortion, stance_atheism, stance_feminist, stance_hillary, match, goemo_soft, reason, formality, politeness, strategyqa, vitaminc, ruletaker, proofwriter, folio, logiqa, tracie, temporal_nli, piqa, siqa, clutrr, gsm8k, svamp, aqua | 16 | 0.792 |
| `v0007-reason` | reason | traps, reason, skills | 16 | 0.943 |
| `v0007-math` | math | math, gsm8k, svamp, aqua | 32 | 0.690 |
| `v0007-language` | language | language | 8 | 1.000 |
| `v0007-support` | support | banking77, clinc150, massive_intent | 8 | – |
| `v0007-sentiment` | sentiment | sst2, sst5, yelp, tweet_sentiment, emotion, goemo_soft, tweet_irony, imdb, formality, politeness, sarcasm | 8 | 0.776 |
| `v0007-safety` | safety | jailbreak, toxic, tweet_hate, tweet_offensive | 8 | 0.844 |
| `v0007-spam` | spam | spam | 8 | 0.990 |
| `v0007-typed` | workflow | typed_decisions | 16 | 0.850 |

Serving: the trunk adapter is merged into the base model (base speed); the domain adapters stay separate and are
switched per request. `model: "auto"` runs the router on the **full request** (every question with its type and
options, then the state), applies every adapter whose head fires above 0.5 (at most two, stacked), and answers with
the trunk when none does. Adding an adapter later means adding one folder and one head file — no retraining.

## Results (v0007, hand-written held-out sets, never trained on)

| set | trunk `v0007` | `auto` (router + adapters) |
|---|---:|---:|
| 196-question test bench, 20 categories | 0.842 | **0.949** |
| typed decisions (Laya benchmark, 1 200-case format) | – | 0.796 (Jev 0.727, Laya 0.766) |
| 10 hard yes/no logic traps (noul10) | 4/10 | 9/10 (Jev 10/10) |
| 15-task unseen public suite, mean acc / ECE | 0.736 / 0.074 | – |

Calibration: temperature scaling per option-count bucket (`config.json`); the trunk's ECE on its validation set is
0.010 before scaling.

## Run it yourself (weights + API server in this repo)

```bash
pip install "huggingface_hub<1.0"
hf download ait-hf/certus-jev-like-v0007 --local-dir certus      # 442 MB: adapters, router heads, registry, serve/
cd certus/serve
pip install -r requirements.txt                                  # GPU: install a CUDA build of torch first
python app.py                                                    # first start downloads Qwen2.5-1.5B-Instruct (3.1 GB)
```

That starts the same server as the Space on http://localhost:7860: the playground page at `/`, the API below and
`GET /health`. Environment: `CERTUS_DEVICE=cpu` to run without a GPU (fp32, ~8 GB RAM), `PORT`, and
`CERTUS_MAX_QUESTIONS` / `CERTUS_MAX_STATE_CHARS` / `CERTUS_MAX_OPTIONS` to lift the demo limits. The server is a plain
FastAPI app (`serve/app.py`) on top of `transformers` + `peft`; the whole model family sits in ~3.5 GB of GPU memory.

## API — Jev-compatible JSON

`POST /v1/systemone` takes the same body as TypeSafe's Jev: a `state` (string or JSON), a dict of named `questions`,
each with a `type` (`choice`, `score`, `noul`), free-text `instructions` and, for `choice` / `score`, the `criteria`
(named options with optional descriptions, or an ordered list of levels). `model` is `auto` (router), `v0007` (trunk)
or one adapter such as `v0007-math`; `GET /v1/models` lists them.

```bash
curl -X POST http://localhost:7860/v1/systemone -H "content-type: application/json" -d '{
  "model": "auto",
  "state": "Help! My payouts have been failing for 3 days and nobody answers the phone.",
  "questions": {
    "team":   {"type": "choice", "instructions": "Which team should handle this?",
               "criteria": {"billing": "payments, payouts, invoices", "technical": "bugs, outages", "sales": "pricing, plans"}},
    "urgent": {"type": "noul",   "instructions": "Does the message convey urgency?"},
    "mood":   {"type": "score",  "instructions": "How frustrated is the customer?", "criteria": ["calm", "annoyed", "angry"]}
  }
}'
```

Response (real output of this model):

```json
{
  "model": "v0007",
  "answers": {
    "team": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {
        "billing": 0.7145,
        "technical": 0.2762,
        "sales": 0.0093
      },
      "confidence": 0.7145
    },
    "urgent": {
      "type": "noul",
      "noul": 0.9201
    },
    "mood": {
      "type": "score",
      "score": 1.7707,
      "legend": {
        "0": "calm",
        "1": "annoyed",
        "2": "angry"
      },
      "probabilities": {
        "0": 0.0124,
        "1": 0.2044,
        "2": 0.7831
      },
      "confidence": 0.551
    }
  },
  "usage": {
    "input_tokens": 686,
    "output_tokens": 0
  },
  "reasoning": {
    "_route": "general"
  }
}
```

- `choice` → the chosen option, a probability per option and `confidence` (= the top probability);
- `score` → the expected level as a number (1.77 sits between *annoyed* and *angry*), the legend, a probability per level;
- `noul` → `P(yes)`;
- `model` → which adapter answered; `reasoning._route` → what the router decided (`general` = the trunk);
- `usage.output_tokens` is always 0: nothing is generated.

Python:

```python
import requests
r = requests.post("http://localhost:7860/v1/systemone", json={
    "model": "auto",
    "state": {"subject": "Invoice 4471 overdue", "body": "Third reminder. Pay within 5 days or we suspend the account."},
    "questions": {"tone": {"type": "score", "instructions": "How threatening is the message?",
                           "criteria": ["neutral", "firm", "threatening"]}}})
print(r.json()["answers"]["tone"]["score"])
```

The same calls work against the Space (`https://ait-hf-certus-jev-like-playground.hf.space/v1/systemone`; add
`Authorization: Bearer <your HF token>` so the GPU time is charged to your own ZeroGPU quota) and, wire for wire,
against Jev.

How an answer is produced: the request is rendered as one prompt with lettered options; the model's logits for the
option letters at the assistant turn are soft-maxed with a temperature calibrated per option count. One forward pass per
question, all questions of a request batched; more than 26 options go through a two-stage pass (chunks of 25 plus
"none of the above", then the top candidates).

## Speed and hardware requirements

The whole point of a small decision model: one forward pass per request, no decoding, and it runs on ordinary hardware.
Measured on our development box (RTX 2080 Ti, a 2018 consumer GPU; Intel i9-9900KF with 4 threads for the CPU rows):

| setting | latency per request (1–3 short questions) | memory | notes |
|---|---:|---:|---|
| GPU, trunk only (`v0007`, LoRA merged) | **37–52 ms** | 3.5 GB VRAM | fp16; no bf16 or flash-attention needed (works on Turing) |
| GPU, one domain adapter by name (`v0007-spam`, …) | 61–73 ms | +8–140 MB per adapter | adapters stay unmerged on top of the merged trunk; the whole family fits in ~3.5 GB |
| GPU, `auto` (router + adapter, two passes) | 113–200 ms | same | a second stacked adapter adds ~10–20 ms |
| GPU, 77-option choice (two-stage path) | ~50–95 ms | same | > 26 options are scored in chunks of 25 + none-of-the-above |
| CPU, fp32, 4 threads, trunk | 0.8 s (short text) / 3.2 s (~400-token text) | 6.8 GB RAM | no GPU at all; cost scales with text length |

Minimum hardware: **any CUDA GPU with 6 GB** (4 GB works for the trunk alone) for real-time use, or **a CPU with 8 GB
of free RAM** for batch/offline use. Disk: 3.1 GB for the base model (downloaded once from `Qwen/Qwen2.5-1.5B-Instruct`)
plus 442 MB for everything in this repo. Throughput scales with batching: all questions of one request are one batch.
The [Space](https://huggingface.co/spaces/ait-hf/certus-jev-like-playground) runs on ZeroGPU, where the model is
materialised on a shared GPU per request.

## Train an adapter for your own use case

The family is built to be specialised without touching the trunk. A new domain = one small LoRA adapter trained on top
of `v0007` + one router head, and everything else keeps working exactly as before:

- **Data**: labelled examples in the same shape as the API — a `state`, a question, the options and the gold answer
  (soft labels allowed). A CSV of `text,label` is enough for a classifier. Our spam adapter in the playground was built
  from **600 labelled emails**; the sentiment adapter from ~2 000 examples per source task; the workflow adapter from
  the 1 200 Laya training cases (5 400 decisions).
- **Cost**: LoRA r=8 on the attention projections (r=16–32 with the MLPs for reasoning-heavy domains), 1–2 epochs,
  a KL anchor to the trunk so the general behaviour is kept. A few hundred examples train in **minutes** on one
  consumer GPU (the 5 400-decision workflow adapter took 10 minutes on an RTX 2080 Ti; 22 000 examples ≈ 20 minutes).
  The result is an **8–140 MB** file, not a new model.
- **Routing**: the router is a set of independent per-adapter heads on the trunk's hidden state, so adding your adapter
  means fitting one more head (about a minute) — no retraining of the router or the other adapters. Requests that
  do not look like your domain still go to the trunk or the other adapters.
- **Serving**: adapters stay unmerged on top of the merged trunk, so a whole family of them fits in ~3.5 GB of GPU
  memory and switching per request costs ~10–20 ms. Call your adapter by name (`"model": "v0007-yourdomain"`) or let
  `auto` pick it.

This is the mechanism behind every adapter in this repo (`tools/train_family.py`, `tools/router_heads.py add` in the
Certus toolkit; the Space ships the inference code only). Measured effect of such an adapter on held-out data: spam
detection 3/6 → 6/6 on our bench, typed workflow decisions 0.526 → 0.796, arithmetic and unit questions 3/8 → 7/8.
Open a discussion on this repo if you want an adapter built for your domain.

## Training data

Trunk: 61 public datasets (sentiment, topic, NLI, boolean QA, intents, commonsense/science QA, logic and word
problems, safety) at ≤ 3 000 examples each, plus synthetic reasoning probes, with LoRA r=16, a proper-scoring
(NLL + Brier) loss, yes/no balancing and a KL anchor to the base model. Adapters: see the table above. No benchmark
item in the results section was used for training. Datasets with non-commercial licences were kept out of this release.

## Limits

- A 1.5B single-pass reader: pure computation (prices × quantities, unit conversions) and multi-step logic traps are
  hit-and-miss even with the math and reasoning adapters.
- States are truncated to 700 tokens.
- English first; a multilingual adapter is planned for the next version.