ait-hf/certus-jev-like-v0007

Certus v0007 — a Jev-like "System One" decision model

Try it: ait-hf/certus-jev-like-playground (free, no account).

Certus is a self-hosted decision model in the spirit of TypeSafe's Jev / System One models. It does not generate text: it reads a state (any text or JSON) plus typed questions and returns calibrated probability distributions over answer sets the caller defines — a choice among named options, a score on an ordered rubric, or a yes/no. Every answer is the soft-max over the logits of the option letters at the assistant turn, so the output is always one of your options and the cost is one forward pass, no decoding.

What is in this repo

registry.json                 version registry (lineage, tasks, calibration temperatures, metrics)
v0007/                        trunk: LoRA r=16 adapter on Qwen2.5-1.5B-Instruct, 61 public tasks, 1 epoch
v0007-<domain>/               domain adapters (LoRA, parents = [v0007]); applied unmerged on top of the merged trunk
family_v0007/family.json      domain -> adapter map, routing threshold, max stack
family_v0007/heads/           router: one logistic head per adapter on the trunk's last hidden state + normalisation
serve/                        the API server + playground page (`python app.py`, see below)

version	domain	trained on	LoRA r	val acc (mean over its tasks)
v0007	trunk	ag_news, banking77, boolq, clinc150, commonsense_qa, copa, dbpedia, emotion, facts, fits, imdb, jailbreak, massive_intent, mnli, mrpc, openbookqa, paws, qnli, read, rte, sciq, sst2, swag, tweet_emoji, tweet_hate, tweet_irony, tweet_offensive, tweet_sentiment, yahoo, yelp, anli, winogrande, hellaswag, race, scitail, qqp, stsb, toxic, stance_abortion, stance_atheism, stance_feminist, stance_hillary, match, goemo_soft, reason, formality, politeness, strategyqa, vitaminc, ruletaker, proofwriter, folio, logiqa, tracie, temporal_nli, piqa, siqa, clutrr, gsm8k, svamp, aqua	16	0.792
v0007-reason	reason	traps, reason, skills	16	0.943
v0007-math	math	math, gsm8k, svamp, aqua	32	0.690
v0007-language	language	language	8	1.000
v0007-support	support	banking77, clinc150, massive_intent	8	–
v0007-sentiment	sentiment	sst2, sst5, yelp, tweet_sentiment, emotion, goemo_soft, tweet_irony, imdb, formality, politeness, sarcasm	8	0.776
v0007-safety	safety	jailbreak, toxic, tweet_hate, tweet_offensive	8	0.844
v0007-spam	spam	spam	8	0.990
v0007-typed	workflow	typed_decisions	16	0.850

Serving: the trunk adapter is merged into the base model (base speed); the domain adapters stay separate and are switched per request. model: "auto" runs the router on the full request (every question with its type and options, then the state), applies every adapter whose head fires above 0.5 (at most two, stacked), and answers with the trunk when none does. Adding an adapter later means adding one folder and one head file — no retraining.

Results (v0007, hand-written held-out sets, never trained on)

set	trunk v0007	auto (router + adapters)
196-question test bench, 20 categories	0.842	0.949
typed decisions (Laya benchmark, 1 200-case format)	–	0.796 (Jev 0.727, Laya 0.766)
10 hard yes/no logic traps (noul10)	4/10	9/10 (Jev 10/10)
15-task unseen public suite, mean acc / ECE	0.736 / 0.074	–

Calibration: temperature scaling per option-count bucket (config.json); the trunk's ECE on its validation set is 0.010 before scaling.

Run it yourself (weights + API server in this repo)

pip install "huggingface_hub<1.0"
hf download ait-hf/certus-jev-like-v0007 --local-dir certus      # 442 MB: adapters, router heads, registry, serve/
cd certus/serve
pip install -r requirements.txt                                  # GPU: install a CUDA build of torch first
python app.py                                                    # first start downloads Qwen2.5-1.5B-Instruct (3.1 GB)

That starts the same server as the Space on http://localhost:7860: the playground page at /, the API below and GET /health. Environment: CERTUS_DEVICE=cpu to run without a GPU (fp32, ~8 GB RAM), PORT, and CERTUS_MAX_QUESTIONS / CERTUS_MAX_STATE_CHARS / CERTUS_MAX_OPTIONS to lift the demo limits. The server is a plain FastAPI app (serve/app.py) on top of transformers + peft; the whole model family sits in ~3.5 GB of GPU memory.

API — Jev-compatible JSON

POST /v1/systemone takes the same body as TypeSafe's Jev: a state (string or JSON), a dict of named questions, each with a type (choice, score, noul), free-text instructions and, for choice / score, the criteria (named options with optional descriptions, or an ordered list of levels). model is auto (router), v0007 (trunk) or one adapter such as v0007-math; GET /v1/models lists them.

curl -X POST http://localhost:7860/v1/systemone -H "content-type: application/json" -d '{
  "model": "auto",
  "state": "Help! My payouts have been failing for 3 days and nobody answers the phone.",
  "questions": {
    "team":   {"type": "choice", "instructions": "Which team should handle this?",
               "criteria": {"billing": "payments, payouts, invoices", "technical": "bugs, outages", "sales": "pricing, plans"}},
    "urgent": {"type": "noul",   "instructions": "Does the message convey urgency?"},
    "mood":   {"type": "score",  "instructions": "How frustrated is the customer?", "criteria": ["calm", "annoyed", "angry"]}
  }
}'

Response (real output of this model):

{
  "model": "v0007",
  "answers": {
    "team": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {
        "billing": 0.7145,
        "technical": 0.2762,
        "sales": 0.0093
      },
      "confidence": 0.7145
    },
    "urgent": {
      "type": "noul",
      "noul": 0.9201
    },
    "mood": {
      "type": "score",
      "score": 1.7707,
      "legend": {
        "0": "calm",
        "1": "annoyed",
        "2": "angry"
      },
      "probabilities": {
        "0": 0.0124,
        "1": 0.2044,
        "2": 0.7831
      },
      "confidence": 0.551
    }
  },
  "usage": {
    "input_tokens": 686,
    "output_tokens": 0
  },
  "reasoning": {
    "_route": "general"
  }
}

• choice → the chosen option, a probability per option and confidence (= the top probability);
• score → the expected level as a number (1.77 sits between annoyed and angry), the legend, a probability per level;
• noul → P(yes);
• model → which adapter answered; reasoning._route → what the router decided (general = the trunk);
• usage.output_tokens is always 0: nothing is generated.

Python:

import requests
r = requests.post("http://localhost:7860/v1/systemone", json={
    "model": "auto",
    "state": {"subject": "Invoice 4471 overdue", "body": "Third reminder. Pay within 5 days or we suspend the account."},
    "questions": {"tone": {"type": "score", "instructions": "How threatening is the message?",
                           "criteria": ["neutral", "firm", "threatening"]}}})
print(r.json()["answers"]["tone"]["score"])

The same calls work against the Space (https://ait-hf-certus-jev-like-playground.hf.space/v1/systemone; add Authorization: Bearer <your HF token> so the GPU time is charged to your own ZeroGPU quota) and, wire for wire, against Jev.

How an answer is produced: the request is rendered as one prompt with lettered options; the model's logits for the option letters at the assistant turn are soft-maxed with a temperature calibrated per option count. One forward pass per question, all questions of a request batched; more than 26 options go through a two-stage pass (chunks of 25 plus "none of the above", then the top candidates).

Speed and hardware requirements

The whole point of a small decision model: one forward pass per request, no decoding, and it runs on ordinary hardware. Measured on our development box (RTX 2080 Ti, a 2018 consumer GPU; Intel i9-9900KF with 4 threads for the CPU rows):

setting	latency per request (1–3 short questions)	memory	notes
GPU, trunk only (v0007, LoRA merged)	37–52 ms	3.5 GB VRAM	fp16; no bf16 or flash-attention needed (works on Turing)
GPU, one domain adapter by name (v0007-spam, …)	61–73 ms	+8–140 MB per adapter	adapters stay unmerged on top of the merged trunk; the whole family fits in ~3.5 GB
GPU, auto (router + adapter, two passes)	113–200 ms	same	a second stacked adapter adds ~10–20 ms
GPU, 77-option choice (two-stage path)	~50–95 ms	same	> 26 options are scored in chunks of 25 + none-of-the-above
CPU, fp32, 4 threads, trunk	0.8 s (short text) / 3.2 s (~400-token text)	6.8 GB RAM	no GPU at all; cost scales with text length

Minimum hardware: any CUDA GPU with 6 GB (4 GB works for the trunk alone) for real-time use, or a CPU with 8 GB of free RAM for batch/offline use. Disk: 3.1 GB for the base model (downloaded once from Qwen/Qwen2.5-1.5B-Instruct) plus 442 MB for everything in this repo. Throughput scales with batching: all questions of one request are one batch. The Space runs on ZeroGPU, where the model is materialised on a shared GPU per request.

Train an adapter for your own use case

The family is built to be specialised without touching the trunk. A new domain = one small LoRA adapter trained on top of v0007 + one router head, and everything else keeps working exactly as before:

• Data: labelled examples in the same shape as the API — a state, a question, the options and the gold answer (soft labels allowed). A CSV of text,label is enough for a classifier. Our spam adapter in the playground was built from 600 labelled emails; the sentiment adapter from ~2 000 examples per source task; the workflow adapter from the 1 200 Laya training cases (5 400 decisions).
• Cost: LoRA r=8 on the attention projections (r=16–32 with the MLPs for reasoning-heavy domains), 1–2 epochs, a KL anchor to the trunk so the general behaviour is kept. A few hundred examples train in minutes on one consumer GPU (the 5 400-decision workflow adapter took 10 minutes on an RTX 2080 Ti; 22 000 examples ≈ 20 minutes). The result is an 8–140 MB file, not a new model.
• Routing: the router is a set of independent per-adapter heads on the trunk's hidden state, so adding your adapter means fitting one more head (about a minute) — no retraining of the router or the other adapters. Requests that do not look like your domain still go to the trunk or the other adapters.
• Serving: adapters stay unmerged on top of the merged trunk, so a whole family of them fits in ~3.5 GB of GPU memory and switching per request costs ~10–20 ms. Call your adapter by name ("model": "v0007-yourdomain") or let auto pick it.

This is the mechanism behind every adapter in this repo (tools/train_family.py, tools/router_heads.py add in the Certus toolkit; the Space ships the inference code only). Measured effect of such an adapter on held-out data: spam detection 3/6 → 6/6 on our bench, typed workflow decisions 0.526 → 0.796, arithmetic and unit questions 3/8 → 7/8. Open a discussion on this repo if you want an adapter built for your domain.

Training data

Trunk: 61 public datasets (sentiment, topic, NLI, boolean QA, intents, commonsense/science QA, logic and word problems, safety) at ≤ 3 000 examples each, plus synthetic reasoning probes, with LoRA r=16, a proper-scoring (NLL + Brier) loss, yes/no balancing and a KL anchor to the base model. Adapters: see the table above. No benchmark item in the results section was used for training. Datasets with non-commercial licences were kept out of this release.

Limits

• A 1.5B single-pass reader: pure computation (prices × quantities, unit conversions) and multi-step logic traps are hit-and-miss even with the math and reasoning adapters.
• States are truncated to 700 tokens.
• English first; a multilingual adapter is planned for the next version.