---
license: apache-2.0
base_model: Qwen/Qwen3-1.7B-Base
library_name: pytorch
tags:
  - decision-model
  - system-one
  - calibrated-probabilities
  - typed-decisions
  - jev-like
pipeline_tag: text-classification
datasets:
  - stanfordnlp/snli
  - nyu-mll/multi_nli
  - facebook/anli
  - google/boolq
  - cais/mmlu
  - allenai/ai2_arc
  - allenai/openbookqa
  - tau/commonsense_qa
  - allenai/sciq
  - allenai/qasc
  - jeggers/logiqa2_formatted
  - deepmind/aqua_rat
  - openlifescienceai/medmcqa
  - cua-ai/cua-s1-forms
  - dwidlee/systemone-lite-general
  - MagaBitmex/jev-4b-distill-data
---

# typical-small-preview

**Typical is an open System-One decision model family, built by OzLabs.**

Give it a state — a support ticket, a document, a form, a log line, any text — a question with its
rubric, and a label set defined at runtime, and it returns a calibrated probability over those
labels plus an explicit abstain option. One forward pass, no generated text to parse, about 45 ms
per decision on an H100. Every decision is one of three typed primitives: **Choice** (categorical),
**Score** (ordinal), **Noul** (yes/no).

`typical-small-preview` is **Release 0**: a frozen checkpoint (`nc_v3_tap20_wf`) that establishes
the architecture — native contextual readout, factored abstention, rubric-conditioned workflow
training. It is the reference point for [`typical-small`](https://huggingface.co/OzLabs/typical-small)
(Release 1, dedicated Score/Noul heads, 1,024-token states) and
[`typical-medium`](https://huggingface.co/OzLabs/typical-medium) (Release 1 at 4B). In this preview,
Score and Noul are trained as plain Choice, not as dedicated ordinal/Bernoulli heads.

## Why it's different

- **The decision is read directly from the model's own candidate-conditioned computation.** The
  state and candidates are encoded once; the answer is read off that computation instead of being
  sampled one token at a time.
- **The state is KV-cached**, so asking many questions about one state is cheap — each additional
  question is a short suffix against a cache that's already built.
- **Latency is flat up to ~64 options** (45 ms at K = 2, 46 ms at K = 32); 106 ms at K = 256 — the
  state dominates until the rendered option list does.
- **Abstention is a first-class output**, not a parsed refusal: every decision carries a calibrated
  P(∅) alongside the label probabilities.
- **Probabilities, not verbalized confidence.** No "I'm 80% sure" string to parse — an actual
  calibrated number.

## Model family

| | **`typical-small-preview`** | [`typical-small`](https://huggingface.co/OzLabs/typical-small) | [`typical-medium`](https://huggingface.co/OzLabs/typical-medium) |
|---|---|---|---|
| Backbone | Qwen3-1.7B-Base | Qwen3-1.7B-Base | Qwen3-4B-Base |
| Tap layer | 20 / 28 | 20 / 28 | 26 / 36 |
| Per-decision latency, K=2 (one H100) | 45 ms | 45 ms | 57 ms |
| Peak memory, K=2 | 6.4 GB | 6.4 GB | 14.6 GB |
| JevBench standard / easy / hard | .750 / 1.00 / .387 | .694 / 1.00 / .432 | .806 / 1.00 / .423 |
| MMLU-Pro among-K | .330 | .343 | .458 |
| Held-out workflow noul / score | .699 / .498 | .715 / .520 | .811 / .528 |
| Calibration (typed-decisions NLL) | 2.06 | 1.25 | 1.18 |

Full ablations and matched-baseline tables: REPORT.md §3t / §3w (this release).

## Results

| | `typical-small-preview` (`nc_v3_tap20_wf`) |
|---|---|
| SNLI / MNLI / BoolQ / ANLI | .894 / .849 / .831 / .481 |
| CLINC-150 / TREC-fine / HWU64 / 20NG | .734 / .372 / .735 / .522 |
| Held-out workflow: noul / score / style | .699 / .498 / .901 |
| JevBench standard / easy / hard (Brier std / hard) | .750 / 1.00 / .387 (.40 / .88) |

**JevBench standard, same 231 public ids, next to the open field:**

| **typical-small-preview** | jeff (GLiFormer 400M) | Laya (ModernBERT) | open-alternative-jev (Qwen3.5-4B) | system-one-open (Gemma E2B LoRA) |
|---|---|---|---|---|
| **.750** | .750 | .694 | .833 | .931 |

Public-subset per-item comparison on the 231 public ids; not a ranked leaderboard entry.

## Quickstart

```bash
pip install typical-ai
```

```python
from typical_ai import Typical

m = Typical.from_pretrained("OzLabs/typical-small-preview", device="auto")

ticket = ("Customer reports repeated 500 errors on checkout after the last deploy; "
          "asking for a refund.")
m.choice(ticket, "Which team should handle this?", ["billing", "infra", "support", "security"])
m.score(ticket, "How severe is this ticket?", ["1", "2", "3", "4"])
m.noul(ticket, "Does this ticket violate the refund-abuse policy (3+ refunds in 30 days)?")
```

Verified: this exact sequence (download, install, run) works on CPU (checked against `typical-small`'s
identical inference package).

`choice`/`score` return `{label: probability, ...}` plus `p_null`; `noul` returns `P(yes)` (routed
through the K-way path in this preview — see Limitations). `decide` mirrors the internal
`PCDMDecider.decide` exactly, including the runtime block (latency, `p_null`, truncation flags) —
see `inference/README.md` for the full API and `inference/example.py` for a runnable script.

## Training recipe

- **Data mix** E .35 / K .25 / W .40 of every batch (evidence, knowledge-MCQ, rubric-conditioned
  workflow) — no soft-target uncertainty corpus and no null augmentation in this release.
- **256-token decision states.**
- **Unified Choice head:** Score is trained as a K-way categorical Choice; Noul is a 2-way Choice
  over `["no","yes"]` — neither has a dedicated typed head yet (see `typical-small`).
- **12,000 steps, effective batch 64** (`bs 64`), LoRA rank 16 on the top 8 layers of a
  20-of-28-layer truncated backbone.
- **Held-out by construction:** one rubric style per workflow family is held out of train/val
  entirely, to measure rubric dependence rather than pattern-matching a fixed decision.

**Data sources and licenses:**

| Source | Role | License |
|---|---|---|
| SNLI, MultiNLI, ANLI, BoolQ, MMLU-aux, ARC, OpenBookQA, CommonsenseQA, SciQ, QASC, LogiQA2, AQuA-RAT, MedMCQA | evidence + knowledge-MCQ | permissive research licenses (CC-BY-SA / MIT / Apache-2.0 family) |
| `cua-ai/cua-s1-forms` | workflow (GUI-form synth) | MIT |
| `dwidlee/systemone-lite-general` | workflow (rule-labeled typed decisions) | MIT |
| `MagaBitmex/jev-4b-distill-data` | workflow (programmatic gold) | Apache-2.0 |
| in-repo rubric-conditioned generator (`data_wf`) | workflow | generated, no external license |

Evaluation-only, never trained on: JevBench (public subset), `typed-decisions`, PagerDuty, tree-choice,
jevlogs (Loghub research license, caveated), Mind2Web.

## Limitations

- Hard tier is .387 (chance .336, SE ≈ .045 at n = 111) — the least reliable tier; the open field's
  best is .73. Long policy states are the weakest slice (≤ 256 tokens .64 vs. > 1,024 tokens .23).
- Score and Noul are trained as plain Choice, not dedicated ordinal/Bernoulli heads — order
  sensitivity on Noul reaches Δp ≈ .12–.55 under label reversal (fixed in `typical-small`).
- Workflow-mix training induces over-abstention on CLINC-150 and TREC-fine, a diagnosed
  null-weighting/sampling interaction, not a discrimination loss.
- Very large candidate sets (K in the hundreds to thousands) need the energy → top-r → native path,
  not this native head directly.
- **Put the case facts AFTER the policy text on long documents.** This checkpoint was trained on a
  corpus whose long states rendered the case last, and it inherited that ordering. Measured on 605
  held-out long policy states (identical items, only the position of the case block differs, no
  truncation at scoring time): **0.744** with the facts last versus **0.534** with the facts first —
  a **21-point** drop. On the yes/no (Noul) subset the facts-first score is 0.430, *below* the 0.612
  majority-class floor. Short states are unaffected. Retrained checkpoints without this bias exist
  (`ts1c` at 1.7B, `tm2` at 4B) and will supersede this line.
- Inference code is included in this repo; training code release to follow.

## License

Apache-2.0 (backbone `Qwen/Qwen3-1.7B-Base` is Apache-2.0; this release's weights are a LoRA adapter
plus a small decision head over that backbone). Training-data licenses above; JevBench numbers are a
public-subset run against `fstandhartinger/jevbench` v1.2.1, not a submitted or ranked leaderboard
entry.

## Files in this repository

- `best.pt` — this checkpoint (`nc_v3_tap20_wf`).
- `results.json` / `eval_wf.json` / `probe_results.json` / `jevbench_summary.json` — full eval
  suite, post-hoc held-out/external eval, probe results, JevBench run summary.
- `baseline_nc_v3_tap20/` — the matched pre-workflow baseline, same five files.
- `inference/` — the self-contained `Typical` inference package (see Quickstart above).
- `MANIFEST.json` — checksum + provenance record for this release.

## Lineage

**`typical-small-preview`** (Release 0, this repo) →
[`typical-small`](https://huggingface.co/OzLabs/typical-small) (Release 1, dedicated typed heads) →
[`typical-medium`](https://huggingface.co/OzLabs/typical-medium) (Release 1 at 4B).

Coming: `typical-large` — a 14B ladder checkpoint scores JevBench .875/.468 (standard/hard), but
probability calibration at that size is not yet fixed (§3ab). Trained, not released.

Typical is built by [OzLabs](https://huggingface.co/OzLabs).