---
license: cc-by-nc-4.0
base_model: Qwen/Qwen3.5-4B-Base
pipeline_tag: text-classification
library_name: transformers.js
tags:
- qwen3_5
- system-one
- onnx
- transformers.js
---

> This is a Transformers.js-ready ONNX conversion of the original Hugging Face model [pngwn/system-one-qwen3.5-4b-scorer](https://huggingface.co/pngwn/system-one-qwen3.5-4b-scorer).
> The original model card content follows below.

## Usage with Transformers.js

This repository contains the merged model (LoRA adapter and `score` head folded
into the Qwen3.5-4B-Base text backbone). Variants under `onnx/`: `q4f16`
(default, 2.4 GB) and `q4` (2.6 GB). Each (state, question, option) triple is
one right-padded sequence; the graph returns one logit per sequence
(`logits[i, 0]`), read at its last non-pad token. Divide a question's option
logits by the temperature (1.75) and softmax them.

Transformers.js has no sequence-classification class for `qwen3_5_text` yet,
so load the graph with the base `PreTrainedModel` (its encoder-only fallback
matches this graph; the one-time "assuming encoder-only architecture" warning
is expected).

```js
import { AutoTokenizer, PreTrainedModel, Tensor } from "@huggingface/transformers";

const repo = "onnx-community/system-one-qwen3.5-4b-scorer-ONNX";
const tokenizer = await AutoTokenizer.from_pretrained(repo);
const model = await PreTrainedModel.from_pretrained(repo, { dtype: "q4f16", device: "webgpu" });

const state = "I was charged twice for the same order and nobody answers my emails.";
const question = "Which team should handle this ticket?";
const options = ["billing", "technical support", "sales"];

const encode = (text) => Array.from(tokenizer(text, { add_special_tokens: false }).input_ids.data, Number);
const sequences = options.map((option) => {
  const tail = encode(`\n\nQuestion:\n${question}\n\nOption:\n${option}`);
  if (tail.length >= 384) return tail.slice(-384);
  const head = encode(`State:\n${state}`);
  return head.slice(0, 384 - tail.length).concat(tail);
});
const length = Math.max(...sequences.map((s) => s.length));
const pad = 248044n; // <|endoftext|>
const input_ids = new Tensor("int64", BigInt64Array.from(sequences.flatMap((s) => [...s.map(BigInt), ...Array(length - s.length).fill(pad)])), [sequences.length, length]);
const attention_mask = new Tensor("int64", BigInt64Array.from(sequences.flatMap((s) => [...Array(s.length).fill(1n), ...Array(length - s.length).fill(0n)])), [sequences.length, length]);

const { logits } = await model({ input_ids, attention_mask });
const scaled = Array.from(logits.to("float32").data, (v) => v / 1.75);
const max = Math.max(...scaled);
const exps = scaled.map((v) => Math.exp(v - max));
const probabilities = exps.map((v) => v / exps.reduce((a, b) => a + b, 0));
// choice: argmax over options; score: expected level index; yes/no: two options
```

The graph uses ONNX Runtime contrib operators and needs onnxruntime-web /
onnxruntime-node 1.26 or newer (bundled with Transformers.js 4.2). Sequences
are limited to 384 tokens as in training.

---

## Original model card

# System One scorer — Qwen/Qwen3.5-4B-Base + scalar scoring head

A single-pass "System One" decision model in the shape of TypeSafe's Jev: it takes
unstructured state plus a set of typed questions (yes/no, Choice, numeric Score) and
returns a **probability distribution over exactly the options the caller supplied** —
in one forward pass, with no autoregressive generation.

Each (state, question, option) triple is scored by a sequence-classification head and
the per-question logits are softmaxed. Because the output space *is* the option set,
the output is type-safe by construction rather than by post-hoc parsing: there is no
token stream that can drift outside the schema.

Training code: [`system_one.py`](./system_one.py) in this repository.

## Intended use

Route, classify, prioritise or score a decision whose options you already know.
This is not a chat model and not a generator — it cannot produce free text.

## Results

Held-out test split, option sets **uncapped at evaluation**, temperature 1.75 fitted on val.

| task | n | accuracy | ECE | Brier |
| --- | --- | --- | --- | --- |
| ag_news | 64 | 0.922 | 0.021 | 0.131 |
| banking77 | 64 | 0.891 | 0.049 | 0.176 |
| go_emotions | 64 | 0.859 | 0.062 | 0.202 |
| tickets_language | 64 | 0.891 | 0.069 | 0.189 |
| tickets_type | 64 | 0.750 | 0.052 | 0.329 |
| mmlu | 64 | 0.703 | 0.188 | 0.407 |
| yelp_score | 64 | 0.641 | 0.117 | 0.460 |
| tickets_priority | 64 | 0.469 | 0.058 | 0.612 |
| tickets_queue | 64 | 0.234 | 0.244 | 0.848 |
| **ALL** | **576** | **0.707** | **0.044** | **0.373** |

Validation split, all 9 families, per-task cap 64 (n=537): accuracy 0.752, ECE 0.032.

### Calibration is the point

| model | accuracy | ECE | Brier |
| --- | --- | --- | --- |
| this model, raw head (test, n=576) | 0.707 | 0.135 | 0.415 |
| this model, T=1.75 fitted on val (test, n=576) | 0.707 | **0.044** | 0.373 |
| prompted Qwen3.5-4B-Base, 26-letter answer (val, n=112) | 0.679 | 0.093 | 0.439 |

Temperature scaling cuts ECE from 0.135 to 0.044 — about 2× better calibrated than the
prompted baseline, at identical accuracy. The raw head is *over*confident; that gap is
the whole reason a calibration stage exists.

The baseline row is **not** measured on the same rows: it covers 7 of 9 families
(112 questions, 16 per family), because banking77 (77 options) and ticket routing (52)
do not fit its 26-letter answer alphabet. On those 7 families this scorer averages
0.748 test accuracy against the baseline's 0.679.

### Latency

112.3 ms per question at 4 options — one forward pass, option batch scored together.
A smoke run measured 559.7 ms at 77 options. Latency scales with option count and
sequence length, not with output length, because nothing is generated.

## Training

- data: `pngwn/system-one-decisions` — 12,913 train questions across 9 task families
- 2,200 optimizer steps (step-capped, ~1.4 epochs), batch 8 questions, max_len 384,
  option cap 16, lr 1e-4 cosine with 3% warmup
- LoRA r=16 over all linear projections plus a new scalar `score` head
  (30.5M trainable of 4.24B)
- bf16 with gradient checkpointing; a100-large; 2h58m wall clock including eval and push
- train loss 1.682 (step 25) → 0.302 (step 2200); loss plateaued near 0.5 by step ~500

## Limitations

- **Option-cardinality mismatch.** High-cardinality tasks are trained with a cap of 16
  options but evaluated over all of them (banking77 77, ticket routing 52), because the
  cap is what keeps batching tractable. `tickets_queue` is the visible casualty at 0.234
  accuracy, and it is also the worst-calibrated task (ECE 0.244).
- **384-token truncation.** Long states (MMLU questions, long reviews) are truncated, so
  MMLU and `yelp_score` are the weakest non-routing tasks.
- **Knowledge-heavy multiple choice is not the strength** of encoder-style single-pass
  scoring. It trades world knowledge for latency and schema safety.
- **The ticket component of the training data is CC-BY-NC-4.0**, so this model inherits a
  non-commercial restriction.
- **A 4-epoch run was attempted and cancelled** around step 500: the measured 0.18–0.22
  steps/s could not finish inside the timeout, and the script only pushes after training
  completes, so continuing would have produced nothing.
- Accuracy is far below frontier models. The claims here are type-safe output, calibrated
  distributions and single-pass latency — not intelligence.