S1MB: comparing System One Decision Models across 100+ benchmarks

Since TypeSafe AI introduced Jev, its System One Decision Model, other developers have been working toward similar capabilities. I built a small model of my own, bekko-system-one-v0, and wanted to see how it compared with other models under the same evaluation conditions.

There are many ways to design a more comprehensive evaluation. What I needed first, though, was a practical reference point for comparing models. I put together tasks drawn from existing public NLP datasets and called the result System One Mosaic Benchmark, or S1MB. That collection of pieces is where “Mosaic” comes from.

• S1MB Leaderboard

I plan to expand both the benchmark suite and model coverage. The leaderboard shows the latest results; the screenshot and tables in this article capture a snapshot collected on September 30, 2026.

The evaluation code, dataset, and results are also available:

• Evaluation code
• Evaluation dataset
• Evaluation results

What it evaluates

System One Decision Models use text and instructions to make yes/no judgments, select an option, or assign a numeric score. S1MB evaluates these as Noul, Choice, and Score decisions, respectively.

This release covers English-language tasks. The evaluation data includes tasks from existing NLP datasets, tasks available through Open-Jev and Laya, and synthetic datasets I prepared for this benchmark. It is not exclusively a collection of the original datasets' official test sets. Within S1MB, however, the distributed data is an evaluation-only test split.

The English evaluation suite in this snapshot has the following composition:

In this snapshot, the data is organized into 106 subsets, containing 14,009 cases and 26,269 judgments. A subset is a collection of data that can contain more than one decision type. Evaluating each subset separately for its Noul, Choice, and Score tasks gives 137 benchmarks. A case can contain more than one judgment, so those counts differ. The six generalization benchmarks described later are included in the 137 benchmarks.

Overall results

These results cover 23 models and were collected on September 30, 2026. Every model completed all 137 benchmarks. The table is sorted by Borda Score, which converts relative model rankings into points. Task Avg averages the adjusted scores for the three decision types. Neither is raw accuracy; I explain both below the table.

The models were evaluated at different times. As models and benchmarks are added, the comparison scope and scores can change; these tables retain the scope of the snapshot above.

TP is the total parameter count. AP excludes embeddings used only for lookup. This is not the MoE meaning of active parameters, where a different set of experts may be used for each input.

Borda Score and Task Avg

For each benchmark, Borda Score assigns 100 points to first place and 0 to last place, with evenly spaced points between them. Ties use average ranks. The points are then averaged equally across all benchmarks in the selected evaluation scope. It measures how consistently a model ranks near the top of the comparison, so changing the model roster or benchmark set can change its value.

Task Avg and the Noul, Choice, and Score columns instead use scores adjusted against simple predictions that ignore the input. A model performing at or below a baseline such as always choosing the same option or returning the same number receives 0; the reference ceiling is 100. A score of 50 therefore means closing half the gap from the baseline to the reference ceiling, not getting half the answers right.

The adjustment differs by task. For Noul, balanced accuracy of 50% maps to 0, and 100% maps to 100; values below the baseline are clipped to 0. Choice compares against the best of uniform random and fixed-answer baselines. Score compares mean absolute error (MAE) against a constant prediction.

The adjusted values are clipped to 0–100 for each benchmark and averaged within each decision type. Task Avg is then the equal-weighted mean of Noul, Choice, and Score. The details are in the scoring definitions.

Borda gives equal weight to each benchmark. In this snapshot, with 59 Noul benchmarks and 21 Score benchmarks, those decision types therefore have different weights in the overall result. Task Avg weights the three types equally, so the rankings need not agree. For example, Open-Jev-27B-v1.1 (ZefanCai) ranks second by Task Avg at 56.88, but fourth by Borda Score. The individual task columns help show what an overall ranking leaves out.

Reading the overall results

Jev 1.13 ranks first with a Borda Score of 86.51. My bekko-system-one-v0-400m is second at 76.21.

There is relevant training history behind that result. My model uses training sets from datasets that also appear in this evaluation. It has not trained on the test sets, but familiarity with the tasks and domains makes high scores on their test sets more likely. The training manifest and S1MB evaluation manifest have 77 subset names in common. That is worth keeping in mind when reading the results.

Even with that familiarity, Jev 1.13 scores higher on both overall Borda Score and Task Avg. Seeing that gap, despite my model having trained on many of the same domains, was one of the surprises of running this comparison.

An overall ranking does not tell us whether a model can handle unfamiliar instructions, though. I wanted to look at that separately.

Evaluating generalization

Performance on an existing dataset can depend on how much experience a model has with that task. What I also want from a System One Decision Model is the ability to adapt when instructions, context, or decision criteria change. To probe that, I used GPT-6-Astra to create a synthetic evaluation set.

For each of Noul, Choice, and Score, I prepared two kinds of benchmarks: Diverse and Contextual.

Each Contextual task contains 20 groups of five questions. For Choice and Score, the question and criteria stay fixed while the state changes. For Noul, both the state and the false/true criteria change. The intention is to see whether a model adapts its judgment to the supplied conditions instead of relying on isolated words.

Here is a summarized example from Diverse Choice:

The same model generated the questions and intended answers and checked its own work. They have not undergone independent human validation. Nor do I guarantee that the evaluated models' training data contains no overlap. What this set measures is adaptation to the particular instructions, contexts, and criteria supplied here.

Generalization results

This table contains the same 23 models, evaluated only on those six benchmarks. Borda Score is recomputed over the six benchmarks, so its values differ from the overall table. The rows are again sorted by Borda Score.

Jev 1.13 reaches a Task Avg of 96.27 and matches the intended answers on almost every Noul and Choice question. Checking the saved predictions, Noul matches on 199 of 200 questions (99.5%) and Choice on 198 of 200 (99.0%). The table's General Noul score of 99.00 and General Choice score of 98.68 are adjusted scores, so they differ from those raw accuracies.

That consistency as instructions and context change is what stands out to me about Jev 1.13.

Open-Jev-27B-v1.1 (ZefanCai) and Open-Jev-9B (ZefanCai) also perform well. Both score 93.00 on General Noul, and their General Choice scores are 100.00 and 96.70, respectively. Their General Score results are 55.91 and 67.21, however, so neither has reached the ceiling across all three decision types.

How I would read these scores

This is a synthetic set I prepared with GPT-6-Astra, and the questions are not particularly complex. As its author, I expect models with some generalization capability to approach the ceiling, especially on Noul and Choice. I would use it as a reference point for asking: can a model get close to the maximum score on this level of variation in instructions and context? Small gaps between the leading models are not a basis for finely ranking their generalization abilities.

Here, approaching the maximum means getting close to 100 on the adjusted task scores, not on Borda's relative ranking points. A high score does not mean a model can handle any unfamiliar task. These 600 questions also do not tell us how the models would separate on harder problems.

For Score, the evaluation measures agreement with criteria and intended scores generated by GPT-6-Astra. Jev 1.13's 91.14 is not an accuracy percentage: it compares error in the expected numeric score against a constant prediction. Exact agreement with an intended score is not necessarily the only valid real-world judgment, either.

Meanwhile, bekko-system-one-v0-400m, second in the overall Borda ranking, has a Task Avg of 54.48 on these six benchmarks. Scoring well on familiar task families and adapting to freely specified instructions and changing contexts are worth examining separately.

Closing thoughts

S1MB started from wanting to evaluate my own small model. Running the comparisons showed me how different the picture can look between an overall ranking and the generalization tests.

Both the collection of existing datasets and these relatively simple synthetic tasks have limited scope. Even so, comparing models on the same inputs and metrics, and seeing which decision types or tasks they struggle with, gives me a more concrete basis for deciding what to improve next.

Instructions for evaluating your own model and adding its results to the leaderboard are available. After validating and exporting the results, you can submit them through a pull request to the results dataset.

• Evaluation guide
• Submitting evaluation results
• Request a model evaluation

I hope S1MB gives other people building System One Decision Models a useful reference point for their own experiments.

References

• TypeSafe AI — System One / Jev
• TypeSafe AI — Models
• S1MB: evaluation code
• S1MB: scoring definitions
• S1MB: evaluation dataset
• S1MB: evaluation results
• S1MB: leaderboard
• bekko-system-one collection
• bekko-system-one: training and inference code