README.md

---
title: Typed Decision Leaderboard
emoji: 🎯
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 6.27.0
python_version: '3.10'
app_file: app.py
pinned: false
license: apache-2.0
short_description: 'JEV & open-Jev answer verifiers, one identical test set'
tags:
- leaderboard
- answer-verification
- hallucination-detection
- llm-evaluation
- confidence-estimation
- calibration
- jev
- open-jev
- typed-decisions
- decision-model
- verifier
- system-one
- llm-judge
- factuality
- uncertainty-estimation
- zero-token
- AUC
- ECE
- benchmark
- ztc
models:
- FINAL-Bench/Darwin-397B-ZTC
- FINAL-Bench/ZTC-Judge-27B
- FINAL-Bench/ZTC-Judge-9B
- FINAL-Bench/ZTC-Judge-4B
- bespokelabs/Bespoke-Nimble-9B
- Mapika/decider-2b
- Contrastive-LM/CLM-v0.1-8B
- pngwn/system-one-qwen3.5-4b-scorer
- openjev/openjev
- AlexWortega/openjev
- jaredpalmer/kev-4b
- wfzyx/von
- ZefanCai/Open-Jev-9B
- apus-ailab/APUS-OpenJev-v1
- com-kotobalabs/open-jev-deberta-v3-large
- convaiinnovations/laya
- convaiinnovations/laya-typed-decisions
- convaiinnovations/laya-multilingual
- PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct
- vectara/hallucination_evaluation_model
- Qwen/Qwen3-Next-80B-A3B-Instruct
---

# Typed Decision Leaderboard — JEV & open-Jev answer verifiers, one identical test set

**An independent, side-by-side benchmark of answer verifiers / typed-decision models / System-1
scorers — the models that read an LLM's answer and decide, with zero generated tokens, whether it
can be trusted.** Every system is scored on the **same 2,018 items with the same labels**; the
scores, labels and grading code are published, and each competitor is run on **its own maker's code**.

If you searched for a **JEV alternative**, an **open-Jev leaderboard**, **Bespoke Nimble vs JEV**,
**CLM-8B benchmark**, **decider-2b AUC**, a **hallucination-detection / answer-verification
leaderboard**, or **calibrated confidence (ECE) for LLM answers** — this is that table.

## Current ranking — weighted AUC (higher is better)

| # | System | AUC | Division | Notes |
|---|--------|-----|----------|-------|
| 1 | **JEV** (TypeSafe AI) | **0.7350** | API | commercial API; three-way tie for first |
| 1 | **ZTC-Judge-27B** (VIDRAFT / FINAL-Bench) | **0.7289** | local | open weights; tie for first |
| 1 | **Darwin-397B-ZTC** (VIDRAFT) | **0.7272** | local | MoE; tie for first |
| 4 | ZTC-Judge-9B | 0.6506 | local | open weights |
| 5 | ZTC-Judge-4B | 0.6360 | local | runs on a laptop |
| 6 | **Bespoke-Nimble-9B** (Bespoke Labs) | 0.6222 | local | ties the length/format baseline |
| — | *length & format baseline* | *0.6223* | — | content-blind reference |
| 7 | open-jev 4B (pngwn) | 0.6101 | local | below baseline |
| 8 | **decider-2b** (Mapika) | 0.6028 | local | below baseline |
| 9 | **CLM-8B** (Stanford · NVIDIA) | 0.5704 | local | below baseline |
| 10 | Patronus Lynx 8B | 0.5179 | local | evidence-grounded design (different axis) |
| 11 | Laya-Typed-Decisions (Convai) | 0.5144 | local | English-only checkpoint |
| 12 | Laya-Multilingual (Convai) | 0.4796 | local | below chance on this set |

The top three sit **inside the confidence interval** (paired bootstrap; ZTC−JEV 95% CI
[−0.034, +0.020]), so they are marked a tie. On a **held-out set of answers from a never-seen model
(Claude Haiku 4.5, 1,939 items)**, **ZTC-Judge-27B v2 = 0.7752 beats JEV = 0.7521** (Δ +0.023,
CI [+0.006, +0.041], excludes zero).

## What is measured (the axis)

- **Question:** *"Is the ANSWER factually correct for the QUESTION?"* → a probability of *true*.
- **Zero generation:** verifiers emit a probability without writing a new answer. Token-spending
  **LLM judges** (GPT-5.2, Gemini 2.5 Flash-Lite, GPT-4o-mini, Qwen3-Next-80B) are kept in an
  off-axis reference, not the ranking.
- **A content-blind baseline** (answer length & formatting only, AUC 0.6223) sits on the same board.
  A verifier below it did not read the content.
- **Divisions:** open-weight (local, self-hostable) systems are never mixed with commercial APIs.
- **Fairness:** every competitor is run on **its own published code / head**, never marked down by a
  re-implementation. A label-shuffle negative control reads 0.4996 (chance), validating the harness.
- **Calibration (ECE):** does "0.8" mean 80% correct? Reported alongside AUC.
- The 2,018 item texts stay private per source licenses (KMMLU CC BY-ND, CLIcK, GPQA); **scores,
  labels and grading code are fully open.**

## The field — the "open-Jev" ecosystem

TypeSafe's **JEV** ("Decisions, Not Strings") started a category that is now a whole ecosystem:
a CMU paper (**JEV-as-a-Judge**), dozens of open reproductions (**open-jev**, **Bespoke Nimble**,
**CLM-8B**, **decider**, **kev**, **von**, **ZefanCai Open-Jev**, **APUS-OpenJev**, **Laya**,
**Manchego**, **Eikos**, **JevK5**, **Winnow**, and 20+ more), competing leaderboards, and live
Spaces. This board tracks 40+ distinct families and ranks the ones that are actually measurable on a
neutral, labeled test set. Systems whose axis differs (game-playing bots, routers, constrained
decoders, entity-extraction encoders) are listed off-ranking with the reason.

**ZTC (Zero-Token Confidence)** by VIDRAFT / FINAL-Bench is an open-weight verifier that reads a
model's own hidden state to judge its answer — no generated tokens, no separate API.

## 한국어 요약

**답변 검증기(타입드 디시전·System-1 스코어러) 리더보드.** LLM의 답이 맞는지 **토큰 생성 없이**
판정하는 모델들을 **같은 2,018문항·같은 라벨**로 재고, 점수·라벨·채점 코드를 공개합니다. 각 경쟁
모델은 **제작자 자신의 코드**로 측정합니다.

- **부문 분리**: 오픈 웨이트(로컬) vs 상용 API
- **기준선 동봉**: 답 길이·서식만 보는 기준선(0.6223)을 못 넘으면 내용을 못 읽는 것
- **현재 1위 그룹(무승부)**: JEV 0.7350 · ZTC-27B 0.7289 · ZTC-397B 0.7272
- **처음 보는 답(Haiku)에선 ZTC가 JEV를 이김** (0.7752 vs 0.7521)
- JEV·open-jev·Bespoke Nimble·CLM-8B·decider·Laya 등 40여 계열 추적

Keywords: JEV alternative, open-jev leaderboard, answer verification benchmark, typed decisions,
hallucination detection, LLM judge, calibrated confidence, ECE, AUC, zero-token verifier, System-1
decision model, Bespoke Nimble, CLM-8B, decider-2b, Laya, ZTC, factuality checker, 답변 검증기,
환각 탐지, 리더보드.