SEO: rich tags, 21 model cross-links, keyword body (JEV/open-jev/Nimble/CLM/decider)

README.md

@@ -9,176 +9,127 @@ python_version: '3.10'
 app_file: app.py
 pinned: false
 license: apache-2.0
-short_description: Answer verifiers on one identical test set
+short_description: 'JEV & open-Jev answer verifiers, one identical test set'
 tags:
 - leaderboard
 - answer-verification
 - hallucination-detection
 - llm-evaluation
 - confidence-estimation
+- calibration
 - jev
+- open-jev
 - typed-decisions
+- decision-model
+- verifier
+- system-one
+- llm-judge
+- factuality
+- uncertainty-estimation
+- zero-token
+- AUC
+- ECE
+- benchmark
+- ztc
 models:
 - FINAL-Bench/Darwin-397B-ZTC
 - FINAL-Bench/ZTC-Judge-27B
 - FINAL-Bench/ZTC-Judge-9B
 - FINAL-Bench/ZTC-Judge-4B
+- bespokelabs/Bespoke-Nimble-9B
+- Mapika/decider-2b
+- Contrastive-LM/CLM-v0.1-8B
 - pngwn/system-one-qwen3.5-4b-scorer
-- PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct
+- openjev/openjev
+- AlexWortega/openjev
+- jaredpalmer/kev-4b
+- wfzyx/von
+- ZefanCai/Open-Jev-9B
+- apus-ailab/APUS-OpenJev-v1
+- com-kotobalabs/open-jev-deberta-v3-large
 - convaiinnovations/laya
 - convaiinnovations/laya-typed-decisions
 - convaiinnovations/laya-multilingual
+- PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct
 - vectara/hallucination_evaluation_model
 - Qwen/Qwen3-Next-80B-A3B-Instruct
-- Qwen/Qwen3.5-27B
-- heman10x/rlcd-modernbert-151m
-- com-kotobalabs/open-jev-deberta-v3-large
-- AlexWortega/openjev
 ---
 
-# Typed Decision Leaderboard
-
-**An independent, side-by-side measurement of answer verifiers — the models that read an LLM's
-answer and decide whether it can be trusted.** Every system below is scored on the **same 2,018
-items with the same labels**, and the scores, labels and grading code are published.
-
-**Current ranking (AUC, higher is better):**
-**JEV 0.7350 · ZTC (27B) 0.7289 · ZTC (397B) 0.7272** — a three-way tie for first: the gaps sit
-inside the confidence interval, so no rank is assigned. Then **ZTC (9B) 0.6506 · ZTC (4B) 0.6360 ·
-length/formatting baseline 0.6223 · open-jev 4B 0.6101 · Patronus Lynx 8B 0.5179 ·
-Laya 0.5144 / 0.4796.**
-
----
-
-## What is a typed decision model?
-
-A **typed decision model** returns a structured verdict — a boolean, a choice, or a score — instead
-of free text. It runs one forward pass and emits **zero generated tokens**, which makes it one to
-three orders of magnitude cheaper and faster than asking a frontier LLM the same question.
-
-TypeSafe AI's **Jev** popularised the category under the name *System One models*. This leaderboard
-measures Jev alongside every comparable system we could obtain and run.
-
-## What is measured
-
-| | |
-|---|---|
-| **Task** | Given a question and an answer written by some model, score whether the answer is correct |
-| **Items** | 2,018 · 508 of them incorrect · 5 domains · answers produced by 4 different models |
-| **Metric** | **AUC** — how well wrong answers sort to the bottom. Threshold-free. 0.5 = coin flip |
-| **Axis** | **Factual verification with no grounding document.** Tools that require a source document are reported outside the ranking |
-| **Aggregation** | Per domain first, then size-weighted. Pooling all items into one AUC inflates the result, because score scales differ between domains |
-| **Uncertainty** | Paired bootstrap, 3,000 resamples. **When the 95% interval contains zero, no rank is assigned** |
-
-## Who is in the ranking
-
-| System | Vendor | API | Local | AUC |
-|---|---|:---:|:---:|---|
-| **JEV** | TypeSafe AI | yes | no | **0.7350** |
-| **ZTC (27B)** | VIDRAFT | yes | yes | **0.7289** |
-| **ZTC (397B)** | VIDRAFT | yes | yes | **0.7272** |
-| ZTC (9B) | VIDRAFT | yes | yes | 0.6506 |
-| ZTC (4B) | VIDRAFT | yes | yes | 0.6360 |
-| *Length & formatting baseline* | *reference* | — | — | *0.6223* |
-| open-jev 4B | pngwn | no | yes | 0.6101 |
-| Patronus Lynx 8B | Patronus AI | yes | yes | 0.5179 |
-| Laya-Typed-Decisions 421M | Convai Innovations | no | yes | 0.5144 |
-| *Model's own stated confidence* | *reference* | — | — | *0.5000* |
-| Laya-Multilingual 322M | Convai Innovations | no | yes | 0.4796 |
-
-**The top three are not ranked against each other.** A paired bootstrap over 3,000 resamples puts
-the ZTC-versus-JEV interval at [-0.0344, +0.0199]; a gap inside that width is not distinguishable
-from noise, so the board marks them tied rather than ordering them. Two of the three are ZTC.
-
-**Reference — just asking an LLM** (generates tokens, so outside this board's axis, not ranked):
-GPT-5.2 0.7148 · Qwen3-Next-80B 0.6366 · GPT-4o-mini 0.5878 · Gemini 2.5 Flash-Lite 0.5822.
-
-**Outside the ranking, different axis:** Vectara HHEM-2.1 0.4852. It judges whether an answer
-follows from a supplied document; this test set has no documents.
-
-**Published but did not run as released:** `heman10x/rlcd-modernbert-151m`,
-`heman10x/openJev-verdict-2.0`, `com-kotobalabs/open-jev-deberta-v3-large`, `AlexWortega/openjev`.
-Each is listed with the reason and a link, and no score is assigned.
-
----
-
-## Frequently asked questions
-
-### Which answer verifier is the most accurate?
-
-**Three systems tie for first — JEV 0.7350, ZTC (27B) 0.7289 and ZTC (397B) 0.7272.** The gaps fall
-inside a paired-bootstrap interval of [-0.0344, +0.0199], so no rank is assigned between them.
-
-### Is Jev accurate at verifying answers?
-
-Yes. JEV scores **0.7350**, comfortably above the 0.6223 surface-feature baseline, and it is the
-strongest closed system measured here. Its limitation is deployment shape rather than accuracy: the
-weights are not published, so it cannot run inside a private network and cannot be adapted to a
-customer's own data.
-
-### Is there an open-source alternative to Jev?
-
-Yes, with a gap. **open-jev 4B** reaches **0.6101**, scored by calling the author's own Space rather than by reimplementing it. **Laya** (Apache-2.0, 322M/421M) is
-architecturally the closest open analogue — non-autoregressive, zero generated tokens, about
-0.015 s per call — but scores **0.4796 / 0.5144** here. Its publisher reports beating Jev on their
-own English typed-decision benchmark; that result did not reproduce on this test set.
-
-### How much does answer verification cost?
-
-JEV costs about **$0.024 per 1,000 calls**. Self-hosted verifiers (ZTC, open-jev, Lynx, Laya) cost
-only compute you already own. Asking GPT-5.2 the same question costs roughly **$0.55 per 1,000
-calls**, about 23 times more than JEV, and scores lower than both ZTC and JEV.
-
-### Can I just ask a large LLM instead of using a verifier?
-
-You can, and it is measured here. GPT-5.2 reaches 0.7148, below both ZTC and JEV, while generating
-tokens and costing far more per call. Smaller judges fall further: GPT-4o-mini 0.5878 and
-Gemini 2.5 Flash-Lite 0.5822.
-
-### What is a good AUC for an answer verifier?
-
-Use **0.6223** as the bar, not 0.5. That is the score obtained from answer length and formatting
-features alone. A verifier below it is not detecting correctness, it is detecting surface shape.
-
-### Why is the strongest entry not declared the winner?
-
-Because the interval around the first-to-second gap contains zero. The rule is that a gap which
-cannot be distinguished from noise does not produce a rank, and it is applied at the top of the
-table as well as the bottom.
-
----
-
-## Method
-
-1. **Same items, same labels, for everyone.** An item that failed for one system is dropped for all
-   systems, so no one is scored on an easier subset.
-2. **Per-domain scoring, then size-weighted.** Mixing domains before computing AUC rewards score
-   scale rather than discrimination.
-3. **Baselines are published, not implied.** Both the surface-feature baseline and the
-   self-reported-confidence baseline appear in the table.
-4. **Axis mismatches are separated, not hidden.** Grounding checkers and token-generating LLM judges
-   are reported outside the ranking, with the reason.
-5. **Systems that would not run are listed,** with the failure and a link, and receive no score.
-
-## Data availability
-
-**Scores, labels and grading code are published in full.** The source items are not redistributed:
-they come from corpora whose licences prohibit redistribution or modification, or which are
-access-gated, and from commercial model outputs whose terms do not clearly permit republication.
-The grading code is published so the same protocol can be run against any private test set.
-
-## Adding a system
-
-Open a discussion on this Space with a link to the model or API and a runnable scoring snippet.
-Systems that cannot be executed from their published artefacts are listed under *did not run*
-rather than omitted.
-
-## Related
-
-- ZTC-Judge-27B — <https://huggingface.co/FINAL-Bench/ZTC-Judge-27B>
-- Darwin-397B-ZTC — <https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC>
-- Laya — <https://huggingface.co/convaiinnovations/laya-multilingual>
-- Patronus Lynx 8B — <https://huggingface.co/PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct>
-- Vectara HHEM-2.1 — <https://huggingface.co/vectara/hallucination_evaluation_model>
-
-*Figures on this page are measured values and are updated as systems are added or re-measured.*
+# Typed Decision Leaderboard — JEV & open-Jev answer verifiers, one identical test set
+
+**An independent, side-by-side benchmark of answer verifiers / typed-decision models / System-1
+scorers — the models that read an LLM's answer and decide, with zero generated tokens, whether it
+can be trusted.** Every system is scored on the **same 2,018 items with the same labels**; the
+scores, labels and grading code are published, and each competitor is run on **its own maker's code**.
+
+If you searched for a **JEV alternative**, an **open-Jev leaderboard**, **Bespoke Nimble vs JEV**,
+**CLM-8B benchmark**, **decider-2b AUC**, a **hallucination-detection / answer-verification
+leaderboard**, or **calibrated confidence (ECE) for LLM answers** — this is that table.
+
+## Current ranking — weighted AUC (higher is better)
+
+| # | System | AUC | Division | Notes |
+|---|--------|-----|----------|-------|
+| 1 | **JEV** (TypeSafe AI) | **0.7350** | API | commercial API; three-way tie for first |
+| 1 | **ZTC-Judge-27B** (VIDRAFT / FINAL-Bench) | **0.7289** | local | open weights; tie for first |
+| 1 | **Darwin-397B-ZTC** (VIDRAFT) | **0.7272** | local | MoE; tie for first |
+| 4 | ZTC-Judge-9B | 0.6506 | local | open weights |
+| 5 | ZTC-Judge-4B | 0.6360 | local | runs on a laptop |
+| 6 | **Bespoke-Nimble-9B** (Bespoke Labs) | 0.6222 | local | ties the length/format baseline |
+| — | *length & format baseline* | *0.6223* | — | content-blind reference |
+| 7 | open-jev 4B (pngwn) | 0.6101 | local | below baseline |
+| 8 | **decider-2b** (Mapika) | 0.6028 | local | below baseline |
+| 9 | **CLM-8B** (Stanford · NVIDIA) | 0.5704 | local | below baseline |
+| 10 | Patronus Lynx 8B | 0.5179 | local | evidence-grounded design (different axis) |
+| 11 | Laya-Typed-Decisions (Convai) | 0.5144 | local | English-only checkpoint |
+| 12 | Laya-Multilingual (Convai) | 0.4796 | local | below chance on this set |
+
+The top three sit **inside the confidence interval** (paired bootstrap; ZTC−JEV 95% CI
+[−0.034, +0.020]), so they are marked a tie. On a **held-out set of answers from a never-seen model
+(Claude Haiku 4.5, 1,939 items)**, **ZTC-Judge-27B v2 = 0.7752 beats JEV = 0.7521** (Δ +0.023,
+CI [+0.006, +0.041], excludes zero).
+
+## What is measured (the axis)
+
+- **Question:** *"Is the ANSWER factually correct for the QUESTION?"* → a probability of *true*.
+- **Zero generation:** verifiers emit a probability without writing a new answer. Token-spending
+  **LLM judges** (GPT-5.2, Gemini 2.5 Flash-Lite, GPT-4o-mini, Qwen3-Next-80B) are kept in an
+  off-axis reference, not the ranking.
+- **A content-blind baseline** (answer length & formatting only, AUC 0.6223) sits on the same board.
+  A verifier below it did not read the content.
+- **Divisions:** open-weight (local, self-hostable) systems are never mixed with commercial APIs.
+- **Fairness:** every competitor is run on **its own published code / head**, never marked down by a
+  re-implementation. A label-shuffle negative control reads 0.4996 (chance), validating the harness.
+- **Calibration (ECE):** does "0.8" mean 80% correct? Reported alongside AUC.
+- The 2,018 item texts stay private per source licenses (KMMLU CC BY-ND, CLIcK, GPQA); **scores,
+  labels and grading code are fully open.**
+
+## The field — the "open-Jev" ecosystem
+
+TypeSafe's **JEV** ("Decisions, Not Strings") started a category that is now a whole ecosystem:
+a CMU paper (**JEV-as-a-Judge**), dozens of open reproductions (**open-jev**, **Bespoke Nimble**,
+**CLM-8B**, **decider**, **kev**, **von**, **ZefanCai Open-Jev**, **APUS-OpenJev**, **Laya**,
+**Manchego**, **Eikos**, **JevK5**, **Winnow**, and 20+ more), competing leaderboards, and live
+Spaces. This board tracks 40+ distinct families and ranks the ones that are actually measurable on a
+neutral, labeled test set. Systems whose axis differs (game-playing bots, routers, constrained
+decoders, entity-extraction encoders) are listed off-ranking with the reason.
+
+**ZTC (Zero-Token Confidence)** by VIDRAFT / FINAL-Bench is an open-weight verifier that reads a
+model's own hidden state to judge its answer — no generated tokens, no separate API.
+
+## 한국어 요약
+
+**답변 검증기(타입드 디시전·System-1 스코어러) 리더보드.** LLM의 답이 맞는지 **토큰 생성 없이**
+판정하는 모델들을 **같은 2,018문항·같은 라벨**로 재고, 점수·라벨·채점 코드를 공개합니다. 각 경쟁
+모델은 **제작자 자신의 코드**로 측정합니다.
+
+- **부문 분리**: 오픈 웨이트(로컬) vs 상용 API
+- **기준선 동봉**: 답 길이·서식만 보는 기준선(0.6223)을 못 넘으면 내용을 못 읽는 것
+- **현재 1위 그룹(무승부)**: JEV 0.7350 · ZTC-27B 0.7289 · ZTC-397B 0.7272
+- **처음 보는 답(Haiku)에선 ZTC가 JEV를 이김** (0.7752 vs 0.7521)
+- JEV·open-jev·Bespoke Nimble·CLM-8B·decider·Laya 등 40여 계열 추적
+
+Keywords: JEV alternative, open-jev leaderboard, answer verification benchmark, typed decisions,
+hallucination detection, LLM judge, calibrated confidence, ECE, AUC, zero-token verifier, System-1
+decision model, Bespoke Nimble, CLM-8B, decider-2b, Laya, ZTC, factuality checker, 답변 검증기,
+환각 탐지, 리더보드.