card: ladder position, per-domain vs baseline, both probes

README.md

@@ -0,0 +1,174 @@
+---
+license: apache-2.0
+language:
+- en
+- ko
+tags:
+- ztc
+- answer-verification
+- hallucination-detection
+- zero-token
+- confidence-estimation
+library_name: transformers
+pipeline_tag: text-classification
+---
+
+# ZTC-Judge-9B
+
+**Answer verification from a single forward pass, with zero generated tokens — at 9B.**
+
+ZTC-Judge-9B takes a question and an answer written by *any* model and scores whether that
+answer can be trusted. It is the third rung of a four-point size ladder measured under one
+identical protocol, and it is published so the shape of that ladder can be checked rather than
+asserted.
+
+> **ZTC** — Zero-Token Confidence · **Judge** — it evaluates *someone else's* answer, not its own
+
+---
+
+## Read this before deploying
+
+This model is **not** the strongest member of the family, and the card says so with numbers.
+
+| Model | Leaderboard AUC |
+|---|---|
+| Darwin-397B-ZTC | 0.7364 |
+| ZTC-Judge-27B | 0.7282 |
+| **ZTC-Judge-9B** | **0.6506** |
+| **ZTC-Judge-4B** | **0.6360** |
+| *Answer length and formatting only* | *0.6223* |
+
+**The ladder does not decline smoothly — it steps.** Between 9B and 27B the score moves 0.078,
+while between 4B and 9B it moves 0.015. Whatever carries verification quality is largely absent
+below 27B on this axis.
+
+**Where this model is worth deploying is one specific place**, and it is a real one:
+
+| Domain | Surface baseline | **9B** | Margin |
+|---|---|---|---|
+| **Professional exams (law · math · biology)** | 0.7138 | **0.7623** | ****+0.0485**** |
+| Scientific reasoning | 0.7272 | 0.6764 | 🔴 -0.0508 |
+| Biology & medicine | 0.5908 | 0.6310 | **+0.0402** |
+| Disaster & safety procedures | 0.5949 | 0.5862 | 🔴 -0.0087 |
+| General multi-step reasoning | 0.5420 | 0.5881 | **+0.0461** |
+| **Size-weighted mean** | **0.6223** | **0.6506** | ****+0.0283**** |
+
+🔴 **Do not use this model for disaster and safety content.** In that domain it does not clear the
+surface baseline, which means it is reading answer shape rather than correctness there.
+
+✅ **Professional-exam style content is where it earns its size.** It runs on a laptop, on CPU, and
+inside networks that never reach the internet — places a hosted API cannot go.
+
+## How it works
+
+```
+[question + answer]  →  one forward pass
+                     →  final-layer hidden state at the last position (4096-d)
+                     →  probe
+                     →  score
+```
+
+**Generated tokens: 0.** No access to the answering model's weights or logits is required; the text
+of the answer is the only input. Latency is one forward pass, and batching converts directly into
+throughput.
+
+## Usage
+
+```python
+import json
+import numpy as np, torch
+from huggingface_hub import hf_hub_download, snapshot_download
+from transformers import AutoModel, AutoTokenizer
+
+REPO = "FINAL-Bench/ZTC-Judge-9B"
+cfg  = json.load(open(hf_hub_download(REPO, "ztc_config.json"), encoding="utf-8"))
+path = snapshot_download(REPO)
+
+tok   = AutoTokenizer.from_pretrained(path)
+model = AutoModel.from_pretrained(path, dtype=torch.bfloat16, low_cpu_mem_usage=True).eval()
+
+def hidden(question, answer):
+    b = tok([cfg["template"] % (question, answer)], return_tensors="pt",
+            truncation=True, max_length=cfg["max_length"])
+    dev = next(model.parameters()).device
+    with torch.no_grad():
+        h = model(input_ids=b["input_ids"].to(dev),
+                  attention_mask=b["attention_mask"].to(dev)).last_hidden_state
+    return h[0, int(b["attention_mask"].sum()) - 1].float().numpy().astype(np.float64)
+
+# linear probe — one dot product
+p = np.load(hf_hub_download(REPO, "ztc_probe.npz"))
+v = hidden("Which defensive chemical does an insect release?", "C. Allomone")
+print(float(((v - p["mu"]) / p["sd"]) @ p["w"]))
+```
+
+The score is an unbounded real number; higher means more likely correct. It is a **ranking signal**,
+not a calibrated probability — choose a threshold from your own review budget.
+
+## Two probes ship with this model
+
+| File | Produces |
+|---|---|
+| `ztc_probe.npz` | linear readout — one dot product |
+| `ztc_curve_probe.npz` | **the reported figure 0.6506** — 256 anchors, RBF kernel |
+
+Both read the same input. The curved probe is the one to use when the number matters.
+
+## Protocol
+
+| | |
+|---|---|
+| Items | 2,018 · 508 incorrect · 5 domains · answers written by 4 different models |
+| Metric | AUC — how well wrong answers sort to the bottom. Threshold-free. 0.5 = coin flip |
+| Selection | **Leave-one-domain-out.** Every figure comes from a domain the probe never saw; hyper-parameters are chosen inside the training domains only |
+| Aggregation | Per domain, then size-weighted. Pooling all items into one AUC inflates the result |
+
+The same protocol, item set and grading code are applied to every rung of the ladder and to the
+other systems on the independent leaderboard:
+<https://huggingface.co/spaces/mayafree/typed-decision-leaderboard>
+
+## Out of scope
+
+- **Not a grounding checker.** It does not take a source document and decide whether the answer
+  follows from it.
+- **Not a safety, toxicity or policy classifier.**
+- **Not a calibrated probability.** Use it to rank and threshold.
+- **Not a general-purpose verifier at this size.** See the domain table above.
+
+## Limitations
+
+- **Domain coverage.** Scores are meaningful only for the five domains listed. Outside them nothing
+  has been measured and no guarantee is published.
+- **Below the surface baseline on disaster and safety.** Stated in the table rather than omitted.
+- **Revision lock.** The probe is fitted to one specific revision of the base model. Running it on a
+  different revision produces **no error and silently wrong scores**; this repository ships the
+  matching weights so that failure mode cannot occur.
+- **It reports the verifier's judgement**, which is not the answering model's own confidence — that
+  quantity measures 0.5000 on this set.
+
+## Lineage
+
+| | |
+|---|---|
+| Base model | `Qwen/Qwen3.5-9B`, Apache-2.0 |
+| Modification to base weights | none — the probes are separate files |
+| Added by FINAL-Bench | probes, inference code, evaluation protocol and tables |
+
+## What this repository contains
+
+| | |
+|---|---|
+| Included | Base weights · tokenizer · linear probe · curved probe · configuration |
+| Not included | Training corpus · hidden-state matrices · fitting pipeline |
+
+## The rest of the ladder
+
+[ZTC-Judge-27B](https://huggingface.co/FINAL-Bench/ZTC-Judge-27B) ·
+[Darwin-397B-ZTC](https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC) ·
+[ZTC-Judge-9B](https://huggingface.co/FINAL-Bench/ZTC-Judge-9B) ·
+[ZTC-Judge-4B](https://huggingface.co/FINAL-Bench/ZTC-Judge-4B)
+
+## License
+
+The base model is Apache-2.0 and redistributable. The probes, the inference code and the evaluation
+tables are assets of FINAL-Bench / VIDRAFT.