Model card: add handoff benchmarks and routing-quality (AUROC) results

README.md

@@ -15,8 +15,7 @@ A small, on-device model is fast and private, but sometimes wrong. At Cactus we
 post-train models to *know when they are wrong*: we ship probes inside the
 checkpoint that score every answer with a **confidence** between 0 and 1,
 returned as structured data (never parsed out of the answer text). Answer
-on-device when confidence is high; re-route to a bigger model when it's low —
-`0.85` is a good threshold:
+on-device when confidence is high; re-route to a bigger model when it's low:
 
 ```python
 if confidence < 0.85:
@@ -30,6 +29,26 @@ checkpoint (same keys); the repo adds eleven probe tensors, a remote-code model
 class and a `custom_generate` recipe. **Stock engine commands work unchanged** —
 you only add `--trust-remote-code` / `trust_remote_code=True`.
 
+## Benchmarks
+
+Gemma 4 E2B Hybrid, the smallest Gemma model, matches Gemini 3.1 Flash-Lite on
+most benchmarks by routing only 15–35% of queries to Flash-Lite and running the
+rest itself:
+
+| Benchmark | Handoff to match Flash-Lite (FP16) | At 4-bit | At 3-bit |
+|---|---|---|---|
+| ChartQA | 15–20% | 25–30% | 40–50% |
+| MMBench | 30–35% | 40–45% | 50–55% |
+| LibriSpeech | 25–30% | 35–40% | 55–65% |
+| GigaSpeech | 30–35% | 40–45% | 50–55% |
+| MMAU | 30–35% | 35–40% | 50–55% |
+| MMLU-Pro | 45–55% | ~90% | n/a |
+
+Quantisation quality is measured on
+[Cactus Quants](https://github.com/cactus-compute/cactus/blob/main/docs/cactus_quants.md),
+which performs well at uniform quantization; developers are encouraged to
+benchmark Unsloth, GGUF, and MLX quantization independently.
+
 ## Quickstart
 
 ```python
@@ -135,6 +154,33 @@ sequences = model.generate(**inputs, max_new_tokens=512, emit_trailer=False)
 | `model*.safetensors` | base weights (identical keys) + `handoff_probe.*` tensors |
 | `gemma_4_e2b_it_hybrid.py` | single-file `mlx-lm` model, wired via config.json's `model_file` |
 
+## Routing quality (AUROC)
+
+AUROC measures how well the probe separates wrong answers from right ones
+(higher = better, 0.5 is random, 1.0 is perfect):
+
+| Hold-out | Modality | Cactus Hybrid | Token Entropy |
+|---|---|---|---|
+| MMLU | text MCQ | **0.770** | 0.697 |
+| MMLU-Pro | text MCQ | **0.771** | 0.692 |
+| ARC-Easy | text MCQ | **0.888** | 0.655 |
+| ARC-Challenge | text MCQ | **0.834** | 0.646 |
+| GSM8K (3-shot) | text gen | **0.782** | 0.731 |
+| MMBench-EN-Dev | vision MCQ | **0.840** | 0.435 |
+| ChartQA | vision QA | **0.779** | 0.615 |
+| DocVQA | vision QA | **0.781** | 0.512 |
+| MMAU | audio MCQ | **0.789** | 0.517 |
+| GigaSpeech | audio | **0.876** | 0.343 |
+| Earnings-22 | audio | **0.839** | 0.323 |
+| LibriSpeech | audio | **0.822** | 0.427 |
+| **Mean** | | **0.814** | **0.549** |
+
+The strongest result: the probe was trained on **zero audio data**, yet achieves
+0.79–0.88 AUROC on four audio benchmarks (two transcription, one audio MCQ, one
+out-of-domain transcription). This rules out surface-level explanations: the
+probe is reading a modality-independent correctness signal from the hidden
+state, not memorizing patterns from training data.
+
 ## All formats
 
 All Cactus Hybrid builds live in the