card: the problem film, tested on the live model (178 requests with known answers), model page link

README.md

@@ -12,7 +12,26 @@ tags: ["decision-model", "calibration", "conformal-prediction", "uncertainty", "
 
 An English decision model with the Jev API. You send a state and typed questions (Choice, Score, Noul) and get a calibrated probability for every option. It can think before it answers, it can say "unknown", and the Q4_K_M build runs on your own machine in about 3 GB of RAM.
 
-**[Try it](#try-it) · [Run it](#run-it) · [Results](#results) · [Use and limits](#use-and-limits) · [Code and links](#code-and-links)**
+**[Try it](#try-it) · [Model page](https://jevalt.mertkayacs.com/models/deem-4b/) · [Run it](#run-it) · [Results](#results) · [Use and limits](#use-and-limits) · [Code and links](#code-and-links)**
+
+## What it fixes
+
+<video controls playsinline preload="metadata" poster="https://huggingface.co/datasets/mertkayacs/emberwick-videos/resolve/main/problems/jevalt-problems-en.jpg" src="https://huggingface.co/datasets/mertkayacs/emberwick-videos/resolve/main/problems/jevalt-problems-en-1080p.mp4"></video>
+
+The 103-second film, sound on. Also in [Türkçe](https://huggingface.co/datasets/mertkayacs/emberwick-videos/resolve/main/problems/jevalt-problems-tr-1080p.mp4) and [Deutsch](https://huggingface.co/datasets/mertkayacs/emberwick-videos/resolve/main/problems/jevalt-problems-de-1080p.mp4).
+
+## Tested on the live model
+
+We sent Deem-4B 178 requests in English with known answers on 4 October 2026; every request and answer is in [results/tested](https://huggingface.co/datasets/mertkayacs/jevalt-bench/tree/main/results/tested).
+
+| Case | What was sent | Result |
+|---|---|---|
+| Planted instructions | 30 phishing emails, each with a different planted line, plus the same 10 without it | 26 of 30 quarantined; 10 of 10 without the line |
+| Long policies | 20 customers against one six-rule return policy | 17 of 20 matched the answer computed from the rules |
+| Negations | 15 short facts, each asked plain and negated | 29 of 30 correct |
+| Missing facts | 10 situations without the deciding fact, plus the same 10 with it | answered `unknown` in 10 of 10; 10 of 10 correct with the fact |
+| Option order | 2 support tickets, each with the options in all 24 orders | the same answer in 48 of 48 orders |
+| Casual messages | 20 casual messages written in English, with typos and slang | 20 of 20 routed to the right team |
 
 ![Emberwick: every villager asks Deem-4B what to do next.](https://huggingface.co/datasets/mertkayacs/emberwick-videos/resolve/main/gifs/emberwick-en.gif)
 
@@ -75,14 +94,14 @@ Same items and client for every model, each as shipped: [Kev-4B](https://hugging
 
 ![What Jev 1.13 lacks and JevAlt has: thinking when unsure, coverage sets, models made for Turkish and German, open weights](https://huggingface.co/mertkayacs/Deem-4B/resolve/main/assets/jev.png)
 
-The three test splits went through the same pipeline as the training rows, so they measure what the training aimed at. On the English split Deem-4B gains 4.4 accuracy points (paired bootstrap, 95% interval +3.4 to +5.5) and lowers Brier by 0.075; the Turkish and German splits move by +4.8 and +12.1 points. On JevBench-hard, TurkishMMLU and GermEval, which the training never saw, and on the typed-decisions test split (its train split was in the mix), accuracy does not change significantly, and Brier gets slightly worse on typed-decisions (+0.013) and 10kGNAD (+0.047). With `reasoning: "auto"` the English date, number and policy test rows go from 0.761 to 0.769 accuracy, an interval that touches zero.
+The three test splits went through the same pipeline as the training rows, so they measure what the training aimed at. On the English split Deem-4B gains 4.4 accuracy points (paired bootstrap, 95% interval +3.5 to +5.4) and lowers Brier by 0.075; the Turkish and German splits move by +5.1 and +11.5 points. On JevBench-hard, TurkishMMLU and GermEval, which the training never saw, and on the typed-decisions test split (its train split was in the mix), accuracy does not change significantly, and Brier gets slightly worse on typed-decisions (+0.013) and 10kGNAD (+0.047). With `reasoning: "auto"` the English date, number and policy test rows go from 0.761 to 0.769 accuracy, an interval that touches zero.
 
 </details>
 
 <details>
 <summary><b>How we fixed each problem</b></summary>
 
-Most fixes are a set of training rows aimed at one weak spot. Every number compares a model with its start checkpoint, Intern-Decision-4B, on rows held out from training. Across all of it, Deem-4B's held-out accuracy in English rose from 90.2% to 94.7%.
+Most fixes are a set of training rows aimed at one weak spot. Every number compares a model with its start checkpoint, Intern-Decision-4B, on rows held out from training. Across all of it, Deem-4B's held-out accuracy in English rose from 90.3% to 94.7%.
 
 - **The data.** About 23,900 training rows in English, Turkish and German. Public sets with known answers (MASSIVE, Open-Jev, PAWS-X, typed-decisions); everyday situations written directly in each language by other open models; requests from the Emberwick game; and the fix sets below. Two teacher models from labs other than the writer give every written row a probability per option, and an answer counts only when both teachers and the writer agree on it. Those probabilities, the soft labels, are what the models learn. Test rows were split off by group, and their checksums recorded, before the final training runs.
 - **Hidden instructions.** A fix set of 827 rows hides a hostile line in the text (an order to the AI filter, a fake rule) at the start, the middle or the end, with the right answer unchanged. On our probe, hidden lines now change 14.0% of Deem-4B's answers; the start checkpoint follows 41.5% of them. Held-out rows of this kind: 80.8% → 90.1%. Our target is under 10%.
@@ -147,5 +166,6 @@ Most fixes are a set of training rows aimed at one weak spot. Every number compa
 - Try it online: [Space](https://huggingface.co/spaces/mertkayacs/JevAlt)
 - The village game: [Emberwick](https://emberwick.mertkayacs.com)
 - Project site: [jevalt.mertkayacs.com](https://jevalt.mertkayacs.com)
+- This model's page: [jevalt.mertkayacs.com/models/deem-4b](https://jevalt.mertkayacs.com/models/deem-4b/)
 
 If this is useful to you, a star on [GitHub](https://github.com/mertkayacs/jevalt) helps other people find it.