Model card: lead with the model's value and the live-web results

README.md

@@ -23,24 +23,44 @@ datasets:
 
 # WebJev-35B-A3B
 
-**A one-pass decision model for web agents and structured decisions, 35B mixture of experts (3B active).**
+**The decision model that more than doubles what web agents achieve on real websites.**
+
+Put in charge of every step of a browser agent, WebJev-35B-A3B completes **38.5%** of 125 live-website tasks, **2.3×**
+the 16.7% of jev-1.13, with the same agent, the same tasks and the same budget. At each step it reads the page and
+decides what to do next and which element to act on, among up to 255 on-page candidates, in a single forward pass.
 
 [**Code**](https://github.com/lexmount/WebJev) · [**Dataset**](https://huggingface.co/datasets/Lexmount/WebJev) ·
 [**Training**](https://github.com/lexmount/WebJev/tree/main/train)
 
-WebJev-35B-A3B reads a *state* and a *question* with an explicit list of options, and returns a probability
-distribution over the options. It does not generate text: each question is answered with a single forward pass that
-reads the logits of the option labels. There is no decoding, no output parsing, and no answer outside the options you
-list.
+![End-to-end success of WebJev-35B-A3B and jev-1.13 on 125 live-website tasks](assets/liveweb.png)
 
-The model is built for the decision loop of browser agents:
+## Highlights
 
-- **Action prediction:** what to do next on the current page (click, type, select, scroll, wait, submit, go back,
-  done, blocked).
-- **Element grounding:** which of the page's elements to act on, among up to 255 candidates.
+- **2.3× end-to-end success on the live web.** WebJev solves 38.5% of 125 real-website tasks, against 16.7% for
+  jev-1.13. It leads on every task family:
+  - Online-Mind2Web: 38.4% against 16.9%;
+  - WebGym: 32.0% against 8.0%, four times as many;
+  - WebVoyager: 45.8% against 25.0%.
+- **Built for the agent's decision loop.** WebJev answers the two questions a browser agent faces at every step
+  directly from the page state:
+  - *action prediction*: click, type, select, scroll, wait, submit, go back, done or blocked;
+  - *element grounding*: which of up to 255 on-page elements to act on.
+
+  It is trained on tens of thousands of execution-verified decisions from live websites
+  ([Lexmount/WebJev](https://huggingface.co/datasets/Lexmount/WebJev)).
+- **Strong general decision-making.** WebJev also leads jev-1.13 on JevBench public (87.9 against 85.7) and on
+  multi-class classification (86.3 against 83.3). It is ahead on Nimble evidence checking and customer-ticket triage
+  too.
+- **Fast and exact.** Each decision is one forward pass with no decoding: a web-page decision takes about a third of a
+  second on one A100. The answer is always one of the listed options, and it comes with the full probability
+  distribution.
+
+## Model overview
 
-It also handles general typed decisions: routing, classification, extraction choices, and policy and evidence
-judgments.
+WebJev-35B-A3B reads a *state* and a *question* with an explicit list of options, and returns a probability
+distribution over the options. It does not generate text, so there is no output parsing and no answer outside the
+options you list. It also handles general typed decisions: routing, classification, extraction choices, and policy
+and evidence judgments.
 
 | | |
 |---|---|
@@ -59,50 +79,20 @@ judgments.
 
 ![WebJev-35B-A3B turns a state and typed questions into option probabilities](assets/overview.png)
 
-## Highlights
-
-- **Web agents.** Plugged into the same browser agent, WebJev-35B-A3B solves **38.5%** of 125 real-website tasks,
-  against **16.7%** for jev-1.13.
-- **General decisions.** On eight structured-decision benchmarks WebJev averages **83.7%**. It leads jev-1.13 on four
-  of them: JevBench, Nimble, customer-ticket triage and multi-class classification.
-- **One forward pass, no decoding.** Each question is answered by a single forward pass that returns a full
-  distribution over its options. The answer is always one of the options you listed.
-
 ## Evaluation
 
 All numbers are measured with the released BF16 weights served by vLLM, at temperature 1.0. The jev-1.13 numbers were
 measured through its official API on the same inputs.
 
-### Eight structured-decision benchmarks
-
-Each item gives a state and a set of candidate answers. The model's choice is correct when it equals the reference
-label. The table reports accuracy.
-
-![Accuracy of WebJev-35B-A3B and jev-1.13 on eight structured-decision benchmarks](assets/benchmarks.png)
-
-| Benchmark (items) | What it measures | jev-1.13 | **WebJev-35B-A3B** |
-|---|---|---:|---:|
-| JevBench public (231) | general structured decisions (intent, extraction, tool choice, policy); the hard tier has long policies, multi-hop, temporal and numeric reasoning, and trap items | 85.71 | **87.88** |
-| Nimble held-out (324) | fine-grained evidence checking with minimal pairs: one fact changes and the answer flips | 92.59 | **92.90** |
-| SemIf external (252) | claim verification: supported, refuted or not enough evidence | **98.41** | 98.02 |
-| Customer-ticket triage (873) | routing queue, anger and priority of support tickets (partly Korean) | 74.91 | **76.29** |
-| Cross-task transfer, dev (764) | MMLU, Emotion, TweetEval, QNLI, PAWS, SciQ and programmatic policy-rule questions | **85.21** | 85.08 |
-| Multi-class decisions, dev (1,468) | news topic, review sentiment and stars, 77-way banking intent, question and entity type, yes/no reading comprehension, entailment, policy rules | 83.31 | **86.31** |
-| MMLU-Pro, 10 options (1,000) | college-level knowledge and reasoning across subjects | **83.40** | 69.40 |
-| Typed business decisions, test (2,000) | agent-trajectory stop and escalation, customer requests, invoice approval, security-alert severity; teacher labels | **74.05** | 73.90 |
-| **Average (equal weights)** | | **84.70** | 83.72 |
-
 ### End-to-end web tasks
 
-The model serves as the decision component of the same browser agent on 125 real-website tasks: 75 from
-Online-Mind2Web, 26 from WebGym and 24 from WebVoyager. Each run has a budget of 900 seconds and 60 actions. A
-deterministic grader checks the final page state and answer.
+WebJev serves as the decision component of the same browser agent on 125 real-website tasks: 75 from Online-Mind2Web,
+26 from WebGym and 24 from WebVoyager. Each run has a budget of 900 seconds and 60 actions. A deterministic grader
+checks the final page state and answer.
 
 - **Success rate** = solved ÷ evaluable tasks. Tasks lost to browser infrastructure or grader errors are excluded.
 - **Strict rate** counts all 125 tasks.
 
-![End-to-end success of WebJev-35B-A3B and jev-1.13 on 125 live-website tasks](assets/liveweb.png)
-
 | Task set | jev-1.13 | **WebJev-35B-A3B** |
 |---|---:|---:|
 | All tasks | 16.67% (20/120) | **38.52% (47/122)** |
@@ -111,8 +101,27 @@ deterministic grader checks the final page state and answer.
 | WebVoyager | 25.00% (6/24) | **45.83% (11/24)** |
 | Strict, all 125 tasks | 16.00% | **37.60%** |
 
-Live websites change between runs, and one task is worth about 0.8 points. Treat small differences as noise; the gap
-between the two models is well beyond it.
+The 95% confidence intervals of the two models do not overlap (see the figure at the top). Live websites change
+between runs, and one task is worth about 0.8 points.
+
+### General structured decisions
+
+Eight benchmarks of structured decisions, beyond the web. Each item gives a state and a set of candidate answers, and
+the model's choice is correct when it equals the reference label. The table reports accuracy.
+
+![Accuracy of WebJev-35B-A3B and jev-1.13 on eight structured-decision benchmarks](assets/benchmarks.png)
+
+| Benchmark (items) | What it measures | jev-1.13 | **WebJev-35B-A3B** |
+|---|---|---:|---:|
+| JevBench public (231) | general structured decisions (intent, extraction, tool choice, policy); the hard tier has long policies, multi-hop, temporal and numeric reasoning, and trap items | 85.71 | **87.88** |
+| Multi-class decisions, dev (1,468) | news topic, review sentiment and stars, 77-way banking intent, question and entity type, yes/no reading comprehension, entailment, policy rules | 83.31 | **86.31** |
+| Customer-ticket triage (873) | routing queue, anger and priority of support tickets (partly Korean) | 74.91 | **76.29** |
+| Nimble held-out (324) | fine-grained evidence checking with minimal pairs: one fact changes and the answer flips | 92.59 | **92.90** |
+| SemIf external (252) | claim verification: supported, refuted or not enough evidence | **98.41** | 98.02 |
+| Cross-task transfer, dev (764) | MMLU, Emotion, TweetEval, QNLI, PAWS, SciQ and programmatic policy-rule questions | **85.21** | 85.08 |
+| Typed business decisions, test (2,000) | agent-trajectory stop and escalation, customer requests, invoice approval, security-alert severity; teacher labels | **74.05** | 73.90 |
+| MMLU-Pro, 10 options (1,000) | college-level knowledge and reasoning across subjects | **83.40** | 69.40 |
+| **Average (equal weights)** | | **84.70** | 83.72 |
 
 ### Speed