Single model: v15s (clean retrain, suite B 100 %); remove v10, v10s, v11s, v14s (kept in history)

.gitattributes

@@ -38,3 +38,4 @@ assets/laya_browser_demo.gif filter=lfs diff=lfs merge=lfs -text
 assets/laya_browser_demo.mp4 filter=lfs diff=lfs merge=lfs -text
 v11s/tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 v14s/tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+v15s/tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -10,182 +10,120 @@ tags:
 - browser-agent
 - web-navigation
 - decision-model
-- modernbert
+- mmbert
 - mind2web
 - tilelang
 datasets:
 - osunlp/Mind2Web
+- stanfordnlp/nnetnav-live
 ---
 
 # laya-browser — laya fine-tuned as a browser-agent decision head (drop-in replacement for TypeSafe Jev)
 
 ![laya driving a real browser: 6 tasks, 12 decisions, median 22 ms per decision](assets/laya_browser_demo.gif)
 
-*46-second demo ([mp4](assets/laya_browser_demo.mp4)): v10s drives headless Chromium through jev-ultrafast, one encoder pass per step; recorded with `code/apps/make_demo.py`.*
-
 **laya** ([convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya)) is a non-autoregressive "System 1" decision model:
-one bidirectional encoder pass answers several typed questions (`choice` / `score` / `noul`) with calibrated probabilities, no text generation.
-Out of the box it is near chance at browser decisions ("which element should I click for this goal?" — top-1 0.10 among ~45 candidates).
-
-This repo is what it took to turn it into a usable decision head for [browser-use/jev-ultrafast](https://github.com/browser-use/jev-ultrafast),
-whose `/v1/systemone` request format is identical to laya's `predict(state, questions)`. Everything was done locally on one RTX 4070 Ti SUPER (16 GB),
-no paid API: the text helper and the DAgger teacher are a local Qwen3-8B-AWQ served by sglang.
-
-## Progress update (2026-09-26): v12s–v14s, a held-out suite, and a data-leak correction
-
-**New checkpoint `v14s/`** (mmBERT-base 322M, format v3, run the server with a 60-option chunk threshold:
-`python apps/systemone_server.py 8791 /path/to/v14s 60`).
-
-| | v10s | v12s | v13s | **v14s** |
-|---|---|---|---|---|
-| **Suite B: 18 tasks on 18 sites that appear in no training source** (×3, `code/apps/browser_suite_b.py`) | – | 94 % (34/36, ×2) | 93 % (50/54) | **100 % (54/54)** |
-| Suite A: the original 16 tasks (×3, patched harness) — *contaminated, see below* | 62 % | 75 % | 75 % | 81 % (39/48) |
-| Held-out long-horizon synthetic tasks (flight / hotel / shop, 8 each) | – | – | – | 0 / 0 / 50 % |
-
-Suite B checks the held-out claim at start-up against `results/round3/train_domains.json` (every domain of every training source,
-682 domains). Its tasks are mostly 1–2 steps: navigation, site search, man pages, RFC / package search, a shop's search, page 2 and
-sort-by-price. Suite A still fails `books-page2` (0/3), Google Flights (0/3) and is flaky on two search tasks.
-
-**Data leak found and removed.** `dagger_cases.jsonl` (the "177 on-policy DAgger corrections" in the table below, 331 cases by
-now) came from running the Qwen teacher on the suite itself: 152 of its cases are suite-A goals, often with the teacher's *wrong*
-label (e.g. "Go to page 2 of the catalogue" → click the first book), oversampled ×5. Every checkpoint from v10 to v14s was trained on
-it, so **all suite-A numbers in this README, including v10s's 62 %, are contaminated** (and `books-page2` failing 0/3 is partly
-the model faithfully reproducing a wrong label). Suite B is clean. From v15s on, `build_items.py` drops every DAgger case and every
-case recorded on a suite A/B start page.
-
-**What changed in the harness** (`code/jev-ultrafast.patch`, applies to jev-ultrafast `1231850`). Half of the original suite
-failures were harness bugs, not model errors:
-- a `PRESS_ENTER` control, offered while a non-empty text field is focused (arXiv's search overlay has no submit button);
-- elements covered by an unrelated element (modals, overlays, banners) are no longer offered; a target covered by its own ancestor
-  is clicked through;
-- observation retries while the document is navigating; a choice that fails 3× is excluded, as is the same action repeated on the
-  same URL (reload loops);
-- an optional sub-goal planner (`JEV_PLANNER=1`: the text model splits the task once, DONE advances to the next sub-goal, a
-  text-model check rejects a sub-goal DONE the page does not show).
-Server side: wide choices are split into chunks above 60 options (a 160-link page gave each label ~4 tokens and a flat
-distribution), and a one-option choice no longer crashes the fast path.
-
-**What changed in the data** (all collected in the real browser, no suite pages):
-- `rollouts2.py`: scripted search → `PRESS_ENTER`/submit → DONE, "open the article about X", scroll-to-next-page, scroll-then-click,
-  `<select>` with intent-style goals ("Sort by price: low to high."), type-then-pick-a-suggestion; 2.3k cases on 206 sites.
-- `convert_nnetnav.py`: 8.9k steps from [stanfordnlp/nnetnav-live](https://huggingface.co/datasets/stanfordnlp/nnetnav-live)
-  (element tables capped at 45, held out by site).
-- target lists sampled to the server's chunk width (≤ 40) and final-round width (2–6) so labels are never truncated in training.
-
-**Where it breaks: long horizons.** On held-out synthetic forms (below) v14s ends flight and hotel searches after 1–2 steps (DONE
-too early) or loops. Jev's own Google Flights run needs 11 actions: one-way, type origin → pick the suggestion, type destination →
-pick the suggestion, open the calendar, pick the day, Done, Search.
-
-**In progress: v15s** (clean retrain from the base, no DAgger, no suite pages) adds:
-- `code/finetune/webgym/`: a local synthetic web environment (flight / hotel search forms, shop listings) with randomized widget
-  implementations — autocomplete comboboxes inline or behind a trigger button, month-grid date pickers with Done/Apply, radio /
-  segmented / native / custom dropdowns, steppers and popups, cookie banners, forms pushed below the fold. A scripted expert
-  reads the page's own state, acts through jev's observations (so the format is exactly what the agent sees), and records each
-  step twice: under the full task and under the current sub-goal, plus a *verified* sub-goal DONE whenever a sub-goal is
-  satisfied. 12 % harmless detours teach recovery (including going back after a wrong submission). 1,408 successful episodes,
-  32k cases; seeds `900000 + 10·i` are held out for `eval_gym.py`.
-- `annotate_subgoals.py`: Mind2Web trajectories segmented into planner-style sub-goals by the local Qwen
-  (trajectory-conditioned, Plan-and-Act style), 5.1k sub-goal-conditioned steps. Sub-goal DONE labels come only from webgym,
-  where they are checked against page state.
-Results will be added here when the run finishes.
-
-## What changed relative to the original laya
-
-| | original laya (typed-decisions) | this repo |
-|---|---|---|
-| browser decision quality (16 real tasks × 3 runs, corrected checks) | 0 % | **v10s: 62 %**, v11s: 56 %, v10: 50 % (10 tasks pass 3/3, 6 fail 3/3; see below) |
-| element top-1 on held-out pages (2,734 decisions, ~45 candidates) | 0.10 | **0.66** (v10), 0.63 (v10s), 0.62 (v11s) |
-| operation accuracy (CLICK / TYPE_TEXT / SELECT / DONE) | 0.54 | 0.88–0.89 |
-| latency per browser step (3 questions, 30–65 candidates) | 50–200 ms | 41–50 ms (v10), **17–23 ms** (v10s / v11s) |
-| backbone | ModernBERT-large 421M | v10: same; v10s / v11s: mmBERT-base 322M |
-| input format | jev's state verbatim (element table as JSON inside the state, truncated by the 1024-token window) | **format v2/v3**: elements live only in the option list (full label + role + current value), state keeps title / URL / history / 1.2–1.5k chars of text, `head_max_len` 512 → 768 |
-| training data | LocalLLaMA/typed-decisions | 5,244 reverse-generated goals on 421 crawled pages (Qwen writes "the goal a user would state to need this element"), 700 real DONE states (clicks actually executed), 659 step-2 negatives, [Mind2Web](https://huggingface.co/datasets/osunlp/Mind2Web) train (7,296 steps, candidates re-rendered as an element table), 177 on-policy DAgger corrections (later found to contain suite-A tasks, see the progress update) |
-| training | — | laya's RLCD recipe (noisy-logit policy gradient + soft CE), single GPU, no gradient checkpointing, 4 epochs (~2 h for v10, ~1 h for v10s), post-hoc temperature |
-| inference | HF eager + autocast | optional TileLang fast path ([PR #25 to laya](https://github.com/NandhaKishorM/laya/pull/25)): fused GEMM/GEGLU/LayerNorm/RoPE, sliding-window flash attention, bf16-resident weights, CUDA graphs — 4–5× lower per-call latency, identical answers |
-
-### Things that did **not** work (so you don't repeat them)
-- Templated DONE goals ("Open the page titled X, stop once it is open") leak phrasing: the model learns *stop when ⇒ DONE*. DONE samples must be real landing pages after an executed action.
-- If every DONE sample has exactly one prior action and every click sample has none, the model learns *any history ⇒ DONE*. Add mid-task negatives (step-2 goals on landing pages).
-- Mind2Web alone kills DONE / TYPE_TEXT (no DONE there, CLICK dominates): re-weight rare operations (DONE ×4, TYPE_TEXT/SELECT ×3).
-- Cutting page text to 3,000 chars saved nothing (the sequence is dominated by the head) and cost 0.04 top-1.
-- `torch.compile` on variable-length batches recompiles per shape: 6× slower.
-- Confidence-gated escalation to Qwen3-8B (System 2) made things *worse* (58 % → 42 %): on these pages the fine-tuned 322M/421M model is a better decider than an 8B general LLM. Use a stronger System 2 or none.
-- jev's DOM reader hides password fields by design (login tasks are impossible) and never sees collapsed menus (Wikipedia's "Random article").
-
-### What still fails
-The suite is bimodal: 10 tasks pass 3/3 (category / tab / page navigation, checkbox, `<select>`, HN pages, DuckDuckGo search in some runs) and 6 fail 3/3:
-"type then submit / pick a suggestion" flows (Wikipedia search ×2, arXiv), pagination that needs a scroll first (the model clicks the first
-visible item instead), and Google Flights. v11s added 682 scripted scroll / search-submit / select trajectories (`code/finetune/rollouts.py`):
-SELECT and DuckDuckGo improved, the scroll case did not — run-to-run variance on live sites (HN front page changes, DDG 50x pages) is larger
-than the v10s ↔ v11s difference, so treat the two as equivalent.
-
-### Teachers vs the fine-tuned student (80 held-out decisions, same element tables)
-| decider | op acc | target top-1 | s / decision |
-|---|---|---|---|
-| Ternary-Bonsai-2-27B (local, thinking off) | 0.825 | 0.554 | 1.56 |
-| Bonsai-27B, thinking budget 300 tokens | 0.861 | 0.603 | 4.7 |
-| Bonsai-27B, thinking unrestricted | 0.625 | 0.474 | 11 |
-| Qwen3-8B-AWQ (35 % of requests failed, survivors only) | 0.904 | 0.617 | 0.38 |
-| **laya v11s (322M, this repo; full 2,734-case set)** | **0.890** | **0.623** | **0.021** |
-
-A 27B general model with a short thinking budget matches the 322M fine-tuned student at 200× the latency; neither 8B nor 27B is a useful
-DAgger teacher or System-2 fallback here. Further gains need a stronger teacher or more targeted trajectories.
-
-## Files
-
-```
-v14s/    mmBERT-base 322M,      format v3, head_max_len 768   (best: suite B 100 %; serve with chunk threshold 60)
-v10/     ModernBERT-large 421M, format v2, head_max_len 768   (best held-out top-1)
-v10s/    mmBERT-base 322M,      format v3, head_max_len 768   (17–23 ms per step; best on the live suite)
-v11s/    v10s data + 682 scripted scroll / search-submit / select trajectories (equivalent to v10s within noise)
-code/    finetune pipeline, laya systemone server, task suite, TileLang kernels, jev-ultrafast patch
-results/ per-run suite JSONs and logs behind every number above (round3/: v12s–v14s, suite B, webgym baseline)
-```
-Each checkpoint is a laya checkpoint directory (`model.safetensors`, `encoder/`, `tokenizer/`, `rl_agent_config.json`); the config records
-`laya_fmt` and `head_max_len_train` so the server applies the matching input format automatically.
+one bidirectional encoder pass answers several typed questions (`choice` / `score` / `noul`) with calibrated probabilities, no text
+generation. Out of the box it is near chance at browser decisions ("which element should I click for this goal?").
+
+This repo fine-tunes it into the decision head of [browser-use/jev-ultrafast](https://github.com/browser-use/jev-ultrafast), whose
+`/v1/systemone` request is exactly laya's `predict(state, questions)`: every step, one ~20 ms forward pass picks the operation
+(CLICK / TYPE_TEXT / SELECT / PRESS_ENTER / SCROLL / DONE …) and its target element. Everything was trained and evaluated locally on
+one RTX 4070 Ti SUPER (16 GB), no paid API; a local Qwen3-8B-AWQ only writes the text that TYPE_TEXT types.
+
+**One model, `v15s/`.** Earlier checkpoints (v10, v10s, v11s, v14s) were removed from the main branch; they remain in the commit
+history. This repo is updated only when a new version is clearly better.
+
+## Results (v15s, mmBERT-base 322M, 17–23 ms per step)
+
+| evaluation | result |
+|---|---|
+| **Suite B: 18 tasks on 18 sites that appear in no training source**, ×3 runs (`code/apps/browser_suite_b.py`) | **100 % (54/54)** |
+| Suite A: the original 16 real-site tasks, ×3 (`code/apps/browser_suite.py`) | 81 % (39/48) |
+| Held-out decisions (6,268: live pages, Mind2Web, NNetNav): operation / target top-1 | 0.757 / 0.593 |
+| Held-out synthetic long-horizon forms (webgym, 10 each): flight / hotel / shop | 2/10 · 0/10 · 8/10 |
+
+- **Suite B** is the headline number. At start-up it checks that none of its domains appear among the 682 domains of every
+  training source (`results/round3/train_domains.json`). Its tasks are mostly one to three steps: navigation, site search, man
+  pages, RFC / package search, a shop's search, page 2, sort by price.
+- **Suite A** (the older suite) still fails `books-page2` (the "next" link is several scrolls down), `internet-dropdown` and Google Flights.
+- **Long multi-field forms are the open problem.** The model now plans the right sequence (trip type, origin with its
+  autocomplete suggestion, passengers popup, submit, "Modify search" after a wrong submission) but confuses a date field labelled
+  "Departure" with the origin, and runs out of steps. See "What still fails".
 
 ## Use
 
 ```bash
 huggingface-cli download cklxx/laya-browser --local-dir laya-browser
 cd laya-browser/code && uv sync --extra fast     # pinned uv.lock (Python 3.12, torch 2.11, tilelang 0.1.14)
-uv run python verify.py v10s                     # downloads v10s if needed, answers one recorded browser step
-uv run python verify.py v10s --fast              # same through the TileLang fast path
-```
-Verified from a clean environment on 2026-09-21 (RTX 4070 Ti SUPER): `TYPE_TEXT → [2] Search Wikipedia (searchbox)`, 35 ms per step
-stock / 28 ms with the fast path on a 65-option, 2.5k-token step. Extras: `--extra data` (Mind2Web conversion, dataset eval),
-`--extra browser` (live suite / crawling; also needs jev-ultrafast with `code/jev-ultrafast.patch` applied and a Chromium with
-`--remote-debugging-port=9222`).
-
-```python
-import laya
-agent = laya.load("laya-browser/v10s")                 # a local laya checkpoint dir
-agent.cfg["head_max_len"] = agent.cfg["head_max_len_train"]
-# state / questions exactly as jev-ultrafast's model.choose() builds them, after the format-v3 transform in code/apps/systemone_server.py
-result = agent.predict(state, questions)
+uv run python verify.py                          # downloads v15s, answers one recorded browser step
+uv run python verify.py --fast                   # same through the TileLang fast path
 ```
 
-As a TypeSafe replacement for jev-ultrafast:
+As a TypeSafe replacement for jev-ultrafast (apply `code/jev-ultrafast.patch` to jev-ultrafast `1231850`):
 
 ```bash
-# in code/: laya systemone-compatible server (format transform + optional gating + DAgger logging)
-python apps/systemone_server.py 8791 /path/to/laya-browser/v10s 999
-# in jev-ultrafast (apply code/jev-ultrafast.patch): TYPESAFE_BASE_URL=http://127.0.0.1:8791
+python code/apps/systemone_server.py 8791 /path/to/laya-browser/v15s 60   # 60 = split choices wider than 60 options
+# jev-ultrafast: TYPESAFE_BASE_URL=http://127.0.0.1:8791
 ```
 
-`code/apps/browser_suite.py` runs the 16-task real-browser suite with automatic outcome checks (`REPEATS=3`).
+The checkpoint records `laya_fmt` (v3) and `head_max_len_train` (768); the server applies the matching input format.
+The chunk threshold matters: without it a 160-link page (Hacker News) leaves each option ~4 tokens and the choice becomes a coin
+toss.
+
+## What went into v15s
+
+**Harness fixes** (`code/jev-ultrafast.patch`). Half of the original failures were harness bugs, not model errors:
+a `PRESS_ENTER` control while a filled text field is focused (arXiv's search overlay has no submit button); elements covered by an
+unrelated element are not offered, a target covered by its own ancestor is clicked through; observation retries while a page is
+navigating; a choice that fails 3× or repeats on the same URL is excluded.
 
-## Reproduce
+**Training data** (clean retrain from the mmBERT-base checkpoint; no suite start page and no DAgger data):
+- 421 crawled pages with reverse-generated goals, 700 real DONE states, step-2 negatives;
+- [Mind2Web](https://huggingface.co/datasets/osunlp/Mind2Web) train (7.3k steps) plus 5.1k steps re-labelled with planner-style
+  sub-goals by a local Qwen (trajectory-conditioned, Plan-and-Act style);
+- 8.9k steps from [NNetNav-live](https://huggingface.co/datasets/stanfordnlp/nnetnav-live);
+- 2.3k scripted real-browser trajectories on 206 sites (search → Enter, open an article, scroll to page 2, `<select>` with
+  intent-style goals, pick an autocomplete suggestion);
+- 32k steps from **webgym** (`code/finetune/webgym/`): a local synthetic environment of flight / hotel search forms and shop
+  listings with randomized widgets (autocomplete inline or behind a trigger, calendars with month navigation and Done/Apply,
+  radio / segmented / native / custom dropdowns, steppers, popups, cookie banners, forms below the fold). A scripted expert acts
+  through jev's own observations and records each step under the full task and under its current sub-goal, plus a sub-goal DONE
+  verified against page state; 12 % harmless detours teach recovery.
 
-`code/finetune/README.md` documents every step (crawl → reverse-generate goals → execute clicks for DONE → step-2 → Mind2Web conversion →
-DAgger → build → train → calibrate → eval → suite) with the exact scripts (`run_v10.sh`, `run_v10s.sh`, `run_final.sh`) and all intermediate
-numbers from v1 to v10s.
+Recipe: laya's RLCD (noisy-logit policy gradient + soft CE), 134k items, 3 epochs (~4 h), post-hoc temperature.
 
-## GPU cost
+## What still fails
 
-v10s: ~0.65 GB weights, ~1.5 GB VRAM resident with CUDA graphs, 17–23 ms per 3-question browser step, 3 ms for a single-question call.
-v10: ~0.85 GB weights, ~1.8 GB VRAM, 41–50 ms per step. The Qwen text helper (only needed for TYPE_TEXT values) is separate.
+- Long forms: "Departure" (a date) vs origin; the 60-step budget runs out after a correction.
+- Pages where the target is several screens down behind many distracting links (`books-page2`).
+- Sites that block headless Chromium (DuckDuckGo, Bing, most airline and hotel sites) cannot be used at all.
+- jev's DOM reader hides password fields by design, so logins are impossible.
+
+Things that did not help: confidence-gated escalation to Qwen3-8B (worse: the fine-tuned model is the better decider on these
+pages), a run-time sub-goal planner on top of v15s (20 % vs 33 % on the synthetic forms), torch.compile on variable shapes.
+
+## Changelog
+
+| version | date | change |
+|---|---|---|
+| v10s | 2026-09-21 | first usable model: format v3, 17–23 ms per step |
+| v14s | 2026-09-26 | harness fixes, Enter / scroll / select data, NNetNav; suite B 100 % |
+| **v15s** | 2026-09-27 | clean retrain without the DAgger set; long-horizon webgym data and sub-goal labels |
+
+Correction: the DAgger set used from v10 to v14s came from running a Qwen teacher on suite A itself (152 of its 331 cases are suite-A
+goals, often with wrong labels), so the suite-A numbers published for those versions were contaminated. v15s is trained without it;
+suite B was never affected.
+
+## Files
+
+```
+v15s/     the model (laya checkpoint dir: model.safetensors, encoder/, tokenizer/, rl_agent_config.json)
+code/     server, suites, finetune pipeline (incl. webgym), TileLang kernels, jev-ultrafast patch, verify.py
+results/  suite JSONs and logs behind the numbers above (results/v15s/, results/round3/)
+assets/   demo video
+```
 
 ## License
 
-Apache-2.0, same as laya. Mind2Web is used under its own license for training only.
+Apache-2.0, same as laya. Mind2Web and NNetNav are used under their own licenses for training only.

code/verify.py

@@ -1,12 +1,12 @@
 """Smoke test: download a checkpoint from the Hub (or use a local dir), load it, answer one recorded browser step.
 
-    python verify.py [v10s|v10|/path/to/checkpoint] [--fast]
+    python verify.py [v15s|/path/to/checkpoint] [--fast]
 """
 import json, os, sys, time
 os.environ.setdefault("USE_TF", "0"); os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
 import torch, laya
 
-which = next((a for a in sys.argv[1:] if not a.startswith("--")), "v10s")
+which = next((a for a in sys.argv[1:] if not a.startswith("--")), "v15s")
 if os.path.isdir(which):
     ck = which
 else:

results/v15s/gym_eval_v15s.json

@@ -0,0 +1,392 @@
+[
+ {
+  "kind": "flight",
+  "seed": 900000,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.4,
+  "url": "http://127.0.0.1:8811/t/flight/900000",
+  "goal": "Show me a return flight from Oslo to Berlin on October 18, returning 23 October 2026 for 3 adults in premium economy. Stop when matching flight options are visible.",
+  "plan": [
+   "Show me a return flight from Oslo to Berlin on October 18, returning 23 October 2026 for 3 adults in premium economy. Stop when matching flight options are visible."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900010,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.6,
+  "url": "http://127.0.0.1:8811/t/flight/900010",
+  "goal": "Look up round-trip flights from Tokyo to Amsterdam on October 13, returning 20 October 2026 for two adults in first class.",
+  "plan": [
+   "Look up round-trip flights from Tokyo to Amsterdam on October 13, returning 20 October 2026 for two adults in first class."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900020,
+  "ok": true,
+  "steps": 12,
+  "status": "done",
+  "wall": 2.7,
+  "url": "http://127.0.0.1:8811/t/flight/900020/results?trip=oneway&origin=rom&destination=sel-ap&date=2026-10-15&adults=2&cabin=economy",
+  "goal": "Search for one-way flights from Roma to Seoul on 15 October 2026 for two adults. Stop when matching flight options are visible.",
+  "plan": [
+   "Search for one-way flights from Roma to Seoul on 15 October 2026 for two adults. Stop when matching flight options are visible."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900030,
+  "ok": true,
+  "steps": 17,
+  "status": "done",
+  "wall": 3.1,
+  "url": "http://127.0.0.1:8811/t/flight/900030/results?trip=oneway&origin=hel&destination=muc&date=2026-10-26&adults=4&cabin=business",
+  "goal": "Show me a one-way flight from Helsinki to Munich on 26 October 2026 for four adults in business class.",
+  "plan": [
+   "Show me a one-way flight from Helsinki to Munich on 26 October 2026 for four adults in business class."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900040,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.4,
+  "url": "http://127.0.0.1:8811/t/flight/900040",
+  "goal": "Show me one way flights from Manchester to Bangkok on November 21 for 3 adults in business class.",
+  "plan": [
+   "Show me one way flights from Manchester to Bangkok on November 21 for 3 adults in business class."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900050,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.5,
+  "url": "http://127.0.0.1:8811/t/flight/900050",
+  "goal": "Show me a one-way flight from Amsterdam to Barcelona on November 8, 2026 for 1 adult in economy.",
+  "plan": [
+   "Show me a one-way flight from Amsterdam to Barcelona on November 8, 2026 for 1 adult in economy."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900060,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.8,
+  "url": "http://127.0.0.1:8811/t/flight/900060",
+  "goal": "Find one-way flights from Seoul to Copenhagen on 10 October 2026 for two adults in business class.",
+  "plan": [
+   "Find one-way flights from Seoul to Copenhagen on 10 October 2026 for two adults in business class."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900070,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.3,
+  "url": "http://127.0.0.1:8811/t/flight/900070",
+  "goal": "Look up a return flight from Barcelona to New York on November 13, returning November 27 for 1 adult. Stop when matching flight options are visible.",
+  "plan": [
+   "Look up a return flight from Barcelona to New York on November 13, returning November 27 for 1 adult. Stop when matching flight options are visible."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900080,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.7,
+  "url": "http://127.0.0.1:8811/t/flight/900080",
+  "goal": "Show me a one-way flight from Miami to Toronto on November 15 for one adult.",
+  "plan": [
+   "Show me a one-way flight from Miami to Toronto on November 15 for one adult."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900090,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.3,
+  "url": "http://127.0.0.1:8811/t/flight/900090",
+  "goal": "Find one-way flights from Barcelona to Bangkok on 4 October 2026 for two adults in first class. Show the results.",
+  "plan": [
+   "Find one-way flights from Barcelona to Bangkok on 4 October 2026 for two adults in first class. Show the results."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900000,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.1,
+  "url": "http://127.0.0.1:8811/t/hotel/900000",
+  "goal": "Find a hotel in Toronto from 31 October 2026 to November 2, sorted by price: low to high.",
+  "plan": [
+   "Find a hotel in Toronto from 31 October 2026 to November 2, sorted by price: low to high."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900010,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.6,
+  "url": "http://127.0.0.1:8811/t/hotel/900010",
+  "goal": "Search for hotels in Chicago from 30 October 2026 to November 1 for three guests, sorted by price: low to high.",
+  "plan": [
+   "Search for hotels in Chicago from 30 October 2026 to November 1 for three guests, sorted by price: low to high."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900020,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.4,
+  "url": "http://127.0.0.1:8811/t/hotel/900020",
+  "goal": "Find a hotel in Oslo from 30 October 2026 to November 3, 2026 for one guest, only showing pet friendly.",
+  "plan": [
+   "Find a hotel in Oslo from 30 October 2026 to November 3, 2026 for one guest, only showing pet friendly."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900030,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.5,
+  "url": "http://127.0.0.1:8811/t/hotel/900030",
+  "goal": "Find stays in Toronto from November 16, 2026 to November 21, 2026, filtered to pet friendly.",
+  "plan": [
+   "Find stays in Toronto from November 16, 2026 to November 21, 2026, filtered to pet friendly."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900040,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.2,
+  "url": "http://127.0.0.1:8811/t/hotel/900040",
+  "goal": "Search for hotels in Tokyo from 14 November 2026 to 15 November 2026, with pet friendly.",
+  "plan": [
+   "Search for hotels in Tokyo from 14 November 2026 to 15 November 2026, with pet friendly."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900050,
+  "ok": false,
+  "steps": 15,
+  "status": "done",
+  "wall": 2.6,
+  "url": "http://127.0.0.1:8811/t/hotel/900050/results?dest=cph&cin=2026-11-07&cout=2026-12-04&guests=1&sort=Guest+rating",
+  "goal": "Find stays in Copenhagen from December 4 to December 7, 2026 for one guest, sorted by guest rating.",
+  "plan": [
+   "Find stays in Copenhagen from December 4 to December 7, 2026 for one guest, sorted by guest rating."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900060,
+  "ok": false,
+  "steps": 25,
+  "status": "done",
+  "wall": 3.6,
+  "url": "http://127.0.0.1:8811/t/hotel/900060/results?dest=rom&cin=2026-11-03&cout=2027-06-10&guests=4",
+  "goal": "Find stays in Rome from November 3, 2026 to 10 November 2026 for 4 guests.",
+  "plan": [
+   "Find stays in Rome from November 3, 2026 to 10 November 2026 for 4 guests."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900070,
+  "ok": false,
+  "steps": 11,
+  "status": "done",
+  "wall": 2.1,
+  "url": "http://127.0.0.1:8811/t/hotel/900070/results?cin=2026-09-27&guests=2&filter=Free+cancellation",
+  "goal": "Find a hotel in Chicago from 27 September 2026 to October 4, only showing free cancellation.",
+  "plan": [
+   "Find a hotel in Chicago from 27 September 2026 to October 4, only showing free cancellation."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900080,
+  "ok": false,
+  "steps": 26,
+  "status": "done",
+  "wall": 3.8,
+  "url": "http://127.0.0.1:8811/t/hotel/900080/results?dest=nyc&cin=2026-11-16&cout=2027-06-20&guests=2&sort=Guest+rating",
+  "goal": "Look for a place to stay in New York from November 16, 2026 to 20 November 2026 for 2 adults, sorted by guest rating.",
+  "plan": [
+   "Look for a place to stay in New York from November 16, 2026 to 20 November 2026 for 2 adults, sorted by guest rating."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900090,
+  "ok": false,
+  "steps": 35,
+  "status": "done",
+  "wall": 5.0,
+  "url": "http://127.0.0.1:8811/t/hotel/900090/results?dest=muc&cin=2026-11-07&cout=2027-06-02&guests=1",
+  "goal": "Search for hotels in Munich from November 2, 2026 to 7 November 2026 for one guest, sorted by price: low to high.",
+  "plan": [
+   "Search for hotels in Munich from November 2, 2026 to 7 November 2026 for one guest, sorted by price: low to high."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900000,
+  "ok": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/shop/900000/p/18",
+  "goal": "Go to page 2 and open 'Hooli Water bottle X'.",
+  "plan": [
+   "Go to page 2 and open 'Hooli Water bottle X'."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900010,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.0,
+  "url": "http://127.0.0.1:8811/t/shop/900010?sort=Price%3A+low+to+high",
+  "goal": "Sort the products by price: low to high.",
+  "plan": [
+   "Sort the products by price: low to high."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900020,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.7,
+  "url": "http://127.0.0.1:8811/t/shop/900020#home",
+  "goal": "Show page 3.",
+  "plan": [
+   "Show page 3."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900030,
+  "ok": false,
+  "steps": 4,
+  "status": "done",
+  "wall": 1.4,
+  "url": "http://127.0.0.1:8811/t/shop/900030?q=water+bottles",
+  "goal": "Find water bottles.",
+  "plan": [
+   "Find water bottles."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900040,
+  "ok": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/shop/900040?brand=Hooli",
+  "goal": "Show only Hooli products.",
+  "plan": [
+   "Show only Hooli products."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900050,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.0,
+  "url": "http://127.0.0.1:8811/t/shop/900050?brand=Acme",
+  "goal": "Filter the list to the brand Acme.",
+  "plan": [
+   "Filter the list to the brand Acme."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900060,
+  "ok": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 1.4,
+  "url": "http://127.0.0.1:8811/t/shop/900060?q=backpack",
+  "goal": "Search the shop for 'backpack'.",
+  "plan": [
+   "Search the shop for 'backpack'."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900070,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.0,
+  "url": "http://127.0.0.1:8811/t/shop/900070?brand=Umbrella",
+  "goal": "I only want to see Umbrella items.",
+  "plan": [
+   "I only want to see Umbrella items."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900080,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.0,
+  "url": "http://127.0.0.1:8811/t/shop/900080?sort=Price%3A+low+to+high",
+  "goal": "Show the products sorted by price: low to high.",
+  "plan": [
+   "Show the products sorted by price: low to high."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900090,
+  "ok": true,
+  "steps": 8,
+  "status": "done",
+  "wall": 1.7,
+  "url": "http://127.0.0.1:8811/t/shop/900090?sort=Price%3A+low+to+high&page=3",
+  "goal": "Go to page 3.",
+  "plan": [
+   "Go to page 3."
+  ]
+ }
+]

results/v15s/gym_eval_v15s_plan.json

@@ -0,0 +1,490 @@
+[
+ {
+  "kind": "flight",
+  "seed": 900000,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 8.3,
+  "url": "http://127.0.0.1:8811/t/flight/900000",
+  "goal": "Show me a return flight from Oslo to Berlin on October 18, returning 23 October 2026 for 3 adults in premium economy. Stop when matching flight options are visible.",
+  "plan": [
+   "Set origin (from) to Oslo",
+   "Set destination (to) to Berlin",
+   "Set departure date to October 18, 2026",
+   "Set return date to October 23, 2026",
+   "Set number of passengers to 3",
+   "Set class to premium economy",
+   "Set trip type to return flight",
+   "Stop when matching flight options are visible"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900010,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.8,
+  "url": "http://127.0.0.1:8811/t/flight/900010",
+  "goal": "Look up round-trip flights from Tokyo to Amsterdam on October 13, returning 20 October 2026 for two adults in first class.",
+  "plan": [
+   "Set where from (origin, departure) to Tokyo",
+   "Set where to (destination, arrival) to Amsterdam",
+   "Set the departure date to October 13, 2026",
+   "Set the return date to October 20, 2026",
+   "Set the number of passengers to 2",
+   "Set the class to first class",
+   "Set trip type to round-trip",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900020,
+  "ok": false,
+  "steps": 23,
+  "status": "blocked",
+  "wall": 5.2,
+  "url": "http://127.0.0.1:8811/t/flight/900020#explore",
+  "goal": "Search for one-way flights from Roma to Seoul on 15 October 2026 for two adults. Stop when matching flight options are visible.",
+  "plan": [
+   "Set where from (origin, departure) to Roma",
+   "Set where to (destination, arrival) to Seoul",
+   "Set the departure date to October 15, 2026",
+   "Set the number of passengers to 2",
+   "Set passenger type to adult",
+   "Set trip type to one-way",
+   "Submit the search and stop when matching flight options are visible"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900030,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 11.7,
+  "url": "http://127.0.0.1:8811/t/flight/900030",
+  "goal": "Show me a one-way flight from Helsinki to Munich on 26 October 2026 for four adults in business class.",
+  "plan": [
+   "Set where from (origin, departure) to Helsinki",
+   "Set where to (destination, arrival) to Munich",
+   "Set the departure date to October 26, 2026",
+   "Set the number of passengers to 4",
+   "Set the class to business",
+   "Set trip type to one-way",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900040,
+  "ok": false,
+  "steps": 7,
+  "status": "blocked",
+  "wall": 2.7,
+  "url": "http://127.0.0.1:8811/t/flight/900040/results?trip=round&adults=1&cabin=economy#about",
+  "goal": "Show me one way flights from Manchester to Bangkok on November 21 for 3 adults in business class.",
+  "plan": [
+   "Set where from (origin, departure) to Manchester",
+   "Set where to (destination, arrival) to Bangkok",
+   "Set the departure date to November 21",
+   "Set the number of adults to 3",
+   "Set the class to business",
+   "Set trip type to one-way",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900050,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.9,
+  "url": "http://127.0.0.1:8811/t/flight/900050",
+  "goal": "Show me a one-way flight from Amsterdam to Barcelona on November 8, 2026 for 1 adult in economy.",
+  "plan": [
+   "Set the origin (from) to Amsterdam",
+   "Set the destination (to) to Barcelona",
+   "Set the departure date to November 8, 2026",
+   "Set the number of passengers to 1",
+   "Set the class to economy",
+   "Set the trip type to one-way",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900060,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.1,
+  "url": "http://127.0.0.1:8811/t/flight/900060",
+  "goal": "Find one-way flights from Seoul to Copenhagen on 10 October 2026 for two adults in business class.",
+  "plan": [
+   "Set origin (from) to Seoul",
+   "Set destination (to) to Copenhagen",
+   "Set departure date to October 10, 2026",
+   "Set number of passengers to 2",
+   "Set class to Business",
+   "Set trip type to One-way",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900070,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.1,
+  "url": "http://127.0.0.1:8811/t/flight/900070",
+  "goal": "Look up a return flight from Barcelona to New York on November 13, returning November 27 for 1 adult. Stop when matching flight options are visible.",
+  "plan": [
+   "Set where from (origin, departure) to Barcelona",
+   "Set where to (destination, arrival) to New York",
+   "Set the departure date to November 13, 2026",
+   "Set the return date to November 27, 2026",
+   "Set the number of passengers to 1",
+   "Set the trip type to return",
+   "Submit the search and stop when matching flight options are visible"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900080,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.3,
+  "url": "http://127.0.0.1:8811/t/flight/900080",
+  "goal": "Show me a one-way flight from Miami to Toronto on November 15 for one adult.",
+  "plan": [
+   "Set the departure city (origin) to Miami",
+   "Set the arrival city (destination) to Toronto",
+   "Set the departure date to November 15",
+   "Set the number of passengers to 1",
+   "Set the passenger type to Adult",
+   "Set the trip type to One-way",
+   "Submit the search and view the flight results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900090,
+  "ok": false,
+  "steps": 17,
+  "status": "blocked",
+  "wall": 4.2,
+  "url": "http://127.0.0.1:8811/t/flight/900090",
+  "goal": "Find one-way flights from Barcelona to Bangkok on 4 October 2026 for two adults in first class. Show the results.",
+  "plan": [
+   "Set where from (origin, departure) to Barcelona",
+   "Set where to (destination, arrival) to Bangkok",
+   "Set the departure date to October 4, 2026",
+   "Set the number of passengers to 2",
+   "Set the class to First Class",
+   "Set trip type to One-way",
+   "Submit the search and show the results"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900000,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 4.9,
+  "url": "http://127.0.0.1:8811/t/hotel/900000",
+  "goal": "Find a hotel in Toronto from 31 October 2026 to November 2, sorted by price: low to high.",
+  "plan": [
+   "Set the departure date to October 31, 2026",
+   "Set the return date to November 2, 2026",
+   "Set the sort order to price: low to high"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900010,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.0,
+  "url": "http://127.0.0.1:8811/t/hotel/900010",
+  "goal": "Search for hotels in Chicago from 30 October 2026 to November 1 for three guests, sorted by price: low to high.",
+  "plan": [
+   "Set the origin (departure) to Chicago",
+   "Set the departure date to October 30, 2026",
+   "Set the return date to November 1, 2026",
+   "Set the number of guests to 3",
+   "Sort the results by price: low to high"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900020,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.6,
+  "url": "http://127.0.0.1:8811/t/hotel/900020",
+  "goal": "Find a hotel in Oslo from 30 October 2026 to November 3, 2026 for one guest, only showing pet friendly.",
+  "plan": [
+   "Set the arrival date to October 30, 2026",
+   "Set the departure date to November 3, 2026",
+   "Set the number of guests to 1",
+   "Set the pet friendly filter to enabled",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900030,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.0,
+  "url": "http://127.0.0.1:8811/t/hotel/900030",
+  "goal": "Find stays in Toronto from November 16, 2026 to November 21, 2026, filtered to pet friendly.",
+  "plan": [
+   "Set the departure date to November 16, 2026",
+   "Set the return date to November 21, 2026",
+   "Set the origin (departure) to Toronto",
+   "Set the destination (arrival) to Toronto",
+   "Set the trip type to stays",
+   "Set the filter to pet friendly"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900040,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.6,
+  "url": "http://127.0.0.1:8811/t/hotel/900040",
+  "goal": "Search for hotels in Tokyo from 14 November 2026 to 15 November 2026, with pet friendly.",
+  "plan": [
+   "Set the departure date to November 14, 2026",
+   "Set the return date to November 15, 2026",
+   "Set the destination (city) to Tokyo",
+   "Set the pet friendly filter to enabled"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900050,
+  "ok": false,
+  "steps": 14,
+  "status": "blocked",
+  "wall": 3.1,
+  "url": "http://127.0.0.1:8811/t/hotel/900050",
+  "goal": "Find stays in Copenhagen from December 4 to December 7, 2026 for one guest, sorted by guest rating.",
+  "plan": [
+   "Set the origin (from) to Copenhagen",
+   "Set the departure date to December 4, 2026",
+   "Set the return date to December 7, 2026",
+   "Set the number of guests to 1",
+   "Sort by guest rating"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900060,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.9,
+  "url": "http://127.0.0.1:8811/t/hotel/900060/results?cin=2026-11-03&cout=2027-06-10&guests=2&filter=Pool%7CPet+friendly",
+  "goal": "Find stays in Rome from November 3, 2026 to 10 November 2026 for 4 guests.",
+  "plan": [
+   "Set the departure date to November 3, 2026",
+   "Set the return date to November 10, 2026",
+   "Set the number of guests to 4",
+   "Set the destination to Rome"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900070,
+  "ok": false,
+  "steps": 4,
+  "status": "blocked",
+  "wall": 1.7,
+  "url": "http://127.0.0.1:8811/t/hotel/900070",
+  "goal": "Find a hotel in Chicago from 27 September 2026 to October 4, only showing free cancellation.",
+  "plan": [
+   "Set the arrival date to September 27, 2026",
+   "Set the departure date to October 4, 2026",
+   "Set the filter to show only free cancellation hotels"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900080,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.8,
+  "url": "http://127.0.0.1:8811/t/hotel/900080/results?cin=2026-11-16&cout=2027-06-20&guests=2&filter=Pool%7CPet+friendly",
+  "goal": "Look for a place to stay in New York from November 16, 2026 to 20 November 2026 for 2 adults, sorted by guest rating.",
+  "plan": [
+   "Set the arrival date to November 16, 2026",
+   "Set the departure date to November 20, 2026",
+   "Set the number of adults to 2",
+   "Set the sort order to guest rating",
+   "Search for places to stay in New York"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900090,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.9,
+  "url": "http://127.0.0.1:8811/t/hotel/900090",
+  "goal": "Search for hotels in Munich from November 2, 2026 to 7 November 2026 for one guest, sorted by price: low to high.",
+  "plan": [
+   "Set the origin (from) to Munich",
+   "Set the departure date to November 2, 2026",
+   "Set the return date to November 7, 2026",
+   "Set the number of guests to 1",
+   "Sort the results by price: low to high"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900000,
+  "ok": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.6,
+  "url": "http://127.0.0.1:8811/t/shop/900000/p/18",
+  "goal": "Go to page 2 and open 'Hooli Water bottle X'.",
+  "plan": [
+   "Go to page 2",
+   "Open 'Hooli Water bottle X'"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900010,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/shop/900010?sort=Price%3A+low+to+high",
+  "goal": "Sort the products by price: low to high.",
+  "plan": [
+   "Sort products by price from low to high"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900020,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 3.9,
+  "url": "http://127.0.0.1:8811/t/shop/900020/p/14",
+  "goal": "Show page 3.",
+  "plan": [
+   "Go to page 3"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900030,
+  "ok": false,
+  "steps": 4,
+  "status": "done",
+  "wall": 1.5,
+  "url": "http://127.0.0.1:8811/t/shop/900030?q=water+bottles",
+  "goal": "Find water bottles.",
+  "plan": [
+   "Find water bottles"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900040,
+  "ok": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 1.2,
+  "url": "http://127.0.0.1:8811/t/shop/900040/p/2",
+  "goal": "Show only Hooli products.",
+  "plan": [
+   "Filter products to show only Hooli products"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900050,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/shop/900050?brand=Acme",
+  "goal": "Filter the list to the brand Acme.",
+  "plan": [
+   "Filter the list to show only items from the brand Acme"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900060,
+  "ok": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 1.4,
+  "url": "http://127.0.0.1:8811/t/shop/900060?q=backpack",
+  "goal": "Search the shop for 'backpack'.",
+  "plan": [
+   "Search the shop for 'backpack'"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900070,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/shop/900070?brand=Umbrella",
+  "goal": "I only want to see Umbrella items.",
+  "plan": [
+   "Set the item type to Umbrella"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900080,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/shop/900080?sort=Price%3A+low+to+high",
+  "goal": "Show the products sorted by price: low to high.",
+  "plan": [
+   "Sort products by price from low to high"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900090,
+  "ok": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 1.2,
+  "url": "http://127.0.0.1:8811/t/shop/900090?sort=Price%3A+low+to+high",
+  "goal": "Go to page 3.",
+  "plan": [
+   "Go to page 3"
+  ]
+ }
+]

results/v15s/suiteA_v15s.json

@@ -0,0 +1,338 @@
+[
+ {
+  "name": "wiki-einstein",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 6.926211595535278
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 6.431009531021118
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.823692560195923
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.112623691558838
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.671370029449463
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.736757516860962
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.099965810775757
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.6844990253448486
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.596674919128418
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5861363410949707
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.3591747283935547
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.479138135910034
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 15.31757926940918
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.4401447772979736
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.368851661682129
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.8873937129974365
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.469536542892456
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.6583445072174072
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.8013386726379395
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8346102237701416
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9873788356781006
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.3997318744659424
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.73085355758667
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8671338558197021
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 6.560246467590332
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0562314987182617
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.17016339302063
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.354671001434326
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.079979419708252
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.158301591873169
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 9,
+  "status": "done",
+  "wall": 5.870171308517456
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 8.680614709854126
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 6,
+  "status": "done",
+  "wall": 5.916048049926758
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.7953104972839355
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.246614933013916
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.7829885482788086
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.1424758434295654
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.512457847595215
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.4985249042510986
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 9.64554238319397
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 4.717194080352783
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 3.879201650619507
+ },
+ {
+  "name": "arxiv-search",
+  "pass": true,
+  "steps": 4,
+  "status": "done",
+  "wall": 8.213128566741943
+ },
+ {
+  "name": "arxiv-search",
+  "pass": true,
+  "steps": 4,
+  "status": "done",
+  "wall": 4.860985040664673
+ },
+ {
+  "name": "arxiv-search",
+  "pass": true,
+  "steps": 4,
+  "status": "done",
+  "wall": 5.041322708129883
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 7.512296438217163
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 3.1583213806152344
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 3.697366237640381
+ }
+]

results/v15s/suiteB_v15s.json

@@ -0,0 +1,380 @@
+[
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.6260435581207275
+ },
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.895298957824707
+ },
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.955277919769287
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 7.174061298370361
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.038809061050415
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.109954595565796
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 6.288775682449341
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9772262573242188
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0207574367523193
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.546006202697754
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.7569341659545898
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.7601943016052246
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.746327638626099
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8911144733428955
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.7690718173980713
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 7.995830774307251
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.563608646392822
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.476851224899292
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.559648275375366
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8806250095367432
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.3005239963531494
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.645474672317505
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.807223320007324
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.1060681343078613
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 8.151905059814453
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.8693556785583496
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.239215850830078
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 6.999143123626709
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.002227306365967
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.620467185974121
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 6.0526509284973145
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 2.5856010913848877
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 3.9109625816345215
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 7.533844470977783
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.053530216217041
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.878587245941162
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 7.374943494796753
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.120349168777466
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.0663528442382812
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.497931241989136
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.707350730895996
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.028721809387207
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 3.677743673324585
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 2.6683971881866455
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 2.8182685375213623
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 21.43064546585083
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 15.432049989700317
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.625671625137329
+ },
+ {
+  "name": "shop-page2",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.8776209354400635
+ },
+ {
+  "name": "shop-page2",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.9768171310424805
+ },
+ {
+  "name": "shop-page2",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.446743726730347
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 6.476119518280029
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.996001720428467
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.738437652587891
+ }
+]

results/v15s/v15s.log

@@ -0,0 +1,210 @@
+== data: rollout2 2343  gym 32051  m2w_sub 5093
+== build laya-browser-v15s
+   (SKIP_BUILD: reusing finetune/out/train_items.pt)
+== train laya-browser-v15s (3 epochs from base)
+=== epoch 1/3 avg loss 1.0861  5018s
+=== epoch 2/3 avg loss 0.9380  9986s
+=== epoch 3/3 avg loss 0.7940  14928s
+saved finetune/out/laya-browser-v15s
+== calibrate
+n=5913 acc=0.669  T=1: nll 3.530 mean conf 0.900  |  T=5.23: nll 1.312 mean conf 0.686
+wrote temperature 5.227105378439709 -> /home/ckl/projects/S/laya/finetune/out/laya-browser-v15s/rl_agent_config.json
+== eval
+/home/ckl/projects/S/laya/finetune/out/laya-browser-v15s/: cases 6268  operation acc 0.757  target top-1 0.593 (n=5560, mean normalized rank 0.287)  20 ms/case
+   live      CLICK     n=1032 op acc 0.96  target top-1 0.85
+   live      DONE      n= 121 op acc 0.91
+   live      SELECT    n=  30 op acc 0.97  target top-1 0.83
+   live      TYPE_TEXT n= 117 op acc 0.80  target top-1 0.96
+   mind2web  CLICK     n=1825 op acc 0.91  target top-1 0.56
+   mind2web  SELECT    n= 213 op acc 0.69  target top-1 0.39
+   mind2web  TYPE_TEXT n= 350 op acc 0.80  target top-1 0.90
+   nnetnav   CLICK     n=1386 op acc 0.66  target top-1 0.25
+   nnetnav   DONE      n= 313 op acc 0.41
+   nnetnav   SCROLL_DOWN n= 224 op acc 0.20  target top-1 0.00
+   nnetnav   SCROLL_UP n=  50 op acc 0.00  target top-1 0.00
+   nnetnav   TYPE_TEXT n= 607 op acc 0.57  target top-1 0.88
+== suite A x3
+PASS  wiki-einstein    steps= 2 status=done        6.9s  https://en.wikipedia.org/wiki/Albert_Einstein
+PASS  wiki-einstein    steps= 3 status=done        6.4s  https://en.wikipedia.org/wiki/Albert_Einstein
+PASS  wiki-einstein    steps= 2 status=done        3.8s  https://en.wikipedia.org/wiki/Albert_Einstein
+PASS  wiki-search      steps= 2 status=done        4.1s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  wiki-search      steps= 2 status=done        4.7s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  wiki-search      steps= 2 status=done        2.7s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  hn-new           steps= 1 status=done        4.1s  https://news.ycombinator.com/newest
+PASS  hn-new           steps= 1 status=done        2.7s  https://news.ycombinator.com/newest
+PASS  hn-new           steps= 1 status=done        2.6s  https://news.ycombinator.com/newest
+PASS  hn-login-page    steps= 1 status=done        2.6s  https://news.ycombinator.com/login?goto=news
+PASS  hn-login-page    steps= 1 status=done        2.4s  https://news.ycombinator.com/login?goto=news
+PASS  hn-login-page    steps= 1 status=done        2.5s  https://news.ycombinator.com/login?goto=news
+PASS  gh-issues        steps= 1 status=done       15.3s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.4s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 2 status=done        4.4s  https://github.com/tile-ai/tilelang/issues?q=is%3Aissue+state%3Aopen
+PASS  py-downloads     steps= 1 status=done        4.9s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        2.5s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        2.7s  https://www.python.org/downloads/
+PASS  books-travel     steps= 1 status=done        5.8s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        1.8s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-open-book  steps= 1 status=done        2.4s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.7s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  internet-dropdown steps= 1 status=done        6.6s  https://the-internet.herokuapp.com/dropdown
+FAIL  internet-dropdown steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/dropdown
+FAIL  internet-dropdown steps= 1 status=done        2.2s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-checkbox steps= 1 status=done        2.4s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.2s  https://the-internet.herokuapp.com/checkboxes
+FAIL  books-page2      steps= 9 status=done        5.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps=20 status=ready       8.7s  https://books.toscrape.com/catalogue/category/books/science-fiction_16
+FAIL  books-page2      steps= 6 status=done        5.9s  https://books.toscrape.com/catalogue/rip-it-up-and-start-again_986/ind
+PASS  quotes-tag-love  steps= 1 status=done        4.8s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        3.2s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        2.8s  https://quotes.toscrape.com/tag/love/
+PASS  hn-past          steps= 1 status=done        3.1s  https://news.ycombinator.com/front
+PASS  hn-past          steps= 1 status=done        2.5s  https://news.ycombinator.com/front
+PASS  hn-past          steps= 1 status=done        2.5s  https://news.ycombinator.com/front
+PASS  web-search       steps= 3 status=done        9.6s  https://marginalia-search.com/search?query=tilelang+github&sst=SE-30ab
+PASS  web-search       steps= 3 status=done        4.7s  https://marginalia-search.com/search?query=tilelang+github&profile=cor
+PASS  web-search       steps= 3 status=done        3.9s  https://marginalia-search.com/search?query=tilelang+github&profile=cor
+PASS  arxiv-search     steps= 4 status=done        8.2s  https://arxiv.org/search/?query=flash+attention&searchtype=all&source=
+PASS  arxiv-search     steps= 4 status=done        4.9s  https://arxiv.org/search/?query=flash+attention&searchtype=all&source=
+PASS  arxiv-search     steps= 4 status=done        5.0s  https://arxiv.org/search/?query=flash+attention&searchtype=all&source=
+FAIL  flights          steps= 3 status=done        7.5s  https://www.google.com/travel/flights?hl=en
+FAIL  flights          steps= 3 status=done        3.2s  https://www.google.com/travel/flights?hl=en
+FAIL  flights          steps= 3 status=done        3.7s  https://www.google.com/travel/flights?hl=en
+== 39/48 passed (81%, 3 run(s) per task) | median wall 3.4s
+   per task: wiki-einstein=3/3  wiki-search=3/3  hn-new=3/3  hn-login-page=3/3  gh-issues=3/3  py-downloads=3/3  books-travel=3/3  books-open-book=3/3  internet-dropdown=0/3  internet-checkbox=3/3  books-page2=0/3  quotes-tag-love=3/3  hn-past=3/3  web-search=3/3  arxiv-search=3/3  flights=0/3
+== suite B x3
+held-out check: OK, no suite-B domain appears in training data
+PASS  sqlite-download      steps= 1 status=done        5.6s  https://www.sqlite.org/download.html
+PASS  sqlite-download      steps= 1 status=done        1.9s  https://www.sqlite.org/download.html
+PASS  sqlite-download      steps= 1 status=done        2.0s  https://www.sqlite.org/download.html
+PASS  haskell-downloads    steps= 1 status=done        7.2s  https://www.haskell.org/downloads/
+PASS  haskell-downloads    steps= 1 status=done        2.0s  https://www.haskell.org/downloads/
+PASS  haskell-downloads    steps= 1 status=done        2.1s  https://www.haskell.org/downloads/
+PASS  zig-download         steps= 1 status=done        6.3s  https://ziglang.org/download/
+PASS  zig-download         steps= 1 status=done        2.0s  https://ziglang.org/download/
+PASS  zig-download         steps= 1 status=done        2.0s  https://ziglang.org/download/
+PASS  openbsd-faq          steps= 1 status=done        5.5s  https://www.openbsd.org/faq/index.html
+PASS  openbsd-faq          steps= 1 status=done        1.8s  https://www.openbsd.org/faq/index.html
+PASS  openbsd-faq          steps= 1 status=done        1.8s  https://www.openbsd.org/faq/index.html
+PASS  nginx-download       steps= 1 status=done        5.7s  https://nginx.org/en/download.html
+PASS  nginx-download       steps= 1 status=done        1.9s  https://nginx.org/en/download.html
+PASS  nginx-download       steps= 1 status=done        1.8s  https://nginx.org/en/download.html
+PASS  xkcd-archive         steps= 1 status=done        8.0s  https://xkcd.com/archive/
+PASS  xkcd-archive         steps= 1 status=done        4.6s  https://xkcd.com/archive/
+PASS  xkcd-archive         steps= 1 status=done        4.5s  https://xkcd.com/archive/
+PASS  elixir-install       steps= 1 status=done        4.6s  https://elixir-lang.org/install/
+PASS  elixir-install       steps= 1 status=done        1.9s  https://elixir-lang.org/install/
+PASS  elixir-install       steps= 1 status=done        2.3s  https://elixir-lang.org/install/
+PASS  prometheus-download  steps= 2 status=done        5.6s  https://prometheus.io/download/
+PASS  prometheus-download  steps= 2 status=done        5.8s  https://prometheus.io/download/
+PASS  prometheus-download  steps= 2 status=done        3.1s  https://prometheus.io/download/
+PASS  ocaml-install        steps= 1 status=done        8.2s  https://ocaml.org/install#linux_mac_bsd
+PASS  ocaml-install        steps= 1 status=done        2.9s  https://ocaml.org/install#linux_mac_bsd
+PASS  ocaml-install        steps= 1 status=done        3.2s  https://ocaml.org/install#linux_mac_bsd
+PASS  rosetta-quicksort    steps= 2 status=done        7.0s  https://rosettacode.org/wiki/Quicksort
+PASS  rosetta-quicksort    steps= 2 status=done        3.0s  https://rosettacode.org/wiki/Quicksort
+PASS  rosetta-quicksort    steps= 2 status=done        2.6s  https://rosettacode.org/wiki/Quicksort
+PASS  arch-man-grep        steps= 3 status=done        6.1s  https://man.archlinux.org/man/grep.1
+PASS  arch-man-grep        steps= 3 status=done        2.6s  https://man.archlinux.org/man/grep.1
+PASS  arch-man-grep        steps= 3 status=done        3.9s  https://man.archlinux.org/man/grep.1
+PASS  debian-man-tar       steps= 2 status=done        7.5s  https://manpages.debian.org/trixie/tar/tar.1.en.html
+PASS  debian-man-tar       steps= 2 status=done        2.1s  https://manpages.debian.org/trixie/tar/tar.1.en.html
+PASS  debian-man-tar       steps= 2 status=done        1.9s  https://manpages.debian.org/trixie/tar/tar.1.en.html
+PASS  rfc-http-semantics   steps= 2 status=done        7.4s  https://www.rfc-editor.org/search/?q=HTTP+semantics
+PASS  rfc-http-semantics   steps= 2 status=done        3.1s  https://www.rfc-editor.org/search/?q=HTTP+semantics
+PASS  rfc-http-semantics   steps= 2 status=done        3.1s  https://www.rfc-editor.org/search/?q=HTTP+semantics
+PASS  nixos-ripgrep        steps= 2 status=done        4.5s  https://search.nixos.org/packages?channel=26.05&query=ripgrep
+PASS  nixos-ripgrep        steps= 2 status=done        2.7s  https://search.nixos.org/packages?channel=26.05&query=ripgrep
+PASS  nixos-ripgrep        steps= 2 status=done        3.0s  https://search.nixos.org/packages?channel=26.05&query=ripgrep
+PASS  ocaml-pkg-yojson     steps= 3 status=done        3.7s  https://ocaml.org/packages/search?q=yojson
+PASS  ocaml-pkg-yojson     steps= 3 status=done        2.7s  https://ocaml.org/packages/search?q=yojson
+PASS  ocaml-pkg-yojson     steps= 3 status=done        2.8s  https://ocaml.org/packages/search?q=yojson
+PASS  shop-search-jacket   steps= 2 status=done       21.4s  https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product
+PASS  shop-search-jacket   steps= 2 status=done       15.4s  https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product
+PASS  shop-search-jacket   steps= 2 status=done        3.6s  https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product
+PASS  shop-page2           steps= 1 status=done        4.9s  https://www.scrapingcourse.com/ecommerce/page/2/
+PASS  shop-page2           steps= 1 status=done        5.0s  https://www.scrapingcourse.com/ecommerce/page/2/
+PASS  shop-page2           steps= 1 status=done        4.4s  https://www.scrapingcourse.com/ecommerce/page/2/
+PASS  shop-sort-price      steps= 1 status=done        6.5s  https://www.scrapingcourse.com/ecommerce/?orderby=price
+PASS  shop-sort-price      steps= 1 status=done        6.0s  https://www.scrapingcourse.com/ecommerce/?orderby=price
+PASS  shop-sort-price      steps= 1 status=done        4.7s  https://www.scrapingcourse.com/ecommerce/?orderby=price
+== 54/54 passed (100%, 3 run(s) per task) | median wall 3.7s
+   per task: sqlite-download=3/3  haskell-downloads=3/3  zig-download=3/3  openbsd-faq=3/3  nginx-download=3/3  xkcd-archive=3/3  elixir-install=3/3  prometheus-download=3/3  ocaml-install=3/3  rosetta-quicksort=3/3  arch-man-grep=3/3  debian-man-tar=3/3  rfc-http-semantics=3/3  nixos-ripgrep=3/3  ocaml-pkg-yojson=3/3  shop-search-jacket=3/3  shop-page2=3/3  shop-sort-price=3/3
+== webgym held-out, planner off
+FAIL  flight seed=900000 steps=40 ready                          6.4s  Show me a return flight from Oslo to Berlin on October 18, returning 2
+FAIL  flight seed=900010 steps=40 ready                          5.6s  Look up round-trip flights from Tokyo to Amsterdam on October 13, retu
+PASS  flight seed=900020 steps=12 done                           2.7s  Search for one-way flights from Roma to Seoul on 15 October 2026 for t
+PASS  flight seed=900030 steps=17 done                           3.1s  Show me a one-way flight from Helsinki to Munich on 26 October 2026 fo
+FAIL  flight seed=900040 steps=40 ready                          6.4s  Show me one way flights from Manchester to Bangkok on November 21 for 
+FAIL  flight seed=900050 steps=40 ready                          5.5s  Show me a one-way flight from Amsterdam to Barcelona on November 8, 20
+FAIL  flight seed=900060 steps=40 ready                          5.8s  Find one-way flights from Seoul to Copenhagen on 10 October 2026 for t
+FAIL  flight seed=900070 steps=40 ready                          5.3s  Look up a return flight from Barcelona to New York on November 13, ret
+FAIL  flight seed=900080 steps=40 ready                          5.7s  Show me a one-way flight from Miami to Toronto on November 15 for one 
+FAIL  flight seed=900090 steps=40 ready                          5.3s  Find one-way flights from Barcelona to Bangkok on 4 October 2026 for t
+FAIL  hotel  seed=900000 steps=40 ready                          5.1s  Find a hotel in Toronto from 31 October 2026 to November 2, sorted by 
+FAIL  hotel  seed=900010 steps=40 ready                          5.6s  Search for hotels in Chicago from 30 October 2026 to November 1 for th
+FAIL  hotel  seed=900020 steps=40 ready                          5.4s  Find a hotel in Oslo from 30 October 2026 to November 3, 2026 for one 
+FAIL  hotel  seed=900030 steps=40 ready                          5.5s  Find stays in Toronto from November 16, 2026 to November 21, 2026, fil
+FAIL  hotel  seed=900040 steps=40 ready                          5.2s  Search for hotels in Tokyo from 14 November 2026 to 15 November 2026, 
+FAIL  hotel  seed=900050 steps=15 done                           2.6s  Find stays in Copenhagen from December 4 to December 7, 2026 for one g
+FAIL  hotel  seed=900060 steps=25 done                           3.6s  Find stays in Rome from November 3, 2026 to 10 November 2026 for 4 gue
+FAIL  hotel  seed=900070 steps=11 done                           2.1s  Find a hotel in Chicago from 27 September 2026 to October 4, only show
+FAIL  hotel  seed=900080 steps=26 done                           3.8s  Look for a place to stay in New York from November 16, 2026 to 20 Nove
+FAIL  hotel  seed=900090 steps=35 done                           5.0s  Search for hotels in Munich from November 2, 2026 to 7 November 2026 f
+PASS  shop   seed=900000 steps= 2 done                           1.1s  Go to page 2 and open 'Hooli Water bottle X'.
+PASS  shop   seed=900010 steps= 1 done                           1.0s  Sort the products by price: low to high.
+FAIL  shop   seed=900020 steps=40 ready                          5.7s  Show page 3.
+FAIL  shop   seed=900030 steps= 4 done                           1.4s  Find water bottles.
+PASS  shop   seed=900040 steps= 2 done                           1.1s  Show only Hooli products.
+PASS  shop   seed=900050 steps= 1 done                           1.0s  Filter the list to the brand Acme.
+PASS  shop   seed=900060 steps= 3 done                           1.4s  Search the shop for 'backpack'.
+PASS  shop   seed=900070 steps= 1 done                           1.0s  I only want to see Umbrella items.
+PASS  shop   seed=900080 steps= 1 done                           1.0s  Show the products sorted by price: low to high.
+PASS  shop   seed=900090 steps= 8 done                           1.7s  Go to page 3.
+== flight: 2/10  median steps 40
+== hotel: 0/10  median steps 40
+== shop: 8/10  median steps 2
+== total 10/30 (33%)
+== webgym held-out, planner on
+FAIL  flight seed=900000 steps=40 ready                          8.3s  Show me a return flight from Oslo to Berlin on October 18, returning 2
+FAIL  flight seed=900010 steps=40 ready                          6.8s  Look up round-trip flights from Tokyo to Amsterdam on October 13, retu
+FAIL  flight seed=900020 steps=23 blocked                        5.2s  Search for one-way flights from Roma to Seoul on 15 October 2026 for t
+FAIL  flight seed=900030 steps=40 ready                         11.7s  Show me a one-way flight from Helsinki to Munich on 26 October 2026 fo
+FAIL  flight seed=900040 steps= 7 blocked                        2.7s  Show me one way flights from Manchester to Bangkok on November 21 for 
+FAIL  flight seed=900050 steps=40 ready                          5.9s  Show me a one-way flight from Amsterdam to Barcelona on November 8, 20
+FAIL  flight seed=900060 steps=40 ready                          6.1s  Find one-way flights from Seoul to Copenhagen on 10 October 2026 for t
+FAIL  flight seed=900070 steps=40 ready                          6.1s  Look up a return flight from Barcelona to New York on November 13, ret
+FAIL  flight seed=900080 steps=40 ready                          6.3s  Show me a one-way flight from Miami to Toronto on November 15 for one 
+FAIL  flight seed=900090 steps=17 blocked                        4.2s  Find one-way flights from Barcelona to Bangkok on 4 October 2026 for t
+FAIL  hotel  seed=900000 steps=40 ready                          4.9s  Find a hotel in Toronto from 31 October 2026 to November 2, sorted by 
+FAIL  hotel  seed=900010 steps=40 ready                          6.0s  Search for hotels in Chicago from 30 October 2026 to November 1 for th
+FAIL  hotel  seed=900020 steps=40 ready                          5.6s  Find a hotel in Oslo from 30 October 2026 to November 3, 2026 for one 
+FAIL  hotel  seed=900030 steps=40 ready                          6.0s  Find stays in Toronto from November 16, 2026 to November 21, 2026, fil
+FAIL  hotel  seed=900040 steps=40 ready                          5.6s  Search for hotels in Tokyo from 14 November 2026 to 15 November 2026, 
+FAIL  hotel  seed=900050 steps=14 blocked                        3.1s  Find stays in Copenhagen from December 4 to December 7, 2026 for one g
+FAIL  hotel  seed=900060 steps=40 ready                          6.9s  Find stays in Rome from November 3, 2026 to 10 November 2026 for 4 gue
+FAIL  hotel  seed=900070 steps= 4 blocked                        1.7s  Find a hotel in Chicago from 27 September 2026 to October 4, only show
+FAIL  hotel  seed=900080 steps=40 ready                          6.8s  Look for a place to stay in New York from November 16, 2026 to 20 Nove
+FAIL  hotel  seed=900090 steps=40 ready                          5.9s  Search for hotels in Munich from November 2, 2026 to 7 November 2026 f
+PASS  shop   seed=900000 steps= 2 done                           1.6s  Go to page 2 and open 'Hooli Water bottle X'.
+PASS  shop   seed=900010 steps= 1 done                           1.1s  Sort the products by price: low to high.
+FAIL  shop   seed=900020 steps=40 ready                          3.9s  Show page 3.
+FAIL  shop   seed=900030 steps= 4 done                           1.5s  Find water bottles.
+FAIL  shop   seed=900040 steps= 3 done                           1.2s  Show only Hooli products.
+PASS  shop   seed=900050 steps= 1 done                           1.1s  Filter the list to the brand Acme.
+PASS  shop   seed=900060 steps= 3 done                           1.4s  Search the shop for 'backpack'.
+PASS  shop   seed=900070 steps= 1 done                           1.1s  I only want to see Umbrella items.
+PASS  shop   seed=900080 steps= 1 done                           1.1s  Show the products sorted by price: low to high.
+FAIL  shop   seed=900090 steps= 3 done                           1.2s  Go to page 3.
+== flight: 0/10  median steps 40
+== hotel: 0/10  median steps 40
+== shop: 6/10  median steps 3
+== total 6/30 (20%)
+== Google Flights x3, planner on
+FAIL  flights          steps=20 status=ready      18.4s  https://www.google.com/travel/flights?tfs=CBwQARoXagwIAxIIL20vMDFfZDRy
+FAIL  flights          steps= 7 status=blocked     9.5s  https://www.google.com/travel/explore?tfs=CBwQAxoQag4IAxIKL20vMDViNHpm
+FAIL  flights          steps= 7 status=blocked    10.4s  https://www.google.com/travel/explore?tfs=CBwQAxoQag4IAxIKL20vMDViNHpm
+== 0/3 passed (0%, 3 run(s) per task) | median wall 10.4s
+V15S_DONE

v10/encoder/config.json

@@ -1,84 +0,0 @@
-{
-  "architectures": [
-    "ModernBertForMaskedLM"
-  ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 50281,
-  "classifier_activation": "gelu",
-  "classifier_bias": false,
-  "classifier_dropout": 0.0,
-  "classifier_pooling": "mean",
-  "cls_token_id": 50281,
-  "decoder_bias": true,
-  "deterministic_flash_attn": false,
-  "dtype": "float32",
-  "embedding_dropout": 0.0,
-  "eos_token_id": 50282,
-  "global_attn_every_n_layers": 3,
-  "gradient_checkpointing": false,
-  "hidden_activation": "gelu",
-  "hidden_size": 1024,
-  "initializer_cutoff_factor": 2.0,
-  "initializer_range": 0.02,
-  "intermediate_size": 2624,
-  "layer_norm_eps": 1e-05,
-  "layer_types": [
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention"
-  ],
-  "local_attention": 128,
-  "max_position_embeddings": 8192,
-  "mlp_bias": false,
-  "mlp_dropout": 0.0,
-  "model_type": "modernbert",
-  "norm_bias": false,
-  "norm_eps": 1e-05,
-  "num_attention_heads": 16,
-  "num_hidden_layers": 28,
-  "pad_token_id": 50283,
-  "position_embedding_type": "absolute",
-  "repad_logits_with_grad": false,
-  "rope_parameters": {
-    "full_attention": {
-      "rope_theta": 160000.0,
-      "rope_type": "default"
-    },
-    "sliding_attention": {
-      "rope_theta": 10000.0,
-      "rope_type": "default"
-    }
-  },
-  "sep_token_id": 50282,
-  "sparse_pred_ignore_index": -100,
-  "sparse_prediction": false,
-  "tie_word_embeddings": true,
-  "transformers_version": "5.17.0",
-  "vocab_size": 50368
-}

v10/model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:0c7f83d5de6aad2e176c03da890e2066f4a027494d751ba6493e60e687bbf9c0
-size 842609220

v10/rl_agent_config.json

@@ -1,31 +0,0 @@
-{
-  "encoder": "answerdotai/ModernBERT-large",
-  "head_layers": 2,
-  "max_len": 1024,
-  "head_max_len": 768,
-  "max_prefixes": 6,
-  "act_costs": {
-    "escalate": 0.5
-  },
-  "cost_wrong_act": 3.0,
-  "amp_dtype": "bf16",
-  "model_name": "laya-browser",
-  "temperature": [
-    5.025579877567767,
-    1.0,
-    1.0
-  ],
-  "temperature_by_options": {},
-  "training": {
-    "updates": 7313,
-    "epochs_completed": 1,
-    "hours": 1.96,
-    "world_size": 1,
-    "fine_tuned_from_checkpoint": true
-  },
-  "gradient_checkpointing": true,
-  "max_tokens_per_batch": 4096,
-  "fine_tuned": true,
-  "laya_fmt": "v2",
-  "head_max_len_train": 768
-}

v10/tokenizer/tokenizer_config.json

@@ -1,15 +0,0 @@
-{
-  "clean_up_tokenization_spaces": true,
-  "cls_token": "[CLS]",
-  "local_files_only": false,
-  "mask_token": "[MASK]",
-  "model_input_names": [
-    "input_ids",
-    "attention_mask"
-  ],
-  "model_max_length": 8192,
-  "pad_token": "[PAD]",
-  "sep_token": "[SEP]",
-  "tokenizer_class": "PreTrainedTokenizerFast",
-  "unk_token": "[UNK]"
-}

v10s/rl_agent_config.json

@@ -1,29 +0,0 @@
-{
-  "encoder": "jhu-clsp/mmBERT-base",
-  "head_layers": 2,
-  "max_len": 1024,
-  "head_max_len": 768,
-  "max_prefixes": 6,
-  "act_costs": {
-    "escalate": 0.5
-  },
-  "cost_wrong_act": 3.0,
-  "amp_dtype": "bf16",
-  "model_name": "laya-browser",
-  "temperature": [
-    4.737765564273641,
-    1.0,
-    1.0
-  ],
-  "temperature_by_options": {},
-  "training": {
-    "updates": 15987,
-    "epochs_completed": 4,
-    "hours": 4.97,
-    "world_size": 1,
-    "fine_tuned_from_checkpoint": false
-  },
-  "fine_tuned": true,
-  "laya_fmt": "v3",
-  "head_max_len_train": 768
-}

v11s/encoder/config.json

@@ -1,79 +0,0 @@
-{
-  "architectures": [
-    "ModernBertForMaskedLM"
-  ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 2,
-  "classifier_activation": "gelu",
-  "classifier_bias": false,
-  "classifier_dropout": 0.0,
-  "classifier_pooling": "mean",
-  "cls_token_id": 1,
-  "decoder_bias": true,
-  "deterministic_flash_attn": false,
-  "dtype": "float32",
-  "embedding_dropout": 0.0,
-  "eos_token_id": 1,
-  "global_attn_every_n_layers": 3,
-  "gradient_checkpointing": false,
-  "hidden_activation": "gelu",
-  "hidden_size": 768,
-  "initializer_cutoff_factor": 2.0,
-  "initializer_range": 0.02,
-  "intermediate_size": 1152,
-  "layer_norm_eps": 1e-05,
-  "layer_types": [
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention"
-  ],
-  "local_attention": 128,
-  "mask_token_id": 4,
-  "max_position_embeddings": 8192,
-  "mlp_bias": false,
-  "mlp_dropout": 0.0,
-  "model_type": "modernbert",
-  "norm_bias": false,
-  "norm_eps": 1e-05,
-  "num_attention_heads": 12,
-  "num_hidden_layers": 22,
-  "pad_token_id": 0,
-  "position_embedding_type": "sans_pos",
-  "repad_logits_with_grad": false,
-  "rope_parameters": {
-    "full_attention": {
-      "rope_theta": 160000,
-      "rope_type": "default"
-    },
-    "sliding_attention": {
-      "rope_theta": 160000,
-      "rope_type": "default"
-    }
-  },
-  "sep_token_id": 1,
-  "sparse_pred_ignore_index": -100,
-  "sparse_prediction": false,
-  "tie_word_embeddings": true,
-  "transformers_version": "5.17.0",
-  "vocab_size": 256000
-}

v11s/model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:42b324b823be353b8295019edbc7ebb1a5acf0b753cd7d9932d8a573e21d8a77
-size 643835524

v11s/rl_agent_config.json

@@ -1,29 +0,0 @@
-{
-  "encoder": "jhu-clsp/mmBERT-base",
-  "head_layers": 2,
-  "max_len": 1024,
-  "head_max_len": 768,
-  "max_prefixes": 6,
-  "act_costs": {
-    "escalate": 0.5
-  },
-  "cost_wrong_act": 3.0,
-  "amp_dtype": "bf16",
-  "model_name": "laya-browser",
-  "temperature": [
-    4.645538132553483,
-    1.0,
-    1.0
-  ],
-  "temperature_by_options": {},
-  "training": {
-    "updates": 15987,
-    "epochs_completed": 4,
-    "hours": 4.97,
-    "world_size": 1,
-    "fine_tuned_from_checkpoint": false
-  },
-  "fine_tuned": true,
-  "laya_fmt": "v3",
-  "head_max_len_train": 768
-}

v11s/tokenizer/tokenizer.json

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f
-size 34363188

v11s/tokenizer/tokenizer_config.json

@@ -1,25 +0,0 @@
-{
-  "bos_token": "<bos>",
-  "clean_up_tokenization_spaces": false,
-  "cls_token": "<bos>",
-  "eos_token": "<eos>",
-  "extra_0": "<start_of_turn>",
-  "extra_1": "<end_of_turn>",
-  "local_files_only": false,
-  "mask_token": "<mask>",
-  "model_input_names": [
-    "input_ids",
-    "attention_mask"
-  ],
-  "model_max_length": 8192,
-  "model_specific_special_tokens": {
-    "extra_0": "<start_of_turn>",
-    "extra_1": "<end_of_turn>"
-  },
-  "pad_token": "<pad>",
-  "padding_side": "right",
-  "sep_token": "<eos>",
-  "spaces_between_special_tokens": false,
-  "tokenizer_class": "PreTrainedTokenizerFast",
-  "unk_token": "<unk>"
-}

v14s/encoder/config.json

@@ -1,79 +0,0 @@
-{
-  "architectures": [
-    "ModernBertForMaskedLM"
-  ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 2,
-  "classifier_activation": "gelu",
-  "classifier_bias": false,
-  "classifier_dropout": 0.0,
-  "classifier_pooling": "mean",
-  "cls_token_id": 1,
-  "decoder_bias": true,
-  "deterministic_flash_attn": false,
-  "dtype": "float32",
-  "embedding_dropout": 0.0,
-  "eos_token_id": 1,
-  "global_attn_every_n_layers": 3,
-  "gradient_checkpointing": false,
-  "hidden_activation": "gelu",
-  "hidden_size": 768,
-  "initializer_cutoff_factor": 2.0,
-  "initializer_range": 0.02,
-  "intermediate_size": 1152,
-  "layer_norm_eps": 1e-05,
-  "layer_types": [
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "full_attention"
-  ],
-  "local_attention": 128,
-  "mask_token_id": 4,
-  "max_position_embeddings": 8192,
-  "mlp_bias": false,
-  "mlp_dropout": 0.0,
-  "model_type": "modernbert",
-  "norm_bias": false,
-  "norm_eps": 1e-05,
-  "num_attention_heads": 12,
-  "num_hidden_layers": 22,
-  "pad_token_id": 0,
-  "position_embedding_type": "sans_pos",
-  "repad_logits_with_grad": false,
-  "rope_parameters": {
-    "full_attention": {
-      "rope_theta": 160000,
-      "rope_type": "default"
-    },
-    "sliding_attention": {
-      "rope_theta": 160000,
-      "rope_type": "default"
-    }
-  },
-  "sep_token_id": 1,
-  "sparse_pred_ignore_index": -100,
-  "sparse_prediction": false,
-  "tie_word_embeddings": true,
-  "transformers_version": "5.17.0",
-  "vocab_size": 256000
-}

v14s/model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:c4e8b144c47e7fa97b128b4447e29cb0540c156fa50b5958200bb15b6a9c1801
-size 643835524

v14s/tokenizer/tokenizer.json

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f
-size 34363188

v14s/tokenizer/tokenizer_config.json

@@ -1,25 +0,0 @@
-{
-  "bos_token": "<bos>",
-  "clean_up_tokenization_spaces": false,
-  "cls_token": "<bos>",
-  "eos_token": "<eos>",
-  "extra_0": "<start_of_turn>",
-  "extra_1": "<end_of_turn>",
-  "local_files_only": false,
-  "mask_token": "<mask>",
-  "model_input_names": [
-    "input_ids",
-    "attention_mask"
-  ],
-  "model_max_length": 8192,
-  "model_specific_special_tokens": {
-    "extra_0": "<start_of_turn>",
-    "extra_1": "<end_of_turn>"
-  },
-  "pad_token": "<pad>",
-  "padding_side": "right",
-  "sep_token": "<eos>",
-  "spaces_between_special_tokens": false,
-  "tokenizer_class": "PreTrainedTokenizerFast",
-  "unk_token": "<unk>"
-}

{v10s → v15s}/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b11217df18bf79cfcd4ab639caf1ae8652b91c9c44fcb9457fbe480237332335
+oid sha256:ef5fe5cb43a6564e9da4eb42936bcc58a254691d021e85055a851b125bc5242e
 size 643835524

{v14s → v15s}/rl_agent_config.json

@@ -11,7 +11,7 @@
   "amp_dtype": "bf16",
   "model_name": "laya-browser",
   "temperature": [
-    5.025579877567767,
+    5.227105378439709,
     1.0,
     1.0
   ],