v14s checkpoint, held-out suite B (100 %), harness fixes, webgym + sub-goal data pipeline, DAgger leak correction

.gitattributes

@@ -37,3 +37,4 @@ v10s/tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 assets/laya_browser_demo.gif filter=lfs diff=lfs merge=lfs -text
 assets/laya_browser_demo.mp4 filter=lfs diff=lfs merge=lfs -text
 v11s/tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+v14s/tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -31,6 +31,64 @@ This repo is what it took to turn it into a usable decision head for [browser-us
 whose `/v1/systemone` request format is identical to laya's `predict(state, questions)`. Everything was done locally on one RTX 4070 Ti SUPER (16 GB),
 no paid API: the text helper and the DAgger teacher are a local Qwen3-8B-AWQ served by sglang.
 
+## Progress update (2026-09-26): v12s–v14s, a held-out suite, and a data-leak correction
+
+**New checkpoint `v14s/`** (mmBERT-base 322M, format v3, run the server with a 60-option chunk threshold:
+`python apps/systemone_server.py 8791 /path/to/v14s 60`).
+
+| | v10s | v12s | v13s | **v14s** |
+|---|---|---|---|---|
+| **Suite B: 18 tasks on 18 sites that appear in no training source** (×3, `code/apps/browser_suite_b.py`) | – | 94 % (34/36, ×2) | 93 % (50/54) | **100 % (54/54)** |
+| Suite A: the original 16 tasks (×3, patched harness) — *contaminated, see below* | 62 % | 75 % | 75 % | 81 % (39/48) |
+| Held-out long-horizon synthetic tasks (flight / hotel / shop, 8 each) | – | – | – | 0 / 0 / 50 % |
+
+Suite B checks the held-out claim at start-up against `results/round3/train_domains.json` (every domain of every training source,
+682 domains). Its tasks are mostly 1–2 steps: navigation, site search, man pages, RFC / package search, a shop's search, page 2 and
+sort-by-price. Suite A still fails `books-page2` (0/3), Google Flights (0/3) and is flaky on two search tasks.
+
+**Data leak found and removed.** `dagger_cases.jsonl` (the "177 on-policy DAgger corrections" in the table below, 331 cases by
+now) came from running the Qwen teacher on the suite itself: 152 of its cases are suite-A goals, often with the teacher's *wrong*
+label (e.g. "Go to page 2 of the catalogue" → click the first book), oversampled ×5. Every checkpoint from v10 to v14s was trained on
+it, so **all suite-A numbers in this README, including v10s's 62 %, are contaminated** (and `books-page2` failing 0/3 is partly
+the model faithfully reproducing a wrong label). Suite B is clean. From v15s on, `build_items.py` drops every DAgger case and every
+case recorded on a suite A/B start page.
+
+**What changed in the harness** (`code/jev-ultrafast.patch`, applies to jev-ultrafast `1231850`). Half of the original suite
+failures were harness bugs, not model errors:
+- a `PRESS_ENTER` control, offered while a non-empty text field is focused (arXiv's search overlay has no submit button);
+- elements covered by an unrelated element (modals, overlays, banners) are no longer offered; a target covered by its own ancestor
+  is clicked through;
+- observation retries while the document is navigating; a choice that fails 3× is excluded, as is the same action repeated on the
+  same URL (reload loops);
+- an optional sub-goal planner (`JEV_PLANNER=1`: the text model splits the task once, DONE advances to the next sub-goal, a
+  text-model check rejects a sub-goal DONE the page does not show).
+Server side: wide choices are split into chunks above 60 options (a 160-link page gave each label ~4 tokens and a flat
+distribution), and a one-option choice no longer crashes the fast path.
+
+**What changed in the data** (all collected in the real browser, no suite pages):
+- `rollouts2.py`: scripted search → `PRESS_ENTER`/submit → DONE, "open the article about X", scroll-to-next-page, scroll-then-click,
+  `<select>` with intent-style goals ("Sort by price: low to high."), type-then-pick-a-suggestion; 2.3k cases on 206 sites.
+- `convert_nnetnav.py`: 8.9k steps from [stanfordnlp/nnetnav-live](https://huggingface.co/datasets/stanfordnlp/nnetnav-live)
+  (element tables capped at 45, held out by site).
+- target lists sampled to the server's chunk width (≤ 40) and final-round width (2–6) so labels are never truncated in training.
+
+**Where it breaks: long horizons.** On held-out synthetic forms (below) v14s ends flight and hotel searches after 1–2 steps (DONE
+too early) or loops. Jev's own Google Flights run needs 11 actions: one-way, type origin → pick the suggestion, type destination →
+pick the suggestion, open the calendar, pick the day, Done, Search.
+
+**In progress: v15s** (clean retrain from the base, no DAgger, no suite pages) adds:
+- `code/finetune/webgym/`: a local synthetic web environment (flight / hotel search forms, shop listings) with randomized widget
+  implementations — autocomplete comboboxes inline or behind a trigger button, month-grid date pickers with Done/Apply, radio /
+  segmented / native / custom dropdowns, steppers and popups, cookie banners, forms pushed below the fold. A scripted expert
+  reads the page's own state, acts through jev's observations (so the format is exactly what the agent sees), and records each
+  step twice: under the full task and under the current sub-goal, plus a *verified* sub-goal DONE whenever a sub-goal is
+  satisfied. 12 % harmless detours teach recovery (including going back after a wrong submission). 1,408 successful episodes,
+  32k cases; seeds `900000 + 10·i` are held out for `eval_gym.py`.
+- `annotate_subgoals.py`: Mind2Web trajectories segmented into planner-style sub-goals by the local Qwen
+  (trajectory-conditioned, Plan-and-Act style), 5.1k sub-goal-conditioned steps. Sub-goal DONE labels come only from webgym,
+  where they are checked against page state.
+Results will be added here when the run finishes.
+
 ## What changed relative to the original laya
 
 | | original laya (typed-decisions) | this repo |
@@ -41,7 +99,7 @@ no paid API: the text helper and the DAgger teacher are a local Qwen3-8B-AWQ ser
 | latency per browser step (3 questions, 30–65 candidates) | 50–200 ms | 41–50 ms (v10), **17–23 ms** (v10s / v11s) |
 | backbone | ModernBERT-large 421M | v10: same; v10s / v11s: mmBERT-base 322M |
 | input format | jev's state verbatim (element table as JSON inside the state, truncated by the 1024-token window) | **format v2/v3**: elements live only in the option list (full label + role + current value), state keeps title / URL / history / 1.2–1.5k chars of text, `head_max_len` 512 → 768 |
-| training data | LocalLLaMA/typed-decisions | 5,244 reverse-generated goals on 421 crawled pages (Qwen writes "the goal a user would state to need this element"), 700 real DONE states (clicks actually executed), 659 step-2 negatives, [Mind2Web](https://huggingface.co/datasets/osunlp/Mind2Web) train (7,296 steps, candidates re-rendered as an element table), 177 on-policy DAgger corrections |
+| training data | LocalLLaMA/typed-decisions | 5,244 reverse-generated goals on 421 crawled pages (Qwen writes "the goal a user would state to need this element"), 700 real DONE states (clicks actually executed), 659 step-2 negatives, [Mind2Web](https://huggingface.co/datasets/osunlp/Mind2Web) train (7,296 steps, candidates re-rendered as an element table), 177 on-policy DAgger corrections (later found to contain suite-A tasks, see the progress update) |
 | training | — | laya's RLCD recipe (noisy-logit policy gradient + soft CE), single GPU, no gradient checkpointing, 4 epochs (~2 h for v10, ~1 h for v10s), post-hoc temperature |
 | inference | HF eager + autocast | optional TileLang fast path ([PR #25 to laya](https://github.com/NandhaKishorM/laya/pull/25)): fused GEMM/GEGLU/LayerNorm/RoPE, sliding-window flash attention, bf16-resident weights, CUDA graphs — 4–5× lower per-call latency, identical answers |
 
@@ -76,11 +134,12 @@ DAgger teacher or System-2 fallback here. Further gains need a stronger teacher
 ## Files
 
 ```
+v14s/    mmBERT-base 322M,      format v3, head_max_len 768   (best: suite B 100 %; serve with chunk threshold 60)
 v10/     ModernBERT-large 421M, format v2, head_max_len 768   (best held-out top-1)
 v10s/    mmBERT-base 322M,      format v3, head_max_len 768   (17–23 ms per step; best on the live suite)
 v11s/    v10s data + 682 scripted scroll / search-submit / select trajectories (equivalent to v10s within noise)
 code/    finetune pipeline, laya systemone server, task suite, TileLang kernels, jev-ultrafast patch
-results/ per-run suite JSONs and logs behind every number above
+results/ per-run suite JSONs and logs behind every number above (round3/: v12s–v14s, suite B, webgym baseline)
 ```
 Each checkpoint is a laya checkpoint directory (`model.safetensors`, `encoder/`, `tokenizer/`, `rl_agent_config.json`); the config records
 `laya_fmt` and `head_max_len_train` so the server applies the matching input format automatically.

code/apps/browser_suite.py

@@ -41,8 +41,9 @@ TASKS = [
      lambda u, t, x: "/tag/love" in u),
     ("hn-past", "https://news.ycombinator.com/", "Open the 'past' page (front pages from previous days).",
      lambda u, t, x: "/front" in u),
-    ("ddg-search", "https://duckduckgo.com/", "Search for 'tilelang github' and show the results.",
-     lambda u, t, x: "q=" in u and "tilelang" in u.lower()),
+    # NOTE: DuckDuckGo (all front ends), Bing, Ecosia and Startpage block headless Chromium (bot pages); Marginalia does not.
+    ("web-search", "https://search.marginalia.nu/", "Search for 'tilelang github' and show the results.",
+     lambda u, t, x: "query=" in u and "tilelang" in u.lower()),
     ("arxiv-search", "https://arxiv.org/", "Search arXiv for 'flash attention' papers and show the results list.",
      lambda u, t, x: "search" in u and "flash" in u.lower()),
     ("flights", "https://www.google.com/travel/flights?hl=en", "Find one-way flights from Zurich to London on September 28, 2026, for one adult in economy. Stop when matching flight options are visible.",

code/apps/browser_suite_b.py

@@ -0,0 +1,60 @@
+"""Held-out suite B: 18 tasks on sites that appear in NO training source (crawl, rollouts, Mind2Web, NNetNav; checked
+against finetune/out/train_domains.json at start).  Same runner and services as apps/browser_suite.py.
+
+    python apps/browser_suite_b.py [name-filter]        REPEATS=3 SUITE_OUT=... for repeated runs
+"""
+import json, os, sys, time
+from urllib.parse import urlparse
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from browser_suite import run  # noqa: E402  (sets up the jev env vars)
+
+TASKS = [
+    # navigation
+    ("sqlite-download", "https://www.sqlite.org/", "Go to the Download page.", lambda u, t, x: "download.html" in u),
+    ("haskell-downloads", "https://www.haskell.org/", "Go to the Downloads page.", lambda u, t, x: "/downloads" in u),
+    ("zig-download", "https://ziglang.org/", "Open the download page.", lambda u, t, x: "/download" in u),
+    ("openbsd-faq", "https://www.openbsd.org/", "Open the FAQ.", lambda u, t, x: "faq" in u.lower()),
+    ("nginx-download", "https://nginx.org/", "Go to the download page.", lambda u, t, x: "download" in u),
+    ("xkcd-archive", "https://xkcd.com/", "Open the archive of all comics.", lambda u, t, x: "/archive" in u),
+    ("elixir-install", "https://elixir-lang.org/", "Open the installation instructions.", lambda u, t, x: "install" in u),
+    ("prometheus-download", "https://prometheus.io/", "Go to the Download page.", lambda u, t, x: "/download" in u),
+    ("ocaml-install", "https://ocaml.org/", "Open the page that explains how to install OCaml.", lambda u, t, x: "install" in u),
+    # site search
+    ("rosetta-quicksort", "https://rosettacode.org/wiki/Rosetta_Code", "Search Rosetta Code for 'quicksort'.",
+     lambda u, t, x: "quicksort" in u.lower() or "quicksort" in t.lower()),
+    ("arch-man-grep", "https://man.archlinux.org/", "Look up the manual page for 'grep'.", lambda u, t, x: "grep" in u.lower()),
+    ("debian-man-tar", "https://manpages.debian.org/", "Look up the manual page for 'tar'.", lambda u, t, x: "tar" in urlparse(u).path.lower() + urlparse(u).query.lower()),
+    ("rfc-http-semantics", "https://www.rfc-editor.org/search/rfc_search.php", "Search for RFCs about 'HTTP semantics' and show the results.",
+     lambda u, t, x: "semantics" in u.lower()),
+    ("nixos-ripgrep", "https://search.nixos.org/packages", "Search for the package 'ripgrep'.", lambda u, t, x: "ripgrep" in u.lower()),
+    ("ocaml-pkg-yojson", "https://ocaml.org/", "Search OCaml packages for 'yojson'.", lambda u, t, x: "yojson" in u.lower()),
+    ("shop-search-jacket", "https://www.scrapingcourse.com/ecommerce/", "Search the shop for 'jacket'.", lambda u, t, x: "s=jacket" in u.lower()),
+    # listing
+    ("shop-page2", "https://www.scrapingcourse.com/ecommerce/", "Go to page 2 of the products.", lambda u, t, x: "page/2" in u),
+    ("shop-sort-price", "https://www.scrapingcourse.com/ecommerce/", "Sort the products by price, from low to high.",
+     lambda u, t, x: "orderby=price" in u and "price-desc" not in u),
+]
+
+if __name__ == "__main__":
+    dom = lambda u: urlparse(u).netloc.lower().removeprefix("www.")
+    try:
+        seen = set(json.load(open(os.path.join(os.path.dirname(__file__), "..", "finetune", "out", "train_domains.json"))))
+        leak = sorted({dom(u) for _, u, _, _ in TASKS} & seen)
+        print("held-out check:", "OK, no suite-B domain appears in training data" if not leak else f"LEAK {leak}", flush=True)
+    except FileNotFoundError:
+        print("held-out check skipped (no finetune/out/train_domains.json)", flush=True)
+    flt = sys.argv[1] if len(sys.argv) > 1 else ""
+    repeats = int(os.environ.get("REPEATS", "1"))
+    rows = []
+    for name, url, goal, check in TASKS:
+        if flt and flt not in name: continue
+        for rep in range(repeats):
+            ok, steps, status, wall, final = run(name, url, goal, check)
+            rows.append((name, ok, steps, status, wall))
+            print(f"{'PASS' if ok else 'FAIL'}  {name:20s} steps={steps:2d} status={status:9s} {wall:5.1f}s  {final[:70]}", flush=True)
+    n = sum(r[1] for r in rows)
+    print(f"\n== {n}/{len(rows)} passed ({100*n/len(rows):.0f}%, {repeats} run(s) per task) | median wall {sorted(r[4] for r in rows)[len(rows)//2]:.1f}s")
+    per = {}
+    for r in rows: per.setdefault(r[0], []).append(r[1])
+    print("   per task: " + "  ".join(f"{k}={sum(v)}/{len(v)}" for k, v in per.items()))
+    json.dump([dict(zip(("name", "pass", "steps", "status", "wall"), r)) for r in rows], open(os.environ.get("SUITE_OUT", "/tmp/suite_b.json"), "w"), indent=1)

code/apps/browser_trace.py

@@ -0,0 +1,39 @@
+"""Run suite tasks (by name filter) with a per-step trace: operation, target label, typed text, page change, url.
+    python apps/browser_trace.py [name-filter] [max_steps=12]
+"""
+import json, os, sys, time
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from browser_suite import TASKS, Agent
+flt = sys.argv[1] if len(sys.argv) > 1 else ""; max_steps = int(sys.argv[2]) if len(sys.argv) > 2 else 12
+for name, url, goal, check in TASKS:
+    if flt and flt not in name.split(",") and flt not in name: continue
+    t0 = time.time(); page = None; status = "error"; err = ""
+    print(f"\n### {name}: {goal}", flush=True)
+    try:
+        with Agent(url, goal) as agent:
+            page = agent.state["page"]
+            print(f"  start: {len(page['actions'])} actions, title={page['title'][:50]!r}", flush=True)
+            for state in agent.run():
+                status = state["status"]; page = state["page"]
+                h = state["history"][-1] if state["history"] else None
+                d = state["decisions"][-1] if state["decisions"] else None
+                if state.get("plan") and len(state["plan"]) > 1 and state.get("_shown_plan") is None:
+                    state["_shown_plan"] = 1; print("  plan:", state["plan"], flush=True)
+                if state.get("subgoals_done") and state.get("_shown_sg") != state["subgoals_done"]:
+                    state["_shown_sg"] = state["subgoals_done"]; print(f"  -- sub-goal {state['subgoals_done']} done, now: {state['goal']!r}", flush=True)
+                if h and d and h["step"] == len(state["history"]):
+                    tp = d.get("target_probabilities") or {}
+                    top = sorted(tp.items(), key=lambda kv: -kv[1])[:3]
+                    print(f"  step {h['step']:2d} {d['operation']:10s} p_op={d['operation_probabilities'].get(d['operation'],0):.2f} "
+                          f"target={h['action'][:60]!r} text={h['text']!r} changed={h['page_changed']} -> {page['url'][:80]}", flush=True)
+                if len(state["history"]) >= max_steps: break
+            if status in ("done", "blocked"):
+                d = state["decisions"][-1] if state["decisions"] else None
+                print(f"  final decision: {status} p={d['operation_probabilities'].get(d['operation'],0) if d else 0:.2f}", flush=True)
+            time.sleep(1.5)
+            try: page = agent.browser.observe(screenshot=False)
+            except Exception: pass
+    except Exception as e:
+        status = f"error:{type(e).__name__}"; err = str(e)[:200]
+    ok = bool(page and check(page["url"], page["title"], page.get("text", "")))
+    print(f"  => {'PASS' if ok else 'FAIL'} status={status} {err} url={(page or {}).get('url','')[:90]} wall={time.time()-t0:.1f}s", flush=True)

code/finetune/annotate_subgoals.py

@@ -0,0 +1,117 @@
+"""Sub-goal annotation of real trajectories (Plan-and-Act style), with the SAME planner prompt the agent uses at run time.
+
+    python finetune/annotate_subgoals.py finetune/out/m2w_cases.jsonl finetune/out/m2w_sub_cases.jsonl [max_tasks=0] [workers=12]
+
+Per task (grouped by task_id, steps ordered by history length):
+  local Qwen (TEXT_MODEL_BASE_URL) segments the recorded steps, in order, into sub-goals written in the run-time
+  planner's style (trajectory-conditioned, so plan and steps agree).
+Emits, per step, a copy of the case with goal = its sub-goal (same gold).  With EMIT_DONE=1 it also emits, at every
+boundary, the next step's page with goal = the finished sub-goal and gold = DONE.  Single-sub-goal tasks are skipped.
+"""
+import json, os, sys, threading
+from collections import defaultdict
+from concurrent.futures import ThreadPoolExecutor
+sys.path.insert(0, "/home/ckl/projects/S/jev-ultrafast")
+import importlib.util
+_spec = importlib.util.spec_from_file_location("jq", "/home/ckl/projects/S/jev-ultrafast/jev_ultrafast/questions.py")
+jq = importlib.util.module_from_spec(_spec); _spec.loader.exec_module(jq)
+import httpx
+
+URL = os.environ.get("TEXT_MODEL_BASE_URL", "http://127.0.0.1:30000/v1").rstrip("/") + "/chat/completions"
+MODEL = os.environ.get("TEXT_MODEL", "Qwen/Qwen3-8B-AWQ")
+CLIENT = httpx.Client(timeout=120, trust_env=False)      # local server: never through a proxy
+# segment boundaries are right most of the time but not always; a wrong boundary turns into a wrong DONE label, so
+# sub-goal DONE cases come only from webgym (verified by page state) unless EMIT_DONE=1
+EMIT_DONE = os.environ.get("EMIT_DONE") == "1"
+
+SEGMENT = """You annotate a recorded browser trajectory with the sub-goals a planner would have written for it.
+Split the recorded steps, in their recorded order, into consecutive segments; each segment is one sub-goal.
+Write each sub-goal as an instruction for the part of the task those steps accomplish, in the task's own words: name
+what is set and repeat the value (patterns: "Set the <thing> to <value>", "Search for <query> and show the results",
+"Sort the results by <order>", "Open <item>"). Steps that only open a menu or a picker belong to the sub-goal they
+serve; a sub-goal ends with the step that actually commits its value. A task done in one go is one segment.
+Return JSON {"segments": [{"subgoal": "...", "last_step": k}, ...]} where last_step is the 0-based index of the segment's
+last step; last_step values increase and the final one is the last step index."""
+
+
+def chat(system, user, max_tokens=400):
+    r = CLIENT.post(URL, json={"model": MODEL, "max_tokens": max_tokens, "temperature": 0, "response_format": {"type": "json_object"},
+                               "chat_template_kwargs": {"enable_thinking": False},
+                               "messages": [{"role": "system", "content": system}, {"role": "user", "content": user}]})
+    r.raise_for_status()
+    return json.loads(r.json()["choices"][0]["message"]["content"])
+
+
+def step_repr(c):
+    t = f" '{c['history'][-1]['text']}'" if False else ""
+    lab = (c.get("label") or "")[:80]
+    return f"{c['gold_op']} {lab}".strip()
+
+
+def annotate(task_cases):
+    goal = task_cases[0]["goal"]
+    steps = [step_repr(c) for c in task_cases]
+    # typed / selected values are only visible in the next case's history; attach them for context
+    for i, c in enumerate(task_cases[:-1]):
+        h = task_cases[i + 1]["history"][-1] if task_cases[i + 1]["history"] else None
+        if h and h.get("text") and c["gold_op"] in ("TYPE_TEXT", "SELECT"):
+            steps[i] += f" = '{h['text']}'"
+    r = chat(SEGMENT, json.dumps({"task": goal, "steps": [f"{i}: {s}" for i, s in enumerate(steps)]}, ensure_ascii=False), max_tokens=600)
+    segs = r.get("segments") or []
+    plan, a, start = [], [None] * len(task_cases), 0
+    for sg in segs:
+        try:
+            last = int(sg["last_step"]); text = str(sg["subgoal"]).strip()
+        except Exception:
+            return None
+        if not text or last < start or last >= len(task_cases):
+            return None
+        for k in range(start, last + 1): a[k] = len(plan)
+        plan.append(text); start = last + 1
+    if start != len(task_cases) or len(plan) < 2:
+        return None
+    return plan, a
+
+
+def main():
+    src, dst = sys.argv[1], sys.argv[2]
+    max_tasks = int(sys.argv[3]) if len(sys.argv) > 3 else 0
+    workers = int(sys.argv[4]) if len(sys.argv) > 4 else 12
+    by = defaultdict(list)
+    for l in open(src):
+        c = json.loads(l); by[c.get("task_id") or c["goal"]].append(c)
+    tasks = [sorted(v, key=lambda c: len(c["history"])) for v in by.values()]
+    tasks = [t for t in tasks if len(t) >= 2]
+    if max_tasks: tasks = tasks[:max_tasks]
+    lock = threading.Lock(); stats = {"tasks": 0, "skipped": 0, "sub": 0, "done": 0, "err": 0}
+    out = open(dst, "w")
+
+    def work(tc):
+        try:
+            r = annotate(tc)
+        except Exception:
+            with lock: stats["err"] += 1
+            return
+        if r is None:
+            with lock: stats["skipped"] += 1
+            return
+        plan, a = r
+        rows = []
+        for i, c in enumerate(tc):
+            rows.append({**c, "goal": plan[a[i]], "skill": "m2w@sub", "plan": plan, "sub_index": a[i]})
+            if EMIT_DONE and i > 0 and a[i] > a[i - 1]:
+                rows.append({**c, "goal": plan[a[i - 1]], "gold_op": "DONE", "gold_id": "DONE", "kind": "done", "label": "",
+                             "skill": "subgoal_done", "plan": plan, "sub_index": a[i - 1]})
+        with lock:
+            for r_ in rows: out.write(json.dumps(r_, ensure_ascii=False) + "\n")
+            stats["tasks"] += 1; stats["sub"] += sum(r_["skill"] == "m2w@sub" for r_ in rows); stats["done"] += sum(r_["skill"] == "subgoal_done" for r_ in rows)
+            if stats["tasks"] % 50 == 0: print(" ", stats, flush=True)
+
+    with ThreadPoolExecutor(workers) as ex:
+        list(ex.map(work, tasks))
+    out.close()
+    print("done", stats, "->", dst)
+
+
+if __name__ == "__main__":
+    main()

code/finetune/build_items.py

@@ -2,7 +2,7 @@
 
     python finetune/build_items.py out/pages.jsonl out/cases.jsonl out/
 """
-import json, os, sys
+import json, random, os, sys
 import torch
 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
 from common_ft import build_request, gold_for
@@ -11,6 +11,8 @@ from laya.common import QTYPES, build_sequence, render_options
 
 MAX_LEN, HEAD_MAX_LEN = 1024, int(os.environ.get('LAYA_HEAD', '512'))
 EVAL_EVERY = 5   # pages with index % 5 == 0 are held out
+MAX_TARGETS = int(os.environ.get('MAX_TARGETS', '40'))
+FINAL_P = float(os.environ.get('FINAL_P', '0.2'))   # share of target items built like the server's final round
 
 
 def main():
@@ -24,18 +26,31 @@ def main():
     items, n_skip, ev = [], 0, []
     import hashlib
     STOPS = [" Stop when it is open.", " Stop once that page is visible.", " Then stop.", " Finish when it has loaded.", ""]
+    # evaluation hygiene: never train on a suite task's start page (suite A and B start URLs), whatever the source
+    import re as _re
+    def _norm(u): return (u or "").split("#")[0].rstrip("/").replace("://www.", "://")
+    suite_urls = set()
+    for f in ("apps/browser_suite.py", "apps/browser_suite_b.py"):
+        if os.path.exists(f):
+            suite_urls |= {_norm(u) for u in _re.findall(r'\("[a-z0-9-]+", "(https?://[^"]+)"', open(f).read())}
+    n_suite = 0
     for c in cases:
+        if _norm(c.get("url") or (c.get("page_obj") or {}).get("url")) in suite_urls or (c.get("page", -1) >= 0 and _norm(pages[c["page"]]["url"]) in suite_urls):
+            n_suite += 1; continue
+        if c.get("source") == "dagger":
+            continue
         if c["kind"] == "done" and "page_obj" not in c:
             continue  # template DONE cases leak phrasing; real ones come from make_done_cases.py
         page = c.get("page_obj") or pages[c["page"]]
         h = int(hashlib.md5(c["goal"].encode()).hexdigest(), 16)
-        goal = c["goal"] + STOPS[h % len(STOPS)]
+        is_sub = str(c.get("skill", "")).endswith("@sub") or c.get("skill") == "subgoal_done"
+        goal = c["goal"] + ("" if is_sub or c.get("source") == "webgym" else STOPS[h % len(STOPS)])
         c = {**c, "goal": goal}
         state, questions, targets, controls = build_request(page, goal, c.get("history", []))
         gop, gidx = gold_for(c, targets, controls)
         if gop is None:
             n_skip += 1; continue
-        held = (hashlib.md5(c["website"].encode()).digest()[0] % EVAL_EVERY == 0) if c.get("source") == "mind2web" else (c["page"] % EVAL_EVERY == 0)
+        held = (hashlib.md5(c["website"].encode()).digest()[0] % EVAL_EVERY == 0) if c.get("source") in ("mind2web", "nnetnav") else (c["page"] % EVAL_EVERY == 0)
         if held:
             ev.append({**c, "gold_index": gidx}); continue
         golds = {"operation": gop}
@@ -43,6 +58,17 @@ def main():
             golds[gop.lower() + "_target"] = gidx
         for qid, gold in golds.items():
             q = questions[qid]; keys = list(q["criteria"])
+            rs = random.Random(h + len(keys) + len(qid))
+            if qid != "operation" and len(keys) > 8 and rs.random() < FINAL_P:
+                width = rs.randint(2, 6)          # the server's final round: only the chunk winners (2-6 options)
+            else:
+                width = MAX_TARGETS
+            if qid != "operation" and len(keys) > width:
+                # inference splits wide choices into chunks of <= MAX_TARGETS (systemone_server MAXOPT); train on the same
+                # widths, so option labels are never truncated to a few tokens: gold + a random sample, page order kept
+                keep = set(rs.sample([k for k in keys if k != gold], width - 1)) | {gold}
+                keys = [k for k in keys if k in keep]
+                q = {**q, "criteria": {k: q["criteria"][k] for k in keys}}
             target = [1.0 if k == gold else 0.0 for k in keys]
             qq = {"t": "choice", "ins": json.dumps(q["instructions"]), "crit": q["criteria"]}
             seq, markers = build_sequence(tok, state, qq, MAX_LEN, HEAD_MAX_LEN)
@@ -50,13 +76,18 @@ def main():
                 n_skip += 1; continue
             item = {"ids": seq, "markers": markers, "qtype": QTYPES["choice"], "target": target, "label": keys.index(gold), "qid": qid, "gold_op": gop}
             # class balance: CLICK dominates the operation question, so repeat the rare operations
-            reps = {"DONE": 4, "TYPE_TEXT": 3, "SELECT": 3}.get(gop, 1) if qid == "operation" else 1
+            reps = {"DONE": 4, "TYPE_TEXT": 3, "SELECT": 3, "PRESS_ENTER": 4, "SCROLL_DOWN": 2}.get(gop, 1) if qid == "operation" else 1
             if c.get("source") == "dagger":
-                reps *= 5   # on-policy teacher corrections from real tasks: few but exactly where the policy fails
+                reps *= 5
+            if c.get("source") == "webgym" or str(c.get("skill", "")).endswith("@sub"):
+                reps = 1      # webgym already pairs every DONE with its "keep going" twin; no op-level oversampling
+            if c.get("source") == "rollout":
+                reps *= 3   # scripted scroll / search-submit / select trajectories: the skills the model lacked   # on-policy teacher corrections from real tasks: few but exactly where the policy fails
             items.extend([item] * reps)
     torch.save(items, os.path.join(out, "train_items.pt"))
     with open(os.path.join(out, "eval_cases.jsonl"), "w") as f:
         for c in ev: f.write(json.dumps(c, ensure_ascii=False) + "\n")
+    print(f"dropped {n_suite} cases on suite start pages; dagger cases are never used")
     lens = [len(i["ids"]) for i in items]
     import collections
     print("operation label counts:", dict(collections.Counter(i["gold_op"] for i in items if i["qid"] == "operation")))

code/finetune/common_ft.py

@@ -69,7 +69,7 @@ def gold_for(case, targets, controls):
     op = case["gold_op"]
     if op == "DONE":
         return "DONE", None
-    if op in ("SCROLL_DOWN", "SCROLL_UP", "WAIT"):          # page-level controls: operation question only
+    if op in ("SCROLL_DOWN", "SCROLL_UP", "WAIT", "PRESS_ENTER"):   # page-level controls: operation question only
         return (op, None) if op in {k.upper() for k in controls} else (None, None)
     for index, a in targets.get(op, {}).items():
         if a["id"] == case["gold_id"]:

code/finetune/convert_nnetnav.py

@@ -0,0 +1,186 @@
+"""NNetNav-live (stanfordnlp/nnetnav-live, WebArena-style accessibility-tree trajectories on live sites) -> jev-format cases.
+
+    python finetune/convert_nnetnav.py finetune/data/nnetnav/train.jsonl finetune/out/nnetnav_cases.jsonl [max_cases=20000] [seed=3]
+
+Each record is one step: the user message holds OBSERVATION (accessibility tree), URL, OBJECTIVE, PREVIOUS ACTIONS; the
+assistant message ends with ```action```.  Mapping to jev's action space:
+  click [id]                  -> CLICK  on the element  (clickable link/button/checkbox/... nodes)
+  type [id] [text] [enter]    -> TYPE_TEXT on the element (textbox/searchbox/combobox); the implicit Enter is dropped
+  scroll [down|up]            -> SCROLL_DOWN / SCROLL_UP (page-level controls)
+  stop [...]                  -> DONE
+  hover / press / goto / go_back / new_tab / tab_focus / close_tab -> skipped
+The element table is the tree's interactive nodes in document order (labels = accessible name, role kept), the page text
+is the tree's static text.  Cases carry `website` so build_items.py can hold out whole sites, and `source: "nnetnav"`.
+Sampling: the action mix is capped so CLICK does not swamp TYPE_TEXT / SCROLL / DONE (see CAP).
+"""
+import json, random, re, sys
+from collections import Counter
+from urllib.parse import urlparse
+
+NODE = re.compile(r"^\s*\[(\d+)\] (\w+) '((?:[^'\\]|\\.)*)'(.*)$")
+STATIC = re.compile(r"^\s*StaticText '((?:[^'\\]|\\.)*)'")
+ACTION = re.compile(r"```\s*(\w+)(?:\s+\[([^\]]*)\])?(?:\s+\[((?:[^\]\\]|\\.)*)\])?(?:\s+\[([^\]]*)\])?\s*```")
+PREV = re.compile(r"^(\d+): (.*)$")
+CLICK_ROLES = {"link", "button", "checkbox", "radio", "menuitem", "tab", "option", "switch", "menuitemcheckbox", "menuitemradio", "treeitem", "combobox", "listbox", "img", "image", "gridcell", "cell", "row", "listitem", "heading", "generic", "LabelText", "DisclosureTriangle", "Details", "Summary", "slider", "spinbutton"}
+FILL_ROLES = {"textbox", "searchbox", "textarea"}
+ROLE_OUT = {"img": "image", "image": "image", "LabelText": "label", "DisclosureTriangle": "button", "Summary": "button", "Details": "group", "combobox": "combobox"}
+CAP = {"CLICK": 0.55, "TYPE_TEXT": 0.2, "SCROLL_DOWN": 0.1, "SCROLL_UP": 0.03, "DONE": 0.12}
+TEXT_CHARS = 6000
+MAX_ELEMENTS = 45
+
+
+def parse_obs(user):
+    """-> (tree_lines, url, objective, previous action strings)"""
+    m = re.search(r"OBSERVATION:\n(.*?)\nURL: (\S+)\nOBJECTIVE: (.*?)\nPREVIOUS ACTIONS:\n(.*)$", user, re.S)
+    if not m:
+        return None
+    tree, url, objective, prev = m.groups()
+    prevs = [PREV.match(l).group(2).strip() for l in prev.strip().splitlines() if PREV.match(l)]
+    prevs = [p for p in prevs if p and p != "None"]
+    return tree.splitlines(), url, objective.strip(), prevs
+
+
+def elements_of(tree_lines):
+    """Interactive nodes -> jev actions; static text -> page text.  Returns (actions, text, title, id->action)."""
+    actions, texts, by_id, title = [], [], {}, ""
+    for line in tree_lines:
+        if line.startswith("RootWebArea"):
+            m = re.match(r"RootWebArea '((?:[^'\\]|\\.)*)'", line)
+            title = m.group(1) if m else ""
+            continue
+        sm = STATIC.match(line)
+        if sm:
+            texts.append(sm.group(1)); continue
+        nm = NODE.match(line)
+        if not nm:
+            continue
+        nid, role, name, rest = nm.groups()
+        name = name.replace("\\'", "'").strip()
+        clickable = "clickable" in rest
+        if role in FILL_ROLES or (role == "combobox" and "hasPopup" not in rest and "autocomplete" in rest):
+            val = re.search(r"value='((?:[^'\\]|\\.)*)'", rest)
+            a = {"id": f"e{len(actions)+1}", "node": int(nid), "kind": "fill", "label": name or role, "role": "textbox" if role == "textarea" else role,
+                 "value": val.group(1) if val else ""}
+            actions.append(a); by_id[nid] = a
+            texts.append(name)
+            continue
+        if role in CLICK_ROLES and (clickable or role in ("link", "button", "checkbox", "radio", "tab", "menuitem", "option")):
+            if not name and role in ("generic", "listitem", "row", "cell", "gridcell", "heading"):
+                continue
+            a = {"id": f"e{len(actions)+1}", "node": int(nid), "kind": "click", "label": name or role, "role": ROLE_OUT.get(role, role)}
+            for attr in ("checked", "expanded", "selected"):
+                mm = re.search(attr + r"=(\w+)", rest)
+                if mm:
+                    a[attr] = mm.group(1) == "True"
+            actions.append(a); by_id[nid] = a
+            texts.append(name)
+            continue
+        if name and role in ("heading", "paragraph", "cell", "gridcell", "LabelText", "caption", "time", "code", "emphasis", "strong"):
+            texts.append(name)
+    return actions, "\n".join(t for t in texts if t)[:TEXT_CHARS], title, by_id
+
+
+def parse_action(assistant):
+    m = ACTION.findall(assistant)
+    if not m:
+        return None
+    verb, a1, a2, a3 = m[-1]
+    return verb.lower(), a1, a2.replace("\\]", "]") if a2 else a2, a3
+
+
+def history_of(prevs, tree_by_id_stack):
+    """jev history from the textual previous actions.  Labels come from 'where [id] is ...' suffixes when present."""
+    hist = []
+    for p in prevs[-10:]:
+        pa = ACTION.findall("```" + p + "```")
+        if not pa:
+            continue
+        verb, a1, a2, _ = pa[0]
+        verb = verb.lower()
+        where = re.search(r"where \[\d+\] is (\w+) '((?:[^'\\]|\\.)*)'", p)
+        label = where.group(2) if where else (f"[{a1}]" if a1 else "")
+        if verb == "click":
+            hist.append({"action": label, "kind": "click", "text": None, "page_changed": None})
+        elif verb == "type":
+            hist.append({"action": label, "kind": "fill", "text": a2, "page_changed": None})
+        elif verb == "scroll":
+            hist.append({"action": "Scroll down" if "down" in a1 else "Scroll up", "kind": "scroll", "text": None, "page_changed": False})
+        elif verb == "hover":
+            hist.append({"action": label, "kind": "click", "text": None, "page_changed": None})
+        else:
+            hist.append({"action": p[:60], "kind": verb, "text": None, "page_changed": None})
+    return hist
+
+
+def main():
+    src, dst = sys.argv[1], sys.argv[2]
+    max_cases = int(sys.argv[3]) if len(sys.argv) > 3 else 20000
+    rng = random.Random(int(sys.argv[4]) if len(sys.argv) > 4 else 3)
+    counts, kept, skipped = Counter(), [], Counter()
+    with open(src) as f:
+        for line in f:
+            try:
+                d = json.loads(line)
+            except Exception:
+                skipped["json"] += 1; continue
+            msgs = d.get("messages") or []
+            user = next((m["content"] for m in msgs if m["role"] == "user"), None)
+            asst = next((m["content"] for m in reversed(msgs) if m["role"] == "assistant"), None) or d.get("output", "")
+            if not user or not asst:
+                skipped["msgs"] += 1; continue
+            obs = parse_obs(user)
+            act = parse_action(asst)
+            if not obs or not act:
+                skipped["parse"] += 1; continue
+            tree, url, objective, prevs = obs
+            verb, a1, a2, a3 = act
+            actions, text, title, by_id = elements_of(tree)
+            if len(actions) < 2:
+                skipped["few_elements"] += 1; continue
+            # keep the element table within laya's option budget (head_max_len 768 ~ 45 options, like the Mind2Web
+            # conversion): the gold element plus a random sample of the others, document order preserved
+            if len(actions) > MAX_ELEMENTS:
+                gold_a = by_id.get(a1) if verb in ("click", "type") else None
+                pool = [a for a in actions if a is not gold_a]
+                keep = set(id(a) for a in rng.sample(pool, MAX_ELEMENTS - (1 if gold_a else 0)))
+                actions = [a for a in actions if id(a) in keep or a is gold_a]
+                for j, a in enumerate(actions):
+                    a["id"] = f"e{j+1}"
+            # page-level controls, as jev would offer them (scroll offered unless the tree is tiny)
+            if len(actions) > 8:
+                actions.append({"id": "scroll_down", "kind": "scroll", "label": "Scroll down", "delta": 560})
+            if any(p.startswith("scroll [down") for p in prevs):
+                actions.append({"id": "scroll_up", "kind": "scroll", "label": "Scroll up", "delta": -560})
+            actions.append({"id": "wait", "kind": "wait", "label": "Wait for the page to update"})
+            if verb == "click" and a1 in by_id and by_id[a1]["kind"] == "click":
+                gop, gid, kind, label = "CLICK", by_id[a1]["id"], "click", by_id[a1]["label"]
+            elif verb == "type" and a1 in by_id and by_id[a1]["kind"] == "fill":
+                gop, gid, kind, label = "TYPE_TEXT", by_id[a1]["id"], "fill", by_id[a1]["label"]
+            elif verb == "scroll" and "down" in (a1 or "") and any(a["id"] == "scroll_down" for a in actions):
+                gop, gid, kind, label = "SCROLL_DOWN", "scroll_down", "scroll", "Scroll down"
+            elif verb == "scroll" and "up" in (a1 or "") and any(a["id"] == "scroll_up" for a in actions):
+                gop, gid, kind, label = "SCROLL_UP", "scroll_up", "scroll", "Scroll up"
+            elif verb == "stop":
+                gop, gid, kind, label = "DONE", "DONE", "done", ""
+            else:
+                skipped["verb:" + verb] += 1; continue
+            site = urlparse(url).netloc
+            kept.append({"page": -1, "url": url, "title": title, "goal": objective, "gold_op": gop, "gold_id": gid, "kind": kind, "label": label,
+                         "history": history_of(prevs, None), "source": "nnetnav", "website": site, "skill": "nnetnav",
+                         "page_obj": {"url": url, "title": title, "text": text, "actions": actions}})
+            counts[gop] += 1
+    print("parsed", len(kept), dict(counts), "skipped", dict(skipped))
+    # cap the mix: sample per operation up to CAP share of max_cases
+    rng.shuffle(kept)
+    out, taken = [], Counter()
+    for c in kept:
+        if taken[c["gold_op"]] < CAP[c["gold_op"]] * max_cases:
+            out.append(c); taken[c["gold_op"]] += 1
+    with open(dst, "w") as f:
+        for c in out:
+            f.write(json.dumps(c, ensure_ascii=False) + "\n")
+    print("wrote", len(out), dict(taken), "->", dst)
+
+
+if __name__ == "__main__":
+    main()

code/finetune/infra.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+# infra.sh start_chrome | start_sglang [model] | start_s1 <ckpt_dir> [maxopt] | stop_s1 | stop_all | status
+S=${INFRA_DIR:-/tmp/laya-infra}; mkdir -p $S
+L=~/projects/S/laya
+case "$1" in
+  start_chrome)
+    curl -s -m 3 http://127.0.0.1:9222/json/version >/dev/null && { echo chrome up; exit 0; }
+    nohup chromium --headless=new --remote-debugging-port=9222 --user-data-dir=$S/chrome-profile --window-size=1120,780 --no-first-run --lang=en-US --accept-lang=en-US,en about:blank >$S/chrome.log 2>&1 &
+    sleep 3; curl -s -m 3 http://127.0.0.1:9222/json/version | head -c 120; echo ;;
+  start_sglang)
+    M=${2:-Qwen/Qwen3-8B-AWQ}
+    curl -s -m 3 http://127.0.0.1:30000/health >/dev/null && { echo sglang up; exit 0; }
+    cd $L; HF_HUB_OFFLINE=1 nohup ~/sglang-venv/bin/python -m sglang.launch_server --model-path $M --port 30000 --mem-fraction-static ${SGL_MEM:-0.35} --context-length 8192 --reasoning-parser qwen3 >$S/sglang.log 2>&1 &
+    echo "sglang starting (pid $!)";;
+  start_s1)
+    bash $0 stop_s1
+    cd $L; source env.sh; nohup env ESCALATE_TAU=${ESCALATE_TAU:-0} .venv/bin/python apps/systemone_server.py 8791 "$2" ${3:-999} >$S/s1.log 2>&1 &
+    for i in $(seq 1 90); do curl -s -m 2 http://127.0.0.1:8791/ >/dev/null 2>&1 && { echo "s1 up ($2)"; exit 0; }; sleep 2; done; echo "s1 FAILED"; tail -5 $S/s1.log;;
+  stop_chrome) pkill -f 'remote-debugging-port=922[2]' 2>/dev/null; sleep 2;;
+  stop_s1) pkill -f 'apps/systemone_serve[r]' 2>/dev/null; sleep 1;;
+  stop_all) bash $0 stop_s1; pkill -f 'sglang.launch_serve[r]' 2>/dev/null; pkill -f 'remote-debugging-port=9222' 2>/dev/null; echo stopped;;
+  status) for p in 9222 8791 30000; do (ss -ltn | grep -q ":$p ") && echo "$p up" || echo "$p down"; done; nvidia-smi --query-gpu=memory.used --format=csv,noheader;;
+esac

code/finetune/rollouts2.py

@@ -0,0 +1,255 @@
+"""Scripted multi-step trajectories in the real browser, round 2: the skills the 16-task suite still fails on.
+
+  search    : TYPE_TEXT query -> PRESS_ENTER (or CLICK a visible submit button) -> DONE on the results page
+  open      : "Open the article/page about X": TYPE_TEXT -> PRESS_ENTER -> (CLICK the matching result if a results
+              list appears) -> DONE
+  next-page : "Go to page 2 / next page": SCROLL_DOWN until a next/2/older link is visible -> CLICK -> DONE
+  scroll    : goal names an element only visible after scrolling -> SCROLL_DOWN -> CLICK -> DONE
+  select    : choose an option of a <select> -> SELECT -> DONE
+
+    python finetune/rollouts2.py sites.txt out/rollout2_cases.jsonl [max_sites=400] [seed=7]
+
+sites.txt: one URL per line (a "#" starts a comment).  Suite URLs are skipped.  Goals are templated (varied phrasing),
+no LLM needed.  Every case stores the observed page (page_obj) so build_items.py can rebuild the exact request.
+Runs against the headless Chromium on BU_CDP_URL (default 127.0.0.1:9222) and needs the patched jev-ultrafast
+(press_enter control, observe retry).
+"""
+import json, os, random, re, sys, time
+sys.path.insert(0, "/home/ckl/projects/S/jev-ultrafast")
+os.environ.setdefault("BU_CDP_URL", "http://127.0.0.1:9222")
+from jev_ultrafast.browser import Browser, StalePage
+
+SUITE_URLS = {"https://en.wikipedia.org/wiki/Main_Page", "https://news.ycombinator.com/", "https://github.com/tile-ai/tilelang", "https://www.python.org/",
+              "https://books.toscrape.com/", "https://the-internet.herokuapp.com/dropdown", "https://the-internet.herokuapp.com/checkboxes",
+              "https://quotes.toscrape.com/", "https://duckduckgo.com/", "https://search.marginalia.nu/", "https://arxiv.org/",
+              "https://www.google.com/travel/flights?hl=en"}
+SEARCH_T = ["Search for '{q}'.", "Search for '{q}' and show the results.", "Look up '{q}' using the search box and show the results.",
+            "Find results for '{q}'.", "Use the site search to find '{q}'.", "Search this site for {q}.", "Run a search for \"{q}\".",
+            "Show me search results for {q}.", "Search {site} for '{q}' and show the results list."]
+OPEN_T = ["Open the page about {q}.", "Open the article about {q}.", "Go to the entry for '{q}'.", "Find and open '{q}'.",
+          "Open '{q}' on this site.", "Search for {q} and open the matching page."]
+NEXT_T = ["Go to page 2.", "Go to page 2 of the list.", "Show the next page of results.", "Open the next page.", "Go to the next page of the catalogue.",
+          "Show page 2.", "Move on to the second page.", "Open the second page of the listing."]
+SCROLL_T = ["Open '{x}'.", "I want to see '{x}', take me there.", "Go to {x}.", "Find and click '{x}' on this page.",
+            "Open the '{x}' link further down the page.", "Navigate to '{x}'.", "Click '{x}'."]
+INTENT_T = ["{o}.", "{o}, please.", "I want to {ol}.", "Change it to {o}.", "Show everything with {f} set to {o}.", "Use '{o}'.", "Switch to {o}."]
+SELECT_T = ["Select '{o}' in the '{f}' dropdown.", "Choose {o} for {f}.", "Set '{f}' to '{o}'.", "Pick the option '{o}'.", "In the {f} dropdown, select {o}."]
+QUERIES = ["python tutorial", "flash attention", "climate change", "rust async", "linear algebra", "tilelang", "modernbert", "black holes",
+           "sourdough bread", "gpu kernels", "machine learning", "world war ii", "electric cars", "photosynthesis", "quantum computing",
+           "jazz history", "docker compose", "typescript generics", "renewable energy", "albert einstein", "marie curie", "solar system",
+           "coffee", "yoga", "kubernetes", "chess openings", "vitamin d", "roman empire", "neural networks", "recycling"]
+OPEN_Q = ["Albert Einstein", "Python programming language", "Marie Curie", "Photosynthesis", "Black hole", "Isaac Newton", "Quantum computing",
+          "Rust programming language", "Great Wall of China", "Ada Lovelace", "Solar System", "Machine learning", "Docker", "Jazz", "Coffee"]
+
+
+def lab(a): return a["label"].split(" → ")[0].strip()
+def site_of(url): return re.sub(r"^www\.", "", url.split("/")[2])
+def case(page, goal, gold_op, gold_id, kind, label, hist, tag):
+    return {"page": -1, "url": page["url"], "title": page["title"], "goal": goal, "gold_op": gold_op, "gold_id": gold_id, "kind": kind, "label": label,
+            "history": hist, "source": "rollout", "skill": tag, "page_obj": {k: page[k] for k in ("url", "title", "text", "actions")}}
+def hist_of(a, text=None, changed=True): return {"action": a["label"], "kind": a["kind"], "text": text, "page_changed": changed}
+def find(page, pred): return next((a for a in page["actions"] if pred(a)), None)
+def ctl(page, cid): return find(page, lambda a: a["id"] == cid)
+def observe(b, tries=8):
+    for i in range(tries):
+        try: return b.observe(screenshot=False)
+        except StalePage:
+            if i == tries - 1: raise
+            time.sleep(0.5)
+def act(b, a, page, text=None, wait=0.5):
+    b.act(a, page, text=text); time.sleep(wait); return observe(b)
+def changed(p, q): return p["url"] != q["url"] or p["title"] != q["title"]
+def search_field(page):
+    return find(page, lambda a: a["kind"] == "fill" and re.search(r"search|query|find|keyword|suche|rechercher|buscar|cerca|zoek|検索|搜索", a["label"], re.I)) \
+        or find(page, lambda a: a["kind"] == "fill" and a.get("role") in ("searchbox",))
+def submit_button(page, q):
+    return find(page, lambda a: a["kind"] == "click" and a.get("role") == "button" and re.search(r"^(search|go|submit|find|suchen|rechercher|buscar)\b", lab(a), re.I))
+
+
+def do_search(b, page, rng, out, site):
+    field = search_field(page)
+    if not field: return 0
+    q = rng.choice(QUERIES); goal = rng.choice(SEARCH_T).format(q=q, site=site)
+    out.write(json.dumps(case(page, goal, "TYPE_TEXT", field["id"], "fill", field["label"], [], "search"), ensure_ascii=False) + "\n"); n = 1
+    p2 = act(b, field, page, text=q)
+    pe, sub = ctl(p2, "press_enter"), submit_button(p2, q)
+    if not pe and not sub: return n
+    # after typing the goal is not done: submit.  Enter is the default; a visible submit button is used 30% of the time
+    use_click = bool(sub) and (not pe or rng.random() < 0.3)
+    if use_click:
+        out.write(json.dumps(case(p2, goal, "CLICK", sub["id"], "click", sub["label"], [hist_of(field, q, False)], "search"), ensure_ascii=False) + "\n")
+        step = hist_of(sub); p3 = act(b, sub, p2, wait=1.0)
+    else:
+        out.write(json.dumps(case(p2, goal, "PRESS_ENTER", "press_enter", "key", pe["label"], [hist_of(field, q, False)], "search"), ensure_ascii=False) + "\n")
+        step = hist_of(pe); p3 = act(b, pe, p2, wait=1.0)
+    n += 1
+    if changed(p3, p2) or q.split()[0].lower() in p3["text"].lower()[:3000]:
+        out.write(json.dumps(case(p3, goal, "DONE", "DONE", "done", "", [hist_of(field, q, False), step], "search"), ensure_ascii=False) + "\n"); n += 1
+    return n
+
+
+def do_open(b, page, rng, out, site):
+    field = search_field(page)
+    if not field: return 0
+    q = rng.choice(OPEN_Q); goal = rng.choice(OPEN_T).format(q=q)
+    out.write(json.dumps(case(page, goal, "TYPE_TEXT", field["id"], "fill", field["label"], [], "open"), ensure_ascii=False) + "\n"); n = 1
+    p2 = act(b, field, page, text=q)
+    pe = ctl(p2, "press_enter")
+    if not pe: return n
+    out.write(json.dumps(case(p2, goal, "PRESS_ENTER", "press_enter", "key", pe["label"], [hist_of(field, q, False)], "open"), ensure_ascii=False) + "\n"); n += 1
+    p3 = act(b, pe, p2, wait=1.2)
+    if not changed(p3, p2): return n
+    hist = [hist_of(field, q, False), hist_of(pe)]
+    words = [w.lower() for w in q.split() if len(w) > 2]
+    tl = p3["title"].lower()
+    title_hit = all(w in tl for w in words[:2]) and not re.search(r"search|results|suche|recherche", tl)
+    if title_hit:   # landed on the page itself (Wikipedia-style direct hit)
+        out.write(json.dumps(case(p3, goal, "DONE", "DONE", "done", "", hist, "open"), ensure_ascii=False) + "\n"); return n + 1
+    # a results list: click the result whose label matches the query best
+    res = [a for a in p3["actions"] if a["kind"] == "click" and a.get("role") == "link" and sum(w in a["label"].lower() for w in words) >= max(1, len(words) - 1)]
+    if not res: return n
+    tgt = res[0]
+    out.write(json.dumps(case(p3, goal, "CLICK", tgt["id"], "click", tgt["label"], hist, "open"), ensure_ascii=False) + "\n"); n += 1
+    try:
+        p4 = act(b, tgt, p3, wait=1.2)
+        if changed(p4, p3):
+            out.write(json.dumps(case(p4, goal, "DONE", "DONE", "done", "", hist + [hist_of(tgt)], "open"), ensure_ascii=False) + "\n"); n += 1
+    except Exception: pass
+    return n
+
+
+NEXT_RE = re.compile(r"^(next\b.*|[»›>→]+\s*next.*|2|page 2|go to page 2|older( posts| entries| \d+)?|older\s*[»›>→]|more|more results|show more|load more|see more|[›»]|next »|next page.*|nächste.*|suivant.*|siguiente.*|次へ|下一页)$", re.I)
+def next_link(page):
+    return find(page, lambda a: a["kind"] == "click" and a.get("role") in ("link", "button") and NEXT_RE.match(lab(a)) is not None)
+
+def do_next(b, page, rng, out, site):
+    if not page.get("scroll") or page["scroll"]["height"] < 1000: return 0
+    goal = rng.choice(NEXT_T); hist = []; p = page; n = 0; buf = []
+    for _ in range(int(os.environ.get("MAX_SCROLLS", "10"))):
+        nl = next_link(p)
+        if nl: break
+        sd = ctl(p, "scroll_down")
+        if not sd: return 0
+        buf.append(json.dumps(case(p, goal, "SCROLL_DOWN", "scroll_down", "scroll", "Scroll down", list(hist), "next"), ensure_ascii=False) + "\n")
+        p = act(b, sd, p, wait=0.4); hist.append(hist_of(sd, changed=False))
+    else:
+        return 0              # no next-page link on this page: the scroll steps would teach scrolling for nothing
+    out.writelines(buf); n = len(buf)     # (a visible next link without scrolling still teaches the 'next' semantics)
+    out.write(json.dumps(case(p, goal, "CLICK", nl["id"], "click", nl["label"], list(hist), "next"), ensure_ascii=False) + "\n"); n += 1
+    try:
+        p2 = act(b, nl, p, wait=1.0)
+        if changed(p2, p):
+            out.write(json.dumps(case(p2, goal, "DONE", "DONE", "done", "", hist + [hist_of(nl)], "next"), ensure_ascii=False) + "\n"); n += 1
+    except Exception: pass
+    return n
+
+
+def do_scroll(b, page, rng, out, site):
+    if not page.get("scroll") or page["scroll"]["height"] < 1300: return 0
+    before = {a["label"] for a in page["actions"] if a["kind"] == "click"}
+    sd = ctl(page, "scroll_down")
+    if not sd: return 0
+    p2 = act(b, sd, page, wait=0.4)
+    new = [a for a in p2["actions"] if a["kind"] == "click" and a["label"] not in before and 3 <= len(lab(a)) <= 60 and a.get("role") in ("link", "button")]
+    if not new: return 0
+    n = 0
+    for tgt in rng.sample(new, min(2, len(new))):
+        goal = rng.choice(SCROLL_T).format(x=lab(tgt))
+        out.write(json.dumps(case(page, goal, "SCROLL_DOWN", "scroll_down", "scroll", "Scroll down", [], "scroll"), ensure_ascii=False) + "\n")
+        out.write(json.dumps(case(p2, goal, "CLICK", tgt["id"], "click", tgt["label"], [hist_of(sd, changed=False)], "scroll"), ensure_ascii=False) + "\n"); n += 2
+    tgt = new[0]
+    try:
+        p3 = act(b, tgt, p2, wait=0.8)
+        if changed(p3, p2):
+            goal = rng.choice(SCROLL_T).format(x=lab(tgt))
+            out.write(json.dumps(case(p3, goal, "DONE", "DONE", "done", "", [hist_of(sd, changed=False), hist_of(tgt)], "scroll"), ensure_ascii=False) + "\n"); n += 1
+    except Exception: pass
+    return n
+
+
+def do_select(b, page, rng, out, site):
+    sels = [a for a in page["actions"] if a["kind"] == "select" and a.get("value") and a["value"] != a.get("current_value")]
+    if not sels: return 0
+    by_field = {}
+    for a in sels: by_field.setdefault(a["node"], []).append(a)
+    node, opts = rng.choice(list(by_field.items()))
+    opt = rng.choice(opts); f = lab(opt); o = opt["label"].split(" → ")[-1].strip()
+    # half the goals state the intent in the option's own words ("Sort by price: low to high."), the way users phrase it
+    if rng.random() < 0.5 and 3 <= len(o) <= 60:
+        goal = rng.choice(INTENT_T).format(o=o, ol=o[0].lower() + o[1:], f=f)
+    else:
+        goal = rng.choice(SELECT_T).format(o=o, f=f)
+    out.write(json.dumps(case(page, goal, "SELECT", opt["id"], "select", opt["label"], [], "select"), ensure_ascii=False) + "\n")
+    try:
+        p2 = act(b, opt, page, wait=0.3)
+        out.write(json.dumps(case(p2, goal, "DONE", "DONE", "done", "", [hist_of(opt, changed=False)], "select"), ensure_ascii=False) + "\n"); return 2
+    except Exception: return 1
+
+
+
+CITIES = ["Zurich", "London", "Paris", "Berlin", "Tokyo", "New York", "Madrid", "Rome", "Amsterdam", "Vienna", "Munich", "Singapore",
+          "Sydney", "Toronto", "Chicago", "Barcelona", "Lisbon", "Prague", "Dublin", "Seoul", "Bangkok", "Istanbul", "Oslo", "Helsinki"]
+AUTO_T = ["Set the destination to {x}.", "Choose {x} as the departure city.", "Set where from to {x}.", "Set where to to {x}.",
+          "Pick {x} from the suggestions.", "Enter {x} and select it from the list.", "Search for {x} and choose the matching suggestion.",
+          "Select {x} as the location."]
+
+def is_combo(a):
+    return a["kind"] == "fill" and (a.get("role") in ("combobox", "searchbox") or re.search(r"from|to\b|where|destination|origin|city|location|airport|search", a["label"], re.I))
+
+def do_autocomplete(b, page, rng, out, site):
+    """TYPE_TEXT a partial value into a combobox -> CLICK the suggestion that appeared -> DONE."""
+    field = find(page, is_combo)
+    if not field: return 0
+    travel = re.search(r"from|to\b|where|destination|origin|city|location|airport|hotel|stay", field["label"], re.I)
+    full = rng.choice(CITIES) if travel else rng.choice(QUERIES)
+    typed = full if rng.random() < 0.6 else full[:max(3, len(full) // 2)]
+    goal = (rng.choice(AUTO_T) if travel else rng.choice(["Search for {x} and choose the matching suggestion.",
+            "Type {x} and pick it from the suggestions.", "Find {x} using the search suggestions."])).format(x=full)
+    before = {a["label"] for a in page["actions"] if a["kind"] == "click"}
+    p2 = act(b, field, page, text=typed, wait=1.2)
+    key = full.split()[0].lower()[:5]
+    sugg = [a for a in p2["actions"] if a["kind"] == "click" and a["label"] not in before and key in a["label"].lower()
+            and a.get("role") in ("option", "listitem", "link", "button", "menuitem", "gridcell", "treeitem")]
+    if not sugg: return 0
+    tgt = sugg[0]
+    out.write(json.dumps(case(page, goal, "TYPE_TEXT", field["id"], "fill", field["label"], [], "autocomplete"), ensure_ascii=False) + "\n")
+    out.write(json.dumps(case(p2, goal, "CLICK", tgt["id"], "click", tgt["label"], [hist_of(field, typed, True)], "autocomplete"), ensure_ascii=False) + "\n")
+    n = 2
+    try:
+        p3 = act(b, tgt, p2, wait=1.0)
+        out.write(json.dumps(case(p3, goal, "DONE", "DONE", "done", "", [hist_of(field, typed, True), hist_of(tgt)], "autocomplete"), ensure_ascii=False) + "\n"); n += 1
+    except Exception: pass
+    return n
+
+SKILLS = (("autocomplete", do_autocomplete), ("search", do_search), ("open", do_open), ("next", do_next), ("scroll", do_scroll), ("select", do_select))
+if os.environ.get("SKILLS"):                     # SKILLS=next,search : run only these skills
+    SKILLS = tuple(x for x in SKILLS if x[0] in os.environ["SKILLS"].split(","))
+
+def main():
+    sites = [l.strip() for l in open(sys.argv[1]) if l.strip() and not l.startswith("#")]
+    out_f = sys.argv[2]; mx = int(sys.argv[3]) if len(sys.argv) > 3 else 400; seed = int(sys.argv[4]) if len(sys.argv) > 4 else 7
+    rng = random.Random(seed); rng.shuffle(sites)
+    if len(sys.argv) > 5:                      # "i/n": this worker takes every n-th site starting at i (parallel workers)
+        i, n = map(int, sys.argv[5].split("/")); sites = sites[i::n]
+    counts = {k: 0 for k, _ in SKILLS}; fails = {}; done = 0; t0 = time.time()
+    with open(out_f, "w") as out:
+        for url in sites:
+            if done >= mx: break
+            if url in SUITE_URLS: continue
+            site = site_of(url)
+            for skill, fn in SKILLS:
+                try:
+                    b = Browser(url)
+                    try:
+                        page = observe(b); n = fn(b, page, rng, out, site)
+                    finally:
+                        b.close()
+                    counts[skill] += n
+                except Exception as e:
+                    fails[type(e).__name__] = fails.get(type(e).__name__, 0) + 1
+            done += 1
+            if done % 20 == 0: print(f"  {done} sites {time.time()-t0:5.0f}s  {counts}  fails={fails}", flush=True)
+    print("wrote", counts, "fails", fails, "->", out_f)
+
+if __name__ == "__main__":
+    main()

code/finetune/run_suite_v12s.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# After v12s training: 16-task suite x3 for v12s and v10s under the patched jev harness (PRESS_ENTER, observe retry, stall guard).
+cd /home/ckl/projects/S/laya; O=finetune/out; I="bash finetune/infra.sh"
+until grep -q "V12S_DONE" $O/v12s.log; do sleep 60; done
+pkill -f 'rollouts2[.]py' 2>/dev/null
+$I start_chrome >/dev/null; SGL_MEM=0.5 $I start_sglang Qwen/Qwen3-8B-AWQ >/dev/null
+until curl -s -m 3 http://127.0.0.1:30000/health >/dev/null; do sleep 5; done
+for ck in laya-browser-v12s laya-browser-v10s; do
+  echo "== suite $ck x3 (patched harness)"
+  $I start_s1 $PWD/$O/$ck 999 >/dev/null
+  (cd ../jev-ultrafast && REPEATS=3 SUITE_OUT=$PWD/../laya/$O/suite_h2_$ck.json timeout 3600 .venv/bin/python ../laya/apps/browser_suite.py 2>&1 | grep -E "^(PASS|FAIL)|^==|per task")
+done
+echo SUITE_V12S_DONE

code/finetune/run_v12s.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+# v12s: v10s recipe (mmBERT-base, format v3, head 768) + rollouts2 (search/open/next-page/scroll/select with PRESS_ENTER) + NNetNav-live.
+# usage: bash finetune/run_v12s.sh [epochs=4]     (stop sglang first: training needs the GPU)
+cd /home/ckl/projects/S/laya && source env.sh
+export LAYA_BASE=$(ls -d ~/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/*/multilingual | head -1)
+export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CKPT=${CKPT:-0} COMPILE=0 LAYA_FMT=v3 LAYA_HEAD=768
+P=.venv/bin/python; O=finetune/out; EP=${1:-4}
+cat $O/rollout2_cases.*.jsonl > $O/rollout2_cases.jsonl 2>/dev/null
+echo "== rollout2 cases: $(wc -l < $O/rollout2_cases.jsonl)   nnetnav cases: $(wc -l < $O/nnetnav_cases.jsonl 2>/dev/null || echo 0)"
+echo "== build v12s"
+$P finetune/build_items.py $O/pages.jsonl $O/cases.jsonl $O/ $O/done_cases.jsonl $O/step2_cases.jsonl $O/m2w_cases.jsonl $O/dagger_cases.jsonl $O/rollout_cases.jsonl $O/rollout2_cases.jsonl $O/nnetnav_cases.jsonl 2>&1 | grep -v Warn | tail -5
+echo "== train v12s ($EP epochs)"; $P finetune/train.py $O/train_items.pt $O/laya-browser-v12s $EP 2>&1 | grep --line-buffered -E "=== epoch|saved|Error|Traceback|loss"
+echo "== calibrate"; $P finetune/calibrate.py $O/pages.jsonl $O/eval_cases.jsonl $PWD/$O/laya-browser-v12s 2>&1 | grep -vE "TileLang|Warn|warn|Fetch"
+echo "== eval v12s"; $P finetune/eval.py $O/pages.jsonl $O/eval_cases.jsonl $PWD/$O/laya-browser-v12s 2>&1 | grep -vE "TileLang|Warn|warn|Fetch"
+echo V12S_DONE

code/finetune/run_v13s.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+# v13s: continue from v12s for 1-2 epochs on v12s data + next-page/select rollouts, with target lists sampled to the
+# server's chunk width (MAX_TARGETS=40) and final-round width (2-6, FINAL_P=0.2).  Then suites A and B.
+cd /home/ckl/projects/S/laya && source env.sh
+export LAYA_BASE=$PWD/finetune/out/laya-browser-v12s
+export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CKPT=${CKPT:-1} COMPILE=0 LAYA_FMT=v3 LAYA_HEAD=768 MAX_TARGETS=40 FINAL_P=0.2
+P=.venv/bin/python; O=finetune/out; EP=${1:-1}; I="bash finetune/infra.sh"
+pkill -f 'systemone_serve[r]' ; pkill -f 'sglang.launch_serve[r]' ; sleep 5
+cat $O/rollout2_cases.*.jsonl > $O/rollout2_cases.jsonl
+echo "== rollout2 cases: $(wc -l < $O/rollout2_cases.jsonl)"
+echo "== build v13s"
+$P finetune/build_items.py $O/pages.jsonl $O/cases.jsonl $O/ $O/done_cases.jsonl $O/step2_cases.jsonl $O/m2w_cases.jsonl $O/dagger_cases.jsonl $O/rollout_cases.jsonl $O/rollout2_cases.jsonl $O/nnetnav_cases.jsonl 2>&1 | grep -v Warn | tail -3
+echo "== train v13s ($EP epochs from v12s)"; $P finetune/train.py $O/train_items.pt $O/laya-browser-v13s $EP 2>&1 | grep --line-buffered -E "=== epoch|saved|Error|Traceback"
+echo "== calibrate"; $P finetune/calibrate.py $O/pages.jsonl $O/eval_cases.jsonl $PWD/$O/laya-browser-v13s 2>&1 | grep -vE "TileLang|Warn|warn|Fetch"
+echo "== eval"; $P finetune/eval.py $O/pages.jsonl $O/eval_cases.jsonl $PWD/$O/laya-browser-v13s 2>&1 | grep -vE "TileLang|Warn|warn|Fetch|return Agent"
+$I start_chrome >/dev/null; SGL_MEM=0.5 $I start_sglang Qwen/Qwen3-8B-AWQ >/dev/null
+until curl -s -m 3 http://127.0.0.1:30000/health >/dev/null; do sleep 5; done
+$I start_s1 $PWD/$O/laya-browser-v13s 40 >/dev/null
+echo "== suite A x3 (v13s, chunk 40)"
+(cd ../jev-ultrafast && REPEATS=3 SUITE_OUT=$PWD/../laya/$O/suiteA_v13s.json timeout 3600 .venv/bin/python ../laya/apps/browser_suite.py 2>&1 | grep -E "^(PASS|FAIL)|^==|per task")
+echo "== suite B x3 (v13s, chunk 40)"
+(cd ../jev-ultrafast && REPEATS=3 SUITE_OUT=$PWD/../laya/$O/suiteB_v13s.json timeout 3600 .venv/bin/python ../laya/apps/browser_suite_b.py 2>&1 | grep -E "^(PASS|FAIL)|^==|per task|held-out")
+echo V13S_DONE

code/finetune/run_v14s.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+# v14s: continue from v13s for 1-2 epochs on v12s data + next-page/select rollouts, with target lists sampled to the
+# server's chunk width (MAX_TARGETS=40) and final-round width (2-6, FINAL_P=0.2).  Then suites A and B.
+cd /home/ckl/projects/S/laya && source env.sh
+export LAYA_BASE=$PWD/finetune/out/laya-browser-v13s
+export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CKPT=${CKPT:-1} COMPILE=0 LAYA_FMT=v3 LAYA_HEAD=768 MAX_TARGETS=40 FINAL_P=0.2
+P=.venv/bin/python; O=finetune/out; EP=${1:-1}; I="bash finetune/infra.sh"
+pkill -f 'systemone_serve[r]' ; pkill -f 'sglang.launch_serve[r]' ; sleep 5
+cat $O/rollout2_cases.*.jsonl > $O/rollout2_cases.jsonl
+echo "== rollout2 cases: $(wc -l < $O/rollout2_cases.jsonl)"
+echo "== build v14s"
+$P finetune/build_items.py $O/pages.jsonl $O/cases.jsonl $O/ $O/done_cases.jsonl $O/step2_cases.jsonl $O/m2w_cases.jsonl $O/dagger_cases.jsonl $O/rollout_cases.jsonl $O/rollout2_cases.jsonl $O/nnetnav_cases.jsonl 2>&1 | grep -v Warn | tail -3
+echo "== train v14s ($EP epochs from v13s)"; $P finetune/train.py $O/train_items.pt $O/laya-browser-v14s $EP 2>&1 | grep --line-buffered -E "=== epoch|saved|Error|Traceback"
+echo "== calibrate"; $P finetune/calibrate.py $O/pages.jsonl $O/eval_cases.jsonl $PWD/$O/laya-browser-v14s 2>&1 | grep -vE "TileLang|Warn|warn|Fetch"
+echo "== eval"; $P finetune/eval.py $O/pages.jsonl $O/eval_cases.jsonl $PWD/$O/laya-browser-v14s 2>&1 | grep -vE "TileLang|Warn|warn|Fetch|return Agent"
+$I start_chrome >/dev/null; SGL_MEM=0.5 $I start_sglang Qwen/Qwen3-8B-AWQ >/dev/null
+until curl -s -m 3 http://127.0.0.1:30000/health >/dev/null; do sleep 5; done
+$I start_s1 $PWD/$O/laya-browser-v14s 60 >/dev/null
+echo "== suite A x3 (v14s, chunk 60)"
+(cd ../jev-ultrafast && REPEATS=3 SUITE_OUT=$PWD/../laya/$O/suiteA_v14s.json timeout 3600 .venv/bin/python ../laya/apps/browser_suite.py 2>&1 | grep -E "^(PASS|FAIL)|^==|per task")
+echo "== suite B x3 (v14s, chunk 60)"
+(cd ../jev-ultrafast && REPEATS=3 SUITE_OUT=$PWD/../laya/$O/suiteB_v14s.json timeout 3600 .venv/bin/python ../laya/apps/browser_suite_b.py 2>&1 | grep -E "^(PASS|FAIL)|^==|per task|held-out")
+echo V14S_DONE

code/finetune/run_v15s.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+# v15s: CLEAN retrain from the mmBERT-base checkpoint (no DAgger data, no suite start pages) on
+#   crawl + DONE + step2 + Mind2Web + rollouts + rollouts2 + NNetNav + webgym (long-horizon forms/listings, sub-goal
+#   conditioned, verified sub-goal DONE) + Mind2Web sub-goal annotation.
+# Then: suite A x3, suite B x3, webgym held-out (planner off / on), Google Flights x3 with the planner.
+cd /home/ckl/projects/S/laya && source env.sh
+export LAYA_BASE=$(ls -d ~/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/*/multilingual | head -1)
+export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CKPT=${CKPT:-1} COMPILE=0 LAYA_FMT=v3 LAYA_HEAD=768 MAX_TARGETS=40 FINAL_P=0.2
+P=.venv/bin/python; O=finetune/out; EP=${1:-3}; I="bash finetune/infra.sh"; CK=laya-browser-v15s
+pkill -f 'systemone_serve[r]' ; pkill -f 'sglang.launch_serve[r]' ; sleep 5
+cat $O/rollout2_cases.*.jsonl > $O/rollout2_cases.jsonl; cat $O/gym/*.jsonl > $O/gym_cases.jsonl
+echo "== data: rollout2 $(wc -l < $O/rollout2_cases.jsonl)  gym $(wc -l < $O/gym_cases.jsonl)  m2w_sub $(wc -l < $O/m2w_sub_cases.jsonl)"
+echo "== build $CK"
+[ -n "$SKIP_BUILD" ] && [ -f $O/train_items.pt ] && echo "   (SKIP_BUILD: reusing $O/train_items.pt)" || \
+$P finetune/build_items.py $O/pages.jsonl $O/cases.jsonl $O/ $O/done_cases.jsonl $O/step2_cases.jsonl $O/m2w_cases.jsonl $O/rollout_cases.jsonl \
+   $O/rollout2_cases.jsonl $O/nnetnav_cases.jsonl $O/gym_cases.jsonl $O/m2w_sub_cases.jsonl 2>&1 | grep -v Warn | tail -4
+echo "== train $CK ($EP epochs from base)"; $P finetune/train.py $O/train_items.pt $O/$CK $EP 2>&1 | grep --line-buffered -E "=== epoch|saved|Error|Traceback"
+echo "== calibrate"; $P finetune/calibrate.py $O/pages.jsonl $O/eval_cases.jsonl $PWD/$O/$CK 2>&1 | grep -vE "TileLang|Warn|warn|Fetch"
+echo "== eval"; $P finetune/eval.py $O/pages.jsonl $O/eval_cases.jsonl $PWD/$O/$CK 2>&1 | grep -vE "TileLang|Warn|warn|Fetch|return Agent"
+$I start_chrome >/dev/null; SGL_MEM=0.5 $I start_sglang Qwen/Qwen3-8B-AWQ >/dev/null; bash finetune/webgym/serve.sh restart >/dev/null
+until curl -s -m 3 http://127.0.0.1:30000/health >/dev/null; do sleep 5; done
+$I start_s1 $PWD/$O/$CK 60 >/dev/null
+J=../jev-ultrafast/.venv/bin/python
+echo "== suite A x3"; (cd ../jev-ultrafast && REPEATS=3 SUITE_OUT=$PWD/../laya/$O/suiteA_v15s.json timeout 3600 .venv/bin/python ../laya/apps/browser_suite.py 2>&1 | grep -E "^(PASS|FAIL)|^==|per task")
+echo "== suite B x3"; (cd ../jev-ultrafast && REPEATS=3 SUITE_OUT=$PWD/../laya/$O/suiteB_v15s.json timeout 3600 .venv/bin/python ../laya/apps/browser_suite_b.py 2>&1 | grep -E "^(PASS|FAIL)|^==|per task|held-out")
+echo "== webgym held-out, planner off"; (cd ../jev-ultrafast && GYM_OUT=$PWD/../laya/$O/gym_eval_v15s.json timeout 3600 .venv/bin/python ../laya/finetune/webgym/eval_gym.py 10 flight,hotel,shop 40 2>&1 | grep -E "^(PASS|FAIL|==)")
+echo "== webgym held-out, planner on"; (cd ../jev-ultrafast && JEV_PLANNER=1 GYM_OUT=$PWD/../laya/$O/gym_eval_v15s_plan.json timeout 3600 .venv/bin/python ../laya/finetune/webgym/eval_gym.py 10 flight,hotel,shop 40 2>&1 | grep -E "^(PASS|FAIL|==)")
+echo "== Google Flights x3, planner on"; (cd ../jev-ultrafast && JEV_PLANNER=1 REPEATS=3 SUITE_OUT=$PWD/../laya/$O/suiteA_flights_plan_v15s.json timeout 1800 .venv/bin/python ../laya/apps/browser_suite.py flights 2>&1 | grep -E "^(PASS|FAIL)|^==")
+echo V15S_DONE

code/finetune/sites.txt

@@ -0,0 +1,340 @@
+# Sites with a search box and/or long lists, for finetune/rollouts2.py.  Suite URLs are skipped by the script.
+# wikis / reference
+https://en.wikipedia.org/wiki/Special:Random
+https://en.wikipedia.org/wiki/Portal:Science
+https://de.wikipedia.org/wiki/Wikipedia:Hauptseite
+https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Accueil_principal
+https://es.wikipedia.org/wiki/Wikipedia:Portada
+https://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3%83%9A%E3%83%BC%E3%82%B8
+https://zh.wikipedia.org/wiki/Wikipedia:%E9%A6%96%E9%A1%B5
+https://simple.wikipedia.org/wiki/Main_Page
+https://en.wiktionary.org/wiki/Wiktionary:Main_Page
+https://www.wikidata.org/wiki/Wikidata:Main_Page
+https://en.wikivoyage.org/wiki/Main_Page
+https://commons.wikimedia.org/wiki/Main_Page
+https://en.wikibooks.org/wiki/Main_Page
+https://en.wikiquote.org/wiki/Main_Page
+https://wiki.archlinux.org/
+https://wiki.debian.org/
+https://help.ubuntu.com/
+https://www.wikihow.com/Main-Page
+https://www.britannica.com/
+https://www.merriam-webster.com/
+https://dictionary.cambridge.org/
+https://www.dictionary.com/
+https://www.thefreedictionary.com/
+https://www.etymonline.com/
+https://en.wikisource.org/wiki/Main_Page
+# open libraries / archives
+https://archive.org/
+https://openlibrary.org/
+https://www.gutenberg.org/
+https://www.gutenberg.org/ebooks/search/?query=science
+https://scholar.archive.org/
+https://www.loc.gov/
+https://www.europeana.eu/en
+https://digital.library.upenn.edu/books/
+https://www.worldcat.org/
+https://doaj.org/
+https://www.biodiversitylibrary.org/
+# papers / science
+https://arxiv.org/list/cs.LG/recent
+https://arxiv.org/abs/2205.14135
+https://www.biorxiv.org/
+https://www.medrxiv.org/
+https://dblp.org/
+https://www.semanticscholar.org/
+https://openreview.net/
+https://aclanthology.org/
+https://journals.plos.org/plosone/
+https://pubmed.ncbi.nlm.nih.gov/
+https://www.ncbi.nlm.nih.gov/
+https://paperswithcode.com/
+https://zenodo.org/
+https://figshare.com/
+https://osf.io/
+https://eric.ed.gov/
+https://core.ac.uk/
+https://www.jstor.org/
+https://inspirehep.net/
+https://ui.adsabs.harvard.edu/
+https://mathworld.wolfram.com/
+https://oeis.org/
+https://www.ncbi.nlm.nih.gov/pmc/
+# package registries / code
+https://pypi.org/
+https://pypi.org/project/requests/
+https://crates.io/
+https://crates.io/crates/serde
+https://www.npmjs.com/
+https://rubygems.org/
+https://packagist.org/
+https://hex.pm/
+https://pub.dev/
+https://hub.docker.com/
+https://huggingface.co/models
+https://huggingface.co/datasets
+https://github.com/explore
+https://github.com/search?q=tilelang&type=repositories
+https://github.com/pytorch/pytorch
+https://github.com/rust-lang/rust/issues
+https://gitlab.com/explore
+https://codeberg.org/explore/repos
+https://sourceforge.net/
+https://www.apache.org/
+https://gitee.com/explore
+https://mvnrepository.com/
+https://search.maven.org/
+https://cran.r-project.org/
+https://www.cpan.org/
+https://metacpan.org/
+https://docs.rs/
+https://godocs.io/
+https://pkg.go.dev/
+https://formulae.brew.sh/
+https://repology.org/
+https://packages.debian.org/
+https://archlinux.org/packages/
+https://launchpad.net/
+# documentation
+https://docs.python.org/3/
+https://developer.mozilla.org/en-US/
+https://doc.rust-lang.org/book/
+https://doc.rust-lang.org/std/
+https://go.dev/doc/
+https://nodejs.org/en/docs
+https://www.php.net/
+https://www.ruby-lang.org/en/
+https://docs.djangoproject.com/en/stable/
+https://flask.palletsprojects.com/
+https://numpy.org/doc/stable/
+https://pandas.pydata.org/docs/
+https://pytorch.org/docs/stable/index.html
+https://www.tensorflow.org/api_docs
+https://scikit-learn.org/stable/
+https://docs.docker.com/
+https://kubernetes.io/docs/home/
+https://docs.readthedocs.io/en/stable/
+https://readthedocs.org/
+https://www.w3schools.com/
+https://www.tutorialspoint.com/index.htm
+https://www.geeksforgeeks.org/
+https://cppreference.com/
+https://en.cppreference.com/w/
+https://learn.microsoft.com/en-us/docs/
+https://docs.oracle.com/en/java/
+https://www.postgresql.org/docs/
+https://dev.mysql.com/doc/
+https://redis.io/docs/
+https://docs.gitlab.com/
+https://git-scm.com/doc
+https://www.gnu.org/manual/
+https://www.kernel.org/doc/html/latest/
+https://docs.ansible.com/
+https://www.terraform.io/docs
+https://wiki.python.org/moin/
+https://devdocs.io/
+https://www.freecodecamp.org/news/
+https://stackoverflow.com/questions
+https://askubuntu.com/
+https://superuser.com/
+https://unix.stackexchange.com/
+https://math.stackexchange.com/
+https://serverfault.com/
+# news / media
+https://www.bbc.com/
+https://www.bbc.com/news
+https://www.theguardian.com/international
+https://apnews.com/
+https://www.npr.org/
+https://www.aljazeera.com/
+https://www.dw.com/en/top-stories/s-9097
+https://www.france24.com/en/
+https://www.euronews.com/
+https://www.cbc.ca/news
+https://www.abc.net.au/news
+https://www.nytimes.com/
+https://www.washingtonpost.com/
+https://www.economist.com/
+https://www.wired.com/
+https://arstechnica.com/
+https://www.theverge.com/
+https://techcrunch.com/
+https://news.ycombinator.com/newest
+https://news.ycombinator.com/ask
+https://lobste.rs/
+https://slashdot.org/
+https://www.phoronix.com/
+https://lwn.net/
+https://www.zdnet.com/
+https://www.engadget.com/
+https://www.reuters.com/
+https://www.scmp.com/
+https://www.japantimes.co.jp/
+https://www.thehindu.com/
+https://timesofindia.indiatimes.com/
+https://www.lemonde.fr/
+https://www.spiegel.de/
+https://elpais.com/
+https://www.corriere.it/
+# government / institutions
+https://www.usa.gov/
+https://www.gov.uk/
+https://www.gov.uk/search/all
+https://europa.eu/european-union/index_en
+https://www.nasa.gov/
+https://www.nih.gov/
+https://www.cdc.gov/
+https://www.who.int/
+https://www.un.org/en/
+https://data.gov/
+https://www.census.gov/
+https://www.noaa.gov/
+https://www.epa.gov/
+https://www.fda.gov/
+https://www.canada.ca/en.html
+https://www.australia.gov.au/
+https://www.bundesregierung.de/breg-en
+https://www.service-public.fr/
+https://www.gov.cn/
+https://www.esa.int/
+https://www.cern.ch/
+https://www.ipcc.ch/
+https://www.imf.org/en/Home
+https://www.worldbank.org/en/home
+https://www.oecd.org/
+# education
+https://ocw.mit.edu/
+https://www.khanacademy.org/
+https://www.coursera.org/
+https://www.edx.org/
+https://www.mit.edu/
+https://www.stanford.edu/
+https://www.ox.ac.uk/
+https://www.cam.ac.uk/
+https://www.harvard.edu/
+https://www.berkeley.edu/
+https://www.tsinghua.edu.cn/
+https://www.ethz.ch/en.html
+https://www.utoronto.ca/
+https://plato.stanford.edu/
+https://www.openculture.com/
+https://www.ted.com/talks
+# demo / practice sites (crawler friendly)
+https://books.toscrape.com/catalogue/page-2.html
+https://books.toscrape.com/catalogue/category/books/mystery_3/index.html
+https://quotes.toscrape.com/page/2/
+https://quotes.toscrape.com/tag/humor/
+https://the-internet.herokuapp.com/
+https://the-internet.herokuapp.com/tables
+https://the-internet.herokuapp.com/login
+https://automationexercise.com/
+https://automationexercise.com/products
+https://www.saucedemo.com/
+https://demoqa.com/
+https://practice.expandtesting.com/
+https://webscraper.io/test-sites/e-commerce/allinone
+https://webscraper.io/test-sites/e-commerce/static
+https://demo.opencart.com/
+https://www.demoblaze.com/
+https://magento.softwaretestingboard.com/
+https://practicesoftwaretesting.com/
+https://demo.nopcommerce.com/
+https://www.globalsqa.com/angularJs-protractor/WebTable/
+https://scrapeme.live/shop/
+https://realpython.github.io/fake-jobs/
+https://www.scrapethissite.com/pages/
+https://parabank.parasoft.com/parabank/index.htm
+https://opensource-demo.orangehrmlive.com/
+# shops / catalogues / listings
+https://www.ebay.com/
+https://www.etsy.com/
+https://www.aliexpress.com/
+https://www.ikea.com/us/en/
+https://www.bestbuy.com/
+https://www.newegg.com/
+https://www.walmart.com/
+https://www.target.com/
+https://www.homedepot.com/
+https://www.wayfair.com/
+https://www.zalando.de/
+https://www.otto.de/
+https://www.jd.com/
+https://www.taobao.com/
+https://www.rakuten.co.jp/
+https://www.mercadolibre.com.ar/
+https://www.thingiverse.com/
+https://www.goodreads.com/
+https://www.imdb.com/
+https://www.rottentomatoes.com/
+https://www.allrecipes.com/
+https://www.bbcgoodfood.com/
+https://www.epicurious.com/
+https://www.foodnetwork.com/
+https://www.yelp.com/
+https://www.tripadvisor.com/
+https://www.booking.com/
+https://www.airbnb.com/
+https://www.kayak.com/
+https://www.skyscanner.net/
+https://www.expedia.com/
+https://www.craigslist.org/about/sites
+https://www.autotrader.com/
+https://www.zillow.com/
+https://www.indeed.com/
+https://www.linkedin.com/jobs/
+https://www.glassdoor.com/
+https://weworkremotely.com/
+https://remoteok.com/
+https://www.discogs.com/
+https://bandcamp.com/
+https://www.last.fm/
+https://musicbrainz.org/
+https://www.metacritic.com/
+https://store.steampowered.com/
+https://www.gog.com/
+https://itch.io/
+https://boardgamegeek.com/
+https://www.chess.com/
+https://lichess.org/
+https://openweathermap.org/
+https://www.timeanddate.com/
+https://www.openstreetmap.org/
+https://www.geonames.org/
+https://www.flickr.com/
+https://unsplash.com/
+https://www.pexels.com/
+https://pixabay.com/
+https://fontawesome.com/
+https://fonts.google.com/
+https://www.reddit.com/
+https://www.quora.com/
+https://medium.com/
+https://dev.to/
+https://hashnode.com/
+https://www.producthunt.com/
+https://alternativeto.net/
+https://www.g2.com/
+https://www.capterra.com/
+https://www.trustpilot.com/
+https://www.kickstarter.com/
+https://www.indiegogo.com/
+https://www.patreon.com/
+https://www.twitch.tv/
+https://www.bilibili.com/
+https://www.zhihu.com/
+https://www.csdn.net/
+https://www.douban.com/
+https://www.jianshu.com/
+https://www.baidu.com/
+https://www.sogou.com/
+https://www.so.com/
+https://www.mojeek.com/
+https://yandex.com/
+https://www.qwant.com/
+https://search.brave.com/
+https://www.wolframalpha.com/
+https://mathworld.wolfram.com/
+https://www.khanacademy.org/math
+https://www.desmos.com/
+https://www.geogebra.org/

code/finetune/sites_autocomplete.txt

@@ -0,0 +1,69 @@
+https://www.kayak.com/flights
+https://www.skyscanner.net/
+https://www.momondo.com/
+https://www.kiwi.com/en/
+https://www.expedia.com/Flights
+https://www.trip.com/flights/
+https://flights.ctrip.com/
+https://www.qunar.com/
+https://www.rome2rio.com/
+https://www.booking.com/
+https://www.airbnb.com/
+https://www.hotels.com/
+https://www.agoda.com/
+https://www.trivago.com/
+https://www.flightradar24.com/
+https://www.lufthansa.com/
+https://www.britishairways.com/
+https://www.ryanair.com/
+https://www.easyjet.com/
+https://www.aa.com/
+https://www.united.com/
+https://www.delta.com/
+https://www.airfrance.com/
+https://www.klm.com/
+https://www.emirates.com/
+https://www.qatarairways.com/
+https://www.sbb.ch/en
+https://www.bahn.com/en
+https://www.thetrainline.com/
+https://www.omio.com/
+https://www.flixbus.com/
+https://www.openstreetmap.org/
+https://www.accuweather.com/
+https://weather.com/
+https://www.timeanddate.com/weather/
+https://www.wunderground.com/
+https://de.wikipedia.org/wiki/Wikipedia:Hauptseite
+https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Accueil_principal
+https://es.wikipedia.org/wiki/Wikipedia:Portada
+https://en.wiktionary.org/wiki/Wiktionary:Main_Page
+https://www.wikidata.org/wiki/Wikidata:Main_Page
+https://commons.wikimedia.org/wiki/Main_Page
+https://developer.mozilla.org/en-US/
+https://www.imdb.com/
+https://www.goodreads.com/
+https://www.yelp.com/
+https://www.tripadvisor.com/
+https://www.zillow.com/
+https://www.indeed.com/
+https://www.glassdoor.com/
+https://www.ikea.com/us/en/
+https://www.bestbuy.com/
+https://www.etsy.com/
+https://www.ebay.com/
+https://www.amazon.com/
+https://www.walmart.com/
+https://www.target.com/
+https://www.homedepot.com/
+https://www.baidu.com/
+https://www.bilibili.com/
+https://www.zhihu.com/
+https://www.douban.com/
+https://www.jd.com/
+https://www.taobao.com/
+https://map.baidu.com/
+https://www.amap.com/
+https://www.12306.cn/index/
+https://www.ly.com/
+https://www.mafengwo.cn/

code/finetune/sites_paged.txt

@@ -0,0 +1,57 @@
+# paginated listings for SKILLS=next rollouts (suite start URLs are skipped by the script)
+https://lobste.rs/
+https://lobste.rs/recent
+https://arxiv.org/list/cs.LG/recent
+https://arxiv.org/list/cs.CL/recent
+https://arxiv.org/list/cs.CV/recent
+https://pypi.org/search/?q=http
+https://pypi.org/search/?q=json
+https://pypi.org/search/?q=cli
+https://crates.io/search?q=serde
+https://www.npmjs.com/search?q=react
+https://github.com/search?q=attention&type=repositories
+https://github.com/pytorch/pytorch/issues
+https://github.com/python/cpython/pulls
+https://stackoverflow.com/questions?tab=newest
+https://stackoverflow.com/questions/tagged/python
+https://askubuntu.com/questions
+https://superuser.com/questions
+https://unix.stackexchange.com/questions
+https://math.stackexchange.com/questions
+https://www.gutenberg.org/ebooks/search/?sort_order=downloads
+https://www.gutenberg.org/ebooks/search/?query=history
+https://openlibrary.org/search?q=science
+https://en.wikipedia.org/wiki/Special:AllPages
+https://en.wikipedia.org/w/index.php?search=physics&fulltext=1
+https://en.wikipedia.org/wiki/Special:NewPages
+https://webscraper.io/test-sites/e-commerce/static/computers/laptops
+https://webscraper.io/test-sites/e-commerce/static/computers/tablets
+https://webscraper.io/test-sites/e-commerce/static/phones/touch
+https://www.scrapethissite.com/pages/forms/
+https://scrapeme.live/shop/
+https://scrapeme.live/shop/page/2/
+https://automationexercise.com/products
+https://practicesoftwaretesting.com/
+https://dev.to/
+https://hashnode.com/
+https://lwn.net/Archives/
+https://www.phoronix.com/
+https://slashdot.org/
+https://www.bbc.com/news/world
+https://apnews.com/hub/technology
+https://huggingface.co/models
+https://huggingface.co/datasets
+https://huggingface.co/spaces
+https://paperswithcode.com/latest
+https://dblp.org/search?q=attention
+https://aclanthology.org/events/acl-2024/
+https://openreview.net/group?id=ICLR.cc/2025/Conference
+https://www.biorxiv.org/content/early/recent
+https://archive.org/search?query=science
+https://musicbrainz.org/search?query=beatles&type=artist
+https://www.discogs.com/search/?q=jazz
+https://boardgamegeek.com/browse/boardgame
+https://itch.io/games
+https://store.steampowered.com/search/
+https://www.goodreads.com/list/show/1.Best_Books_Ever
+https://www.imdb.com/chart/top/

code/finetune/sites_select.txt

@@ -0,0 +1,38 @@
+https://about.gitlab.com/events/
+https://accounts.google.com/v3/signin/identifier?continue=https://translate.google.com/history&followup=https://translate.google.com/history&passive=1209600&flowName=WebLiteSignIn&flowEntry=ServiceLogin&dsh=S235265407:1789917099781115
+https://accounts.google.com/v3/signin/identifier?continue=https://translate.google.com/saved&followup=https://translate.google.com/saved&passive=1209600&flowName=WebLiteSignIn&flowEntry=ServiceLogin&dsh=S-686375364:1789917900173883
+https://accounts.google.com/v3/signin/identifier?continue=https://translate.google.com/saved&followup=https://translate.google.com/saved&passive=1209600&flowName=WebLiteSignIn&flowEntry=ServiceLogin&dsh=S-801860207:1789917894266607
+https://accounts.google.com/v3/signin/identifier?continue=https://translate.google.com/saved&followup=https://translate.google.com/saved&passive=1209600&flowName=WebLiteSignIn&flowEntry=ServiceLogin&dsh=S271895034:1789917813501863
+https://archlinux.org/packages/
+https://dmytro-shyshkin.kit.com/ai-workshop
+https://docs.python.org/3/
+https://docs.python.org/3/download.html
+https://docs.python.org/3/library/
+https://docs.python.org/3/library/faulthandler.html
+https://docs.python.org/3/library/numbers.html
+https://docs.python.org/3/library/xml.sax.handler.html
+https://docs.python.org/3/py-modindex.html
+https://docs.python.org/3/tutorial/
+https://docs.python.org/3/tutorial/appetite.html
+https://docs.python.org/3/tutorial/floatingpoint.html
+https://formy-project.herokuapp.com/form
+https://getbootstrap.com/docs/5.3/forms/select/
+https://huggingface.co/docs/huggingface_hub/en/guides/cli
+https://pkg.go.dev/image
+https://rust-lang.org/
+https://rust-lang.org/funding/
+https://rust-lang.org/governance/
+https://rust-lang.org/learn/
+https://rust-lang.org/policies/
+https://rust-lang.org/policies/code-of-conduct/
+https://rust-lang.org/policies/licenses/
+https://rust-lang.org/policies/security/
+https://rust-lang.org/tools/
+https://rust-lang.org/tools/install/
+https://rust-lang.org/what/embedded/
+https://www.calculator.net/auto-loan-calculator.html
+https://www.calculator.net/calorie-calculator.html
+https://www.ebay.com/
+https://www.ebay.com/b/Coffee-Tea-Espresso-Makers/38250/bn_2710762
+https://www.ebay.com/ebaylive/sellers/0stkntuiq8o
+https://www.selenium.dev/selenium/web/web-form.html

code/finetune/sites_sort.txt

@@ -0,0 +1,15 @@
+https://scrapeme.live/shop/
+https://scrapeme.live/shop/page/3/
+https://demo.opencart.com/index.php?route=product/category&path=20
+https://demo.opencart.com/index.php?route=product/category&path=57
+https://demo.nopcommerce.com/computers
+https://demo.nopcommerce.com/notebooks
+https://demo.nopcommerce.com/books
+https://practicesoftwaretesting.com/
+https://magento.softwaretestingboard.com/women/tops-women.html
+https://magento.softwaretestingboard.com/men/bottoms-men.html
+https://automationexercise.com/products
+https://www.saucedemo.com/
+https://webscraper.io/test-sites/e-commerce/allinone/computers/laptops
+https://demoblaze.com/
+https://www.demoblaze.com/

code/finetune/train.py

@@ -41,10 +41,27 @@ def main():
     total = max(1, (len(items) // (MICRO * ACCUM)) * epochs)
     sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=total, eta_min=1e-6)
     t0 = time.time(); step = 0
-    for ep in range(epochs):
+    # ---- resumable: <out>/resume.pt holds model/optimizer/scheduler + (epoch, micro-batch offset); written every
+    # SAVE_EVERY optimizer steps and at each epoch end.  Re-running the same command continues from it.
+    ck_path = os.path.join(out, "resume.pt"); os.makedirs(out, exist_ok=True)
+    start_ep, start_b = 0, 0
+    if os.path.exists(ck_path):
+        ck = torch.load(ck_path, map_location=dev, weights_only=False)
+        tgt = model.encoder._orig_mod if hasattr(model.encoder, "_orig_mod") else None
+        model.load_state_dict({k.replace("encoder._orig_mod.", "encoder."): v for k, v in ck["model"].items()} if tgt is None else ck["model"], strict=False)
+        opt.load_state_dict(ck["opt"]); sched.load_state_dict(ck["sched"]); step = ck["step"]; start_ep, start_b = ck["epoch"], ck["b"]
+        print(f"resumed from {ck_path}: epoch {start_ep+1}, item {start_b}, step {step}", flush=True)
+    SAVE_EVERY = int(os.environ.get("SAVE_EVERY", "300"))
+    def save_resume(ep_, b_):
+        tmp = ck_path + ".tmp"
+        torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "sched": sched.state_dict(), "step": step, "epoch": ep_, "b": b_}, tmp)
+        os.replace(tmp, ck_path)
+    for ep0 in range(start_ep):          # replay the shuffles of finished epochs so the order matches the original run
+        random.seed(42 + ep0); random.shuffle(items)
+    for ep in range(start_ep, epochs):
         random.seed(42 + ep); random.shuffle(items)
         sigma = S0 + (S1 - S0) * ep / max(1, epochs - 1); tot_loss, nb = 0.0, 0; opt.zero_grad(set_to_none=True)
-        for b in range(0, len(items), MICRO):
+        for b in range((start_b if ep == start_ep else 0), len(items), MICRO):
             batch = collate(items[b:b + MICRO], tok.pad_token_id); batch = {k: v.to(dev) for k, v in batch.items()}
             with torch.autocast("cuda", dtype=torch.bfloat16):
                 logits, act = model(batch["input_ids"], batch["attention_mask"], batch["marker_pos"], batch["marker_mask"], batch["qtype"])
@@ -58,8 +75,10 @@ def main():
             loss = (loss_rl + loss_ce) / ACCUM + 0.0 * act.sum(); loss.backward(); nb += 1; tot_loss += loss.item() * ACCUM
             if nb % ACCUM == 0 or b + MICRO >= len(items):
                 torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step(); sched.step(); opt.zero_grad(set_to_none=True); step += 1
+                if step % SAVE_EVERY == 0: save_resume(ep, b + MICRO)
                 if step % 20 == 0: print(f"  ep {ep+1} step {step}/{total} loss {loss.item()*ACCUM:.3f} ce {loss_ce.item():.3f} reward {r.mean().item():.3f} {time.time()-t0:.0f}s", flush=True)
         print(f"=== epoch {ep+1}/{epochs} avg loss {tot_loss/max(1,nb):.4f}  {time.time()-t0:.0f}s", flush=True)
+        save_resume(ep + 1, 0)
     os.makedirs(out, exist_ok=True); model.eval()
     if hasattr(model.encoder, "_orig_mod"):
         model.encoder = model.encoder._orig_mod
@@ -68,6 +87,7 @@ def main():
     cfg.update(fine_tuned=True, model_name="laya-browser", temperature=[1.0, 1.0, 1.0], temperature_by_options={},
                laya_fmt=os.environ.get("LAYA_FMT", "v1"), head_max_len_train=cfg["head_max_len"])
     json.dump(cfg, open(os.path.join(out, "rl_agent_config.json"), "w"), indent=2); print("saved", out)
+    if os.path.exists(ck_path): os.remove(ck_path)
 
 if __name__ == "__main__":
     main()

code/finetune/webgym/app.py

@@ -0,0 +1,169 @@
+"""webgym HTTP server: renders spec.py tasks as real pages for headless Chromium.
+
+    python finetune/webgym/app.py [port=8811]
+
+  /t/<kind>/<seed>                 the task's start page (flight / hotel search form, or shop listing)
+  /t/<kind>/<seed>/results?...     results after submitting a form (hotel: filters + sort that reload with params)
+  /t/shop/<seed>?page=&sort=&brand=&q=   listing;   /t/shop/<seed>/p/<id>  product page
+"""
+import html, json, os, sys, urllib.parse
+from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+import spec as S
+
+WIDGETS = open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "widgets.js")).read()
+CSS = """body{font-family:system-ui,sans-serif;margin:0;background:#fafafa;color:#222}header{display:flex;gap:18px;padding:12px 24px;background:#fff;border-bottom:1px solid #ddd}
+header a{color:#246;text-decoration:none}main{max-width:1000px;margin:0 auto;padding:20px}.hero{background:linear-gradient(#cde,#eef);border-radius:12px;margin-bottom:16px;display:flex;align-items:end;padding:16px;box-sizing:border-box}
+form,.panel{background:#fff;border:1px solid #ddd;border-radius:12px;padding:16px;display:flex;flex-wrap:wrap;gap:14px;align-items:flex-start}
+.f{position:relative;min-width:200px}.list{position:absolute;z-index:20;background:#fff;border:1px solid #bbb;list-style:none;margin:0;padding:4px;min-width:240px}
+.opt{padding:6px 8px;cursor:pointer}.opt:hover{background:#eef}.dlg{position:absolute;z-index:30;background:#fff;border:1px solid #999;border-radius:8px;padding:12px;box-shadow:0 4px 18px #0003;min-width:260px}
+.cal .grid{display:grid;grid-template-columns:repeat(7,34px);gap:3px}.cal .hdr{display:flex;justify-content:space-between;align-items:center;margin-bottom:6px}
+button{cursor:pointer}.seg button[aria-pressed=true]{background:#246;color:#fff}.cookie{position:fixed;bottom:0;left:0;right:0;background:#223;color:#fff;padding:18px 24px;z-index:50;display:flex;gap:16px;align-items:center}
+.card{background:#fff;border:1px solid #ddd;border-radius:10px;padding:12px;margin:10px 0}.grid3{display:grid;grid-template-columns:repeat(4,1fr);gap:10px}
+.pager{display:flex;gap:10px;margin:24px 0}footer{padding:30px;color:#777}.side{min-width:180px}"""
+
+
+def page(title, body, gym=None, cookie=False, decoys=()):
+    nav = "".join(f'<a href="#{html.escape(d.lower().replace(" ", "-"))}">{html.escape(d)}</a>' for d in decoys)
+    ck = ('<div class="cookie" id="cookie"><span>We use cookies to improve your experience.</span>'
+          '<button type="button" data-g="cookie-accept" onclick="document.getElementById(\'cookie\').remove()">Accept all</button>'
+          '<button type="button" data-g="cookie-reject" onclick="document.getElementById(\'cookie\').remove()">Reject all</button></div>') if cookie else ""
+    return (f"<!doctype html><html lang=en><head><meta charset=utf-8><title>{html.escape(title)}</title><style>{CSS}</style>"
+            f"<script>{WIDGETS}</script></head><body><header><a href='#home'><b>{html.escape(title.split(' ')[0])}</b></a>{nav}</header>"
+            f"<main>{body}</main><footer>© 2026 · <a href='#about'>About</a> · <a href='#privacy'>Privacy</a> · <a href='#contact'>Contact</a></footer>{ck}"
+            f"<script>window.__gym.meta={json.dumps(gym or {})};</script></body></html>")
+
+
+def form_page(sp):
+    hosts = "".join(f'<div class="f" id="host-{k}"></div>' for k in sp["order"])
+    hero = f'<div class="hero" style="height:{sp["hero"]}px"><h1>{"Find your next flight" if sp["kind"] == "flight" else "Find a place to stay"}</h1></div>' if sp["hero"] else ""
+    fields = [f for f in sp["fields"]]
+    js = f"""
+    const SP={json.dumps({"fields": fields, "kind": sp["kind"], "seed": sp["seed"]})};
+    for (const f of SP.fields) {{ const h=document.getElementById('host-'+f.key); ({{combo:__gym.combo,choice:__gym.choice,date:__gym.date,count:__gym.count}})[f.w](h,f); }}
+    const ret=SP.fields.find(f=>f.only_round);
+    if (ret) setInterval(()=>{{ const h=document.getElementById('host-'+ret.key); const show=__gym.state.trip==='round'; if (h.hidden===show) h.hidden=!show; }},100);
+    document.getElementById('go').addEventListener('click',e=>{{ e.preventDefault();
+      location.href=location.pathname+'/results?'+new URLSearchParams(Object.entries(__gym.state).filter(([k,v])=>v!==null&&v!==undefined).map(([k,v])=>[k,String(v)])); }});
+    """
+    body = (f"{hero}<form onsubmit='return false'>{hosts}<button id='go' type='submit' data-g='submit'>{html.escape(sp['submit'])}</button></form>"
+            f"<div class='panel' style='margin-top:18px'><p>Popular: " + ", ".join(f"<a href='#pop-{c[0]}'>{c[1]}</a>" for c in S.CITIES[:5]) + "</p></div>"
+            f"<script>{js}</script>")
+    return page(("SkyFinder" if sp["kind"] == "flight" else "StayFinder") + " – search", body, gym={"kind": sp["kind"], "seed": sp["seed"]},
+                cookie=sp["cookie"], decoys=sp["decoys"])
+
+
+def results_page(sp, q):
+    city = {c[0]: c[1] for c in S.CITIES}
+    k = lambda x: (q.get(x) or "").replace("-ap", "")
+    if sp["kind"] == "flight":
+        head = f"{city.get(k('origin'), '?')} to {city.get(k('destination'), '?')}"
+        chips = [("One way" if q.get("trip") == "oneway" else "Round trip"), f"{q.get('adults', '1')} adult(s)", (q.get("cabin") or "economy").title(),
+                 f"Departs {q.get('date', '?')}"] + ([f"Returns {q.get('ret')}"] if q.get("ret") else [])
+        cards = "".join(f"<div class='card'><b>{7 + i}:{15 * (i % 4):02d} – {9 + i}:{(15 * i + 40) % 60:02d}</b> · {['Nonstop', '1 stop'][i % 2]} · "
+                        f"<a href='#flight-{i}'>Select ${120 + 37 * i}</a></div>" for i in range(5))
+        body = f"<h1>{head}</h1><p>{' · '.join(chips)}</p>{cards}<p><a href='{sp_url(sp)}'>Modify search</a></p>"
+        return page("SkyFinder – results", body, gym={"kind": "flight", "results": dict(q)}, decoys=sp["decoys"])
+    # hotel results: filters + sort reload the page with params
+    title = f"Stays in {city.get(k('dest'), '?')}"
+    filt = [f for f in ["Free cancellation", "Breakfast included", "Pool", "Pet friendly"]]
+    on = set((q.get("filter") or "").split("|")) - {""}
+    boxes = "".join(f"<label><input type=checkbox data-g='filter-{i}' {'checked' if f in on else ''} onchange=\"setF('{f}',this.checked)\"> {f}</label><br>"
+                    for i, f in enumerate(filt))
+    sorts = ["Recommended", "Price: low to high", "Guest rating", "Distance to center"]
+    cur = q.get("sort") or "Recommended"
+    sel = "".join(f"<option {'selected' if s == cur else ''}>{s}</option>" for s in sorts)
+    cards = "".join(f"<div class='card'><a href='#hotel-{i}'>Hotel {['Aurora', 'Belvedere', 'Casa Flora', 'Duomo', 'Elba', 'Fjord'][i]}</a> · "
+                    f"{8 + (i % 3) * 0.5}/10 · ${90 + 23 * i}/night</div>" for i in range(6))
+    js = """function go(p){const u=new URL(location.href);for(const[k,v]of Object.entries(p)){if(v)u.searchParams.set(k,v);else u.searchParams.delete(k)}location.href=u}
+    function setF(f,on){const u=new URL(location.href);const s=new Set((u.searchParams.get('filter')||'').split('|').filter(Boolean));on?s.add(f):s.delete(f);go({filter:[...s].join('|')})}"""
+    body = (f"<h1>{title}</h1><p>{q.get('cin', '?')} → {q.get('cout', '?')} · {q.get('guests', '2')} guests</p><div style='display:flex;gap:18px'>"
+            f"<div class='panel side'><b>Filters</b><br>{boxes}</div><div style='flex:1'><label for=sort>Sort by</label> <select id=sort data-g=sort "
+            f"onchange=\"go({{sort:this.value}})\">{sel}</select>{cards}</div></div><script>{js}</script>")
+    return page("StayFinder – results", body, gym={"kind": "hotel", "results": dict(q)}, decoys=sp["decoys"])
+
+
+def sp_url(sp):
+    return f"/t/{sp['kind']}/{sp['seed']}"
+
+
+def shop_page(sp, q):
+    items = sp["items"]; L = sp["labels"]
+    if q.get("q"):
+        items = [i for i in items if q["q"] in i["name"].lower()]
+    if q.get("brand"):
+        items = [i for i in items if i["brand"] == q["brand"]]
+    s = q.get("sort") or "Featured"
+    if s.startswith("Price: low"): items = sorted(items, key=lambda i: i["price"])
+    elif s.startswith("Price: high"): items = sorted(items, key=lambda i: -i["price"])
+    elif s == "Newest": items = sorted(items, key=lambda i: -i["id"])
+    elif s == "Top rated": items = sorted(items, key=lambda i: (i["id"] * 7) % 13)
+    n = int(q.get("page") or 1); per = sp["per_page"]; pages = max(1, -(-len(items) // per))
+    shown = items[(n - 1) * per: n * per]
+    base = sp_url(sp)
+    def link(**kw):
+        p = {k: v for k, v in {**{k: q.get(k) for k in ("q", "brand", "sort")}, **kw}.items() if v}
+        return base + ("?" + urllib.parse.urlencode(p) if p else "")
+    grid = "".join(f"<div class='card'><a href='{base}/p/{i['id']}'>{html.escape(i['name'])}</a><br>${i['price']:.2f}</div>" for i in shown)
+    sorts = ["Featured", "Price: low to high", "Price: high to low", "Newest", "Top rated"]
+    if sp["sort_impl"] == "select":
+        sort_ui = (f"<label for=sort>{L['sort']}</label> <select id=sort data-g=sort onchange=\"location.href=this.value\">" +
+                   "".join(f"<option value='{link(sort=x if x != 'Featured' else None, page=None)}' {'selected' if x == s else ''}>{x}</option>" for x in sorts) + "</select>")
+    else:
+        sort_ui = (f"<button type=button data-g=sort-trigger aria-haspopup=listbox aria-label='{L['sort']}: {s}' onclick=\"const l=document.getElementById('sl');l.hidden=!l.hidden\">"
+                   f"{L['sort']}: {s} ▾</button><div id=sl role=listbox class=list hidden>" +
+                   "".join(f"<a role=option class=opt href='{link(sort=x if x != 'Featured' else None, page=None)}'>{x}</a>" for x in sorts) + "</div>")
+    brands = "".join(f"<label><input type=checkbox data-g='brand-{b}' {'checked' if q.get('brand') == b else ''} "
+                     f"onchange=\"location.href=this.checked?'{link(brand=b, page=None)}':'{link(brand=None, page=None)}'\"> {b}</label><br>" for b in S.BRANDS)
+    btn = f"<button type=submit data-g=search-btn>{'Search' if sp['search_button'] else ''}</button>" if sp["search_button"] else ""
+    search = (f"<form action='{base}' method=get style='display:inline-flex;gap:6px'><input type=search name=q aria-label=\"{L['search']}\" "
+              f"placeholder=\"{L['search']}\" value=\"{html.escape(q.get('q') or '')}\" data-g=search>{btn}</form>")
+    pager = []
+    if sp["pager"] in ("numbers", "both"):
+        pager += [f"<a href='{link(page=p if p > 1 else None)}' {'aria-current=page' if p == n else ''}>{p}</a>" for p in range(1, pages + 1)]
+    if sp["pager"] in ("next", "both") and n < pages:
+        pager.append(f"<a href='{link(page=n + 1)}' data-g=next>{L['next']}</a>")
+    hero = f"<div class='hero' style='height:{sp['hero']}px'><h1>{sp['category']}</h1></div>" if sp["hero"] else ""
+    body = (f"{hero}<div class=panel>{search} {sort_ui}</div><div style='display:flex;gap:18px;margin-top:14px'><div class='panel side'><b>{L['brand']}</b><br>{brands}</div>"
+            f"<div style='flex:1'><p>Showing {len(shown)} of {len(items)} products · page {n} of {pages}</p><div class=grid3>{grid}</div>"
+            f"<nav class=pager aria-label=Pagination>{' '.join(pager)}</nav></div></div>")
+    state = {"page": n, "sort": q.get("sort"), "brand": q.get("brand"), "q": q.get("q")}
+    return page(f"{sp['category']} – ShopRight", body, gym={"kind": "shop", "listing": state}, cookie=sp["cookie"] and not q, decoys=["Deals", "Help", "Sign in", "Cart"])
+
+
+def product_page(sp, pid):
+    it = sp["items"][pid]
+    body = f"<h1>{html.escape(it['name'])}</h1><p>${it['price']:.2f}</p><button data-g=cart>Add to cart</button> <a href='{sp_url(sp)}'>Back to all products</a>"
+    return page(f"{it['name']} – ShopRight", body, gym={"kind": "shop", "product": pid}, decoys=["Deals", "Help", "Sign in", "Cart"])
+
+
+class H(BaseHTTPRequestHandler):
+    def log_message(self, *a): pass
+    def do_GET(self):
+        u = urllib.parse.urlparse(self.path); parts = [p for p in u.path.split("/") if p]
+        q = {k: v[0] for k, v in urllib.parse.parse_qs(u.query).items()}
+        try:
+            if len(parts) >= 3 and parts[0] == "t":
+                kind, seed = parts[1], int(parts[2]); sp = S.make(kind, seed)
+                if kind == "shop":
+                    out = product_page(sp, int(parts[4])) if len(parts) >= 5 and parts[3] == "p" else shop_page(sp, q)
+                elif len(parts) >= 4 and parts[3] == "results":
+                    out = results_page(sp, q)
+                else:
+                    out = form_page(sp)
+                code = 200
+            elif parts == ["spec"]:
+                out = json.dumps(S.make(q["kind"], int(q["seed"]))); code = 200
+            else:
+                out, code = "not found", 404
+        except Exception as e:
+            out, code = f"error: {type(e).__name__}: {e}", 500
+        data = out.encode()
+        self.send_response(code); self.send_header("Content-Type", "application/json" if parts == ["spec"] else "text/html; charset=utf-8")
+        self.send_header("Content-Length", str(len(data))); self.end_headers(); self.wfile.write(data)
+
+
+if __name__ == "__main__":
+    port = int(sys.argv[1]) if len(sys.argv) > 1 else 8811
+    print(f"webgym on http://127.0.0.1:{port}/t/flight/1", flush=True)
+    ThreadingHTTPServer(("127.0.0.1", port), H).serve_forever()

code/finetune/webgym/eval_gym.py

@@ -0,0 +1,80 @@
+"""Run the real agent (jev loop + the laya systemone server) on HELD-OUT webgym seeds (seed % 10 == 0, never generated
+for training) and judge success from the page's own state.
+
+    python finetune/webgym/eval_gym.py [n_per_kind=10] [kinds=flight,hotel,shop] [max_steps=40]
+    env: JEV_PLANNER=1 to plan sub-goals first;  GYM_OUT=results.json
+
+Success: flight/hotel = on the results page with every target value (+ hotel filter/sort), shop = the listing/product
+state the task asks for.  The agent's DONE is not trusted; a run that ends without reaching the state fails.
+"""
+import json, os, sys, time, urllib.parse
+sys.path.insert(0, "/home/ckl/projects/S/jev-ultrafast")
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..", "apps"))
+import browser_suite  # noqa: F401  (sets the jev env: CDP, systemone URL, text model)
+from jev_ultrafast import Agent
+import spec as S
+
+BASE = os.environ.get("GYM_URL", "http://127.0.0.1:8811")
+
+
+def success(sp, url, meta):
+    if sp["kind"] in ("flight", "hotel"):
+        res = (meta or {}).get("results")
+        if res is None:
+            return False
+        for k, v in sp["target"].items():
+            got = (res.get(k) or "")
+            if isinstance(v, str) and len(v) == 3 and v.isalpha():
+                if got.replace("-ap", "") != v: return False
+            elif str(got) != str(v):
+                return False
+        post = sp.get("post") or {}
+        if "filter" in post and post["filter"] not in (res.get("filter") or "").split("|"): return False
+        if "sort" in post and res.get("sort") != post["sort"]: return False
+        return True
+    post = sp["post"]
+    if "open" in post:
+        return (meta or {}).get("product") == post["open"]
+    lst = (meta or {}).get("listing") or {}
+    return all((lst.get(k) or (1 if k == "page" else None)) == v if k != "q" else (lst.get("q") or "").lower() == v for k, v in post.items())
+
+
+def run(kind, seed, max_steps):
+    sp = S.make(kind, seed)
+    t0 = time.time(); steps = 0; status = "error"; meta = None; url = ""; plan = None
+    try:
+        with Agent(f"{BASE}/t/{kind}/{seed}", sp["goal"]) as agent:
+            plan = agent.state.get("plan")
+            for state in agent.run():
+                steps = len(state["history"]); status = state["status"]
+                if steps >= max_steps: break
+            time.sleep(0.8)
+            meta = json.loads(agent.browser.evaluate("JSON.stringify(window.__gym ? window.__gym.meta : null)") or "null")
+            url = agent.browser.evaluate("location.href")
+    except Exception as e:
+        status = f"error:{type(e).__name__}:{str(e)[:60]}"
+    ok = success(sp, url, meta)
+    return {"kind": kind, "seed": seed, "ok": ok, "steps": steps, "status": status, "wall": round(time.time() - t0, 1), "url": url,
+            "goal": sp["goal"], "plan": plan}
+
+
+def main():
+    n = int(sys.argv[1]) if len(sys.argv) > 1 else 10
+    kinds = (sys.argv[2] if len(sys.argv) > 2 else "flight,hotel,shop").split(",")
+    max_steps = int(sys.argv[3]) if len(sys.argv) > 3 else 40
+    rows = []
+    for kind in kinds:
+        for i in range(n):
+            seed = 900000 + 10 * i          # held out: never generated for training (and disjoint from training ranges)
+            r = run(kind, seed, max_steps); rows.append(r)
+            print(f"{'PASS' if r['ok'] else 'FAIL'}  {kind:6s} seed={seed} steps={r['steps']:2d} {r['status'][:28]:28s} {r['wall']:5.1f}s  {r['goal'][:70]}", flush=True)
+    for kind in kinds:
+        k = [r for r in rows if r["kind"] == kind]
+        print(f"== {kind}: {sum(r['ok'] for r in k)}/{len(k)}  median steps {sorted(r['steps'] for r in k)[len(k)//2]}")
+    print(f"== total {sum(r['ok'] for r in rows)}/{len(rows)} ({100*sum(r['ok'] for r in rows)/max(1,len(rows)):.0f}%)")
+    json.dump(rows, open(os.environ.get("GYM_OUT", "/tmp/gym_eval.json"), "w"), indent=1)
+
+
+if __name__ == "__main__":
+    main()

code/finetune/webgym/expert.py

@@ -0,0 +1,330 @@
+"""Scripted expert for webgym tasks, run in the real headless Chromium through jev's Browser, so every recorded case
+has exactly the observation format the agent sees.  Writes jev-format training cases:
+
+  * goal = the full task       -> gold = the expert's next operation/target
+  * goal = the current sub-goal -> the same gold (sub-goal-conditioned policy)
+  * goal = a sub-goal just satisfied -> gold = DONE (sub-goal completion)
+  * on the final state, goal = full task -> DONE
+
+    python finetune/webgym/expert.py <out.jsonl> <n_episodes> [seed0=1] [kinds=flight,hotel,shop] [noise=0.1]
+
+Held-out seeds (seed % 10 == 0) are skipped here; eval_gym.py runs the real agent on them.
+"""
+import json, os, random, re, sys, time
+sys.path.insert(0, "/home/ckl/projects/S/jev-ultrafast")
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+os.environ.setdefault("BU_CDP_URL", "http://127.0.0.1:9222")
+from jev_ultrafast.browser import Browser, StalePage
+import spec as S
+
+BASE = os.environ.get("GYM_URL", "http://127.0.0.1:8811")
+
+
+def observe(b, tries=10):
+    for i in range(tries):
+        try:
+            return b.observe(screenshot=False)
+        except StalePage:
+            time.sleep(0.3)
+    raise StalePage("never settled")
+
+
+def js(b, expr):
+    return b.evaluate(expr)
+
+
+def status(b):
+    s = js(b, "JSON.stringify({st: window.__gym && window.__gym.status ? window.__gym.status() : null, meta: window.__gym ? window.__gym.meta : null, url: location.href})")
+    return json.loads(s) if s else {}
+
+
+def node_of(b, sel):
+    return js(b, f"window.__gym.node({json.dumps(sel)})")
+
+
+def where(b, sel):
+    return js(b, f"window.__gym.where({json.dumps(sel)})")
+
+
+class Stop(Exception):
+    pass
+
+
+class Expert:
+    def __init__(self, sp, b, rng):
+        self.sp, self.b, self.rng = sp, b, rng
+
+    # ------------------------------------------------------------------ satisfaction per sub-goal
+    def satisfied(self, key, st, meta):
+        sp = self.sp
+        if sp["kind"] in ("flight", "hotel"):
+            res = (meta or {}).get("results")
+            if key == "submit":
+                return res is not None
+            if key in ("filter", "sort"):
+                if res is None: return False
+                want = sp["post"][key]
+                return want in (res.get("filter") or "").split("|") if key == "filter" else res.get("sort") == want
+            if res is not None:       # past the form: the submitted search must carry the target value
+                t = sp["target"][key]
+                v = res.get(key)
+                if isinstance(t, str) and len(t) == 3 and t.isalpha():
+                    return isinstance(v, str) and v.replace("-ap", "") == t
+                return str(v) == str(t)
+            v, t = st["state"].get(key), sp["target"][key]
+            if isinstance(t, str) and len(t) == 3 and t.isalpha():   # city key
+                return isinstance(v, str) and (v == t or v == t + "-ap" or (v.startswith("text:") and S_name(t).lower() in v.lower()))
+            return v == t or str(v) == str(t)
+        lst = (meta or {}).get("listing") or {}
+        post = sp["post"]
+        if key == "open":
+            return (meta or {}).get("product") == post["open"]
+        if (meta or {}).get("product") is not None:
+            return True
+        if key == "page": return lst.get("page") == post["page"]
+        if key == "sort": return lst.get("sort") == post["sort"]
+        if key == "brand": return lst.get("brand") == post["brand"]
+        if key == "q": return (lst.get("q") or "").lower() == post["q"]
+        return False
+
+    def wrong_results(self, meta):
+        """Submitted with wrong or missing values: the form sub-goals are not all satisfied on the results page."""
+        res = (meta or {}).get("results")
+        if res is None or self.sp["kind"] not in ("flight", "hotel"):
+            return False
+        return any(not self.satisfied(k, {"state": {}}, meta) for k in self.sp["target"])
+
+    def current(self, st, meta):
+        for i, (key, text) in enumerate(self.sp["subgoals"]):
+            if key == "ret" and self.sp["target"].get("trip") != "round":
+                continue
+            if not self.satisfied(key, st, meta):
+                return i
+        return None
+
+    # ------------------------------------------------------------------ the next expert step for sub-goal key
+    def step_for(self, key, st, page):
+        sp, b, rng = self.sp, self.b, self.rng
+        open_ = set(st.get("open") or [])
+        if sp["kind"] in ("flight", "hotel"):
+            if key == "submit":
+                return ("CLICK", "[data-g=submit]", None)
+            if key == "filter":
+                i = ["Free cancellation", "Breakfast included", "Pool", "Pet friendly"].index(sp["post"]["filter"])
+                return ("CLICK", f"[data-g=filter-{i}]", None)
+            if key == "sort":
+                return ("SELECT", "[data-g=sort]", sp["post"]["sort"])
+            f = next(x for x in sp["fields"] if x["key"] == key)
+            t = sp["target"][key]
+            # close a popup that belongs to another field first (it may cover what we need)
+            for o in open_:
+                if o and not o.startswith(key) and o.endswith("-dialog"):
+                    other = o[: -len("-dialog")]
+                    for btn in (f"[data-g={other}-close]", f"[data-g={other}-done]"):
+                        if js(b, f"!!document.querySelector({json.dumps(btn)})"):
+                            return ("CLICK", btn, None)
+            if f["w"] == "combo":
+                dlg_open = (key + "-dialog") in open_
+                if f["trigger"] and not dlg_open:
+                    return ("CLICK", f"[data-g={key}-trigger]", None)
+                name = S_name(t)
+                if f["suggest"]:
+                    for k in ([t + "-ap", t] if rng.random() < 0.5 else [t, t + "-ap"]):
+                        sel = f"[data-g={key}-opt-{k}]"
+                        if where(b, sel) is not None:
+                            first = js(b, f"(document.querySelector('#{key}-list [role=option]')||{{}}).getAttribute && document.querySelector('#{key}-list [role=option]').getAttribute('data-g')")
+                            if first == sel[8:-1] and rng.random() < 0.25:
+                                return ("PRESS_ENTER", "press_enter", None)
+                            return ("CLICK", sel, None)
+                    alias = next((c[4] for c in S.CITIES if c[0] == t and c[4]), "")
+                    typed = rng.choice([name, name, name[: max(3, len(name) // 2 + 1)], alias or name])
+                    return ("TYPE_TEXT", f"[data-g={key}-input]", typed)
+                return ("TYPE_TEXT", f"[data-g={key}-input]", name)
+            if f["w"] == "choice":
+                if f["impl"] == "select":
+                    return ("SELECT", f"[data-g={key}-select]", t)
+                if f["impl"] == "dropdown":
+                    if js(b, f"document.querySelector('[data-g={key}-trigger]').getAttribute('aria-expanded')") != "true":
+                        return ("CLICK", f"[data-g={key}-trigger]", None)
+                return ("CLICK", f"[data-g={key}-{t}]", None)
+            if f["w"] == "date":
+                if f["impl"] == "typed":
+                    y, m, d = t.split("-")
+                    return ("TYPE_TEXT", f"[data-g={key}-input]", f"{m}/{d}/{y}")
+                if (key + "-dialog") not in open_:
+                    return ("CLICK", f"[data-g={key}-trigger]", None)
+                day = f"[data-g={key}-day-{t}]"
+                if js(b, f"!!document.querySelector({json.dumps(day)})"):
+                    if js(b, f"document.querySelector({json.dumps(day)}).getAttribute('aria-pressed')") != "true":
+                        return ("CLICK", day, None)
+                    return ("CLICK", f"[data-g={key}-done]", None)
+                shown = js(b, f"document.querySelector('[data-g={key}-dialog] strong').textContent")
+                mon, yr = shown.split()
+                cur = int(yr) * 12 + S.MONTHS.index(mon)
+                want = int(t[:4]) * 12 + int(t[5:7]) - 1
+                return ("CLICK", f"[data-g={key}-{'next' if want > cur else 'prev'}]", None)
+            if f["w"] == "count":
+                v = st["state"].get(key)
+                if f["impl"] == "select":
+                    return ("SELECT", f"[data-g={key}-select]", str(t))
+                if f["impl"] == "popup" and (key + "-dialog") not in open_:
+                    return ("CLICK", f"[data-g={key}-trigger]", None)
+                return ("CLICK", f"[data-g={key}-{'inc' if v < t else 'dec'}]", None)
+        # ------------------------------------------------ shop
+        post = sp["post"]
+        if key == "q":
+            val = js(b, "(document.querySelector('[data-g=search]')||{}).value")
+            if (val or "").strip().lower() != post["q"]:
+                return ("TYPE_TEXT", "[data-g=search]", post["q"])
+            if sp["search_button"] and rng.random() < 0.4:
+                return ("CLICK", "[data-g=search-btn]", None)
+            return ("PRESS_ENTER", "press_enter", None)
+        if key == "sort":
+            if sp["sort_impl"] == "select":
+                return ("SELECT", "[data-g=sort]", post["sort"])
+            if js(b, "document.getElementById('sl').hidden"):
+                return ("CLICK", "[data-g=sort-trigger]", None)
+            return ("CLICK_TEXT", "#sl [role=option]", post["sort"])
+        if key == "brand":
+            return ("CLICK", f"[data-g=brand-{post['brand']}]", None)
+        if key == "page":
+            n = post["page"]
+            if js(b, f"!!document.querySelector('[data-g=next]')") and (sp["pager"] == "next" or rng.random() < 0.5):
+                return ("CLICK", "[data-g=next]", None)
+            return ("CLICK_TEXT", "nav.pager a", str(n))
+        if key == "open":
+            return ("CLICK", f"a[href$='/p/{post['open']}']", None)
+        raise Stop(f"no step for {key}")
+
+
+def S_name(key):
+    return next(c[1] for c in S.CITIES if c[0] == key.replace("-ap", ""))
+
+
+def to_action(b, page, op, sel, val):
+    """Map an expert step to the observed jev action (same node, same kind).  None when it is not on screen."""
+    if op == "PRESS_ENTER":
+        return next((a for a in page["actions"] if a["id"] == "press_enter"), None)
+    if op == "CLICK_TEXT":
+        node = js(b, f"window.__gym.nodeText({json.dumps(sel)}, {json.dumps(val)})")
+        sel_for_where = None
+    else:
+        node = node_of(b, sel)
+    kind = {"CLICK": "click", "CLICK_TEXT": "click", "TYPE_TEXT": "fill", "SELECT": "select"}[op]
+    for a in page["actions"]:
+        if a.get("node") != node or a["kind"] != kind:
+            continue
+        if kind == "select" and not (a.get("value") == val or a["label"].split(" → ")[-1].strip() == val):
+            continue
+        if kind == "click" and a["label"].startswith("Open ") and op == "CLICK" and any(x.get("node") == node and x["kind"] == "fill" for x in page["actions"]):
+            continue      # the "Open <field>" click of an editable field is not the intended click
+        return a
+    return None
+
+
+def hist_entry(a, text, changed):
+    return {"action": a["label"], "kind": a["kind"], "text": text, "page_changed": changed}
+
+
+def case(page, goal, gold_op, gold_id, a, hist, sp, tag):
+    return {"page": -1, "url": page["url"], "title": page["title"], "goal": goal, "gold_op": gold_op, "gold_id": gold_id,
+            "kind": a["kind"] if a else "done", "label": a["label"] if a else "", "history": list(hist)[-10:], "source": "webgym",
+            "skill": tag, "seed": sp["seed"], "gym_kind": sp["kind"], "page_obj": {k: page[k] for k in ("url", "title", "text", "actions")}}
+
+
+def run_episode(kind, seed, out, rng, noise=0.1, max_steps=45):
+    sp = S.make(kind, seed)
+    b = Browser(f"{BASE}/t/{kind}/{seed}")
+    ex = Expert(sp, b, rng)
+    hist, n, prev_sub = [], 0, None
+    try:
+        page = observe(b)
+        for step in range(max_steps):
+            info = status(b); st = info.get("st") or {}; meta = info.get("meta")
+            if st and "open" not in st: st["open"] = []
+            if ex.wrong_results(meta):
+                # recovery: the search was submitted with wrong values -> back to the form
+                a = next((x for x in page["actions"] if x["label"] == "Modify search"), None)
+                if a is None:
+                    raise Stop("wrong results and no way back")
+                out.write(json.dumps(case(page, sp["goal"], "CLICK", a["id"], a, hist, sp, "recover"), ensure_ascii=False) + "\n"); n += 1
+                before = page["fingerprint"]; b.act(a, page); time.sleep(0.3); page = observe(b)
+                hist.append(hist_entry(a, None, page["fingerprint"] != before)); prev_sub = None
+                continue
+            cur = ex.current(st, meta)
+            # sub-goal completion: the previous sub-goal is now satisfied -> DONE under that sub-goal
+            if prev_sub is not None and prev_sub != cur:
+                sg = sp["subgoals"][prev_sub][1]
+                out.write(json.dumps(case(page, sg, "DONE", "DONE", None, hist, sp, "subgoal_done"), ensure_ascii=False) + "\n"); n += 1
+            if cur is None:
+                out.write(json.dumps(case(page, sp["goal"], "DONE", "DONE", None, hist, sp, "task_done"), ensure_ascii=False) + "\n")
+                return n + 1, True
+            key = sp["subgoals"][cur][0]
+            op, sel, val = ex.step_for(key, st, page)
+            # cookie banner in the way of anything below it: accept it
+            a = to_action(b, page, op, sel, val)
+            if a is None and op != "PRESS_ENTER":
+                if js(b, "!!document.getElementById('cookie')"):
+                    op, sel, val = "CLICK", "[data-g=cookie-accept]", None; a = to_action(b, page, op, sel, val)
+                elif op != "CLICK_TEXT" and where(b, sel) in (1, -1):
+                    sid = "scroll_down" if where(b, sel) == 1 else "scroll_up"
+                    a = next((x for x in page["actions"] if x["id"] == sid), None); op = sid.upper()
+                elif op == "CLICK_TEXT":
+                    a = next((x for x in page["actions"] if x["id"] == "scroll_down"), None); op = "SCROLL_DOWN"
+            if a is None:
+                raise Stop(f"step {step}: {op} {sel} not actionable")
+            gold_op = op if op in ("SCROLL_DOWN", "SCROLL_UP", "PRESS_ENTER") else {"CLICK_TEXT": "CLICK"}.get(op, op)
+            text = val if op == "TYPE_TEXT" else None
+            out.write(json.dumps(case(page, sp["goal"], gold_op, a["id"], a, hist, sp, key), ensure_ascii=False) + "\n")
+            out.write(json.dumps(case(page, sp["subgoals"][cur][1], gold_op, a["id"], a, hist, sp, key + "@sub"), ensure_ascii=False) + "\n")
+            n += 2
+            before = page["fingerprint"]
+            b.act(a, page, text=text); time.sleep(0.15 if op != "CLICK" else 0.25)
+            page = observe(b)
+            hist.append(hist_entry(a, text, page["fingerprint"] != before))
+            prev_sub = cur
+            # DART-style noise: a harmless detour the expert then has to recover from (not recorded as a gold step)
+            if rng.random() < noise:
+                submit_node = node_of(b, "[data-g=submit]")
+                decoy = {"Deals", "Help", "Blog", "Careers", "Travel guides", "Gift cards", "Manage booking", "About", "Privacy", "Contact"}
+                cands = [x for x in page["actions"] if x["kind"] == "click" and x.get("node") != submit_node and
+                         ((x["label"].startswith("Open ")) or (x.get("role") == "link" and x["label"] in decoy))]
+                if cands:
+                    z = rng.choice(cands)
+                    try:
+                        before = page["fingerprint"]; b.act(z, page); time.sleep(0.2); page = observe(b)
+                        hist.append(hist_entry(z, None, page["fingerprint"] != before))
+                    except Exception:
+                        page = observe(b)
+        raise Stop("step budget")
+    finally:
+        b.close()
+
+
+def main():
+    out_f, n_ep = sys.argv[1], int(sys.argv[2])
+    seed0 = int(sys.argv[3]) if len(sys.argv) > 3 else 1
+    kinds = (sys.argv[4] if len(sys.argv) > 4 else "flight,hotel,shop").split(",")
+    noise = float(sys.argv[5]) if len(sys.argv) > 5 else 0.1
+    rng = random.Random(seed0)
+    ok = fail = cases = 0; reasons = {}
+    t0 = time.time()
+    with open(out_f, "a") as out:
+        seed = seed0
+        while ok + fail < n_ep:
+            if seed % 10 == 0:
+                seed += 1; continue
+            kind = kinds[(ok + fail) % len(kinds)]
+            try:
+                c, done = run_episode(kind, seed, out, rng, noise)
+                cases += c; ok += 1
+            except Exception as e:
+                fail += 1; r = f"{type(e).__name__}: {str(e)[:60]}"; reasons[r] = reasons.get(r, 0) + 1
+            seed += 1
+            if (ok + fail) % 25 == 0:
+                print(f"  {ok+fail} episodes ok={ok} fail={fail} cases={cases} {time.time()-t0:.0f}s", flush=True)
+    print(f"done: ok={ok} fail={fail} cases={cases}  failures={sorted(reasons.items(), key=lambda kv: -kv[1])[:6]}")
+
+
+if __name__ == "__main__":
+    main()

code/finetune/webgym/serve.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+# serve.sh start|stop|restart  -- webgym server on :8811, tracked by pid file (never pkill by pattern)
+D=$(cd "$(dirname "$0")" && pwd); PIDF=${GYM_PIDFILE:-/tmp/webgym.pid}; LOG=${GYM_LOG:-/tmp/webgym.log}
+stop() { [ -f $PIDF ] && kill $(cat $PIDF) 2>/dev/null; rm -f $PIDF; sleep 0.5; }
+start() { nohup $D/../../.venv/bin/python $D/app.py 8811 > $LOG 2>&1 & echo $! > $PIDF; sleep 1.5; curl -s --noproxy '*' -o /dev/null -w "webgym %{http_code}\n" http://127.0.0.1:8811/t/flight/1; }
+case "$1" in start) start;; stop) stop;; restart) stop; start;; esac

code/finetune/webgym/spec.py

@@ -0,0 +1,222 @@
+"""Task specs for webgym: every (kind, seed) deterministically yields labels, widget implementations, layout, target
+values, the natural-language goal and planner-style sub-goals.  Seeds with seed % 10 == 0 are held out for evaluation.
+"""
+import datetime as dt
+import random
+
+CITIES = [("zrh", "Zürich", "Switzerland", "ZRH", "Zurich"), ("lon", "London", "United Kingdom", "LHR", ""), ("par", "Paris", "France", "CDG", ""),
+          ("ber", "Berlin", "Germany", "BER", ""), ("mad", "Madrid", "Spain", "MAD", ""), ("rom", "Rome", "Italy", "FCO", "Roma"),
+          ("ams", "Amsterdam", "Netherlands", "AMS", ""), ("vie", "Vienna", "Austria", "VIE", "Wien"), ("muc", "Munich", "Germany", "MUC", "München"),
+          ("lis", "Lisbon", "Portugal", "LIS", "Lisboa"), ("prg", "Prague", "Czechia", "PRG", "Praha"), ("dub", "Dublin", "Ireland", "DUB", ""),
+          ("osl", "Oslo", "Norway", "OSL", ""), ("hel", "Helsinki", "Finland", "HEL", ""), ("bcn", "Barcelona", "Spain", "BCN", ""),
+          ("nyc", "New York", "United States", "JFK", ""), ("chi", "Chicago", "United States", "ORD", ""), ("tor", "Toronto", "Canada", "YYZ", ""),
+          ("tyo", "Tokyo", "Japan", "HND", ""), ("sel", "Seoul", "South Korea", "ICN", ""), ("sin", "Singapore", "Singapore", "SIN", ""),
+          ("bkk", "Bangkok", "Thailand", "BKK", ""), ("syd", "Sydney", "Australia", "SYD", ""), ("ist", "Istanbul", "Türkiye", "IST", ""),
+          ("mil", "Milan", "Italy", "MXP", "Milano"), ("man", "Manchester", "United Kingdom", "MAN", ""), ("mia", "Miami", "United States", "MIA", ""),
+          ("ath", "Athens", "Greece", "ATH", ""), ("waw", "Warsaw", "Poland", "WAW", ""), ("cph", "Copenhagen", "Denmark", "CPH", "")]
+TODAY = dt.date(2026, 9, 25)
+MONTHS = ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"]
+
+
+def city_options(rng, airports):
+    out = []
+    for key, name, country, code, alias in CITIES:
+        out.append({"key": key, "label": f"{name}, {country}", "short": name, "alias": alias, "name": name})
+        if airports:
+            out.append({"key": key + "-ap", "label": f"{name} Airport ({code})", "short": f"{name} ({code})", "alias": code, "name": name})
+    return out
+
+
+def say_date(d, rng):
+    return rng.choice([f"{MONTHS[d.month-1]} {d.day}, {d.year}", f"{d.day} {MONTHS[d.month-1]} {d.year}", f"{MONTHS[d.month-1]} {d.day}"])
+
+
+def pick(rng, xs):
+    return rng.choice(xs)
+
+
+def impl(rng, choices, weights=None):
+    return rng.choices(choices, weights=weights)[0]
+
+
+# ------------------------------------------------------------------------------------------------ flight
+def flight(seed):
+    rng = random.Random(seed * 7919 + 1)
+    trigger = rng.random() < 0.4
+    airports = rng.random() < 0.5
+    o, d = rng.sample(CITIES, 2)
+    one_way = rng.random() < 0.65
+    dep = TODAY + dt.timedelta(days=rng.randint(2, 80))
+    ret = dep + dt.timedelta(days=rng.randint(2, 14))
+    adults = rng.choices([1, 2, 3, 4], weights=[5, 3, 1, 1])[0]
+    cabin = rng.choices(["economy", "premium", "business", "first"], weights=[5, 2, 2, 1])[0]
+    lab = dict(origin=pick(rng, ["Where from?", "From", "Origin", "Leaving from", "Departure airport", "Flying from"]),
+               destination=pick(rng, ["Where to?", "To", "Destination", "Going to", "Arrival airport", "Flying to"]),
+               trip=pick(rng, ["Trip type", "Ticket type", "Trip"]), date=pick(rng, ["Departure", "Depart", "Departure date", "Outbound"]),
+               ret=pick(rng, ["Return", "Return date", "Inbound"]), adults=pick(rng, ["Adults", "Passengers", "Travelers"]),
+               cabin=pick(rng, ["Cabin", "Class", "Seating class", "Travel class"]), submit=pick(rng, ["Search", "Search flights", "Find flights", "Explore", "Show flights"]))
+    ow_label = pick(rng, ["One way", "One-way"])
+    date_impl = impl(rng, ["picker", "typed"], [8, 1])
+    date_done = pick(rng, [None, "Done", "Apply"]) if date_impl == "picker" else None
+    fields = [
+        {"w": "choice", "key": "trip", "label": lab["trip"], "impl": impl(rng, ["radio", "segmented", "select", "dropdown"]), "default": "round",
+         "options": [{"key": "round", "label": pick(rng, ["Round trip", "Return"])}, {"key": "oneway", "label": ow_label}]},
+        {"w": "combo", "key": "origin", "label": lab["origin"], "trigger": trigger, "suggest": rng.random() < 0.9,
+         "options": city_options(rng, airports), "placeholder": pick(rng, ["", "City or airport", "Leaving from"])},
+        {"w": "combo", "key": "destination", "label": lab["destination"], "trigger": trigger, "suggest": True,
+         "options": city_options(rng, airports), "placeholder": pick(rng, ["", "City or airport", "Going to"])},
+        {"w": "date", "key": "date", "label": lab["date"], "impl": date_impl, "done": date_done, "start": [TODAY.year, TODAY.month - 1]},
+        {"w": "date", "key": "ret", "label": lab["ret"], "impl": date_impl, "done": date_done, "start": [TODAY.year, TODAY.month - 1], "only_round": True},
+        {"w": "count", "key": "adults", "label": lab["adults"], "impl": impl(rng, ["select", "stepper", "popup"]), "default": 1, "min": 1, "max": 9,
+         "one": "adult", "many": "adults"},
+        {"w": "choice", "key": "cabin", "label": lab["cabin"], "impl": impl(rng, ["radio", "select", "dropdown", "segmented"], [2, 4, 4, 1]), "default": "economy",
+         "options": [{"key": "economy", "label": "Economy"}, {"key": "premium", "label": "Premium economy"}, {"key": "business", "label": "Business"},
+                     {"key": "first", "label": "First"}]},
+    ]
+    target = {"trip": "oneway" if one_way else "round", "origin": o[0], "destination": d[0], "date": dep.isoformat(), "adults": adults, "cabin": cabin}
+    if not one_way:
+        target["ret"] = ret.isoformat()
+    # goal text: every constraint that differs from a default, plus some stated defaults
+    parts_trip = pick(rng, ["one-way flights", "a one-way flight", "one way flights"]) if one_way else pick(rng, ["round-trip flights", "a return flight", "round trip flights"])
+    who = {1: pick(rng, ["for one adult", "for 1 adult", ""]), 2: "for two adults", 3: "for 3 adults", 4: "for four adults"}[adults]
+    cab = {"economy": pick(rng, ["in economy", ""]), "premium": "in premium economy", "business": "in business class", "first": "in first class"}[cabin]
+    when = f"on {say_date(dep, rng)}" + ("" if one_way else f", returning {say_date(ret, rng)}")
+    o_name = o[4] if o[4] and rng.random() < 0.5 else o[1]
+    goal = " ".join(x for x in [pick(rng, ["Find", "Search for", "Look up", "Show me"]), parts_trip, f"from {o_name} to {d[1]}", when, who, cab] if x).strip() + "."
+    if rng.random() < 0.5:
+        goal += pick(rng, [" Stop when matching flight options are visible.", " Show the results.", " Stop at the results list."])
+    subgoals = [("trip", "Set the trip type to one-way" if one_way else "Set the trip type to round trip"),
+                ("origin", f"Set where from (origin, departure) to {o[1]}"), ("destination", f"Set where to (destination, arrival) to {d[1]}"),
+                ("date", f"Set the departure date to {MONTHS[dep.month-1]} {dep.day}, {dep.year}")]
+    if not one_way:
+        subgoals.append(("ret", f"Set the return date to {MONTHS[ret.month-1]} {ret.day}, {ret.year}"))
+    subgoals += [("adults", f"Set the number of adults to {adults}"), ("cabin", f"Set the cabin class to {dict(economy='economy', premium='premium economy', business='business', first='first')[cabin]}"),
+                 ("submit", "Search for flights and show the matching results")]
+    if not one_way and rng.random() < 0.0:
+        pass
+    return common(rng, "flight", seed, fields, target, goal, subgoals, lab["submit"],
+                  results_title=f"{o[1]} to {d[1]}")
+
+
+# ------------------------------------------------------------------------------------------------ hotel
+def hotel(seed):
+    rng = random.Random(seed * 104729 + 3)
+    trigger = rng.random() < 0.3
+    c = rng.choice(CITIES)
+    cin = TODAY + dt.timedelta(days=rng.randint(2, 70)); nights = rng.randint(1, 7); cout = cin + dt.timedelta(days=nights)
+    guests = rng.choices([1, 2, 3, 4], weights=[2, 5, 1, 1])[0]
+    date_done = pick(rng, [None, "Done", "Apply"])
+    lab = dict(dest=pick(rng, ["Destination", "Where are you going?", "City", "Location", "Going to"]),
+               cin=pick(rng, ["Check-in", "Check in", "Arrival"]), cout=pick(rng, ["Check-out", "Check out", "Departure"]),
+               guests=pick(rng, ["Guests", "Adults", "Travelers"]), submit=pick(rng, ["Search", "Search hotels", "Find stays", "Show hotels"]))
+    fields = [
+        {"w": "combo", "key": "dest", "label": lab["dest"], "trigger": trigger, "suggest": True, "options": city_options(rng, False),
+         "placeholder": pick(rng, ["", "Where to?", "Search destinations"])},
+        {"w": "date", "key": "cin", "label": lab["cin"], "impl": "picker", "done": date_done, "start": [TODAY.year, TODAY.month - 1]},
+        {"w": "date", "key": "cout", "label": lab["cout"], "impl": "picker", "done": date_done, "start": [TODAY.year, TODAY.month - 1]},
+        {"w": "count", "key": "guests", "label": lab["guests"], "impl": impl(rng, ["select", "stepper", "popup"]), "default": 2, "min": 1, "max": 8,
+         "one": "guest", "many": "guests"},
+    ]
+    target = {"dest": c[0], "cin": cin.isoformat(), "cout": cout.isoformat(), "guests": guests}
+    post = {}
+    if rng.random() < 0.5:
+        post["filter"] = rng.choice(["Free cancellation", "Breakfast included", "Pool", "Pet friendly"])
+    if rng.random() < 0.4:
+        post["sort"] = rng.choice(["Price: low to high", "Guest rating", "Distance to center"])
+    who = {1: "for one guest", 2: pick(rng, ["for two guests", "for 2 adults", ""]), 3: "for three guests", 4: "for 4 guests"}[guests]
+    goal = f"{pick(rng, ['Find a hotel', 'Search for hotels', 'Look for a place to stay', 'Find stays'])} in {c[1]} from {say_date(cin, rng)} to {say_date(cout, rng)} {who}".strip()
+    if post.get("filter"):
+        goal += f", {pick(rng, ['only showing', 'filtered to', 'with'])} {post['filter'].lower()}"
+    if post.get("sort"):
+        goal += f", sorted by {post['sort'].lower()}"
+    goal += "."
+    subgoals = [("dest", f"Set the destination (city, location) to {c[1]}"), ("cin", f"Set the check-in date to {MONTHS[cin.month-1]} {cin.day}, {cin.year}"),
+                ("cout", f"Set the check-out date to {MONTHS[cout.month-1]} {cout.day}, {cout.year}"), ("guests", f"Set the number of guests to {guests}"),
+                ("submit", "Search for hotels and show the results")]
+    if post.get("filter"):
+        subgoals.append(("filter", f"Turn on the '{post['filter']}' filter"))
+    if post.get("sort"):
+        subgoals.append(("sort", f"Sort the results by {post['sort'].lower()}"))
+    return common(rng, "hotel", seed, fields, target, goal, subgoals, lab["submit"], results_title=f"Stays in {c[1]}", post=post)
+
+
+# ------------------------------------------------------------------------------------------------ shop listing
+BRANDS = ["Acme", "Globex", "Initech", "Umbrella", "Hooli", "Stark", "Wayne", "Tyrell"]
+PRODUCTS = ["Laptop", "Headphones", "Backpack", "Coffee maker", "Desk lamp", "Keyboard", "Monitor", "Water bottle", "Running shoes", "Camera",
+            "Blender", "Jacket", "Tent", "Watch", "Speaker", "Router"]
+
+
+def shop(seed):
+    rng = random.Random(seed * 15485863 + 5)
+    cat = rng.choice(["Electronics", "Home", "Outdoor", "Fashion", "Kitchen"])
+    items = []
+    for i in range(48):
+        r = random.Random(seed * 1000 + i)
+        items.append({"id": i, "name": f"{r.choice(BRANDS)} {r.choice(PRODUCTS)} {r.choice(['Pro', 'Lite', 'Max', 'Mini', 'Plus', 'X', '2', 'S'])}",
+                      "brand": None, "price": round(r.uniform(9, 900), 2)})
+        items[-1]["brand"] = items[-1]["name"].split()[0]
+    per_page = rng.choice([12, 16])
+    task = rng.choice(["page", "sort", "brand", "search", "search_sort", "brand_page", "open_on_page"])
+    post, goal, subgoals = {}, "", []
+    if task == "page":
+        n = rng.choice([2, 2, 3])
+        post["page"] = n
+        goal = pick(rng, [f"Go to page {n} of the products.", f"Show page {n}.", f"Open page {n} of the listing.", "Go to the next page of products." if n == 2 else f"Go to page {n}."])
+        subgoals = [("page", f"Go to page {n} of the product list")]
+    elif task == "sort":
+        s = rng.choice(["Price: low to high", "Price: high to low", "Newest", "Top rated"])
+        post["sort"] = s
+        goal = pick(rng, [f"Sort the products by {s.lower()}.", f"Show the products sorted by {s.lower()}.", f"{s}, please."])
+        subgoals = [("sort", f"Sort the products by {s.lower()}")]
+    elif task == "brand":
+        b = rng.choice(BRANDS); post["brand"] = b
+        goal = pick(rng, [f"Show only {b} products.", f"Filter the list to the brand {b}.", f"I only want to see {b} items."])
+        subgoals = [("brand", f"Turn on the brand filter for {b}")]
+    elif task == "search":
+        q = rng.choice(PRODUCTS); post["q"] = q.lower()
+        goal = pick(rng, [f"Search the shop for '{q.lower()}'.", f"Find {q.lower()}s.", f"Search for {q.lower()} and show the results."])
+        subgoals = [("q", f"Search for '{q.lower()}' and show the results")]
+    elif task == "search_sort":
+        q = rng.choice(PRODUCTS); s = rng.choice(["Price: low to high", "Price: high to low"]); post.update(q=q.lower(), sort=s)
+        goal = f"Search for {q.lower()} and sort the results by {s.lower()}."
+        subgoals = [("q", f"Search for '{q.lower()}' and show the results"), ("sort", f"Sort the results by {s.lower()}")]
+    elif task == "brand_page":
+        counts = {x: sum(i["brand"] == x for i in items) for x in BRANDS}
+        ok = [x for x in BRANDS if counts[x] > per_page] or [max(counts, key=counts.get)]
+        b = rng.choice(ok); post.update(brand=b, page=2)
+        if counts[b] <= per_page:          # not enough items for a second page: make it a plain brand task
+            post.pop("page"); goal = f"Show only {b} products."; subgoals = [("brand", f"Turn on the brand filter for {b}")]
+            return shop_spec(rng, seed, cat, items, per_page, post, goal, subgoals)
+        goal = f"Show only {b} products and go to page 2."
+        subgoals = [("brand", f"Turn on the brand filter for {b}"), ("page", "Go to page 2 of the product list")]
+    else:
+        idx = rng.randint(per_page, min(47, 2 * per_page - 1)); post.update(page=2, open=idx)
+        goal = f"Open the product '{items[idx]['name']}' (it is on page 2)." if rng.random() < 0.5 else f"Go to page 2 and open '{items[idx]['name']}'."
+        subgoals = [("page", "Go to page 2 of the product list"), ("open", f"Open the product page of '{items[idx]['name']}'")]
+    return shop_spec(rng, seed, cat, items, per_page, post, goal, subgoals)
+
+
+def shop_spec(rng, seed, cat, items, per_page, post, goal, subgoals):
+    return {"kind": "shop", "seed": seed, "category": cat, "items": items, "per_page": per_page, "post": post, "goal": goal, "subgoals": subgoals,
+            "search_button": rng.random() < 0.5, "sort_impl": impl(rng, ["select", "dropdown"]), "pager": pick(rng, ["numbers", "next", "both"]),
+            "hero": rng.choice([0, 0, 200, 450]), "cookie": rng.random() < 0.25, "held_out": seed % 10 == 0,
+            "labels": {"search": pick(rng, ["Search products", "Search", "What are you looking for?"]), "sort": pick(rng, ["Sort by", "Sort", "Order by"]),
+                       "brand": pick(rng, ["Brand", "Brands", "Manufacturer"]), "next": pick(rng, ["Next", "Next ›", "Next page", "›"])}}
+
+
+# ------------------------------------------------------------------------------------------------ shared
+def common(rng, kind, seed, fields, target, goal, subgoals, submit, results_title, post=None):
+    rng.shuffle(order := [f["key"] for f in fields])
+    # keep a natural order most of the time (trip type first, then places, dates, counts, class), shuffled otherwise
+    natural = rng.random() < 0.7
+    return {"kind": kind, "seed": seed, "fields": fields, "target": target, "goal": goal, "subgoals": subgoals, "submit": submit,
+            "order": [f["key"] for f in fields] if natural else order, "results_title": results_title, "post": post or {},
+            "hero": rng.choice([0, 0, 0, 300, 650]), "cookie": rng.random() < 0.25, "held_out": seed % 10 == 0,
+            "decoys": rng.sample(["Deals", "Help", "Sign in", "Manage booking", "Explore", "Travel guides", "Gift cards", "Careers", "Blog"], 4)}
+
+
+KINDS = {"flight": flight, "hotel": hotel, "shop": shop}
+
+
+def make(kind, seed):
+    return KINDS[kind](seed)

code/finetune/webgym/widgets.js

@@ -0,0 +1,194 @@
+// webgym widget library: every widget keeps its committed value in window.__gym.state, exactly like a real site
+// (typing into an autocomplete field does NOT commit a value until an option is chosen).
+(function () {
+  const G = window.__gym = window.__gym || { state: {}, fields: {} };
+  const $ = (s, r = document) => r.querySelector(s);
+  const el = (tag, attrs = {}, ...kids) => {
+    const e = document.createElement(tag);
+    for (const [k, v] of Object.entries(attrs)) {
+      if (k === "text") e.textContent = v; else if (k.startsWith("on")) e.addEventListener(k.slice(2), v); else e.setAttribute(k, v);
+    }
+    for (const k of kids) e.append(k);
+    return e;
+  };
+  const norm = s => s.normalize("NFD").replace(/[̀-ͯ]/g, "").toLowerCase();
+  const close = d => { d.hidden = true; };
+
+  // ---------------------------------------------------------------- autocomplete (inline or behind a trigger button)
+  G.combo = function (host, f) {
+    const id = f.key;
+    const input = el("input", { id: id + "-input", "data-g": id + "-input", type: "text", autocomplete: "off", role: "combobox",
+      "aria-autocomplete": "list", "aria-expanded": "false", "aria-controls": id + "-list", placeholder: f.placeholder || "" });
+    const label = el("label", { for: id + "-input", text: f.label });
+    const list = el("ul", { id: id + "-list", role: "listbox", class: "list" }); list.hidden = true;
+    const commit = opt => {
+      G.state[id] = opt.key; input.value = opt.short; list.hidden = true; input.setAttribute("aria-expanded", "false");
+      if (trigger) { trigger.querySelector(".v").textContent = opt.short; close(dialog); }
+    };
+    const render = () => {
+      const q = norm(input.value.trim()); list.innerHTML = "";
+      if (!f.suggest) { G.state[id] = input.value.trim() ? "text:" + input.value.trim() : null; return; }
+      if (q.length < 2) { list.hidden = true; input.setAttribute("aria-expanded", "false"); return; }
+      const hits = f.options.filter(o => norm(o.label).includes(q) || norm(o.alias || "").includes(q)).slice(0, 6);
+      hits.forEach((o, i) => list.append(el("li", { role: "option", tabindex: "-1", "data-g": id + "-opt-" + o.key, class: "opt",
+        onclick: () => commit(o) }, o.label)));
+      list.hidden = !hits.length; input.setAttribute("aria-expanded", String(!!hits.length));
+    };
+    input.addEventListener("input", render);
+    input.addEventListener("blur", () => setTimeout(() => { list.hidden = true; input.setAttribute("aria-expanded", "false"); }, 250));
+    input.addEventListener("focus", () => { if (input.value.trim().length >= 2 && f.suggest) render(); });
+    input.addEventListener("keydown", e => {
+      if (e.key === "Enter") { e.preventDefault(); const first = list.querySelector("[role=option]"); if (first && !list.hidden) first.click(); }
+    });
+    let trigger = null, dialog = null;
+    if (f.trigger) {        // Google-style: a button shows the value; clicking it opens a dialog with the real input
+      trigger = el("button", { type: "button", "data-g": id + "-trigger", "aria-haspopup": "dialog", class: "trig" },
+        el("span", { class: "k", text: f.label + " " }), el("span", { class: "v", text: f.empty || "" }));
+      dialog = el("div", { role: "dialog", "aria-label": f.label, class: "dlg", "data-g": id + "-dialog" }, label, input, list,
+        el("button", { type: "button", "data-g": id + "-close", onclick: () => close(dialog), text: "Close" }));
+      dialog.hidden = true;
+      trigger.addEventListener("click", () => { dialog.hidden = false; input.value = ""; input.focus(); });
+      host.append(trigger, dialog);
+    } else host.append(label, input, list);
+    G.fields[id] = { kind: "combo", trigger: !!f.trigger, suggest: !!f.suggest };
+    G.state[id] = null;
+  };
+
+  // ---------------------------------------------------------------- one-of: radio group, segmented buttons, native select, custom dropdown
+  G.choice = function (host, f) {
+    const id = f.key; G.state[id] = f.default; G.fields[id] = { kind: "choice", impl: f.impl };
+    const legend = f.label;
+    if (f.impl === "radio") {
+      const fs = el("fieldset", {}, el("legend", { text: legend }));
+      f.options.forEach(o => {
+        const r = el("input", { type: "radio", name: id, id: id + "-" + o.key, value: o.key, "data-g": id + "-" + o.key });
+        if (o.key === f.default) r.checked = true;
+        r.addEventListener("change", () => { G.state[id] = o.key; });
+        fs.append(el("span", { class: "ro" }, r, el("label", { for: id + "-" + o.key, text: o.label })));
+      });
+      host.append(fs);
+    } else if (f.impl === "segmented") {
+      const g = el("div", { role: "group", "aria-label": legend, class: "seg" });
+      f.options.forEach(o => {
+        const b = el("button", { type: "button", "data-g": id + "-" + o.key, "aria-pressed": String(o.key === f.default), text: o.label });
+        b.addEventListener("click", () => { G.state[id] = o.key; g.querySelectorAll("button").forEach(x => x.setAttribute("aria-pressed", String(x === b))); });
+        g.append(b);
+      });
+      host.append(el("span", { class: "k", text: legend + ": " }), g);
+    } else if (f.impl === "select") {
+      const s = el("select", { id: id + "-select", "data-g": id + "-select" });
+      f.options.forEach(o => { const op = el("option", { value: o.key, text: o.label }); if (o.key === f.default) op.selected = true; s.append(op); });
+      s.addEventListener("change", () => { G.state[id] = s.value; });
+      host.append(el("label", { for: id + "-select", text: legend }), s);
+    } else {                // custom dropdown: button showing the value -> listbox of options
+      const cur = () => f.options.find(o => o.key === G.state[id]).label;
+      const b = el("button", { type: "button", "data-g": id + "-trigger", "aria-haspopup": "listbox", "aria-expanded": "false",
+        "aria-label": legend + ": " + cur(), text: cur() + " ▾" });
+      const lb = el("div", { role: "listbox", "aria-label": legend, class: "list" }); lb.hidden = true;
+      f.options.forEach(o => lb.append(el("div", { role: "option", "data-g": id + "-" + o.key, tabindex: "-1", class: "opt",
+        onclick: () => { G.state[id] = o.key; b.textContent = o.label + " ▾"; b.setAttribute("aria-label", legend + ": " + o.label);
+          lb.hidden = true; b.setAttribute("aria-expanded", "false"); } }, o.label)));
+      b.addEventListener("click", () => { lb.hidden = !lb.hidden; b.setAttribute("aria-expanded", String(!lb.hidden)); });
+      host.append(el("span", { class: "k", text: legend + " " }), b, lb);
+    }
+  };
+
+  // ---------------------------------------------------------------- date picker (month grid) or a typed date field
+  const MONTHS = ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"];
+  const DAYS = ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"];
+  G.fmt = d => `${DAYS[d.getDay()]}, ${MONTHS[d.getMonth()]} ${d.getDate()}, ${d.getFullYear()}`;
+  G.date = function (host, f) {
+    const id = f.key; G.state[id] = null; G.fields[id] = { kind: "date", impl: f.impl, done: !!f.done };
+    if (f.impl === "typed") {
+      const i = el("input", { id: id + "-input", "data-g": id + "-input", type: "text", placeholder: "MM/DD/YYYY" });
+      i.addEventListener("input", () => {
+        const m = i.value.trim().match(/^(\d{1,2})\/(\d{1,2})\/(\d{4})$/) || i.value.trim().match(/^(\d{4})-(\d{2})-(\d{2})$/);
+        if (!m) { G.state[id] = null; return; }
+        const d = i.value.includes("/") ? new Date(+m[3], +m[1] - 1, +m[2]) : new Date(+m[1], +m[2] - 1, +m[3]);
+        G.state[id] = `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, "0")}-${String(d.getDate()).padStart(2, "0")}`;
+      });
+      host.append(el("label", { for: id + "-input", text: f.label }), i);
+      return;
+    }
+    let view = new Date(f.start[0], f.start[1], 1), picked = null;
+    const shown = el("span", { class: "v", text: f.empty || "" });
+    const trig = el("button", { type: "button", "data-g": id + "-trigger", "aria-haspopup": "dialog", class: "trig" },
+      el("span", { class: "k", text: f.label + " " }), shown);
+    const dlg = el("div", { role: "dialog", "aria-label": f.label + " calendar", class: "dlg cal", "data-g": id + "-dialog" }); dlg.hidden = true;
+    const title = el("strong", {});
+    const grid = el("div", { role: "grid", class: "grid" });
+    const draw = () => {
+      title.textContent = MONTHS[view.getMonth()] + " " + view.getFullYear(); grid.innerHTML = "";
+      const n = new Date(view.getFullYear(), view.getMonth() + 1, 0).getDate();
+      for (let day = 1; day <= n; day++) {
+        const d = new Date(view.getFullYear(), view.getMonth(), day), iso = `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, "0")}-${String(day).padStart(2, "0")}`;
+        const b = el("button", { type: "button", "data-g": id + "-day-" + iso, "aria-label": G.fmt(d), "aria-pressed": String(picked === iso), text: String(day) });
+        b.addEventListener("click", () => {
+          picked = iso; shown.textContent = G.fmt(d).slice(0, -6); draw();
+          if (!f.done) { G.state[id] = iso; dlg.hidden = true; }
+        });
+        grid.append(b);
+      }
+    };
+    const nav = (k, lab) => el("button", { type: "button", "data-g": id + "-" + k, "aria-label": lab, text: k === "prev" ? "‹" : "›",
+      onclick: () => { view = new Date(view.getFullYear(), view.getMonth() + (k === "prev" ? -1 : 1), 1); draw(); } });
+    dlg.append(el("div", { class: "hdr" }, nav("prev", "Previous month"), title, nav("next", "Next month")), grid,
+      el("button", { type: "button", "data-g": id + "-close", "aria-label": "Close calendar", text: "\u2715", onclick: () => { dlg.hidden = true; } }));
+    if (f.done) dlg.append(el("button", { type: "button", "data-g": id + "-done", text: f.done,
+      onclick: () => { if (picked) G.state[id] = picked; dlg.hidden = true; } }));
+    trig.addEventListener("click", () => { dlg.hidden = false; draw(); });
+    host.append(trig, dlg);
+  };
+
+  // ---------------------------------------------------------------- counts: native select, stepper, or a popup with steppers
+  G.count = function (host, f) {
+    const id = f.key; G.state[id] = f.default; G.fields[id] = { kind: "count", impl: f.impl };
+    const noun = n => n + " " + (n === 1 ? f.one : f.many);
+    if (f.impl === "select") {
+      const s = el("select", { id: id + "-select", "data-g": id + "-select" });
+      for (let n = f.min; n <= f.max; n++) { const o = el("option", { value: String(n), text: noun(n) }); if (n === f.default) o.selected = true; s.append(o); }
+      s.addEventListener("change", () => { G.state[id] = +s.value; });
+      host.append(el("label", { for: id + "-select", text: f.label }), s);
+      return;
+    }
+    const val = el("span", { class: "v", "aria-live": "polite", text: String(f.default) });
+    const step = d => { G.state[id] = Math.max(f.min, Math.min(f.max, G.state[id] + d)); val.textContent = String(G.state[id]);
+      if (sum) sum.textContent = noun(G.state[id]); };
+    const box = el("div", { class: "step" }, el("span", { class: "k", text: f.label + " " }),
+      el("button", { type: "button", "data-g": id + "-dec", "aria-label": "Decrease " + f.many, text: "−", onclick: () => step(-1) }), val,
+      el("button", { type: "button", "data-g": id + "-inc", "aria-label": "Increase " + f.many, text: "+", onclick: () => step(1) }));
+    let sum = null;
+    if (f.impl === "popup") {
+      sum = el("span", { class: "v", text: noun(f.default) });
+      const dlg = el("div", { role: "dialog", "aria-label": f.label, class: "dlg", "data-g": id + "-dialog" }, box,
+        el("button", { type: "button", "data-g": id + "-done", text: "Done", onclick: () => { dlg.hidden = true; } }));
+      dlg.hidden = true;
+      const trig = el("button", { type: "button", "data-g": id + "-trigger", "aria-haspopup": "dialog", class: "trig" }, sum);
+      trig.addEventListener("click", () => { dlg.hidden = false; });
+      host.append(trig, dlg);
+    } else host.append(box);
+  };
+
+  // ---------------------------------------------------------------- plumbing for the expert
+  G.status = () => {
+    const open = [...document.querySelectorAll("[role=dialog],[role=listbox]")].filter(d => !d.hidden && d.offsetParent !== null)
+      .map(d => d.getAttribute("data-g") || d.id || d.getAttribute("aria-label"));
+    return { state: G.state, fields: G.fields, open, url: location.href };
+  };
+  G.node = sel => { const e = document.querySelector(sel); const c = window.__jevFast; return e && c ? (c.ids.get(e) || null) : null; };
+  G.nodeText = (sel, text) => {
+    const c = window.__jevFast; if (!c) return null;
+    const e = [...document.querySelectorAll(sel)].find(x => x.textContent.trim() === text);
+    return e ? (c.ids.get(e) || null) : null;
+  };
+  G.where = sel => {                 // -1 above the viewport, 0 inside, 1 below, null missing/hidden
+    const e = document.querySelector(sel); if (!e || !e.checkVisibility()) return null; const r = e.getBoundingClientRect();
+    const cy = r.y + r.height / 2;      // jev offers an element only when its center is inside the viewport
+    return cy < 0 ? -1 : cy >= innerHeight ? 1 : 0;
+  };
+  G.attr = (sel, a) => { const e = document.querySelector(sel); return e ? e.getAttribute(a) : null; };
+  G.visible = sel => {
+    const e = document.querySelector(sel); if (!e) return false; const r = e.getBoundingClientRect();
+    return r.width > 0 && r.bottom > 0 && r.top < innerHeight && e.checkVisibility();
+  };
+})();

code/kernels/fast.py

@@ -190,8 +190,25 @@ class FastLaya:
         p = torch.softmax(logits, -1)
         k = marker_mask.sum(-1).clamp(min=2).float()
         ent = -(p * torch.log(p.clamp_min(1e-9))).sum(-1) / torch.log(k)
-        top2 = p.topk(2, -1).values
+        # a choice with a single option (e.g. only WAIT left after exclusions): pad the second slot, as upstream does
+        top2 = p.topk(2, -1).values if p.shape[-1] >= 2 else torch.stack([p[:, 0], torch.zeros_like(p[:, 0])], dim=-1)
         feats = torch.stack([top2[:, 0], top2[:, 0] - top2[:, 1], ent, k / 255.0], -1)
         pooled = h[:, 0].float()
         act_logits = m.act_head(torch.cat([pooled, feats], -1))
         return logits, act_logits
+
+
+def accelerate(agent, use_graphs=True, verbose=False):
+    """Patch a laya Agent in place so agent.predict() uses the TileLang fast path.  Returns the FastLaya."""
+    fast = FastLaya(agent.model, max_len=agent.cfg.get("max_len", 1024), use_graphs=use_graphs, verbose=verbose)
+    if not hasattr(agent, "_orig_forward"):
+        agent._orig_forward = agent.model.forward
+    agent.model.forward = fast.forward
+    agent.fast = fast
+    return fast
+
+
+def restore(agent):
+    if hasattr(agent, "_orig_forward"):
+        agent.model.forward = agent._orig_forward
+        del agent._orig_forward

results/round3/gym_eval_v14s.json

@@ -0,0 +1,314 @@
+[
+ {
+  "kind": "flight",
+  "seed": 900000,
+  "ok": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.5,
+  "url": "http://127.0.0.1:8811/t/flight/900000#pop-ber",
+  "goal": "Show me a return flight from Oslo to Berlin on October 18, returning 23 October 2026 for 3 adults in premium economy. Stop when matching flight options are visible.",
+  "plan": [
+   "Show me a return flight from Oslo to Berlin on October 18, returning 23 October 2026 for 3 adults in premium economy. Stop when matching flight options are visible."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900010,
+  "ok": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.3,
+  "url": "http://127.0.0.1:8811/t/flight/900010",
+  "goal": "Look up round-trip flights from Tokyo to Amsterdam on October 13, returning 20 October 2026 for two adults in first class.",
+  "plan": [
+   "Look up round-trip flights from Tokyo to Amsterdam on October 13, returning 20 October 2026 for two adults in first class."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900020,
+  "ok": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/flight/900020",
+  "goal": "Search for one-way flights from Roma to Seoul on 15 October 2026 for two adults. Stop when matching flight options are visible.",
+  "plan": [
+   "Search for one-way flights from Roma to Seoul on 15 October 2026 for two adults. Stop when matching flight options are visible."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900030,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.2,
+  "url": "http://127.0.0.1:8811/t/flight/900030",
+  "goal": "Show me a one-way flight from Helsinki to Munich on 26 October 2026 for four adults in business class.",
+  "plan": [
+   "Show me a one-way flight from Helsinki to Munich on 26 October 2026 for four adults in business class."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900040,
+  "ok": false,
+  "steps": 4,
+  "status": "done",
+  "wall": 1.6,
+  "url": "http://127.0.0.1:8811/t/flight/900040/results?trip=round&adults=1&cabin=economy#flight-3",
+  "goal": "Show me one way flights from Manchester to Bangkok on November 21 for 3 adults in business class.",
+  "plan": [
+   "Show me one way flights from Manchester to Bangkok on November 21 for 3 adults in business class."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900050,
+  "ok": false,
+  "steps": 7,
+  "status": "blocked",
+  "wall": 1.9,
+  "url": "http://127.0.0.1:8811/t/flight/900050",
+  "goal": "Show me a one-way flight from Amsterdam to Barcelona on November 8, 2026 for 1 adult in economy.",
+  "plan": [
+   "Show me a one-way flight from Amsterdam to Barcelona on November 8, 2026 for 1 adult in economy."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900060,
+  "ok": false,
+  "steps": 21,
+  "status": "blocked",
+  "wall": 3.2,
+  "url": "http://127.0.0.1:8811/t/flight/900060",
+  "goal": "Find one-way flights from Seoul to Copenhagen on 10 October 2026 for two adults in business class.",
+  "plan": [
+   "Find one-way flights from Seoul to Copenhagen on 10 October 2026 for two adults in business class."
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900070,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 10.3,
+  "url": "http://127.0.0.1:8811/t/flight/900070",
+  "goal": "Look up a return flight from Barcelona to New York on November 13, returning November 27 for 1 adult. Stop when matching flight options are visible.",
+  "plan": [
+   "Look up a return flight from Barcelona to New York on November 13, returning November 27 for 1 adult. Stop when matching flight options are visible."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900000,
+  "ok": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/hotel/900000/results?guests=2&sort=Price%3A+low+to+high",
+  "goal": "Find a hotel in Toronto from 31 October 2026 to November 2, sorted by price: low to high.",
+  "plan": [
+   "Find a hotel in Toronto from 31 October 2026 to November 2, sorted by price: low to high."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900010,
+  "ok": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/hotel/900010/results?guests=2",
+  "goal": "Search for hotels in Chicago from 30 October 2026 to November 1 for three guests, sorted by price: low to high.",
+  "plan": [
+   "Search for hotels in Chicago from 30 October 2026 to November 1 for three guests, sorted by price: low to high."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900020,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.3,
+  "url": "http://127.0.0.1:8811/t/hotel/900020/results?guests=2#home",
+  "goal": "Find a hotel in Oslo from 30 October 2026 to November 3, 2026 for one guest, only showing pet friendly.",
+  "plan": [
+   "Find a hotel in Oslo from 30 October 2026 to November 3, 2026 for one guest, only showing pet friendly."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900030,
+  "ok": false,
+  "steps": 4,
+  "status": "done",
+  "wall": 1.6,
+  "url": "http://127.0.0.1:8811/t/hotel/900030",
+  "goal": "Find stays in Toronto from November 16, 2026 to November 21, 2026, filtered to pet friendly.",
+  "plan": [
+   "Find stays in Toronto from November 16, 2026 to November 21, 2026, filtered to pet friendly."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900040,
+  "ok": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.4,
+  "url": "http://127.0.0.1:8811/t/hotel/900040",
+  "goal": "Search for hotels in Tokyo from 14 November 2026 to 15 November 2026, with pet friendly.",
+  "plan": [
+   "Search for hotels in Tokyo from 14 November 2026 to 15 November 2026, with pet friendly."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900050,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 4.5,
+  "url": "http://127.0.0.1:8811/t/hotel/900050/results?guests=2&sort=Guest+rating#home",
+  "goal": "Find stays in Copenhagen from December 4 to December 7, 2026 for one guest, sorted by guest rating.",
+  "plan": [
+   "Find stays in Copenhagen from December 4 to December 7, 2026 for one guest, sorted by guest rating."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900060,
+  "ok": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.3,
+  "url": "http://127.0.0.1:8811/t/hotel/900060",
+  "goal": "Find stays in Rome from November 3, 2026 to 10 November 2026 for 4 guests.",
+  "plan": [
+   "Find stays in Rome from November 3, 2026 to 10 November 2026 for 4 guests."
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900070,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 4.5,
+  "url": "http://127.0.0.1:8811/t/hotel/900070/results?guests=4&filter=Free+cancellation#home",
+  "goal": "Find a hotel in Chicago from 27 September 2026 to October 4, only showing free cancellation.",
+  "plan": [
+   "Find a hotel in Chicago from 27 September 2026 to October 4, only showing free cancellation."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900000,
+  "ok": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.2,
+  "url": "http://127.0.0.1:8811/t/shop/900000/p/4",
+  "goal": "Go to page 2 and open 'Hooli Water bottle X'.",
+  "plan": [
+   "Go to page 2 and open 'Hooli Water bottle X'."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900010,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/shop/900010?sort=Price%3A+low+to+high",
+  "goal": "Sort the products by price: low to high.",
+  "plan": [
+   "Sort the products by price: low to high."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900020,
+  "ok": false,
+  "steps": 8,
+  "status": "blocked",
+  "wall": 1.9,
+  "url": "http://127.0.0.1:8811/t/shop/900020/p/8#home",
+  "goal": "Show page 3.",
+  "plan": [
+   "Show page 3."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900030,
+  "ok": false,
+  "steps": 0,
+  "status": "done",
+  "wall": 1.0,
+  "url": "http://127.0.0.1:8811/t/shop/900030",
+  "goal": "Find water bottles.",
+  "plan": [
+   "Find water bottles."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900040,
+  "ok": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.2,
+  "url": "http://127.0.0.1:8811/t/shop/900040/p/2",
+  "goal": "Show only Hooli products.",
+  "plan": [
+   "Show only Hooli products."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900050,
+  "ok": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/shop/900050?brand=Acme&sort=Price%3A+low+to+high",
+  "goal": "Filter the list to the brand Acme.",
+  "plan": [
+   "Filter the list to the brand Acme."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900060,
+  "ok": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.4,
+  "url": "http://127.0.0.1:8811/t/shop/900060?q=backpack",
+  "goal": "Search the shop for 'backpack'.",
+  "plan": [
+   "Search the shop for 'backpack'."
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900070,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.1,
+  "url": "http://127.0.0.1:8811/t/shop/900070?brand=Umbrella",
+  "goal": "I only want to see Umbrella items.",
+  "plan": [
+   "I only want to see Umbrella items."
+  ]
+ }
+]

results/round3/gym_eval_v14s.log

@@ -0,0 +1,28 @@
+FAIL  flight seed=900000 steps= 1 done                           1.5s  Show me a return flight from Oslo to Berlin on October 18, returning 2
+FAIL  flight seed=900010 steps= 2 done                           1.3s  Look up round-trip flights from Tokyo to Amsterdam on October 13, retu
+FAIL  flight seed=900020 steps= 1 done                           1.1s  Search for one-way flights from Roma to Seoul on 15 October 2026 for t
+FAIL  flight seed=900030 steps=40 ready                          6.2s  Show me a one-way flight from Helsinki to Munich on 26 October 2026 fo
+FAIL  flight seed=900040 steps= 4 done                           1.6s  Show me one way flights from Manchester to Bangkok on November 21 for 
+FAIL  flight seed=900050 steps= 7 blocked                        1.9s  Show me a one-way flight from Amsterdam to Barcelona on November 8, 20
+FAIL  flight seed=900060 steps=21 blocked                        3.2s  Find one-way flights from Seoul to Copenhagen on 10 October 2026 for t
+FAIL  flight seed=900070 steps=40 ready                         10.3s  Look up a return flight from Barcelona to New York on November 13, ret
+FAIL  hotel  seed=900000 steps= 2 done                           1.1s  Find a hotel in Toronto from 31 October 2026 to November 2, sorted by 
+FAIL  hotel  seed=900010 steps= 1 done                           1.1s  Search for hotels in Chicago from 30 October 2026 to November 1 for th
+FAIL  hotel  seed=900020 steps=40 ready                          5.3s  Find a hotel in Oslo from 30 October 2026 to November 3, 2026 for one 
+FAIL  hotel  seed=900030 steps= 4 done                           1.6s  Find stays in Toronto from November 16, 2026 to November 21, 2026, fil
+FAIL  hotel  seed=900040 steps= 2 done                           1.4s  Search for hotels in Tokyo from 14 November 2026 to 15 November 2026, 
+FAIL  hotel  seed=900050 steps=40 ready                          4.5s  Find stays in Copenhagen from December 4 to December 7, 2026 for one g
+FAIL  hotel  seed=900060 steps= 2 done                           1.3s  Find stays in Rome from November 3, 2026 to 10 November 2026 for 4 gue
+FAIL  hotel  seed=900070 steps=40 ready                          4.5s  Find a hotel in Chicago from 27 September 2026 to October 4, only show
+FAIL  shop   seed=900000 steps= 2 done                           1.2s  Go to page 2 and open 'Hooli Water bottle X'.
+PASS  shop   seed=900010 steps= 1 done                           1.1s  Sort the products by price: low to high.
+FAIL  shop   seed=900020 steps= 8 blocked                        1.9s  Show page 3.
+FAIL  shop   seed=900030 steps= 0 done                           1.0s  Find water bottles.
+FAIL  shop   seed=900040 steps= 1 done                           1.2s  Show only Hooli products.
+PASS  shop   seed=900050 steps= 2 done                           1.1s  Filter the list to the brand Acme.
+PASS  shop   seed=900060 steps= 2 done                           1.4s  Search the shop for 'backpack'.
+PASS  shop   seed=900070 steps= 1 done                           1.1s  I only want to see Umbrella items.
+== flight: 0/8  median steps 7
+== hotel: 0/8  median steps 4
+== shop: 4/8  median steps 2
+== total 4/24 (17%)

results/round3/gym_eval_v14s_plan.json

@@ -0,0 +1,391 @@
+[
+ {
+  "kind": "flight",
+  "seed": 900000,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.5,
+  "url": "http://127.0.0.1:8811/t/flight/900000",
+  "goal": "Show me a return flight from Oslo to Berlin on October 18, returning 23 October 2026 for 3 adults in premium economy. Stop when matching flight options are visible.",
+  "plan": [
+   "Set origin (departure) to Oslo",
+   "Set destination (arrival) to Berlin",
+   "Set departure date to October 18, 2026",
+   "Set return date to October 23, 2026",
+   "Set number of passengers to 3",
+   "Set class to premium economy",
+   "Set trip type to return flight",
+   "Stop when matching flight options are visible"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900010,
+  "ok": false,
+  "steps": 2,
+  "status": "blocked",
+  "wall": 2.6,
+  "url": "http://127.0.0.1:8811/t/flight/900010",
+  "goal": "Look up round-trip flights from Tokyo to Amsterdam on October 13, returning 20 October 2026 for two adults in first class.",
+  "plan": [
+   "Set where from (origin, departure) to Tokyo",
+   "Set where to (destination, arrival) to Amsterdam",
+   "Set the departure date to October 13, 2026",
+   "Set the return date to October 20, 2026",
+   "Set the number of passengers to 2",
+   "Set the class to first class",
+   "Set trip type to round-trip",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900020,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.3,
+  "url": "http://127.0.0.1:8811/t/flight/900020",
+  "goal": "Search for one-way flights from Roma to Seoul on 15 October 2026 for two adults. Stop when matching flight options are visible.",
+  "plan": [
+   "Set where from (origin, departure) to Roma",
+   "Set where to (destination, arrival) to Seoul",
+   "Set the departure date to October 15, 2026",
+   "Set the number of passengers to 2",
+   "Set passenger type to adult",
+   "Set trip type to one-way",
+   "Submit the search and stop when matching flight options are visible"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900030,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.8,
+  "url": "http://127.0.0.1:8811/t/flight/900030",
+  "goal": "Show me a one-way flight from Helsinki to Munich on 26 October 2026 for four adults in business class.",
+  "plan": [
+   "Set where from (origin, departure) to Helsinki",
+   "Set where to (destination, arrival) to Munich",
+   "Set the departure date to October 26, 2026",
+   "Set the number of passengers to 4",
+   "Set the class to business",
+   "Set trip type to one-way",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900040,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 7.6,
+  "url": "http://127.0.0.1:8811/t/flight/900040/results?trip=round&adults=1&cabin=first#flight-1",
+  "goal": "Show me one way flights from Manchester to Bangkok on November 21 for 3 adults in business class.",
+  "plan": [
+   "Set where from (origin, departure) to Manchester",
+   "Set where to (destination, arrival) to Bangkok",
+   "Set the departure date to November 21",
+   "Set the number of passengers to 3",
+   "Set the class to business",
+   "Set trip type to one-way",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900050,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.2,
+  "url": "http://127.0.0.1:8811/t/flight/900050",
+  "goal": "Show me a one-way flight from Amsterdam to Barcelona on November 8, 2026 for 1 adult in economy.",
+  "plan": [
+   "Set the origin (from) to Amsterdam",
+   "Set the destination (to) to Barcelona",
+   "Set the departure date to November 8, 2026",
+   "Set the number of passengers to 1",
+   "Set the class to economy",
+   "Set the trip type to one-way",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900060,
+  "ok": false,
+  "steps": 2,
+  "status": "blocked",
+  "wall": 2.4,
+  "url": "http://127.0.0.1:8811/t/flight/900060",
+  "goal": "Find one-way flights from Seoul to Copenhagen on 10 October 2026 for two adults in business class.",
+  "plan": [
+   "Set origin (from) to Seoul",
+   "Set destination (to) to Copenhagen",
+   "Set departure date to October 10, 2026",
+   "Set number of passengers to 2",
+   "Set class to Business",
+   "Set trip type to One-way",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "flight",
+  "seed": 900070,
+  "ok": false,
+  "steps": 5,
+  "status": "blocked",
+  "wall": 2.7,
+  "url": "http://127.0.0.1:8811/t/flight/900070",
+  "goal": "Look up a return flight from Barcelona to New York on November 13, returning November 27 for 1 adult. Stop when matching flight options are visible.",
+  "plan": [
+   "Set where from (origin, departure) to Barcelona",
+   "Set where to (destination, arrival) to New York",
+   "Set the departure date to November 13, 2026",
+   "Set the return date to November 27, 2026",
+   "Set the number of passengers to 1",
+   "Set the trip type to return",
+   "Submit the search and stop when matching flight options are visible"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900000,
+  "ok": false,
+  "steps": 1,
+  "status": "blocked",
+  "wall": 1.6,
+  "url": "http://127.0.0.1:8811/t/hotel/900000",
+  "goal": "Find a hotel in Toronto from 31 October 2026 to November 2, sorted by price: low to high.",
+  "plan": [
+   "Set the arrival date to October 31, 2026",
+   "Set the departure date to November 2, 2026",
+   "Sort hotels by price: low to high"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900010,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.5,
+  "url": "http://127.0.0.1:8811/t/hotel/900010",
+  "goal": "Search for hotels in Chicago from 30 October 2026 to November 1 for three guests, sorted by price: low to high.",
+  "plan": [
+   "Set the origin (departure) to Chicago",
+   "Set the departure date to October 30, 2026",
+   "Set the return date to November 1, 2026",
+   "Set the number of guests to 3",
+   "Sort the results by price: low to high"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900020,
+  "ok": false,
+  "steps": 4,
+  "status": "blocked",
+  "wall": 3.4,
+  "url": "http://127.0.0.1:8811/t/hotel/900020",
+  "goal": "Find a hotel in Oslo from 30 October 2026 to November 3, 2026 for one guest, only showing pet friendly.",
+  "plan": [
+   "Set the arrival date to October 30, 2026",
+   "Set the departure date to November 3, 2026",
+   "Set the number of guests to 1",
+   "Set the pet friendly filter to enabled",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900030,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.8,
+  "url": "http://127.0.0.1:8811/t/hotel/900030",
+  "goal": "Find stays in Toronto from November 16, 2026 to November 21, 2026, filtered to pet friendly.",
+  "plan": [
+   "Set the arrival date to November 16, 2026",
+   "Set the departure date to November 21, 2026",
+   "Set the destination (city) to Toronto",
+   "Set the filter to pet friendly"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900040,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.6,
+  "url": "http://127.0.0.1:8811/t/hotel/900040",
+  "goal": "Search for hotels in Tokyo from 14 November 2026 to 15 November 2026, with pet friendly.",
+  "plan": [
+   "Set the departure date to November 14, 2026",
+   "Set the return date to November 15, 2026",
+   "Set the destination (city) to Tokyo",
+   "Set the pet friendly filter to enabled",
+   "Submit the search and view the results"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900050,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.4,
+  "url": "http://127.0.0.1:8811/t/hotel/900050",
+  "goal": "Find stays in Copenhagen from December 4 to December 7, 2026 for one guest, sorted by guest rating.",
+  "plan": [
+   "Set the origin (from) to Copenhagen",
+   "Set the departure date to December 4, 2026",
+   "Set the return date to December 7, 2026",
+   "Set the number of guests to 1",
+   "Sort by guest rating"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900060,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 6.0,
+  "url": "http://127.0.0.1:8811/t/hotel/900060",
+  "goal": "Find stays in Rome from November 3, 2026 to 10 November 2026 for 4 guests.",
+  "plan": [
+   "Set the arrival date to November 3, 2026",
+   "Set the departure date to November 10, 2026",
+   "Set the number of guests to 4",
+   "Search for stays in Rome"
+  ]
+ },
+ {
+  "kind": "hotel",
+  "seed": 900070,
+  "ok": false,
+  "steps": 40,
+  "status": "ready",
+  "wall": 5.5,
+  "url": "http://127.0.0.1:8811/t/hotel/900070",
+  "goal": "Find a hotel in Chicago from 27 September 2026 to October 4, only showing free cancellation.",
+  "plan": [
+   "Set the arrival date to September 27, 2026",
+   "Set the departure date to October 4, 2026",
+   "Set the filter to only show free cancellation hotels"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900000,
+  "ok": false,
+  "steps": 5,
+  "status": "blocked",
+  "wall": 2.5,
+  "url": "http://127.0.0.1:8811/t/shop/900000/p/4#home",
+  "goal": "Go to page 2 and open 'Hooli Water bottle X'.",
+  "plan": [
+   "Go to page 2",
+   "Open 'Hooli Water bottle X'"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900010,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.3,
+  "url": "http://127.0.0.1:8811/t/shop/900010?sort=Price%3A+low+to+high",
+  "goal": "Sort the products by price: low to high.",
+  "plan": [
+   "Sort products by price from low to high"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900020,
+  "ok": false,
+  "steps": 8,
+  "status": "blocked",
+  "wall": 1.9,
+  "url": "http://127.0.0.1:8811/t/shop/900020/p/8#home",
+  "goal": "Show page 3.",
+  "plan": [
+   "Go to page 3"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900030,
+  "ok": false,
+  "steps": 0,
+  "status": "done",
+  "wall": 1.3,
+  "url": "http://127.0.0.1:8811/t/shop/900030",
+  "goal": "Find water bottles.",
+  "plan": [
+   "Find water bottles"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900040,
+  "ok": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.4,
+  "url": "http://127.0.0.1:8811/t/shop/900040/p/2",
+  "goal": "Show only Hooli products.",
+  "plan": [
+   "Filter products to show only Hooli products"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900050,
+  "ok": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.5,
+  "url": "http://127.0.0.1:8811/t/shop/900050?brand=Acme",
+  "goal": "Filter the list to the brand Acme.",
+  "plan": [
+   "Filter the list to show only items from the brand Acme"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900060,
+  "ok": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.9,
+  "url": "http://127.0.0.1:8811/t/shop/900060?q=backpack",
+  "goal": "Search the shop for 'backpack'.",
+  "plan": [
+   "Search the shop for 'backpack'"
+  ]
+ },
+ {
+  "kind": "shop",
+  "seed": 900070,
+  "ok": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.5,
+  "url": "http://127.0.0.1:8811/t/shop/900070?brand=Umbrella&sort=Price%3A+low+to+high",
+  "goal": "I only want to see Umbrella items.",
+  "plan": [
+   "Set the item type to Umbrella"
+  ]
+ }
+]

results/round3/gym_eval_v14s_plan.log

@@ -0,0 +1,28 @@
+FAIL  flight seed=900000 steps=40 ready                          6.5s  Show me a return flight from Oslo to Berlin on October 18, returning 2
+FAIL  flight seed=900010 steps= 2 blocked                        2.6s  Look up round-trip flights from Tokyo to Amsterdam on October 13, retu
+FAIL  flight seed=900020 steps=40 ready                          6.3s  Search for one-way flights from Roma to Seoul on 15 October 2026 for t
+FAIL  flight seed=900030 steps=40 ready                          6.8s  Show me a one-way flight from Helsinki to Munich on 26 October 2026 fo
+FAIL  flight seed=900040 steps=40 ready                          7.6s  Show me one way flights from Manchester to Bangkok on November 21 for 
+FAIL  flight seed=900050 steps=40 ready                          6.2s  Show me a one-way flight from Amsterdam to Barcelona on November 8, 20
+FAIL  flight seed=900060 steps= 2 blocked                        2.4s  Find one-way flights from Seoul to Copenhagen on 10 October 2026 for t
+FAIL  flight seed=900070 steps= 5 blocked                        2.7s  Look up a return flight from Barcelona to New York on November 13, ret
+FAIL  hotel  seed=900000 steps= 1 blocked                        1.6s  Find a hotel in Toronto from 31 October 2026 to November 2, sorted by 
+FAIL  hotel  seed=900010 steps=40 ready                          5.5s  Search for hotels in Chicago from 30 October 2026 to November 1 for th
+FAIL  hotel  seed=900020 steps= 4 blocked                        3.4s  Find a hotel in Oslo from 30 October 2026 to November 3, 2026 for one 
+FAIL  hotel  seed=900030 steps=40 ready                          5.8s  Find stays in Toronto from November 16, 2026 to November 21, 2026, fil
+FAIL  hotel  seed=900040 steps=40 ready                          6.6s  Search for hotels in Tokyo from 14 November 2026 to 15 November 2026, 
+FAIL  hotel  seed=900050 steps=40 ready                          6.4s  Find stays in Copenhagen from December 4 to December 7, 2026 for one g
+FAIL  hotel  seed=900060 steps=40 ready                          6.0s  Find stays in Rome from November 3, 2026 to 10 November 2026 for 4 gue
+FAIL  hotel  seed=900070 steps=40 ready                          5.5s  Find a hotel in Chicago from 27 September 2026 to October 4, only show
+FAIL  shop   seed=900000 steps= 5 blocked                        2.5s  Go to page 2 and open 'Hooli Water bottle X'.
+PASS  shop   seed=900010 steps= 1 done                           1.3s  Sort the products by price: low to high.
+FAIL  shop   seed=900020 steps= 8 blocked                        1.9s  Show page 3.
+FAIL  shop   seed=900030 steps= 0 done                           1.3s  Find water bottles.
+FAIL  shop   seed=900040 steps= 1 done                           1.4s  Show only Hooli products.
+PASS  shop   seed=900050 steps= 1 done                           1.5s  Filter the list to the brand Acme.
+PASS  shop   seed=900060 steps= 2 done                           1.9s  Search the shop for 'backpack'.
+PASS  shop   seed=900070 steps= 2 done                           1.5s  I only want to see Umbrella items.
+== flight: 0/8  median steps 40
+== hotel: 0/8  median steps 40
+== shop: 4/8  median steps 2
+== total 4/24 (17%)

results/round3/suiteA_v13s.json

@@ -0,0 +1,338 @@
+[
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 5.852099657058716
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 3.777109146118164
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 3,
+  "status": "blocked",
+  "wall": 3.5345683097839355
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.188246726989746
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.308509111404419
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.4918646812438965
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.145602226257324
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.6322431564331055
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.6054000854492188
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.7457010746002197
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.4926605224609375
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5525999069213867
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 7.7928466796875
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.424391984939575
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.2397661209106445
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.143227577209473
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.3897807598114014
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.97131085395813
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0191097259521484
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9415113925933838
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9320547580718994
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.2813827991485596
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.896589994430542
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8832671642303467
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.4084954261779785
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.18304181098938
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.2253031730651855
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.534019947052002
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.08272123336792
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.201373815536499
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.7590856552124023
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9029622077941895
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9843285083770752
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.8725624084472656
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.835207939147949
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.7795658111572266
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.0871634483337402
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.41726279258728
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.973738670349121
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.196753025054932
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.8438737392425537
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.0845510959625244
+ },
+ {
+  "name": "arxiv-search",
+  "pass": false,
+  "steps": 2,
+  "status": "error:ValueError",
+  "wall": 14.285512685775757
+ },
+ {
+  "name": "arxiv-search",
+  "pass": false,
+  "steps": 2,
+  "status": "error:ValueError",
+  "wall": 12.945858716964722
+ },
+ {
+  "name": "arxiv-search",
+  "pass": false,
+  "steps": 2,
+  "status": "error:ValueError",
+  "wall": 13.426517963409424
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 19,
+  "status": "blocked",
+  "wall": 7.32370924949646
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 5.2465996742248535
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 5.680525064468384
+ }
+]

results/round3/suiteA_v14s.json

@@ -0,0 +1,338 @@
+[
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 7.153369426727295
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.913437843322754
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.909303903579712
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.399548768997192
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.0288166999816895
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.731976270675659
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.587008476257324
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5481419563293457
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.49631929397583
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.745541572570801
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5028481483459473
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.3588883876800537
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 6.790970325469971
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.614898681640625
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.7601253986358643
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.211635589599609
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.601041793823242
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5883960723876953
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.084430694580078
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.985591173171997
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.959421157836914
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.1794636249542236
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.842677116394043
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9249365329742432
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.38923716545105
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0766379833221436
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.203416347503662
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5841612815856934
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.088857412338257
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.1360607147216797
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.98116135597229
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9536120891571045
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9566633701324463
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.7906875610351562
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.9570891857147217
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.9571480751037598
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.0445728302001953
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.6037495136260986
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5883827209472656
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 6.664576530456543
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.646151781082153
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.23597264289856
+ },
+ {
+  "name": "arxiv-search",
+  "pass": false,
+  "steps": 4,
+  "status": "done",
+  "wall": 6.1844401359558105
+ },
+ {
+  "name": "arxiv-search",
+  "pass": true,
+  "steps": 8,
+  "status": "done",
+  "wall": 3.808042526245117
+ },
+ {
+  "name": "arxiv-search",
+  "pass": true,
+  "steps": 14,
+  "status": "done",
+  "wall": 10.105110168457031
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 7.139219760894775
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 5.5144500732421875
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 5.720807313919067
+ }
+]

results/round3/suiteB_v12s_c40.json

@@ -0,0 +1,254 @@
+[
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.6703267097473145
+ },
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8731908798217773
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.509976387023926
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0649473667144775
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.1759567260742188
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.014357089996338
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.8925137519836426
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8947360515594482
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.0906717777252197
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.008801221847534
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.957569599151611
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.6660614013671875
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.626530647277832
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8818230628967285
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 6.250592947006226
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.139995574951172
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.652239084243774
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.2106099128723145
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.763736963272095
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.696882724761963
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.270788192749023
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.530022621154785
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.517338991165161
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.0429556369781494
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.772353887557983
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.16965389251709
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.179338216781616
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.827910900115967
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.698476791381836
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.673051357269287
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.619678258895874
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.966660261154175
+ },
+ {
+  "name": "shop-page2",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.731016159057617
+ },
+ {
+  "name": "shop-page2",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.623927116394043
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": false,
+  "steps": 1,
+  "status": "error:ValueError",
+  "wall": 2.9405176639556885
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 4.098207473754883
+ }
+]

results/round3/suiteB_v13s.json

@@ -0,0 +1,380 @@
+[
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.695581436157227
+ },
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.869412899017334
+ },
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8962252140045166
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.8517751693725586
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.8190269470214844
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.199798345565796
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.331904411315918
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9637272357940674
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.1597702503204346
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0122523307800293
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.7669165134429932
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.949770212173462
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.1853749752044678
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.7542953491210938
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.918290138244629
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 8.402779340744019
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.298560857772827
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.158369779586792
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.858856201171875
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.169969320297241
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9936721324920654
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.214618682861328
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.1169214248657227
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.1795413494110107
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.403536081314087
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.2332334518432617
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.269718885421753
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.526256084442139
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.611555576324463
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.591611862182617
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.365265130996704
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.4871649742126465
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.5202934741973877
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 6.339969158172607
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 2.552432060241699
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 2.5456464290618896
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 15.8356351852417
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.7176716327667236
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.710045099258423
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.8107569217681885
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.727369546890259
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.1598076820373535
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.899163007736206
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.642282485961914
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.9193930625915527
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.8924782276153564
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.8857860565185547
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.837737798690796
+ },
+ {
+  "name": "shop-page2",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.3639087677001953
+ },
+ {
+  "name": "shop-page2",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.182849407196045
+ },
+ {
+  "name": "shop-page2",
+  "pass": false,
+  "steps": 3,
+  "status": "blocked",
+  "wall": 2.9457852840423584
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 6.215277910232544
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.446885585784912
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.414456844329834
+ }
+]

results/round3/suiteB_v14s.json

@@ -0,0 +1,380 @@
+[
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 6.361268043518066
+ },
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9992897510528564
+ },
+ {
+  "name": "sqlite-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.970348596572876
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.513546943664551
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0639684200286865
+ },
+ {
+  "name": "haskell-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.178180694580078
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.86093807220459
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9833033084869385
+ },
+ {
+  "name": "zig-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.164769172668457
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9758775234222412
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.984276533126831
+ },
+ {
+  "name": "openbsd-faq",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9708170890808105
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.156858444213867
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.7520651817321777
+ },
+ {
+  "name": "nginx-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9046032428741455
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 7.542937278747559
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.576859474182129
+ },
+ {
+  "name": "xkcd-archive",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.049166440963745
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.433667898178101
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.2110342979431152
+ },
+ {
+  "name": "elixir-install",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0132791996002197
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 9.333859920501709
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.161487340927124
+ },
+ {
+  "name": "prometheus-download",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.020197629928589
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.945269823074341
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.126190185546875
+ },
+ {
+  "name": "ocaml-install",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.143099784851074
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.9808945655822754
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.70145320892334
+ },
+ {
+  "name": "rosetta-quicksort",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.6665432453155518
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.003252267837524
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.5729293823242188
+ },
+ {
+  "name": "arch-man-grep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.561814308166504
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.128390312194824
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.061967134475708
+ },
+ {
+  "name": "debian-man-tar",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 1.9078538417816162
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.725454807281494
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.664327383041382
+ },
+ {
+  "name": "rfc-http-semantics",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.0745277404785156
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.9704113006591797
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.753455638885498
+ },
+ {
+  "name": "nixos-ripgrep",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.771595001220703
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.7241852283477783
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.646784782409668
+ },
+ {
+  "name": "ocaml-pkg-yojson",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.8170480728149414
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 5.183164119720459
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.852771520614624
+ },
+ {
+  "name": "shop-search-jacket",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.919358253479004
+ },
+ {
+  "name": "shop-page2",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.1722159385681152
+ },
+ {
+  "name": "shop-page2",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.6293818950653076
+ },
+ {
+  "name": "shop-page2",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.4879581928253174
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.8961870670318604
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.308035135269165
+ },
+ {
+  "name": "shop-sort-price",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.4196367263793945
+ }
+]

results/round3/suite_h2_laya-browser-v10s.json

@@ -0,0 +1,338 @@
+[
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.83627986907959
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.2153303623199463
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 2,
+  "status": "done",
+  "wall": 2.7771353721618652
+ },
+ {
+  "name": "wiki-search",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 7.70431661605835
+ },
+ {
+  "name": "wiki-search",
+  "pass": false,
+  "steps": 11,
+  "status": "blocked",
+  "wall": 6.307146072387695
+ },
+ {
+  "name": "wiki-search",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 7.217585802078247
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.1100358963012695
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.744152069091797
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5624217987060547
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.7742652893066406
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.3693900108337402
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5206425189971924
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.734229564666748
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.3147778511047363
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.5175280570983887
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.399106502532959
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.375485897064209
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5823707580566406
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9549450874328613
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0089175701141357
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9885730743408203
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.223175287246704
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.734677791595459
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9055156707763672
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.755065679550171
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0737600326538086
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.218611240386963
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.42691707611084
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.051815986633301
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.1564059257507324
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9430482387542725
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9770045280456543
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9294829368591309
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.630563974380493
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.9477081298828125
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.8947083950042725
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.8496310710906982
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.1609859466552734
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.4889137744903564
+ },
+ {
+  "name": "web-search",
+  "pass": false,
+  "steps": 8,
+  "status": "error:ValueError",
+  "wall": 15.667231559753418
+ },
+ {
+  "name": "web-search",
+  "pass": false,
+  "steps": 9,
+  "status": "error:ValueError",
+  "wall": 13.118170261383057
+ },
+ {
+  "name": "web-search",
+  "pass": false,
+  "steps": 6,
+  "status": "error:ValueError",
+  "wall": 14.582857608795166
+ },
+ {
+  "name": "arxiv-search",
+  "pass": false,
+  "steps": 2,
+  "status": "error:ValueError",
+  "wall": 9.322384357452393
+ },
+ {
+  "name": "arxiv-search",
+  "pass": false,
+  "steps": 2,
+  "status": "error:ValueError",
+  "wall": 10.208577632904053
+ },
+ {
+  "name": "arxiv-search",
+  "pass": false,
+  "steps": 2,
+  "status": "error:ValueError",
+  "wall": 10.149547815322876
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 14,
+  "status": "error:ValueError",
+  "wall": 9.853911638259888
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 14,
+  "status": "error:ValueError",
+  "wall": 9.251529693603516
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 6.629589080810547
+ }
+]

results/round3/suite_h2_laya-browser-v12s.json

@@ -0,0 +1,338 @@
+[
+ {
+  "name": "wiki-einstein",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 6.088091135025024
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 4.1120641231536865
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 4.785888195037842
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.805815935134888
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.1727588176727295
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.9865856170654297
+ },
+ {
+  "name": "hn-new",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.6679415702819824
+ },
+ {
+  "name": "hn-new",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.4920201301574707
+ },
+ {
+  "name": "hn-new",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.4279229640960693
+ },
+ {
+  "name": "hn-login-page",
+  "pass": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 4.908161878585815
+ },
+ {
+  "name": "hn-login-page",
+  "pass": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 3.3291561603546143
+ },
+ {
+  "name": "hn-login-page",
+  "pass": false,
+  "steps": 3,
+  "status": "done",
+  "wall": 3.2981934547424316
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 7.582534551620483
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.5515997409820557
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.4299209117889404
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.59052300453186
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.7227585315704346
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.298168897628784
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.351111650466919
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9312889575958252
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.046699285507202
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.01000714302063
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8002934455871582
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8848111629486084
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 4.636353254318237
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.068424701690674
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.11932373046875
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.576756000518799
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0067596435546875
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.1043567657470703
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9515693187713623
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0128142833709717
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.850271224975586
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.917008399963379
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.8107078075408936
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.968144655227661
+ },
+ {
+  "name": "hn-past",
+  "pass": false,
+  "steps": 7,
+  "status": "blocked",
+  "wall": 4.717610836029053
+ },
+ {
+  "name": "hn-past",
+  "pass": false,
+  "steps": 7,
+  "status": "blocked",
+  "wall": 3.822110176086426
+ },
+ {
+  "name": "hn-past",
+  "pass": false,
+  "steps": 7,
+  "status": "blocked",
+  "wall": 3.8178353309631348
+ },
+ {
+  "name": "web-search",
+  "pass": false,
+  "steps": 14,
+  "status": "done",
+  "wall": 16.275484323501587
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.697402477264404
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.85608983039856
+ },
+ {
+  "name": "arxiv-search",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 5.771880865097046
+ },
+ {
+  "name": "arxiv-search",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 4.881175994873047
+ },
+ {
+  "name": "arxiv-search",
+  "pass": true,
+  "steps": 3,
+  "status": "done",
+  "wall": 3.4859631061553955
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 9.096100568771362
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 14,
+  "status": "blocked",
+  "wall": 5.75098180770874
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 15,
+  "status": "blocked",
+  "wall": 6.6374828815460205
+ }
+]

results/round3/suite_h2c40_laya-browser-v12s.json

@@ -0,0 +1,338 @@
+[
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.807706832885742
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.3761672973632812
+ },
+ {
+  "name": "wiki-einstein",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.567605495452881
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.4094016551971436
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.763357162475586
+ },
+ {
+  "name": "wiki-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.382323265075684
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.7456138134002686
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.794174909591675
+ },
+ {
+  "name": "hn-new",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.594071626663208
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.8987975120544434
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.7102601528167725
+ },
+ {
+  "name": "hn-login-page",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.4696624279022217
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 5.98798680305481
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.4403786659240723
+ },
+ {
+  "name": "gh-issues",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.37373423576355
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.724518299102783
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.5852811336517334
+ },
+ {
+  "name": "py-downloads",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.565865993499756
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0555620193481445
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0139927864074707
+ },
+ {
+  "name": "books-travel",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.9735207557678223
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.1686925888061523
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.7811036109924316
+ },
+ {
+  "name": "books-open-book",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8474400043487549
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.797389507293701
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.0917704105377197
+ },
+ {
+  "name": "internet-dropdown",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.2623064517974854
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.3313634395599365
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.104766368865967
+ },
+ {
+  "name": "internet-checkbox",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.1607117652893066
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.7619709968566895
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.8645665645599365
+ },
+ {
+  "name": "books-page2",
+  "pass": false,
+  "steps": 1,
+  "status": "done",
+  "wall": 1.977370023727417
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 3.4522359371185303
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.9943950176239014
+ },
+ {
+  "name": "quotes-tag-love",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.8459112644195557
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.764420747756958
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.925821304321289
+ },
+ {
+  "name": "hn-past",
+  "pass": true,
+  "steps": 1,
+  "status": "done",
+  "wall": 2.6401472091674805
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 7.507221460342407
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 3.8072285652160645
+ },
+ {
+  "name": "web-search",
+  "pass": true,
+  "steps": 2,
+  "status": "done",
+  "wall": 4.1349287033081055
+ },
+ {
+  "name": "arxiv-search",
+  "pass": false,
+  "steps": 1,
+  "status": "error:ValueError",
+  "wall": 1.9037411212921143
+ },
+ {
+  "name": "arxiv-search",
+  "pass": false,
+  "steps": 1,
+  "status": "error:ValueError",
+  "wall": 0.8892548084259033
+ },
+ {
+  "name": "arxiv-search",
+  "pass": false,
+  "steps": 1,
+  "status": "error:ValueError",
+  "wall": 0.9213480949401855
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 8.24284815788269
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 20,
+  "status": "ready",
+  "wall": 6.250298261642456
+ },
+ {
+  "name": "flights",
+  "pass": false,
+  "steps": 20,
+  "status": "blocked",
+  "wall": 6.608509063720703
+ }
+]

results/round3/suite_v12s.log

@@ -0,0 +1,103 @@
+== suite laya-browser-v12s x3 (patched harness)
+PASS  wiki-einstein    steps= 3 status=done        6.1s  https://en.wikipedia.org/wiki/Albert_Einstein
+PASS  wiki-einstein    steps= 3 status=done        4.1s  https://en.wikipedia.org/wiki/Albert_Einstein
+PASS  wiki-einstein    steps= 3 status=done        4.8s  https://en.wikipedia.org/wiki/Albert_Einstein
+PASS  wiki-search      steps= 2 status=done        4.8s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  wiki-search      steps= 2 status=done        4.2s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  wiki-search      steps= 2 status=done        4.0s  https://en.wikipedia.org/wiki/Python_(programming_language)
+FAIL  hn-new           steps= 1 status=done        3.7s  https://news.ycombinator.com/front
+FAIL  hn-new           steps= 1 status=done        2.5s  https://news.ycombinator.com/front
+FAIL  hn-new           steps= 1 status=done        2.4s  https://news.ycombinator.com/front
+FAIL  hn-login-page    steps= 3 status=done        4.9s  https://news.ycombinator.com/submit
+FAIL  hn-login-page    steps= 3 status=done        3.3s  https://news.ycombinator.com/submit
+FAIL  hn-login-page    steps= 3 status=done        3.3s  https://news.ycombinator.com/submit
+PASS  gh-issues        steps= 1 status=done        7.6s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.6s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.4s  https://github.com/tile-ai/tilelang/issues
+PASS  py-downloads     steps= 1 status=done        4.6s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        2.7s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        2.3s  https://www.python.org/downloads/
+PASS  books-travel     steps= 1 status=done        3.4s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-open-book  steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.8s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  internet-dropdown steps= 1 status=done        4.6s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-dropdown steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-dropdown steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-checkbox steps= 1 status=done        2.6s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.0s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/checkboxes
+FAIL  books-page2      steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  quotes-tag-love  steps= 1 status=done        3.9s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        2.8s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        3.0s  https://quotes.toscrape.com/tag/love/
+FAIL  hn-past          steps= 7 status=blocked     4.7s  https://news.ycombinator.com/login?goto=news
+FAIL  hn-past          steps= 7 status=blocked     3.8s  https://news.ycombinator.com/login?goto=news
+FAIL  hn-past          steps= 7 status=blocked     3.8s  https://news.ycombinator.com/login?goto=news
+FAIL  web-search       steps=14 status=done       16.3s  https://marginalia-search.com/
+PASS  web-search       steps= 2 status=done        4.7s  https://marginalia-search.com/search?query=tilelang+github&profile=cor
+PASS  web-search       steps= 2 status=done        4.9s  https://marginalia-search.com/search?query=tilelang+github&profile=cor
+PASS  arxiv-search     steps= 3 status=done        5.8s  https://arxiv.org/search/?query=flash+attention&searchtype=all&source=
+PASS  arxiv-search     steps= 3 status=done        4.9s  https://arxiv.org/search/?query=flash+attention&searchtype=all&source=
+PASS  arxiv-search     steps= 3 status=done        3.5s  https://arxiv.org/search/?query=flash+attention&searchtype=all&source=
+FAIL  flights          steps=20 status=ready       9.1s  https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI2LTEwL
+FAIL  flights          steps=14 status=blocked     5.8s  https://www.google.com/travel/flights?hl=en
+FAIL  flights          steps=15 status=blocked     6.6s  https://www.google.com/travel/flights?tfs=CBwQARoOcgwIAxIIL20vMDRqcGwa
+== 32/48 passed (67%, 3 run(s) per task) | median wall 3.6s
+   per task: wiki-einstein=3/3  wiki-search=3/3  hn-new=0/3  hn-login-page=0/3  gh-issues=3/3  py-downloads=3/3  books-travel=3/3  books-open-book=3/3  internet-dropdown=3/3  internet-checkbox=3/3  books-page2=0/3  quotes-tag-love=3/3  hn-past=0/3  web-search=2/3  arxiv-search=3/3  flights=0/3
+== suite laya-browser-v10s x3 (patched harness)
+FAIL  wiki-einstein    steps= 2 status=done        3.8s  https://en.wikipedia.org/wiki/Main_Page
+FAIL  wiki-einstein    steps= 2 status=done        3.2s  https://en.wikipedia.org/wiki/Main_Page
+FAIL  wiki-einstein    steps= 2 status=done        2.8s  https://en.wikipedia.org/wiki/Main_Page
+FAIL  wiki-search      steps=20 status=ready       7.7s  https://en.wikipedia.org/w/index.php?search=&title=Special%3ASearch&pr
+FAIL  wiki-search      steps=11 status=blocked     6.3s  https://en.wikipedia.org/w/index.php?search=&title=Special%3ASearch
+FAIL  wiki-search      steps=20 status=ready       7.2s  https://en.wikipedia.org/w/index.php?search=&title=Special%3ASearch
+PASS  hn-new           steps= 1 status=done        3.1s  https://news.ycombinator.com/newest
+PASS  hn-new           steps= 1 status=done        2.7s  https://news.ycombinator.com/newest
+PASS  hn-new           steps= 1 status=done        2.6s  https://news.ycombinator.com/newest
+PASS  hn-login-page    steps= 1 status=done        2.8s  https://news.ycombinator.com/login?goto=news
+PASS  hn-login-page    steps= 1 status=done        2.4s  https://news.ycombinator.com/login?goto=news
+PASS  hn-login-page    steps= 1 status=done        2.5s  https://news.ycombinator.com/login?goto=news
+PASS  gh-issues        steps= 1 status=done        4.7s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.3s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.5s  https://github.com/tile-ai/tilelang/issues
+PASS  py-downloads     steps= 1 status=done        2.4s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        2.4s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        2.6s  https://www.python.org/downloads/
+PASS  books-travel     steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-open-book  steps= 1 status=done        2.2s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.7s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  internet-dropdown steps= 1 status=done        2.8s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-dropdown steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-dropdown steps= 1 status=done        2.2s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-checkbox steps= 1 status=done        2.4s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.2s  https://the-internet.herokuapp.com/checkboxes
+FAIL  books-page2      steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  quotes-tag-love  steps= 1 status=done        3.6s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        2.9s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        2.9s  https://quotes.toscrape.com/tag/love/
+PASS  hn-past          steps= 1 status=done        3.8s  https://news.ycombinator.com/front
+PASS  hn-past          steps= 1 status=done        3.2s  https://news.ycombinator.com/front
+PASS  hn-past          steps= 1 status=done        2.5s  https://news.ycombinator.com/front
+FAIL  web-search       steps= 8 status=error:ValueError  15.7s  https://github.com/MarginaliaSearch/MarginaliaSearch/issues?q=is%3Aiss
+FAIL  web-search       steps= 9 status=error:ValueError  13.1s  https://github.com/MarginaliaSearch/MarginaliaSearch/issues?q=is%3Aiss
+FAIL  web-search       steps= 6 status=error:ValueError  14.6s  https://github.com/MarginaliaSearch/MarginaliaSearch/issues?q=is%3Aiss
+FAIL  arxiv-search     steps= 2 status=error:ValueError   9.3s  https://arxiv.org/
+FAIL  arxiv-search     steps= 2 status=error:ValueError  10.2s  https://arxiv.org/
+FAIL  arxiv-search     steps= 2 status=error:ValueError  10.1s  https://arxiv.org/
+FAIL  flights          steps=14 status=error:ValueError   9.9s  https://www.google.com/travel/flights?tfs=CBwQARocagwIAxIIL20vMDg5NjZy
+FAIL  flights          steps=14 status=error:ValueError   9.3s  https://www.google.com/travel/flights?tfs=CBwQARocagwIAxIIL20vMDg5NjZy
+FAIL  flights          steps=20 status=ready       6.6s  https://www.google.com/travel/flights?tfs=CBwQARoOcgwIAxIIL20vMDRqcGwa
+== 30/48 passed (62%, 3 run(s) per task) | median wall 2.8s
+   per task: wiki-einstein=0/3  wiki-search=0/3  hn-new=3/3  hn-login-page=3/3  gh-issues=3/3  py-downloads=3/3  books-travel=3/3  books-open-book=3/3  internet-dropdown=3/3  internet-checkbox=3/3  books-page2=0/3  quotes-tag-love=3/3  hn-past=3/3  web-search=0/3  arxiv-search=0/3  flights=0/3
+SUITE_V12S_DONE

results/round3/suite_v12s_c40.log

@@ -0,0 +1,50 @@
+FAIL  wiki-einstein    steps= 1 status=done        3.8s  https://en.wikipedia.org/wiki/Main_Page#/media/File:SMRT_C151_set_065-
+FAIL  wiki-einstein    steps= 1 status=done        2.4s  https://en.wikipedia.org/wiki/Main_Page#/media/File:SMRT_C151_set_065-
+FAIL  wiki-einstein    steps= 1 status=done        2.6s  https://en.wikipedia.org/wiki/Main_Page#/media/File:SMRT_C151_set_065-
+PASS  wiki-search      steps= 2 status=done        4.4s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  wiki-search      steps= 2 status=done        4.8s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  wiki-search      steps= 2 status=done        4.4s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  hn-new           steps= 1 status=done        2.7s  https://news.ycombinator.com/newest
+PASS  hn-new           steps= 1 status=done        2.8s  https://news.ycombinator.com/newest
+PASS  hn-new           steps= 1 status=done        2.6s  https://news.ycombinator.com/newest
+PASS  hn-login-page    steps= 1 status=done        2.9s  https://news.ycombinator.com/login?goto=news
+PASS  hn-login-page    steps= 1 status=done        2.7s  https://news.ycombinator.com/login?goto=news
+PASS  hn-login-page    steps= 1 status=done        2.5s  https://news.ycombinator.com/login?goto=news
+PASS  gh-issues        steps= 1 status=done        6.0s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.4s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.4s  https://github.com/tile-ai/tilelang/issues
+PASS  py-downloads     steps= 1 status=done        2.7s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        2.6s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        2.6s  https://www.python.org/downloads/
+PASS  books-travel     steps= 1 status=done        2.1s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-open-book  steps= 1 status=done        2.2s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.8s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.8s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  internet-dropdown steps= 1 status=done        2.8s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-dropdown steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-dropdown steps= 1 status=done        2.3s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-checkbox steps= 1 status=done        2.3s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.2s  https://the-internet.herokuapp.com/checkboxes
+FAIL  books-page2      steps= 1 status=done        1.8s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  quotes-tag-love  steps= 1 status=done        3.5s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        3.0s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        2.8s  https://quotes.toscrape.com/tag/love/
+PASS  hn-past          steps= 1 status=done        2.8s  https://news.ycombinator.com/front
+PASS  hn-past          steps= 1 status=done        2.9s  https://news.ycombinator.com/front
+PASS  hn-past          steps= 1 status=done        2.6s  https://news.ycombinator.com/front
+PASS  web-search       steps= 2 status=done        7.5s  https://marginalia-search.com/search?query=tilelang+github&profile=cor
+PASS  web-search       steps= 2 status=done        3.8s  https://marginalia-search.com/search?query=tilelang+github&sst=SE-655d
+PASS  web-search       steps= 2 status=done        4.1s  https://marginalia-search.com/search?query=tilelang+github&profile=cor
+FAIL  arxiv-search     steps= 1 status=error:ValueError   1.9s  https://arxiv.org/login
+FAIL  arxiv-search     steps= 1 status=error:ValueError   0.9s  https://arxiv.org/login
+FAIL  arxiv-search     steps= 1 status=error:ValueError   0.9s  https://arxiv.org/login
+FAIL  flights          steps=20 status=ready       8.2s  https://www.google.com/travel/flights?tfs=CBwQARoOcgwIAxIIL20vMDRqcGwa
+FAIL  flights          steps=20 status=ready       6.3s  https://www.google.com/travel/flights?tfs=CBwQARoOcgwIBBIIL20vMDJqNzEa
+FAIL  flights          steps=20 status=blocked     6.6s  https://www.google.com/travel/flights?tfs=CBwQARoOcgwIAxIIL20vMDRqcGwa
+== 36/48 passed (75%, 3 run(s) per task) | median wall 2.7s
+   per task: wiki-einstein=0/3  wiki-search=3/3  hn-new=3/3  hn-login-page=3/3  gh-issues=3/3  py-downloads=3/3  books-travel=3/3  books-open-book=3/3  internet-dropdown=3/3  internet-checkbox=3/3  books-page2=0/3  quotes-tag-love=3/3  hn-past=3/3  web-search=3/3  arxiv-search=0/3  flights=0/3

results/round3/train_domains.json

@@ -0,0 +1 @@
+["14945.blackbaudhosting.com", "aa", "aa.com", "abc.net.au", "about.gitlab.com", "about.google", "accessmedicalrehab.com", "account.apple.com", "account.bbc.com", "account.wolfram.com", "accounts.google.com", "aclanthology.org", "admissions.duke.edu", "ads.google.com", "agoda", "agoda.com", "ai.google", "airbnb", "airbnb.com", "aliexpress.com", "aljazeera.com", "alliedartsguild.org", "allrecipes.com", "alternativeto.net", "amazon", "amazon.com", "amctheatres", "amctheatres.com", "amtrak", "amtrak.com", "aoschat.apple.com", "apache.org", "api.bfl.ml", "api.endpoints.huggingface.cloud", "apnews.com", "apple", "apple.com", "apps.apple.com", "archive.org", "archlinux.org", "arrowheadpride.com", "arstechnica.com", "arxiv.org", "askubuntu.com", "aur.archlinux.org", "australia.gov.au", "auth.allrecipes.com", "auth.wikimedia.org", "automationexercise.com", "autotrader.com", "baidu.com", "bandcamp.com", "bbc.co.uk", "bbc.com", "bbcgoodfood.com", "berkeley.edu", "bestbuy.com", "bgr.com", "bilibili.com", "bing.com", "biodiversitylibrary.org", "biorxiv.org", "bits.debian.org", "blanchardgold.com", "blog.coursera.org", "blog.google", "blog.packagist.com", "blog.rust-lang.org", "bluezones.com", "boardgamegeek", "boardgamegeek.com", "bookdepository", "bookdepository.com", "booking", "booking.com", "books.toscrape.com", "britannica.com", "bsky.app", "budget", "budget.com", "budgetbytes.com", "builtinla.com", "bulatov.org", "bundesregierung.de", "business.apple.com", "cabelas", "cabelas.com", "calculator.net", "cam.ac.uk", "canada.ca", "capterra.com", "cargurus", "cargurus.com", "carmax", "carmax.com", "carnival", "carnival.com", "catalog.data.gov", "cbc.ca", "cbssports.com", "cdc.gov", "census.gov", "cern.ch", "checkcoverage.apple.com", "chess.com", "chi2023.acm.org", "chooser-beta.creativecommons.org", "circlecare4kids.com", "cli.github.com", "climate.com", "codeberg.org", "codeql.github.com", "colab.research.google.com", "commons.wikimedia.org", "companiesmarketcap.com", "compass.cornwall.gov.uk", "conservatorysf.com", "contactretail.apple.com", "cookingclassy.com", "copilot.github.trust.page", "core.ac.uk", "cornell.edu", "corriere.it", "coursera.org", "cpan.org", "cppreference.com", "craigslist.org", "cran.r-project.org", "crates.io", "creativecommons.org", "csdn.net", "cvent.com", "cvs", "cvs.com", "d1.supercook.com", "data.gov", "datarepository.wolframcloud.com", "dblp.org", "dblp.uni-trier.de", "de.wikipedia.org", "debian.org", "deepl.com", "delta", "delta.com", "demo.nopcommerce.com", "demo.opencart.com", "demoblaze.com", "demoqa.com", "desmos.com", "dev.mysql.com", "dev.to", "devdocs.io", "developer.apple.com", "developer.hashicorp.com", "developer.mozilla.org", "developer.nvidia.com", "dictionary.cambridge.org", "dictionary.com", "dictionaryblog.cambridge.org", "digital.library.upenn.edu", "dis.acm.org", "discogs", "discogs.com", "discuss.huggingface.co", "discussions.apple.com", "dmytro-shyshkin.kit.com", "doaj.org", "doc.rust-lang.org", "docker.com", "docs.ansible.com", "docs.bfl.ml", "docs.djangoproject.com", "docs.docker.com", "docs.github.com", "docs.gitlab.com", "docs.graphene-python.org", "docs.oracle.com", "docs.python.org", "docs.pytorch.org", "docs.readthedocs.com", "docs.readthedocs.io", "docs.rs", "donate.wikimedia.org", "douban.com", "duckduckgo.com", "duke.edu", "duolingo.com", "dw.com", "earthobservatory.nasa.gov", "ebay", "ebay.com", "ecommerce.apple.com", "economist.com", "edx.org", "elcampesinopgh.com", "elpais.com", "en.cppreference.com", "en.wikibooks.org", "en.wikipedia.org", "en.wikiquote.org", "en.wikisource.org", "en.wikivoyage.org", "en.wiktionary.org", "endpoints.huggingface.co", "engadget.com", "enterprise", "enterprise.com", "enterprise.github.com", "epa.gov", "epicurious.com", "eric.ed.gov", "es.wikipedia.org", "esa.int", "espn", "espn.com", "ethz.ch", "etsy.com", "etymonline.com", "euronews.com", "europa.eu", "european-union.europa.eu", "europeana.eu", "eventbrite", "eventbrite.com", "eventupplanner.com", "expedia", "expedia.com", "exploretock", "exploretock.com", "fda.gov", "figshare.com", "finance.yahoo.com", "flask.palletsprojects.com", "flexera.com", "flickr.com", "flightaware", "flightaware.com", "flights.booking.com", "flyus.com", "fontawesome.com", "fonts.google.com", "foodnetwork.com", "fool.com", "formulae.brew.sh", "formy-project.herokuapp.com", "forums.parasoft.com", "foundation.wikimedia.org", "foxsports", "foxsports.com", "fr.wikipedia.org", "france24.com", "freecodecamp.org", "g2.com", "gamestop", "gamestop.com", "gayoarabicacoffee.or.id", "geeksforgeeks.org", "geogebra.org", "geonames.org", "getbootstrap.com", "getsupport.apple.com", "git-scm.com", "git.kernel.org", "gitee.com", "github.blog", "github.com", "githubstatus.com", "gitlab.archlinux.org", "gitlab.com", "glamour.com", "glassdoor.com", "globalsqa.com", "gnu.org", "go.dev", "godocs.io", "gog.com", "goodreads.com", "google.com", "gov.cn", "gov.uk", "graphql.org", "guggenheim.org", "gutenberg.org", "harvard.edu", "hashnode.com", "healthline.com", "help.github.com", "help.ubuntu.com", "hex.pm", "home.cern", "homedepot.com", "https%3a%2f%2fdictionary.cambridge.org%2fdictionary%2fenglish%2fstatuette&text=", "https%3a%2f%2fwww.google.com%2fmaps%2fpreview%2fplace%3fauthuser%3d0%26hl%3den%26gl%3dus%26pb%3d%211m24%211s0x47e671c8ff6946b9%3a0x1194a918177c2ac8%213m12%211m3%211d39369.73246780893%212d2.3643297605770677%213d48.864906460992195%212m3%211f0%212f0%213f0%213m2%211i440%212i720%214f13.1%214m2%213d48.8535799%214d2.3426408%2115m6%211m5%211s0x47e671c8ff6946b9%3a0x1194a918177c2ac8%214s%2fg%2f11l75rskx1%215schijuuzp_8hx5kcrycp8fxiplbe%216s4571445599278015326%217s100590512120706543623%216srestaurants%2bnear%2bparis%2112m4%212m3%211i360%212i120%214i8%2113m48%212m2%211i203%212i100%213m2%212i4%215b1%216m6%211m2%211i86%212i86%211m2%211i408%212i240%217m33%211m3%211e1%212b0%213e3%211m3%211e2%212b1%213e2%211m3%211e2%212b0%213e3%211m3%211e8%212b0%213e3%211m3%211e10%212b0%213e3%211m3%211e10%212b1%213e2%211m3%211e10%212b0%213e4%211m3%211e9%212b1%213e2%212b1%219b0%2114m2%211sthpuz8yuhune0pepiyxe6q0%217e81%2115m107%211m34%214e2%2113m9%212b1%213b1%214b1%216i1%218b1%219b1%2114b1%2120b1%2125b1%2118m22%213b1%214b1%215b1%216b1%219b1%2112b1%2113b1%2114b1%2117b1%2120b1%2121b1%2122b1%2125b1%2127m1%211b0%2128b0%2130b1%2132b0%2133m1%211b1%2134b0%2136e1%2110m1%218e3%2111m1%213e1%2114m1%213b1%2117b1%2120m2%211e3%211e6%2124b1%2125b1%2126b1%2129b1%2130m1%212b1%2136b1%2139m3%212m2%212i1%213i1%2143b1%2152b1%2154m1%211b1%2155b1%2156m1%211b1%2165m5%213m4%211m3%211m2%211i224%212i298%2171b1%2172m19%211m5%211b1%212b1%213b1%215b1%217b1%214b1%218m10%211m6%214m1%211e1%214m1%211e3%214m1%211e4%213sother_user_reviews%216m1%211e1%219b1%2189b1%2198m3%211b1%212b1%213b1%21103b1%21113b1%21114m3%211b1%212m1%211b1%21117b1%21122m1%211b1%21125b0%21126b1%21127b1%2121m0%2122m2%211e81%218e4%2129m0%2130m6%213b1%216m1%212b1%217m1%212b1%219b1%2134m2%217b1%2110b1%2137i715%2138schzszxn0yxvyyw50cybuzwfyifbhcmlzwhgifnjlc3rhdxjhbnrzig5lyxigcgfyaxosarfpbmrpyw5fcmvzdgf1cmfudoabaa%2139spatang%2140b0%2141b1%26q%3dpatang&ihs=5", "https%3a%2f%2fwww.google.com%2fmaps%2fpreview%2fplace%3fauthuser%3d0%26hl%3den%26gl%3dus%26pb%3d%211m24%211s0x808fa4afd47b8fa3%3a0x8d48a6ae94d4207b%213m12%211m3%211d47502.581597019554%212d-122.16297270467842%213d37.46452844837867%212m3%211f0%212f0%213f0%213m2%211i440%212i720%214f13.1%214m2%213d37.452656%214d-122.18331719999999%2115m6%211m5%211s0x808fa4afd47b8fa3%3a0x8d48a6ae94d4207b%214s%2fg%2f1tfv5c5m%215schijo4971k-kj4areydulk6msi0%216s16632359157944504516%217s106796813392715891691%216swheelchair-accessible%2brestaurants%2bnear%2bme%2112m4%212m3%211i360%212i120%214i8%2113m48%212m2%211i203%212i100%213m2%212i4%215b1%216m6%211m2%211i86%212i86%211m2%211i408%212i240%217m33%211m3%211e1%212b0%213e3%211m3%211e2%212b1%213e2%211m3%211e2%212b0%213e3%211m3%211e8%212b0%213e3%211m3%211e10%212b0%213e3%211m3%211e10%212b1%213e2%211m3%211e10%212b0%213e4%211m3%211e9%212b1%213e2%212b1%219b0%2114m2%211szojxz--ab8hd0pep7qs5ua%217e81%2115m107%211m34%214e2%2113m9%212b1%213b1%214b1%216i1%218b1%219b1%2114b1%2120b1%2125b1%2118m22%213b1%214b1%215b1%216b1%219b1%2112b1%2113b1%2114b1%2117b1%2120b1%2121b1%2122b1%2125b1%2127m1%211b0%2128b0%2130b1%2132b0%2133m1%211b1%2134b0%2136e1%2110m1%218e3%2111m1%213e1%2114m1%213b1%2117b1%2120m2%211e3%211e6%2124b1%2125b1%2126b1%2129b1%2130m1%212b1%2136b1%2139m3%212m2%212i1%213i1%2143b1%2152b1%2154m1%211b1%2155b1%2156m1%211b1%2165m5%213m4%211m3%211m2%211i224%212i298%2171b1%2172m19%211m5%211b1%212b1%213b1%215b1%217b1%214b1%218m10%211m6%214m1%211e1%214m1%211e3%214m1%211e4%213sother_user_reviews%216m1%211e1%219b1%2189b1%2198m3%211b1%212b1%213b1%21103b1%21113b1%21114m3%211b1%212m1%211b1%21117b1%21122m1%211b1%21125b0%21126b1%21127b1%2121m0%2122m2%211e81%218e4%2129m0%2130m6%213b1%216m1%212b1%217m1%212b1%219b1%2134m2%217b1%2110b1%2137i715%2138scilxagvlbgnoywlylwfjy2vzc2libgugcmvzdgf1cmfudhmgbmvhcibtzsjduj4vz2vvl3r5cguvzxn0ywjsaxnobwvudf9wb2kvagfzx3dozwvsy2hhaxjfywnjzxnzawjszv9lbnryyw5jzzabavoriil3agvlbgnoywlyigfjy2vzc2libgugcmvzdgf1cmfudhmgbmvhcibtzzibewzyzw5jaf9yzxn0yxvyyw504aea%2139sleft%2bbank%2140b0%2141b1%26q%3dleft%2bbank&ihs=9", "https%3a%2f%2fwww.google.com%2fmaps%2fpreview%2fplace%3fauthuser%3d0%26hl%3den%26gl%3dus%26pb%3d%211m24%211s0x808fbb39c04bc749%3a0xc5d634786e3aba09%213m12%211m3%211d11878.61345184126%212d-122.1621916%213d37.4458386%212m3%211f0%212f0%213f0%213m2%211i440%212i720%214f13.1%214m2%213d37.445838599999995%214d-122.1621916%2115m6%211m5%211s0x808fbb39c04bc749%3a0xc5d634786e3aba09%214s%2fg%2f11b6jh7xz6%215schijscdlwdm7j4arcbo6bng01su%216s15460965790847260555%217s108538787170261119825%216spalo%2balto%2brestaurants%2112m4%212m3%211i360%212i120%214i8%2113m48%212m2%211i203%212i100%213m2%212i4%215b1%216m6%211m2%211i86%212i86%211m2%211i408%212i240%217m33%211m3%211e1%212b0%213e3%211m3%211e2%212b1%213e2%211m3%211e2%212b0%213e3%211m3%211e8%212b0%213e3%211m3%211e10%212b0%213e3%211m3%211e10%212b1%213e2%211m3%211e10%212b0%213e4%211m3%211e9%212b1%213e2%212b1%219b0%2114m2%211s1zzuz4jzn8m80pepr9c4-qy%217e81%2115m107%211m34%214e2%2113m9%212b1%213b1%214b1%216i1%218b1%219b1%2114b1%2120b1%2125b1%2118m22%213b1%214b1%215b1%216b1%219b1%2112b1%2113b1%2114b1%2117b1%2120b1%2121b1%2122b1%2125b1%2127m1%211b0%2128b0%2130b1%2132b0%2133m1%211b1%2134b0%2136e1%2110m1%218e3%2111m1%213e1%2114m1%213b1%2117b1%2120m2%211e3%211e6%2124b1%2125b1%2126b1%2129b1%2130m1%212b1%2136b1%2139m3%212m2%212i1%213i1%2143b1%2152b1%2154m1%211b1%2155b1%2156m1%211b1%2165m5%213m4%211m3%211m2%211i224%212i298%2171b1%2172m19%211m5%211b1%212b1%213b1%215b1%217b1%214b1%218m10%211m6%214m1%211e1%214m1%211e3%214m1%211e4%213sother_user_reviews%216m1%211e1%219b1%2189b1%2198m3%211b1%212b1%213b1%21103b1%21113b1%21114m3%211b1%212m1%211b1%21117b1%21122m1%211b1%21125b0%21126b1%21127b1%2121m0%2122m2%211e81%218e4%2129m0%2130m6%213b1%216m1%212b1%217m1%212b1%219b1%2134m2%217b1%2110b1%2137i715%2138schvqywxviefsdg8gcmvzdgf1cmfudhnafyivcgfsbybhbhrvihjlc3rhdxjhbnrzkgekcmvzdgf1cmfudoabaa%2139slocal%2bunion%2b271%2141b1%26q%3dlocal%2bunion%2b271&ihs=7", "https%3a%2f%2fwww.google.com%2fsearch%3ftbm%3dmap%26authuser%3d0%26hl%3den%26gl%3dus%26pb%3d%214m12%211m3%211d11329.387714118226%212d-73.99018908577365%213d40.78027538426176%212m3%211f0%212f0%213f0%213m2%211i1280%212i720%214f13.1%217i20%2110b1%2112m22%211m3%2118b1%2130b1%2134e1%212m3%215m1%216e2%2120e3%2110b1%2112b1%2113b1%2116b1%2117m1%213e1%2120m3%215e2%216b1%2114b1%2146m1%211b0%2194b1%2196b1%2119m4%212m3%211i360%212i120%214i8%2120m48%212m2%211i203%212i100%213m2%212i4%215b1%216m6%211m2%211i86%212i86%211m2%211i408%212i240%217m33%211m3%211e1%212b0%213e3%211m3%211e2%212b1%213e2%211m3%211e2%212b0%213e3%211m3%211e8%212b0%213e3%211m3%211e10%212b0%213e3%211m3%211e10%212b1%213e2%211m3%211e10%212b0%213e4%211m3%211e9%212b1%213e2%212b1%219b0%2122m6%211semjvz77uou6z0pepqkdjsqs%3a244%212s1i%3a0%2ct%3a11887%2cp%3aemjvz77uou6z0pepqkdjsqs%3a244%217e81%2112e3%2117semjvz77uou6z0pepqkdjsqs%3a327%2118e15%2124m105%211m32%2113m9%212b1%213b1%214b1%216i1%218b1%219b1%2114b1%2120b1%2125b1%2118m21%213b1%214b1%215b1%216b1%219b1%2112b1%2113b1%2114b1%2117b1%2120b1%2121b1%2122b1%2125b1%2127m1%211b0%2128b0%2132b0%2133m1%211b1%2134b0%2136e1%2110m1%218e3%2111m1%213e1%2114m1%213b1%2117b1%2120m2%211e3%211e6%2124b1%2125b1%2126b1%2129b1%2130m1%212b1%2136b1%2139m3%212m2%212i1%213i1%2143b1%2152b1%2154m1%211b1%2155b1%2156m1%211b1%2165m5%213m4%211m3%211m2%211i224%212i298%2171b1%2172m19%211m5%211b1%212b1%213b1%215b1%217b1%214b1%218m10%211m6%214m1%211e1%214m1%211e3%214m1%211e4%213sother_user_reviews%216m1%211e1%219b1%2189b1%2198m3%211b1%212b1%213b1%21103b1%21113b1%21114m3%211b1%212m1%211b1%21117b1%21122m1%211b1%21125b0%21126b1%21127b1%2126m4%212m3%211i80%212i92%214i8%2127m5%212smanhattan%2c%2bnew%2byork%2c%2bny%213s0x89c2588f046ee661%3a0xa0b3281fcecc08c%214m2%213d40.7685167%214d-73.98219379999999%2130m28%211m6%211m2%211i0%212i0%212m2%211i530%212i720%211m6%211m2%211i1230%212i0%212m2%211i1280%212i720%211m6%211m2%211i0%212i0%212m2%211i1280%212i20%211m6%211m2%211i0%212i700%212m2%211i1280%212i720%2134m18%212b1%213b1%214b1%216b1%218m6%211b1%213b1%214b1%215b1%216b1%217b1%219b1%2112b1%2114b1%2120b1%2123b1%2125b1%2126b1%2137m1%211e81%2142b1%2147m0%2149m9%213b1%216m2%211b1%212b1%217m2%211e3%212b1%218b1%219b1%2150m4%212e2%213m2%211b1%213b1%2167m2%217b1%2110b1%2169i715%26q%3dwheelchair%2baccessible%2brestaurants%2bnear%2bnew%2byork%2bcity%26oq%3dwheelchair%2baccessible%2brestaurants%2bnear%2bnew%2byork%2bcity%26gs_l%3dmaps.12...0.0.6.8288.0.0.....0.0..0.....0......maps..0.0.0.0.%26ech%3d6%26psi%3demjvz77uou6z0pepqkdjsqs.1735377425420.1&ihs=13", "https%3a%2f%2fwww.google.com%2fsearch%3ftbm%3dmap%26authuser%3d0%26hl%3den%26gl%3dus%26pb%3d%214m12%211m3%211d23759.89356993355%212d-122.1820416%213d37.437439999999995%212m3%211f0%212f0%213f0%213m2%211i1280%212i720%214f13.1%217i20%2110b1%2112m22%211m3%2118b1%2130b1%2134e1%212m3%215m1%216e2%2120e3%2110b1%2112b1%2113b1%2116b1%2117m1%213e1%2120m3%215e2%216b1%2114b1%2146m1%211b0%2194b1%2196b1%2119m4%212m3%211i360%212i120%214i8%2120m48%212m2%211i203%212i100%213m2%212i4%215b1%216m6%211m2%211i86%212i86%211m2%211i408%212i240%217m33%211m3%211e1%212b0%213e3%211m3%211e2%212b1%213e2%211m3%211e2%212b0%213e3%211m3%211e8%212b0%213e3%211m3%211e10%212b0%213e3%211m3%211e10%212b1%213e2%211m3%211e10%212b0%213e4%211m3%211e9%212b1%213e2%212b1%219b0%2122m6%211snaduz_n7npfd0pep2cwb8aq%3a2%212s1i%3a0%2ct%3a11886%2cp%3anaduz_n7npfd0pep2cwb8aq%3a2%217e81%2112e5%2117snaduz_n7npfd0pep2cwb8aq%3a33%2118e15%2124m105%211m32%2113m9%212b1%213b1%214b1%216i1%218b1%219b1%2114b1%2120b1%2125b1%2118m21%213b1%214b1%215b1%216b1%219b1%2112b1%2113b1%2114b1%2117b1%2120b1%2121b1%2122b1%2125b1%2127m1%211b0%2128b0%2132b0%2133m1%211b1%2134b0%2136e1%2110m1%218e3%2111m1%213e1%2114m1%213b1%2117b1%2120m2%211e3%211e6%2124b1%2125b1%2126b1%2129b1%2130m1%212b1%2136b1%2139m3%212m2%212i1%213i1%2143b1%2152b1%2154m1%211b1%2155b1%2156m1%211b1%2165m5%213m4%211m3%211m2%211i224%212i298%2171b1%2172m19%211m5%211b1%212b1%213b1%215b1%217b1%214b1%218m10%211m6%214m1%211e1%214m1%211e3%214m1%211e4%213sother_user_reviews%216m1%211e1%219b1%2189b1%2198m3%211b1%212b1%213b1%21103b1%21113b1%21114m3%211b1%212m1%211b1%21117b1%21122m1%211b1%21125b0%21126b1%21127b1%2126m4%212m3%211i80%212i92%214i8%2130m28%211m6%211m2%211i0%212i0%212m2%211i530%212i720%211m6%211m2%211i1230%212i0%212m2%211i1280%212i720%211m6%211m2%211i0%212i0%212m2%211i1280%212i20%211m6%211m2%211i0%212i700%212m2%211i1280%212i720%2134m18%212b1%213b1%214b1%216b1%218m6%211b1%213b1%214b1%215b1%216b1%217b1%219b1%2112b1%2114b1%2120b1%2123b1%2125b1%2126b1%2137m1%211e81%2142b1%2147m0%2149m9%213b1%216m2%211b1%212b1%217m2%211e3%212b1%218b1%219b1%2150m4%212e2%213m2%211b1%213b1%2167m2%217b1%2110b1%2169i715%26q%3drestaurants%2bnear%2bstanford%26oq%3drestaurants%2bnear%2bstanford%26gs_l%3dmaps.3..38i72k1l2j38i377i430i444k1j38i72k1l2.4825.4825.1.5846.1.1.....61.61.1.1.....0....1..maps..0.1.67.0.%26ech%3d1%26psi%3dnaduz_n7npfd0pep2cwb8aq.1735264157340.1&ihs=7", "hub.docker.com", "hud.pytorch.org", "huggingface.co", "idmsa.apple.com", "ign", "ign.com", "ikea", "ikea.com", "imdb", "imdb.com", "imf.org", "indeed.com", "indiegogo.com", "info.arxiv.org", "info.dev.arxiv.org", "inspector.pypi.io", "inspirehep.net", "instacart", "instacart.com", "instructables.com", "ipcc.ch", "ironhorsehome.com", "itch.io", "ithaka.org", "ja.wikipedia.org", "jagranjosh.com", "japantimes.co.jp", "jd.com", "jetblue", "jetblue.com", "jianshu.com", "jmlr.org", "jobs.lever.co", "journals.plos.org", "jstor.org", "kaggle.com", "kayak", "kayak.com", "kbb", "kbb.com", "kernel.org", "khanacademy.org", "kickstarter.com", "kidshealth.org", "kimkim.com", "koa", "koa.com", "kohls", "kohls.com", "kubernetes.io", "last.fm", "last.fm.com", "launchpad.net", "layaface.com", "layarestaurant.com", "learn.microsoft.com", "lemonde.fr", "leviathansecurity.com", "lichess.org", "linkedin.com", "llama.com", "llm-grounded-diffusion.github.io", "lobste.rs", "loc.gov", "lonelyplanet.com", "lumenprize.com", "lwn.net", "magento.softwaretestingboard.com", "managore.itch.io", "marriott", "marriott.com", "math.stackexchange.com", "mathworld.wolfram.com", "mayoclinic.org", "mbta", "mbta.com", "medium.com", "medrxiv.org", "menards", "menards.com", "mercadolibre.com.ar", "merriam-webster.com", "metacpan.org", "metacritic.com", "metrics.lecoq.io", "micronews.debian.org", "midsomersetcameraclub.org.uk", "missionjuno.swri.edu", "mit.edu", "modelscope.cn", "mojeek.com", "monroeville.elcampesinopgh.com", "mtu.edu", "musicbrainz.org", "mvnrepository.com", "my.gov.au", "name.design", "nasa.gov", "nationwidechildrens.org", "nature.com", "ncbi.nlm.nih.gov", "ness.com", "new.mta.info", "new.mta.info.com", "newegg", "newegg.com", "news.bbc.co.uk", "news.google.com", "news.ycombinator.com", "nfl", "nfl.com", "nhl.com", "nih.gov", "ninehours.co.jp", "noaa.gov", "nodejs.org", "notion.com", "npmjs.com", "npr.org", "nps.gov", "nps.gov.com", "numpy.org", "nvd.nist.gov", "nyc", "nyc.com", "nytimes.com", "octoverse.github.com", "ocw.mit.edu", "oecd.org", "oeis.org", "openculture.com", "openlibrary.org", "openreview.net", "opensource-demo.orangehrmlive.com", "openstreetmap.org", "openweathermap.org", "orcid.org", "order.pacificcatch.com", "osf.io", "osmfoundation.org", "otter.ai", "otto.de", "ox.ac.uk", "packages.debian.org", "packagist.org", "pandas.pydata.org", "paperswithcode.com", "parabank.parasoft.com", "parking", "parking.com", "patanghotel.com", "patreon.com", "pbs.org", "pbskids.org", "peps.python.org", "pexels.com", "pharmacy.amazon.com", "phoronix.com", "php.net", "pixabay.com", "pkg.go.dev", "plato.stanford.edu", "platogbr.com", "play.google.com", "play.rust-lang.org", "pmc.ncbi.nlm.nih.gov", "postgresql.org", "practice.expandtesting.com", "practicesoftwaretesting.com", "practicetestautomation.com", "prevention.com", "producthunt.com", "profile.theguardian.com", "projects.apache.org", "proxy.golang.org", "pub.dev", "publicpolicy.google", "pubmed.ncbi.nlm.nih.gov", "pypi.org", "python.org", "pytorch.org", "qatarairways", "qatarairways.com", "quora.com", "quotes.toscrape.com", "qwant.com", "rakuten.co.jp", "react.dev", "readthedocs.org", "realpython.github.io", "realty101.com", "rec.stanford.edu", "redbox", "redbox.com", "reddit.com", "redditinc.com", "redis.io", "reference.wolfram.com", "rei", "rei.com", "remoteok.com", "rentalcars", "rentalcars.com", "repology.org", "research.google", "reservation.patanghotel.com", "reservations.limprimeriehotel.com", "resources.github.com", "resy", "resy.com", "reuters.com", "rottentomatoes", "rottentomatoes.com", "ruby-lang.org", "rubycentral.org", "rubygems.org", "rust-lang.org", "ryanair", "ryanair.com", "safety.google", "saucedemo.com", "sbert.net", "scholar.archive.org", "scholar.google.com", "science.nasa.gov", "scikit-learn.org", "scmp.com", "scrapeme.live", "scrapethissite.com", "search.brave.com", "search.epa.gov", "search.maven.org", "search.usa.gov", "searchengineland.com", "seatgeek", "seatgeek.com", "secure8.store.apple.com", "secure9.store.apple.com", "security.archlinux.org", "securitylab.github.com", "selenium.dev", "semanticscholar.org", "serverfault.com", "service-public.fr", "service-public.gouv.fr", "sfmta.com", "simple.wikipedia.org", "sixflags", "sixflags.com", "skyscanner.net", "slashdot.org", "snlab.bme.cornell.edu", "so.com", "so.gitee.com", "sogou.com", "soundcloud", "soundcloud.com", "sourceforge.net", "southampton.ac.uk", "space.com", "spiegel.de", "sports.yahoo", "sports.yahoo.com", "spothero", "spothero.com", "stablediffusionapi.com", "stackexchange.com", "stackoverflow.blog", "stackoverflow.co", "stackoverflow.com", "stanford.edu", "store.google.com", "store.steampowered", "store.steampowered.com", "supercook.com", "superuser.com", "support.apple.com", "support.google.com", "t-mobile.com", "t10.org", "tailwindcss.com", "taobao.com", "target", "target.com", "techaro.lol", "techcrunch.com", "ted.com", "tensorflow.org", "terraform.io", "tesla", "tesla.com", "the-internet.herokuapp.com", "thebluezonesstore.com", "theclimateweb.com", "thefreedictionary.com", "theguardian.com", "thehindu.com", "themelt.com", "thesaurus.com", "thetrainline", "thetrainline.com", "theverge.com", "theweekendwoodworker.com", "thingiverse.com", "ticket.toureiffel.paris", "ticketcenter", "ticketcenter.com", "timeanddate.com", "timesofindia.indiatimes.com", "toureiffel.paris", "translate.google.com", "travelzoo", "travelzoo.com", "trends.google.com", "trends.google.fr", "tripadvisor.com", "trust.weather.com", "trustpilot.com", "tsinghua.edu.cn", "tutorialspoint.com", "tvguide", "tvguide.com", "twitch.tv", "ubuntu.com", "ui.adsabs.harvard.edu", "ultimate-guitar", "ultimate-guitar.com", "un.org", "underarmour", "underarmour.com", "uniqlo", "uniqlo.com", "united", "united.com", "unix.stackexchange.com", "unsplash.com", "us.megabus", "us.megabus.com", "usa.gov", "utoronto.ca", "viator", "viator.com", "vogue.com", "vuejs.org", "w3schools.com", "walmart.com", "washingtonpost.com", "wayfair.com", "weather.com", "web.dev", "webscraper.io", "wedgewoodweddings.com", "weworkremotely.com", "who.int", "wiki.archlinux.org", "wiki.debian.org", "wiki.osgeo.org", "wiki.python.org", "wikidata.org", "wikihow.com", "wikimedia.org", "wikipedia.org", "wiktionary.org", "wired.com", "wolfram.com", "wolframalpha.com", "worldbank.org", "worldcat.org", "wri.org", "xe.com", "yandex.com", "yellowpages", "yellowpages.com", "yelp", "yelp.com", "youtube.com", "zalando.de", "zdnet.com", "zenodo.org", "zh.airbnb.com", "zh.thefreedictionary.com", "zh.wikipedia.org", "zh.wiktionary.org", "zhihu.com", "zillow.com"]

results/round3/v12s.log

@@ -0,0 +1,314 @@
+== rollout2 cases: 818   nnetnav cases: 8917
+== build v12s
+operation label counts: {'TYPE_TEXT': 10107, 'CLICK': 15208, 'SELECT': 1380, 'DONE': 10364, 'SCROLL_DOWN': 3338, 'PRESS_ENTER': 1620, 'SCROLL_UP': 137}
+train items 61191 (op 42154, target 19037), eval cases 5314, skipped 0, seq len mean 786 max 1024
+== train v12s (3 epochs)
+  ep 1 step 20/5736 loss 1.867 ce 1.661 reward -1.357 26s
+  ep 1 step 40/5736 loss 0.806 ce 1.988 reward -1.635 48s
+  ep 1 step 60/5736 loss 2.244 ce 2.367 reward -2.104 73s
+  ep 1 step 80/5736 loss 1.748 ce 1.633 reward -1.354 100s
+  ep 1 step 100/5736 loss 0.450 ce 0.805 reward -0.451 123s
+  ep 1 step 120/5736 loss 1.838 ce 1.455 reward -1.094 150s
+  ep 1 step 140/5736 loss 1.511 ce 0.991 reward -0.550 176s
+  ep 1 step 160/5736 loss 0.068 ce 0.081 reward 0.647 202s
+  ep 1 step 180/5736 loss 1.662 ce 1.634 reward -1.109 229s
+  ep 1 step 200/5736 loss 0.155 ce 0.486 reward 0.035 262s
+  ep 1 step 220/5736 loss 0.922 ce 0.934 reward -0.485 292s
+  ep 1 step 240/5736 loss -0.582 ce 1.647 reward -1.240 315s
+  ep 1 step 260/5736 loss 0.030 ce 0.086 reward 0.649 338s
+  ep 1 step 280/5736 loss 1.079 ce 0.886 reward -0.323 362s
+  ep 1 step 300/5736 loss 1.009 ce 1.515 reward -1.017 385s
+  ep 1 step 320/5736 loss 0.501 ce 1.460 reward -0.954 407s
+  ep 1 step 340/5736 loss 0.948 ce 1.097 reward -0.542 437s
+  ep 1 step 360/5736 loss 2.070 ce 0.914 reward -0.374 462s
+  ep 1 step 380/5736 loss 1.567 ce 1.001 reward -0.514 491s
+  ep 1 step 400/5736 loss 0.226 ce 0.115 reward 0.625 517s
+  ep 1 step 420/5736 loss 1.699 ce 1.414 reward -1.130 541s
+  ep 1 step 440/5736 loss 0.594 ce 0.051 reward 0.688 566s
+  ep 1 step 460/5736 loss -0.469 ce 0.178 reward 0.498 591s
+  ep 1 step 480/5736 loss 1.374 ce 0.395 reward 0.257 614s
+  ep 1 step 500/5736 loss 2.047 ce 0.957 reward -0.341 636s
+  ep 1 step 520/5736 loss 1.322 ce 1.881 reward -1.510 658s
+  ep 1 step 540/5736 loss 1.677 ce 1.290 reward -0.781 681s
+  ep 1 step 560/5736 loss 0.978 ce 0.955 reward -0.334 703s
+  ep 1 step 580/5736 loss 0.759 ce 1.147 reward -0.779 726s
+  ep 1 step 600/5736 loss 0.153 ce 0.252 reward 0.430 751s
+  ep 1 step 620/5736 loss 1.371 ce 0.907 reward -0.309 775s
+  ep 1 step 640/5736 loss 1.415 ce 1.854 reward -1.247 800s
+  ep 1 step 660/5736 loss 1.116 ce 0.642 reward -0.033 825s
+  ep 1 step 680/5736 loss 2.978 ce 1.458 reward -0.996 850s
+  ep 1 step 700/5736 loss 1.113 ce 1.102 reward -0.652 874s
+  ep 1 step 720/5736 loss 2.186 ce 1.739 reward -1.365 896s
+  ep 1 step 740/5736 loss 0.807 ce 0.314 reward 0.352 918s
+  ep 1 step 760/5736 loss -0.023 ce 0.189 reward 0.520 941s
+  ep 1 step 780/5736 loss 0.104 ce 0.405 reward 0.284 963s
+  ep 1 step 800/5736 loss 0.616 ce 0.335 reward 0.351 985s
+  ep 1 step 820/5736 loss -1.048 ce 1.086 reward -0.633 1008s
+  ep 1 step 840/5736 loss 1.684 ce 0.770 reward -0.112 1031s
+  ep 1 step 860/5736 loss 1.732 ce 1.397 reward -1.155 1055s
+  ep 1 step 880/5736 loss 3.040 ce 3.139 reward -2.820 1080s
+  ep 1 step 900/5736 loss 2.128 ce 2.322 reward -2.160 1105s
+  ep 1 step 920/5736 loss -0.518 ce 0.000 reward 0.750 1129s
+  ep 1 step 940/5736 loss 1.805 ce 2.345 reward -2.121 1154s
+  ep 1 step 960/5736 loss -0.430 ce 0.000 reward 0.750 1177s
+  ep 1 step 980/5736 loss 2.253 ce 1.044 reward -0.507 1199s
+  ep 1 step 1000/5736 loss 0.894 ce 0.654 reward -0.094 1222s
+  ep 1 step 1020/5736 loss 1.455 ce 1.017 reward -0.609 1244s
+  ep 1 step 1040/5736 loss 0.287 ce 0.555 reward 0.075 1266s
+  ep 1 step 1060/5736 loss -0.340 ce 0.005 reward 0.740 1288s
+  ep 1 step 1080/5736 loss 0.915 ce 0.219 reward 0.550 1310s
+  ep 1 step 1100/5736 loss 0.092 ce 0.278 reward 0.335 1334s
+  ep 1 step 1120/5736 loss 2.255 ce 1.808 reward -1.507 1359s
+  ep 1 step 1140/5736 loss 1.179 ce 0.495 reward 0.200 1384s
+  ep 1 step 1160/5736 loss 0.012 ce 0.004 reward 0.747 1408s
+  ep 1 step 1180/5736 loss 0.123 ce 0.754 reward -0.255 1433s
+  ep 1 step 1200/5736 loss 0.198 ce 0.326 reward 0.309 1458s
+  ep 1 step 1220/5736 loss 0.509 ce 0.000 reward 0.750 1482s
+  ep 1 step 1240/5736 loss 0.322 ce 0.074 reward 0.670 1504s
+  ep 1 step 1260/5736 loss 1.003 ce 1.388 reward -0.980 1526s
+  ep 1 step 1280/5736 loss 0.873 ce 4.870 reward -1.776 1548s
+  ep 1 step 1300/5736 loss 0.422 ce 0.007 reward 0.743 1570s
+  ep 1 step 1320/5736 loss 0.311 ce 0.001 reward 0.748 1593s
+  ep 1 step 1340/5736 loss 5.178 ce 4.459 reward -3.079 1615s
+  ep 1 step 1360/5736 loss -0.279 ce 0.000 reward 0.750 1639s
+  ep 1 step 1380/5736 loss -0.840 ce 1.816 reward -1.648 1665s
+  ep 1 step 1400/5736 loss 2.030 ce 1.714 reward -1.225 1689s
+  ep 1 step 1420/5736 loss 1.433 ce 1.163 reward -0.708 1714s
+  ep 1 step 1440/5736 loss 0.244 ce 0.753 reward -0.194 1738s
+  ep 1 step 1460/5736 loss 2.513 ce 0.902 reward -0.237 1763s
+  ep 1 step 1480/5736 loss 1.593 ce 1.058 reward -0.578 1786s
+  ep 1 step 1500/5736 loss 0.870 ce 0.267 reward 0.457 1808s
+  ep 1 step 1520/5736 loss -0.183 ce 0.027 reward 0.714 1831s
+  ep 1 step 1540/5736 loss 1.141 ce 0.604 reward -0.165 1853s
+  ep 1 step 1560/5736 loss 1.778 ce 2.143 reward -1.779 1875s
+  ep 1 step 1580/5736 loss -1.432 ce 0.043 reward 0.695 1897s
+  ep 1 step 1600/5736 loss 1.285 ce 0.893 reward -0.555 1919s
+  ep 1 step 1620/5736 loss -0.448 ce 0.026 reward 0.725 1944s
+  ep 1 step 1640/5736 loss 1.916 ce 0.766 reward -0.141 1968s
+  ep 1 step 1660/5736 loss 0.366 ce 0.288 reward 0.416 1993s
+  ep 1 step 1680/5736 loss 0.359 ce 0.506 reward -0.044 2018s
+  ep 1 step 1700/5736 loss 1.577 ce 1.610 reward -1.254 2043s
+  ep 1 step 1720/5736 loss 0.310 ce 0.000 reward 0.750 2067s
+  ep 1 step 1740/5736 loss -0.173 ce 0.082 reward 0.674 2090s
+  ep 1 step 1760/5736 loss 2.839 ce 2.416 reward -1.750 2112s
+  ep 1 step 1780/5736 loss 0.334 ce 0.251 reward 0.427 2135s
+  ep 1 step 1800/5736 loss 6.210 ce 6.935 reward -4.351 2157s
+  ep 1 step 1820/5736 loss 1.940 ce 1.649 reward -1.230 2179s
+  ep 1 step 1840/5736 loss -0.000 ce -0.000 reward 0.750 2201s
+  ep 1 step 1860/5736 loss 0.137 ce 0.154 reward 0.588 2223s
+  ep 1 step 1880/5736 loss 3.022 ce 3.763 reward -3.697 2249s
+  ep 1 step 1900/5736 loss -0.994 ce 0.000 reward 0.750 2273s
+=== epoch 1/3 avg loss 0.9655  2288s
+  ep 2 step 1920/5736 loss -0.334 ce 0.004 reward 0.745 2297s
+  ep 2 step 1940/5736 loss 2.466 ce 2.807 reward -2.458 2322s
+  ep 2 step 1960/5736 loss -0.088 ce 0.014 reward 0.733 2347s
+  ep 2 step 1980/5736 loss 0.001 ce 0.000 reward 0.750 2371s
+  ep 2 step 2000/5736 loss -1.152 ce 0.257 reward 0.430 2393s
+  ep 2 step 2020/5736 loss 1.120 ce 1.008 reward -0.512 2416s
+  ep 2 step 2040/5736 loss 0.434 ce 0.852 reward -0.291 2438s
+  ep 2 step 2060/5736 loss 0.258 ce 0.137 reward 0.572 2460s
+  ep 2 step 2080/5736 loss -0.961 ce 1.669 reward -1.219 2482s
+  ep 2 step 2100/5736 loss 0.203 ce 0.268 reward 0.453 2505s
+  ep 2 step 2120/5736 loss -2.449 ce 0.003 reward 0.747 2528s
+  ep 2 step 2140/5736 loss -1.220 ce 0.003 reward 0.748 2552s
+  ep 2 step 2160/5736 loss 1.053 ce 0.486 reward 0.137 2577s
+  ep 2 step 2180/5736 loss -0.319 ce 0.342 reward 0.400 2602s
+  ep 2 step 2200/5736 loss -0.000 ce -0.000 reward 0.750 2627s
+  ep 2 step 2220/5736 loss -0.226 ce 0.424 reward 0.254 2651s
+  ep 2 step 2240/5736 loss 4.043 ce 4.104 reward -1.740 2675s
+  ep 2 step 2260/5736 loss 1.498 ce 1.368 reward -0.860 2698s
+  ep 2 step 2280/5736 loss -0.211 ce 0.000 reward 0.750 2720s
+  ep 2 step 2300/5736 loss -0.000 ce -0.000 reward 0.750 2742s
+  ep 2 step 2320/5736 loss 0.910 ce 1.463 reward -0.898 2764s
+  ep 2 step 2340/5736 loss 2.722 ce 2.704 reward -2.044 2786s
+  ep 2 step 2360/5736 loss 4.900 ce 4.703 reward -1.953 2808s
+  ep 2 step 2380/5736 loss -0.120 ce 0.008 reward 0.741 2832s
+  ep 2 step 2400/5736 loss 4.746 ce 4.734 reward -1.740 2857s
+  ep 2 step 2420/5736 loss 0.733 ce 0.408 reward 0.220 2881s
+  ep 2 step 2440/5736 loss 2.885 ce 3.230 reward -2.934 2906s
+  ep 2 step 2460/5736 loss -0.172 ce 0.000 reward 0.750 2930s
+  ep 2 step 2480/5736 loss -0.000 ce -0.000 reward 0.750 2955s
+  ep 2 step 2500/5736 loss -1.332 ce 0.024 reward 0.727 2979s
+  ep 2 step 2520/5736 loss 0.470 ce 1.203 reward -0.676 3002s
+  ep 2 step 2540/5736 loss -0.000 ce -0.000 reward 0.750 3024s
+  ep 2 step 2560/5736 loss 5.343 ce 5.343 reward -1.740 3046s
+  ep 2 step 2580/5736 loss 2.787 ce 1.669 reward -1.162 3068s
+  ep 2 step 2600/5736 loss -2.983 ce 0.709 reward -0.160 3091s
+  ep 2 step 2620/5736 loss 0.900 ce 0.000 reward 0.750 3113s
+  ep 2 step 2640/5736 loss 0.301 ce 0.370 reward 0.234 3136s
+  ep 2 step 2660/5736 loss 0.777 ce 1.571 reward -1.142 3162s
+  ep 2 step 2680/5736 loss 0.597 ce 0.584 reward -0.027 3187s
+  ep 2 step 2700/5736 loss 0.287 ce 0.329 reward 0.222 3212s
+  ep 2 step 2720/5736 loss 1.478 ce 0.387 reward 0.282 3237s
+  ep 2 step 2740/5736 loss 0.738 ce 0.507 reward 0.012 3261s
+  ep 2 step 2760/5736 loss 3.690 ce 4.616 reward -2.242 3284s
+  ep 2 step 2780/5736 loss -0.260 ce 0.193 reward 0.537 3306s
+  ep 2 step 2800/5736 loss 4.752 ce 4.766 reward -1.740 3329s
+  ep 2 step 2820/5736 loss 0.753 ce 0.179 reward 0.517 3351s
+  ep 2 step 2840/5736 loss 0.397 ce 0.280 reward 0.355 3373s
+  ep 2 step 2860/5736 loss 4.371 ce 4.577 reward -4.193 3396s
+  ep 2 step 2880/5736 loss 3.433 ce 3.418 reward -1.740 3418s
+  ep 2 step 2900/5736 loss -0.165 ce 0.302 reward 0.355 3442s
+  ep 2 step 2920/5736 loss -1.110 ce 0.000 reward 0.750 3467s
+  ep 2 step 2940/5736 loss 0.899 ce 2.142 reward -1.766 3491s
+  ep 2 step 2960/5736 loss -0.160 ce 0.002 reward 0.748 3516s
+  ep 2 step 2980/5736 loss 0.397 ce 0.076 reward 0.679 3540s
+  ep 2 step 3000/5736 loss 8.716 ce 9.129 reward -6.550 3565s
+  ep 2 step 3020/5736 loss 0.517 ce 0.504 reward 0.093 3587s
+  ep 2 step 3040/5736 loss 0.571 ce 1.267 reward -0.669 3609s
+  ep 2 step 3060/5736 loss 10.010 ce 9.623 reward -6.253 3631s
+  ep 2 step 3080/5736 loss 0.209 ce 0.092 reward 0.650 3652s
+  ep 2 step 3100/5736 loss 1.857 ce 1.848 reward -1.399 3674s
+  ep 2 step 3120/5736 loss 2.592 ce 1.673 reward -1.339 3695s
+  ep 2 step 3140/5736 loss 0.931 ce 1.065 reward -0.577 3717s
+  ep 2 step 3160/5736 loss -0.050 ce 0.524 reward 0.185 3741s
+  ep 2 step 3180/5736 loss 1.458 ce 0.869 reward -0.260 3766s
+  ep 2 step 3200/5736 loss -0.652 ce 0.678 reward -0.112 3790s
+  ep 2 step 3220/5736 loss 1.152 ce 0.740 reward -0.147 3813s
+  ep 2 step 3240/5736 loss -0.866 ce 1.132 reward -0.573 3837s
+  ep 2 step 3260/5736 loss 0.069 ce 0.347 reward 0.276 3861s
+  ep 2 step 3280/5736 loss 3.832 ce 3.105 reward -2.694 3884s
+  ep 2 step 3300/5736 loss -1.609 ce 0.866 reward -0.277 3906s
+  ep 2 step 3320/5736 loss -1.190 ce 0.540 reward 0.017 3927s
+  ep 2 step 3340/5736 loss 0.095 ce 0.000 reward 0.750 3949s
+  ep 2 step 3360/5736 loss 0.018 ce 0.000 reward 0.750 3971s
+  ep 2 step 3380/5736 loss -0.164 ce 0.000 reward 0.750 3992s
+  ep 2 step 3400/5736 loss 3.871 ce 3.871 reward -1.740 4014s
+  ep 2 step 3420/5736 loss 1.919 ce 1.431 reward -0.920 4037s
+  ep 2 step 3440/5736 loss -0.110 ce 0.000 reward 0.750 4061s
+  ep 2 step 3460/5736 loss 2.759 ce 3.016 reward -1.740 4085s
+  ep 2 step 3480/5736 loss 1.899 ce 0.326 reward 0.379 4109s
+  ep 2 step 3500/5736 loss 0.121 ce 0.015 reward 0.736 4133s
+  ep 2 step 3520/5736 loss 1.250 ce 1.704 reward -1.355 4157s
+  ep 2 step 3540/5736 loss -0.545 ce 0.141 reward 0.629 4180s
+  ep 2 step 3560/5736 loss 1.155 ce 0.648 reward -0.077 4202s
+  ep 2 step 3580/5736 loss 0.130 ce 0.000 reward 0.750 4223s
+  ep 2 step 3600/5736 loss 13.606 ce 13.596 reward -6.720 4245s
+  ep 2 step 3620/5736 loss 2.642 ce 1.323 reward -0.879 4266s
+  ep 2 step 3640/5736 loss 2.688 ce 2.795 reward -1.740 4288s
+  ep 2 step 3660/5736 loss 0.000 ce 0.000 reward 0.750 4309s
+  ep 2 step 3680/5736 loss 0.543 ce 0.526 reward 0.137 4332s
+  ep 2 step 3700/5736 loss 0.859 ce 0.061 reward 0.697 4356s
+  ep 2 step 3720/5736 loss 0.372 ce 0.000 reward 0.750 4380s
+  ep 2 step 3740/5736 loss 0.582 ce 0.795 reward -0.266 4403s
+  ep 2 step 3760/5736 loss -0.116 ce 0.000 reward 0.750 4427s
+  ep 2 step 3780/5736 loss 3.356 ce 3.557 reward -1.740 4451s
+  ep 2 step 3800/5736 loss 2.293 ce 1.484 reward -0.860 4475s
+  ep 2 step 3820/5736 loss -0.101 ce 0.000 reward 0.750 4496s
+=== epoch 2/3 avg loss 1.1094  4502s
+  ep 3 step 3840/5736 loss -0.065 ce 0.102 reward 0.632 4517s
+  ep 3 step 3860/5736 loss -0.032 ce 0.000 reward 0.750 4539s
+  ep 3 step 3880/5736 loss 5.263 ce 4.626 reward -1.795 4560s
+  ep 3 step 3900/5736 loss -0.141 ce 0.752 reward -0.168 4582s
+  ep 3 step 3920/5736 loss -0.000 ce -0.000 reward 0.750 4604s
+  ep 3 step 3940/5736 loss 4.600 ce 4.187 reward -2.491 4625s
+  ep 3 step 3960/5736 loss 9.254 ce 9.254 reward -4.230 4649s
+  ep 3 step 3980/5736 loss 0.000 ce 0.000 reward 0.750 4673s
+  ep 3 step 4000/5736 loss 1.703 ce 2.893 reward -2.497 4697s
+  ep 3 step 4020/5736 loss -0.000 ce -0.000 reward 0.750 4721s
+  ep 3 step 4040/5736 loss 3.809 ce 3.816 reward -1.740 4745s
+  ep 3 step 4060/5736 loss -0.239 ce 0.000 reward 0.750 4769s
+  ep 3 step 4080/5736 loss 2.466 ce 1.558 reward -1.119 4792s
+  ep 3 step 4100/5736 loss 2.049 ce 1.311 reward -0.801 4813s
+  ep 3 step 4120/5736 loss 0.000 ce 0.000 reward 0.750 4835s
+  ep 3 step 4140/5736 loss 0.000 ce 0.000 reward 0.750 4857s
+  ep 3 step 4160/5736 loss -0.029 ce 0.000 reward 0.750 4878s
+  ep 3 step 4180/5736 loss 3.194 ce 2.636 reward -2.267 4900s
+  ep 3 step 4200/5736 loss -0.904 ce 0.138 reward 0.607 4922s
+  ep 3 step 4220/5736 loss 10.090 ce 9.401 reward -4.727 4946s
+  ep 3 step 4240/5736 loss 4.646 ce 4.656 reward -1.740 4970s
+  ep 3 step 4260/5736 loss 0.157 ce 0.000 reward 0.750 4994s
+  ep 3 step 4280/5736 loss 0.922 ce 0.055 reward 0.696 5018s
+  ep 3 step 4300/5736 loss -0.000 ce -0.000 reward 0.750 5042s
+  ep 3 step 4320/5736 loss 0.922 ce 0.000 reward 0.750 5066s
+  ep 3 step 4340/5736 loss 0.076 ce 0.000 reward 0.750 5088s
+  ep 3 step 4360/5736 loss -0.000 ce -0.000 reward 0.750 5110s
+  ep 3 step 4380/5736 loss 1.704 ce 1.207 reward -0.685 5131s
+  ep 3 step 4400/5736 loss 4.283 ce 4.309 reward -1.740 5153s
+  ep 3 step 4420/5736 loss -0.121 ce 2.678 reward -2.319 5175s
+  ep 3 step 4440/5736 loss 1.292 ce 0.812 reward -0.235 5197s
+  ep 3 step 4460/5736 loss 0.000 ce 0.000 reward 0.750 5218s
+  ep 3 step 4480/5736 loss -0.058 ce 0.170 reward 0.512 5242s
+  ep 3 step 4500/5736 loss -0.010 ce 0.000 reward 0.750 5265s
+  ep 3 step 4520/5736 loss 0.579 ce 0.225 reward 0.471 5289s
+  ep 3 step 4540/5736 loss 0.051 ce 0.135 reward 0.576 5313s
+  ep 3 step 4560/5736 loss -0.379 ce 1.452 reward -0.917 5337s
+  ep 3 step 4580/5736 loss -0.011 ce 0.000 reward 0.750 5361s
+  ep 3 step 4600/5736 loss 1.084 ce 0.575 reward 0.081 5384s
+  ep 3 step 4620/5736 loss 2.207 ce 1.260 reward -0.702 5406s
+  ep 3 step 4640/5736 loss 0.395 ce 0.169 reward 0.529 5427s
+  ep 3 step 4660/5736 loss -0.267 ce 0.305 reward 0.364 5449s
+  ep 3 step 4680/5736 loss -0.000 ce -0.000 reward 0.750 5471s
+  ep 3 step 4700/5736 loss -0.872 ce 0.273 reward 0.471 5492s
+  ep 3 step 4720/5736 loss 0.732 ce 1.012 reward -0.467 5514s
+  ep 3 step 4740/5736 loss -0.000 ce -0.000 reward 0.750 5537s
+  ep 3 step 4760/5736 loss 0.043 ce 0.000 reward 0.750 5561s
+  ep 3 step 4780/5736 loss -0.008 ce 0.000 reward 0.750 5584s
+  ep 3 step 4800/5736 loss -0.015 ce 0.000 reward 0.750 5609s
+  ep 3 step 4820/5736 loss 7.047 ce 6.954 reward -4.230 5632s
+  ep 3 step 4840/5736 loss 0.021 ce 0.024 reward 0.723 5657s
+  ep 3 step 4860/5736 loss 2.918 ce 2.794 reward -2.469 5680s
+  ep 3 step 4880/5736 loss -0.000 ce -0.000 reward 0.750 5702s
+  ep 3 step 4900/5736 loss -0.010 ce 0.000 reward 0.750 5724s
+  ep 3 step 4920/5736 loss 0.033 ce 0.000 reward 0.750 5745s
+  ep 3 step 4940/5736 loss 4.543 ce 3.750 reward -1.740 5767s
+  ep 3 step 4960/5736 loss -0.032 ce 0.000 reward 0.750 5789s
+  ep 3 step 4980/5736 loss 1.302 ce 1.697 reward -1.307 5810s
+  ep 3 step 5000/5736 loss -0.000 ce -0.000 reward 0.750 5833s
+  ep 3 step 5020/5736 loss 7.283 ce 6.728 reward -4.230 5857s
+  ep 3 step 5040/5736 loss 0.000 ce 0.000 reward 0.750 5881s
+  ep 3 step 5060/5736 loss 2.784 ce 3.168 reward -1.740 5905s
+  ep 3 step 5080/5736 loss 0.077 ce 0.000 reward 0.750 5929s
+  ep 3 step 5100/5736 loss 4.980 ce 6.121 reward -3.107 5953s
+  ep 3 step 5120/5736 loss -0.000 ce -0.000 reward 0.750 5977s
+  ep 3 step 5140/5736 loss 2.089 ce 1.122 reward -0.550 5999s
+  ep 3 step 5160/5736 loss 1.105 ce 1.833 reward -1.380 6020s
+  ep 3 step 5180/5736 loss 0.000 ce 0.000 reward 0.750 6042s
+  ep 3 step 5200/5736 loss -0.055 ce -0.000 reward 0.750 6063s
+  ep 3 step 5220/5736 loss -0.000 ce -0.000 reward 0.750 6085s
+  ep 3 step 5240/5736 loss -0.090 ce 0.000 reward 0.750 6106s
+  ep 3 step 5260/5736 loss -0.000 ce -0.000 reward 0.750 6128s
+  ep 3 step 5280/5736 loss 3.014 ce 1.651 reward -1.102 6152s
+  ep 3 step 5300/5736 loss 0.000 ce 0.000 reward 0.750 6176s
+  ep 3 step 5320/5736 loss -0.332 ce 0.000 reward 0.750 6200s
+  ep 3 step 5340/5736 loss 4.471 ce 4.873 reward -1.857 6224s
+  ep 3 step 5360/5736 loss 0.047 ce 0.059 reward 0.691 6248s
+  ep 3 step 5380/5736 loss 4.046 ce 4.062 reward -1.740 6272s
+  ep 3 step 5400/5736 loss -0.000 ce -0.000 reward 0.750 6294s
+  ep 3 step 5420/5736 loss 0.000 ce 0.000 reward 0.750 6315s
+  ep 3 step 5440/5736 loss -0.000 ce -0.000 reward 0.750 6337s
+  ep 3 step 5460/5736 loss 9.175 ce 8.045 reward -4.230 6359s
+  ep 3 step 5480/5736 loss 3.850 ce 3.783 reward -1.740 6381s
+  ep 3 step 5500/5736 loss 0.803 ce 0.670 reward -0.024 6402s
+  ep 3 step 5520/5736 loss 0.000 ce 0.000 reward 0.750 6424s
+  ep 3 step 5540/5736 loss 0.488 ce 0.544 reward 0.064 6448s
+  ep 3 step 5560/5736 loss -1.941 ce 0.158 reward 0.581 6472s
+  ep 3 step 5580/5736 loss 0.006 ce 0.000 reward 0.750 6496s
+  ep 3 step 5600/5736 loss 3.625 ce 3.855 reward -2.341 6521s
+  ep 3 step 5620/5736 loss 5.359 ce 5.359 reward -1.740 6545s
+  ep 3 step 5640/5736 loss 0.021 ce 0.000 reward 0.750 6569s
+  ep 3 step 5660/5736 loss 3.300 ce 3.121 reward -1.740 6592s
+  ep 3 step 5680/5736 loss 2.049 ce 1.110 reward -0.529 6613s
+  ep 3 step 5700/5736 loss 0.056 ce 0.000 reward 0.750 6635s
+  ep 3 step 5720/5736 loss -0.148 ce 0.000 reward 0.750 6657s
+=== epoch 3/3 avg loss 1.0639  6676s
+saved finetune/out/laya-browser-v12s
+== calibrate
+n=4957 acc=0.634  T=1: nll 3.288 mean conf 0.850  |  T=5.23: nll 1.397 mean conf 0.640
+wrote temperature 5.227105378439709 -> /home/ckl/projects/S/laya/finetune/out/laya-browser-v12s/rl_agent_config.json
+== eval v12s
+  return Agent(model_id_or_path, device=device, token=token, subfolder=subfolder, fast=fast, compile=compile,
+/home/ckl/projects/S/laya/finetune/out/laya-browser-v12s/: cases 5314  operation acc 0.727  target top-1 0.572 (n=4602, mean normalized rank 0.300)  19 ms/case
+   live      CLICK     n=1050 op acc 0.96  target top-1 0.84
+   live      DONE      n= 125 op acc 0.90
+   live      SELECT    n=  32 op acc 0.91  target top-1 0.84
+   live      TYPE_TEXT n= 117 op acc 0.79  target top-1 0.92
+   mind2web  CLICK     n=1084 op acc 0.90  target top-1 0.49
+   mind2web  SELECT    n= 126 op acc 0.57  target top-1 0.35
+   mind2web  TYPE_TEXT n= 200 op acc 0.73  target top-1 0.83
+   nnetnav   CLICK     n=1386 op acc 0.69  target top-1 0.25
+   nnetnav   DONE      n= 313 op acc 0.36
+   nnetnav   SCROLL_DOWN n= 224 op acc 0.06  target top-1 0.00
+   nnetnav   SCROLL_UP n=  50 op acc 0.00  target top-1 0.00
+   nnetnav   TYPE_TEXT n= 607 op acc 0.57  target top-1 0.88
+V12S_DONE

results/round3/v13s.log

@@ -0,0 +1,134 @@
+== rollout2 cases: 2057
+== build v13s
+operation label counts: {'TYPE_TEXT': 11052, 'CLICK': 16000, 'SELECT': 3090, 'DONE': 14636, 'SCROLL_DOWN': 4844, 'PRESS_ENTER': 2496, 'SCROLL_UP': 137}
+train items 72969 (op 52255, target 20714), eval cases 5314, skipped 0, seq len mean 764 max 1024
+== train v13s (1 epochs from v12s)
+=== epoch 1/1 avg loss 1.0192  2620s
+saved finetune/out/laya-browser-v13s
+== calibrate
+n=4957 acc=0.624  T=1: nll 3.180 mean conf 0.853  |  T=4.93: nll 1.415 mean conf 0.633
+wrote temperature 4.9277497256270095 -> /home/ckl/projects/S/laya/finetune/out/laya-browser-v13s/rl_agent_config.json
+== eval
+/home/ckl/projects/S/laya/finetune/out/laya-browser-v13s/: cases 5314  operation acc 0.712  target top-1 0.572 (n=4602, mean normalized rank 0.299)  20 ms/case
+   live      CLICK     n=1050 op acc 0.95  target top-1 0.83
+   live      DONE      n= 125 op acc 0.90
+   live      SELECT    n=  32 op acc 0.94  target top-1 0.81
+   live      TYPE_TEXT n= 117 op acc 0.85  target top-1 0.97
+   mind2web  CLICK     n=1084 op acc 0.87  target top-1 0.49
+   mind2web  SELECT    n= 126 op acc 0.58  target top-1 0.33
+   mind2web  TYPE_TEXT n= 200 op acc 0.76  target top-1 0.84
+   nnetnav   CLICK     n=1386 op acc 0.67  target top-1 0.25
+   nnetnav   DONE      n= 313 op acc 0.35
+   nnetnav   SCROLL_DOWN n= 224 op acc 0.13  target top-1 0.00
+   nnetnav   SCROLL_UP n=  50 op acc 0.00  target top-1 0.00
+   nnetnav   TYPE_TEXT n= 607 op acc 0.50  target top-1 0.88
+== suite A x3 (v13s, chunk 40)
+FAIL  wiki-einstein    steps= 3 status=done        5.9s  https://en.wikipedia.org/wiki/Main_Page
+FAIL  wiki-einstein    steps= 3 status=done        3.8s  https://en.wikipedia.org/wiki/Main_Page
+FAIL  wiki-einstein    steps= 3 status=blocked     3.5s  https://en.wikipedia.org/wiki/Main_Page
+PASS  wiki-search      steps= 2 status=done        4.2s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  wiki-search      steps= 2 status=done        4.3s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  wiki-search      steps= 2 status=done        4.5s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  hn-new           steps= 1 status=done        4.1s  https://news.ycombinator.com/newest
+PASS  hn-new           steps= 1 status=done        2.6s  https://news.ycombinator.com/newest
+PASS  hn-new           steps= 1 status=done        2.6s  https://news.ycombinator.com/newest
+PASS  hn-login-page    steps= 1 status=done        2.7s  https://news.ycombinator.com/login?goto=news
+PASS  hn-login-page    steps= 1 status=done        2.5s  https://news.ycombinator.com/login?goto=news
+PASS  hn-login-page    steps= 1 status=done        2.6s  https://news.ycombinator.com/login?goto=news
+PASS  gh-issues        steps= 1 status=done        7.8s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.4s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.2s  https://github.com/tile-ai/tilelang/issues
+PASS  py-downloads     steps= 1 status=done        4.1s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        2.4s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        3.0s  https://www.python.org/downloads/
+PASS  books-travel     steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-open-book  steps= 1 status=done        2.3s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  internet-dropdown steps= 1 status=done        3.4s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-dropdown steps= 1 status=done        2.2s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-dropdown steps= 1 status=done        2.2s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-checkbox steps= 1 status=done        2.5s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.2s  https://the-internet.herokuapp.com/checkboxes
+FAIL  books-page2      steps= 1 status=done        1.8s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  quotes-tag-love  steps= 1 status=done        3.9s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        2.8s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        2.8s  https://quotes.toscrape.com/tag/love/
+PASS  hn-past          steps= 1 status=done        3.1s  https://news.ycombinator.com/front
+PASS  hn-past          steps= 1 status=done        4.4s  https://news.ycombinator.com/front
+PASS  hn-past          steps= 1 status=done        3.0s  https://news.ycombinator.com/front
+PASS  web-search       steps= 2 status=done        5.2s  https://marginalia-search.com/search?query=tilelang+github&profile=cor
+PASS  web-search       steps= 2 status=done        3.8s  https://marginalia-search.com/search?query=tilelang+github&sst=SE-472b
+PASS  web-search       steps= 2 status=done        3.1s  https://marginalia-search.com/search?query=tilelang+github&profile=cor
+FAIL  arxiv-search     steps= 2 status=error:ValueError  14.3s  https://arxiv.org/
+FAIL  arxiv-search     steps= 2 status=error:ValueError  12.9s  https://arxiv.org/
+FAIL  arxiv-search     steps= 2 status=error:ValueError  13.4s  https://arxiv.org/
+FAIL  flights          steps=19 status=blocked     7.3s  https://www.google.com/travel/flights?tfs=CBwQARoOagwIAxIIL20vMDRqcGwa
+FAIL  flights          steps=20 status=ready       5.2s  https://www.google.com/travel/flights?tfs=CBwQARoOcgwIBBIIL20vMDJqNzEa
+FAIL  flights          steps=20 status=ready       5.7s  https://www.google.com/travel/flights?tfs=CBwQARoOcgwIBBIIL20vMDJqNzEa
+== 36/48 passed (75%, 3 run(s) per task) | median wall 3.1s
+   per task: wiki-einstein=0/3  wiki-search=3/3  hn-new=3/3  hn-login-page=3/3  gh-issues=3/3  py-downloads=3/3  books-travel=3/3  books-open-book=3/3  internet-dropdown=3/3  internet-checkbox=3/3  books-page2=0/3  quotes-tag-love=3/3  hn-past=3/3  web-search=3/3  arxiv-search=0/3  flights=0/3
+== suite B x3 (v13s, chunk 40)
+held-out check: OK, no suite-B domain appears in training data
+PASS  sqlite-download      steps= 1 status=done        5.7s  https://www.sqlite.org/download.html
+PASS  sqlite-download      steps= 1 status=done        1.9s  https://www.sqlite.org/download.html
+PASS  sqlite-download      steps= 1 status=done        1.9s  https://www.sqlite.org/download.html
+PASS  haskell-downloads    steps= 1 status=done        3.9s  https://www.haskell.org/downloads/
+PASS  haskell-downloads    steps= 1 status=done        2.8s  https://www.haskell.org/downloads/
+PASS  haskell-downloads    steps= 1 status=done        2.2s  https://www.haskell.org/downloads/
+PASS  zig-download         steps= 1 status=done        2.3s  https://ziglang.org/download/
+PASS  zig-download         steps= 1 status=done        2.0s  https://ziglang.org/download/
+PASS  zig-download         steps= 1 status=done        2.2s  https://ziglang.org/download/
+PASS  openbsd-faq          steps= 1 status=done        2.0s  https://www.openbsd.org/faq/index.html
+PASS  openbsd-faq          steps= 1 status=done        1.8s  https://www.openbsd.org/faq/index.html
+PASS  openbsd-faq          steps= 1 status=done        1.9s  https://www.openbsd.org/faq/index.html
+PASS  nginx-download       steps= 1 status=done        2.2s  https://nginx.org/en/download.html
+PASS  nginx-download       steps= 1 status=done        1.8s  https://nginx.org/en/download.html
+PASS  nginx-download       steps= 1 status=done        1.9s  https://nginx.org/en/download.html
+PASS  xkcd-archive         steps= 1 status=done        8.4s  https://xkcd.com/archive/
+PASS  xkcd-archive         steps= 1 status=done        4.3s  https://xkcd.com/archive/
+PASS  xkcd-archive         steps= 1 status=done        4.2s  https://xkcd.com/archive/
+PASS  elixir-install       steps= 1 status=done        3.9s  https://elixir-lang.org/install/
+PASS  elixir-install       steps= 1 status=done        2.2s  https://elixir-lang.org/install/
+PASS  elixir-install       steps= 1 status=done        2.0s  https://elixir-lang.org/install/
+PASS  prometheus-download  steps= 1 status=done        5.2s  https://prometheus.io/download/
+PASS  prometheus-download  steps= 1 status=done        3.1s  https://prometheus.io/download/
+PASS  prometheus-download  steps= 1 status=done        3.2s  https://prometheus.io/download/
+PASS  ocaml-install        steps= 1 status=done        4.4s  https://ocaml.org/install#linux_mac_bsd
+PASS  ocaml-install        steps= 1 status=done        3.2s  https://ocaml.org/install#linux_mac_bsd
+PASS  ocaml-install        steps= 1 status=done        3.3s  https://ocaml.org/install#linux_mac_bsd
+PASS  rosetta-quicksort    steps= 2 status=done        4.5s  https://rosettacode.org/wiki/Quicksort
+PASS  rosetta-quicksort    steps= 2 status=done        2.6s  https://rosettacode.org/wiki/Sorting_algorithms/Quicksort
+PASS  rosetta-quicksort    steps= 2 status=done        2.6s  https://rosettacode.org/wiki/Sorting_algorithms/Quicksort
+PASS  arch-man-grep        steps= 2 status=done        4.4s  https://man.archlinux.org/man/grep.1
+PASS  arch-man-grep        steps= 2 status=done        2.5s  https://man.archlinux.org/man/grep.1
+PASS  arch-man-grep        steps= 2 status=done        2.5s  https://man.archlinux.org/man/grep.1
+PASS  debian-man-tar       steps= 3 status=done        6.3s  https://manpages.debian.org/trixie/tar/tar.1.en.html
+PASS  debian-man-tar       steps= 3 status=done        2.6s  https://manpages.debian.org/trixie/tar/tar.1.en.html
+PASS  debian-man-tar       steps= 3 status=done        2.5s  https://manpages.debian.org/trixie/tar/tar.1.en.html
+PASS  rfc-http-semantics   steps= 2 status=done       15.8s  https://www.rfc-editor.org/search/?q=HTTP+semantics
+PASS  rfc-http-semantics   steps= 2 status=done        2.7s  https://www.rfc-editor.org/search/?q=HTTP+semantics
+PASS  rfc-http-semantics   steps= 2 status=done        2.7s  https://www.rfc-editor.org/search/?q=HTTP+semantics
+PASS  nixos-ripgrep        steps= 2 status=done        4.8s  https://search.nixos.org/packages?channel=26.05&query=ripgrep
+PASS  nixos-ripgrep        steps= 2 status=done        2.7s  https://search.nixos.org/packages?channel=26.05&query=ripgrep
+PASS  nixos-ripgrep        steps= 2 status=done        3.2s  https://search.nixos.org/packages?channel=26.05&query=ripgrep
+PASS  ocaml-pkg-yojson     steps= 2 status=done        3.9s  https://ocaml.org/packages/search?q=yojson
+PASS  ocaml-pkg-yojson     steps= 2 status=done        2.6s  https://ocaml.org/packages/search?q=yojson
+PASS  ocaml-pkg-yojson     steps= 2 status=done        2.9s  https://ocaml.org/packages/search?q=yojson
+PASS  shop-search-jacket   steps= 2 status=done        4.9s  https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product
+PASS  shop-search-jacket   steps= 2 status=done        2.9s  https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product
+PASS  shop-search-jacket   steps= 2 status=done        2.8s  https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product
+PASS  shop-page2           steps= 2 status=done        3.4s  https://www.scrapingcourse.com/ecommerce/page/2/
+PASS  shop-page2           steps= 2 status=done        4.2s  https://www.scrapingcourse.com/ecommerce/page/2/
+FAIL  shop-page2           steps= 3 status=blocked     2.9s  https://www.scrapingcourse.com/ecommerce/
+FAIL  shop-sort-price      steps= 2 status=done        6.2s  https://www.scrapingcourse.com/ecommerce/page/2/
+FAIL  shop-sort-price      steps= 2 status=done        3.4s  https://www.scrapingcourse.com/ecommerce/page/2/
+FAIL  shop-sort-price      steps= 2 status=done        3.4s  https://www.scrapingcourse.com/ecommerce/page/2/
+== 50/54 passed (93%, 3 run(s) per task) | median wall 2.9s
+   per task: sqlite-download=3/3  haskell-downloads=3/3  zig-download=3/3  openbsd-faq=3/3  nginx-download=3/3  xkcd-archive=3/3  elixir-install=3/3  prometheus-download=3/3  ocaml-install=3/3  rosetta-quicksort=3/3  arch-man-grep=3/3  debian-man-tar=3/3  rfc-http-semantics=3/3  nixos-ripgrep=3/3  ocaml-pkg-yojson=3/3  shop-search-jacket=3/3  shop-page2=2/3  shop-sort-price=0/3
+V13S_DONE

results/round3/v14s.log

@@ -0,0 +1,134 @@
+== rollout2 cases: 2343
+== build v14s
+operation label counts: {'TYPE_TEXT': 11106, 'CLICK': 16255, 'SELECT': 3198, 'DONE': 15560, 'SCROLL_DOWN': 5480, 'PRESS_ENTER': 2496, 'SCROLL_UP': 137}
+train items 75255 (op 54232, target 21023), eval cases 5314, skipped 0, seq len mean 765 max 1024
+== train v14s (1 epochs from v13s)
+=== epoch 1/1 avg loss 0.9916  2721s
+saved finetune/out/laya-browser-v14s
+== calibrate
+n=4957 acc=0.636  T=1: nll 3.337 mean conf 0.868  |  T=5.03: nll 1.398 mean conf 0.647
+wrote temperature 5.025579877567767 -> /home/ckl/projects/S/laya/finetune/out/laya-browser-v14s/rl_agent_config.json
+== eval
+/home/ckl/projects/S/laya/finetune/out/laya-browser-v14s/: cases 5314  operation acc 0.724  target top-1 0.575 (n=4602, mean normalized rank 0.298)  19 ms/case
+   live      CLICK     n=1050 op acc 0.96  target top-1 0.85
+   live      DONE      n= 125 op acc 0.87
+   live      SELECT    n=  32 op acc 0.88  target top-1 0.84
+   live      TYPE_TEXT n= 117 op acc 0.84  target top-1 0.96
+   mind2web  CLICK     n=1084 op acc 0.89  target top-1 0.48
+   mind2web  SELECT    n= 126 op acc 0.52  target top-1 0.33
+   mind2web  TYPE_TEXT n= 200 op acc 0.76  target top-1 0.84
+   nnetnav   CLICK     n=1386 op acc 0.71  target top-1 0.25
+   nnetnav   DONE      n= 313 op acc 0.31
+   nnetnav   SCROLL_DOWN n= 224 op acc 0.08  target top-1 0.00
+   nnetnav   SCROLL_UP n=  50 op acc 0.00  target top-1 0.00
+   nnetnav   TYPE_TEXT n= 607 op acc 0.55  target top-1 0.88
+== suite A x3 (v14s, chunk 60)
+FAIL  wiki-einstein    steps= 2 status=done        7.2s  https://en.wikipedia.org/wiki/Wikipedia
+PASS  wiki-einstein    steps= 2 status=done        4.9s  https://en.wikipedia.org/wiki/Albert_Einstein
+FAIL  wiki-einstein    steps= 2 status=done        5.9s  https://en.wikipedia.org/wiki/Wikipedia
+PASS  wiki-search      steps= 2 status=done        5.4s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  wiki-search      steps= 2 status=done        5.0s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  wiki-search      steps= 2 status=done        4.7s  https://en.wikipedia.org/wiki/Python_(programming_language)
+PASS  hn-new           steps= 1 status=done        3.6s  https://news.ycombinator.com/newest
+PASS  hn-new           steps= 1 status=done        2.5s  https://news.ycombinator.com/newest
+PASS  hn-new           steps= 1 status=done        2.5s  https://news.ycombinator.com/newest
+PASS  hn-login-page    steps= 1 status=done        2.7s  https://news.ycombinator.com/login?goto=news
+PASS  hn-login-page    steps= 1 status=done        2.5s  https://news.ycombinator.com/login?goto=news
+PASS  hn-login-page    steps= 1 status=done        2.4s  https://news.ycombinator.com/login?goto=news
+PASS  gh-issues        steps= 1 status=done        6.8s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.6s  https://github.com/tile-ai/tilelang/issues
+PASS  gh-issues        steps= 1 status=done        3.8s  https://github.com/tile-ai/tilelang/issues
+PASS  py-downloads     steps= 1 status=done        5.2s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        3.6s  https://www.python.org/downloads/
+PASS  py-downloads     steps= 1 status=done        2.6s  https://www.python.org/downloads/
+PASS  books-travel     steps= 1 status=done        2.1s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-travel     steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
+PASS  books-open-book  steps= 1 status=done        2.2s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.8s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  books-open-book  steps= 1 status=done        1.9s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  internet-dropdown steps= 1 status=done        3.4s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-dropdown steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-dropdown steps= 1 status=done        2.2s  https://the-internet.herokuapp.com/dropdown
+PASS  internet-checkbox steps= 1 status=done        2.6s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/checkboxes
+PASS  internet-checkbox steps= 1 status=done        2.1s  https://the-internet.herokuapp.com/checkboxes
+FAIL  books-page2      steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+FAIL  books-page2      steps= 1 status=done        2.0s  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
+PASS  quotes-tag-love  steps= 1 status=done        3.8s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        3.0s  https://quotes.toscrape.com/tag/love/
+PASS  quotes-tag-love  steps= 1 status=done        3.0s  https://quotes.toscrape.com/tag/love/
+PASS  hn-past          steps= 1 status=done        3.0s  https://news.ycombinator.com/front
+PASS  hn-past          steps= 1 status=done        2.6s  https://news.ycombinator.com/front
+PASS  hn-past          steps= 1 status=done        2.6s  https://news.ycombinator.com/front
+PASS  web-search       steps= 2 status=done        6.7s  https://marginalia-search.com/search?query=tilelang+github&sst=SE-1e14
+PASS  web-search       steps= 2 status=done        4.6s  https://marginalia-search.com/search?query=tilelang+github&profile=cor
+PASS  web-search       steps= 2 status=done        5.2s  https://marginalia-search.com/search?query=tilelang+github&profile=cor
+FAIL  arxiv-search     steps= 4 status=done        6.2s  https://arxiv.org/search/
+PASS  arxiv-search     steps= 8 status=done        3.8s  https://arxiv.org/search/?query=flash+attention&searchtype=all&source=
+PASS  arxiv-search     steps=14 status=done       10.1s  https://arxiv.org/search/?query=flash+attention&searchtype=all&abstrac
+FAIL  flights          steps=20 status=ready       7.1s  https://www.google.com/travel/flights?tfs=CBwQARoOcgwIAxIIL20vMDRqcGwa
+FAIL  flights          steps=20 status=ready       5.5s  https://www.google.com/travel/flights?tfs=CBwQARoOcgwIAxIIL20vMDRqcGwa
+FAIL  flights          steps=20 status=ready       5.7s  https://www.google.com/travel/flights?tfs=CBwQARoOcgwIAxIIL20vMDRqcGwa
+== 39/48 passed (81%, 3 run(s) per task) | median wall 3.0s
+   per task: wiki-einstein=1/3  wiki-search=3/3  hn-new=3/3  hn-login-page=3/3  gh-issues=3/3  py-downloads=3/3  books-travel=3/3  books-open-book=3/3  internet-dropdown=3/3  internet-checkbox=3/3  books-page2=0/3  quotes-tag-love=3/3  hn-past=3/3  web-search=3/3  arxiv-search=2/3  flights=0/3
+== suite B x3 (v14s, chunk 60)
+held-out check: OK, no suite-B domain appears in training data
+PASS  sqlite-download      steps= 1 status=done        6.4s  https://www.sqlite.org/download.html
+PASS  sqlite-download      steps= 1 status=done        2.0s  https://www.sqlite.org/download.html
+PASS  sqlite-download      steps= 1 status=done        2.0s  https://www.sqlite.org/download.html
+PASS  haskell-downloads    steps= 1 status=done        3.5s  https://www.haskell.org/downloads/
+PASS  haskell-downloads    steps= 1 status=done        2.1s  https://www.haskell.org/downloads/
+PASS  haskell-downloads    steps= 1 status=done        2.2s  https://www.haskell.org/downloads/
+PASS  zig-download         steps= 1 status=done        5.9s  https://ziglang.org/download/
+PASS  zig-download         steps= 1 status=done        2.0s  https://ziglang.org/download/
+PASS  zig-download         steps= 1 status=done        2.2s  https://ziglang.org/download/
+PASS  openbsd-faq          steps= 1 status=done        2.0s  https://www.openbsd.org/faq/index.html
+PASS  openbsd-faq          steps= 1 status=done        2.0s  https://www.openbsd.org/faq/index.html
+PASS  openbsd-faq          steps= 1 status=done        2.0s  https://www.openbsd.org/faq/index.html
+PASS  nginx-download       steps= 1 status=done        2.2s  https://nginx.org/en/download.html
+PASS  nginx-download       steps= 1 status=done        1.8s  https://nginx.org/en/download.html
+PASS  nginx-download       steps= 1 status=done        1.9s  https://nginx.org/en/download.html
+PASS  xkcd-archive         steps= 1 status=done        7.5s  https://xkcd.com/archive/
+PASS  xkcd-archive         steps= 1 status=done        4.6s  https://xkcd.com/archive/
+PASS  xkcd-archive         steps= 1 status=done        4.0s  https://xkcd.com/archive/
+PASS  elixir-install       steps= 1 status=done        4.4s  https://elixir-lang.org/install/
+PASS  elixir-install       steps= 1 status=done        2.2s  https://elixir-lang.org/install/
+PASS  elixir-install       steps= 1 status=done        2.0s  https://elixir-lang.org/install/
+PASS  prometheus-download  steps= 1 status=done        9.3s  https://prometheus.io/download/
+PASS  prometheus-download  steps= 1 status=done        3.2s  https://prometheus.io/download/
+PASS  prometheus-download  steps= 1 status=done        3.0s  https://prometheus.io/download/
+PASS  ocaml-install        steps= 2 status=done        4.9s  https://ocaml.org/docs/installing-ocaml
+PASS  ocaml-install        steps= 2 status=done        4.1s  https://ocaml.org/docs/installing-ocaml
+PASS  ocaml-install        steps= 2 status=done        4.1s  https://ocaml.org/docs/installing-ocaml
+PASS  rosetta-quicksort    steps= 2 status=done        4.0s  https://rosettacode.org/wiki/Quicksort
+PASS  rosetta-quicksort    steps= 2 status=done        2.7s  https://rosettacode.org/wiki/Quicksort
+PASS  rosetta-quicksort    steps= 2 status=done        2.7s  https://rosettacode.org/wiki/Sorting_algorithms/Quicksort
+PASS  arch-man-grep        steps= 2 status=done        4.0s  https://man.archlinux.org/man/grep.1
+PASS  arch-man-grep        steps= 2 status=done        2.6s  https://man.archlinux.org/man/grep.1
+PASS  arch-man-grep        steps= 2 status=done        2.6s  https://man.archlinux.org/man/grep.1
+PASS  debian-man-tar       steps= 2 status=done        5.1s  https://manpages.debian.org/trixie/tar/tar.1.en.html
+PASS  debian-man-tar       steps= 2 status=done        2.1s  https://manpages.debian.org/trixie/tar/tar.1.en.html
+PASS  debian-man-tar       steps= 2 status=done        1.9s  https://manpages.debian.org/trixie/tar/tar.1.en.html
+PASS  rfc-http-semantics   steps= 2 status=done        5.7s  https://www.rfc-editor.org/search/?q=HTTP+semantics
+PASS  rfc-http-semantics   steps= 2 status=done        3.7s  https://www.rfc-editor.org/search/?q=HTTP+semantics
+PASS  rfc-http-semantics   steps= 2 status=done        3.1s  https://www.rfc-editor.org/search/?q=HTTP+semantics
+PASS  nixos-ripgrep        steps= 2 status=done        4.0s  https://search.nixos.org/packages?channel=26.05&query=ripgrep
+PASS  nixos-ripgrep        steps= 2 status=done        2.8s  https://search.nixos.org/packages?channel=26.05&query=ripgrep
+PASS  nixos-ripgrep        steps= 2 status=done        2.8s  https://search.nixos.org/packages?channel=26.05&query=ripgrep
+PASS  ocaml-pkg-yojson     steps= 2 status=done        3.7s  https://ocaml.org/packages/search?q=yojson
+PASS  ocaml-pkg-yojson     steps= 2 status=done        2.6s  https://ocaml.org/packages/search?q=yojson
+PASS  ocaml-pkg-yojson     steps= 2 status=done        2.8s  https://ocaml.org/packages/search?q=yojson
+PASS  shop-search-jacket   steps= 2 status=done        5.2s  https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product
+PASS  shop-search-jacket   steps= 2 status=done        2.9s  https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product
+PASS  shop-search-jacket   steps= 2 status=done        2.9s  https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product
+PASS  shop-page2           steps= 1 status=done        3.2s  https://www.scrapingcourse.com/ecommerce/page/2/
+PASS  shop-page2           steps= 1 status=done        3.6s  https://www.scrapingcourse.com/ecommerce/page/2/
+PASS  shop-page2           steps= 1 status=done        3.5s  https://www.scrapingcourse.com/ecommerce/page/2/
+PASS  shop-sort-price      steps= 1 status=done        3.9s  https://www.scrapingcourse.com/ecommerce/?orderby=price
+PASS  shop-sort-price      steps= 1 status=done        3.3s  https://www.scrapingcourse.com/ecommerce/?orderby=price
+PASS  shop-sort-price      steps= 1 status=done        3.4s  https://www.scrapingcourse.com/ecommerce/?orderby=price
+== 54/54 passed (100%, 3 run(s) per task) | median wall 3.1s
+   per task: sqlite-download=3/3  haskell-downloads=3/3  zig-download=3/3  openbsd-faq=3/3  nginx-download=3/3  xkcd-archive=3/3  elixir-install=3/3  prometheus-download=3/3  ocaml-install=3/3  rosetta-quicksort=3/3  arch-man-grep=3/3  debian-man-tar=3/3  rfc-http-semantics=3/3  nixos-ripgrep=3/3  ocaml-pkg-yojson=3/3  shop-search-jacket=3/3  shop-page2=3/3  shop-sort-price=3/3
+V14S_DONE

v14s/encoder/config.json

@@ -0,0 +1,79 @@
+{
+  "architectures": [
+    "ModernBertForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 1,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 1,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "mask_token_id": 4,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 0,
+  "position_embedding_type": "sans_pos",
+  "repad_logits_with_grad": false,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 1,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "vocab_size": 256000
+}

v14s/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4e8b144c47e7fa97b128b4447e29cb0540c156fa50b5958200bb15b6a9c1801
+size 643835524