Suite C — held-out, multi-step real-website evaluation
apps/browser_suite_c.py runs the jev-ultrafast agent against 27 live tasks on domains that appear in no training
source (finetune/out/train_domains.json, finetune/sites*.txt) and not in suites A/B (checked at start-up, both on the
exact host and on the registrable base domain; prints LEAK otherwise). 19 tasks need 4+ actions.
python apps/browser_suite_c.py [name-filter] REPEATS=3 SUITE_OUT=/tmp/suite_c.json
python apps/suite_c_scripts.py [name-filter] model-free scripted solutions; REPEATS=3 SHARD=i/n
Validation protocol (no model, no GPU): every task has a scripted solution in apps/suite_c_scripts.py
(observe → pick the action by label → act, through jev's Browser). A task is kept only if the script passed its check
3/3 on separate runs and the check failed on the start page (a do-nothing run cannot pass). Checks read the final
URL/title/text or the observed form state (checked radios/checkboxes, current <select> value), never the model.
Because an observation only covers the viewport, the judge (final_state) scrolls to the top and then screen by screen,
merging the actions and visible text of the whole page before the check runs, so a check never depends on where the
agent left the scroll position.
Tasks
Steps = number of browser actions of the scripted solution (a scroll counts as one action).
| # | name | site | goal | scripted steps | check verifies |
|---|---|---|---|---|---|
| 1 | met-sunflowers | metmuseum.org collection search | search "sunflowers", tick "Has image", sort by date oldest first | 4: fill, Enter, checkbox, select | URL q=sunflowers&showOnly=withImage&sortBy=Date (newest gives DateDesc) |
| 2 | nuget-serilog-tool | nuget.org | search "serilog", package type ".NET tool", sort by downloads | 4: fill, Enter, radio, select | URL q=serilog, packagetype=dotnettool, sortby=totalDownloads-desc |
| 3 | alpine-curl-filter | pkgs.alpinelinux.org | package name curl, branch v3.20, repo main, arch aarch64 | 5: fill, 3× select, Enter | URL name=curl&branch=v3.20&repo=main&arch=aarch64 |
| 4 | freesound-rain-cc0 | freesound.org | search "rain", license CC0 facet, category Soundscapes facet | 4: fill, Enter, 2× facet link | URL q=rain and f=license:"Creative Commons 0" category:"Soundscapes" |
| 5 | ats-shampoo-haircare | automationteststore.com | search "shampoo", category Hair Care, "search in descriptions", Search, sort price high→low | 6: fill, Enter, select, checkbox, button, select | URL keyword=shampoo, category_id contains 52, description=1, sort=p.price-DESC |
| 6 | bnf-hugo-printed-p2 | catalogue.bnf.fr | search "victor hugo", facet "Texte imprimé et livre numérique", page 2 | 4: fill, submit, facet, "Page suivante" | URL motRecherche=victor hugo, listeAffinages has FacNatDoc_a, pageEnCours=2 |
| 7 | vsm-python-installs | marketplace.visualstudio.com | search "python", Sort By menu → Installs | 4: fill, Enter, menu, menuitem | URL term=python&sortBy=Installs |
| 8 | wp-cache-commercial-redis | wordpress.org/plugins | search "cache", "Commercial" filter, open Redis Object Cache | 4: fill, Enter, filter, link | URL /plugins/redis-cache |
| 9 | todomvc-active | demo.playwright.dev/todomvc | add "buy milk" and "walk the dog", show Active | 5: fill, Enter, fill, Enter, link | URL ends #/active, both todos in text, "2 items left"; localStorage is cleared before each run |
| 10 | setlist-radiohead-uk | setlist.fm | search "radiohead", Artist → Radiohead, Country → United Kingdom | 4: fill, Enter, 2× select | URL query=radiohead&artist=bd6bd12&country=gb |
| 11 | jetbrains-rust-free | plugins.jetbrains.com | search "rust", "free" filter, open the Rust plugin by JetBrains | 4: fill, Enter, button, link | URL /plugin/22407-rust |
| 12 | luarocks-rapidjson | luarocks.org | search "json", tick "Include non-root", Search, open rapidjson | 5: fill, Enter, checkbox, button, link | URL /modules/xpol/rapidjson |
| 13 | letcode-dropdowns | letcode.in/dropdowns | four <select>s: Apple, Batman, Swift, India (the last one is below the fold) |
5: 3× select, scroll, select | observed current_value of the four selects |
| 14 | letcode-radio | letcode.in/radio | radios Foo + Going, tick "I agree…", untick "Remember me" | 4: 2× radio, 2× checkbox | observed checked state of the four controls |
| 15 | clojars-ring-page3 | clojars.org | search "ring", go to page 3 (pagination is below the fold) | 7: fill, Enter, 4× scroll, link "3" | URL q=ring&page=3 |
| 16 | fred-unemployment-pop | fred.stlouisfed.org | search "unemployment rate", sort by popularity | 4: fill, Enter, sort menu, "Popularity" | URL st=unemployment rate and ob=p |
| 17 | modrinth-sodium | modrinth.com/mods | search "sodium", game version 1.21.11, sort by downloads | 5: fill, Enter, version button, sort menu, option | URL q=sodium&v=1.21.11&s=downloads |
| 18 | tvmaze-friends-episodes | tvmaze.com | search "friends", open show, Episodes tab | 4: fill, Enter, link, tab link | URL /shows/431/friends/episodes |
| 19 | qaclickjet-form | rahulshettyacademy.com/dropdownsPractise | radio Round Trip, tick Senior Citizen, currency USD, type "India" in the country box | 4: radio, checkbox, select, fill | observed radio/checkbox/select state and the country field's value |
| 20 | cocktail-margarita | thecocktaildb.com | search "margarita", open Margarita | 3 | URL /drink/11007 |
| 21 | mealdb-arrabiata | themealdb.com | search "arrabiata", open Spicy Arrabiata Penne | 3 | URL /meal/52771 |
| 22 | gentoo-openrc-talk | wiki.gentoo.org | search "OpenRC" (lands on the article), open Discussion | 3 | URL contains Talk:OpenRC |
| 23 | webkit-css-bugs | bugs.webkit.org | Browse → WebKit product → CSS component | 3 | URL buglist.cgi?...product=WebKit&component=CSS |
| 24 | govdata-wetter-energie | govdata.de | search "Wetter", category Energie | 3 | URL q=Wetter&groups=ener |
| 25 | fedora-vim-common | packages.fedoraproject.org | search "vim", open vim-common | 3 | URL /pkgs/vim/vim-common |
| 26 | racket-argo | pkgs.racket-lang.org | search "json", open argo | 3 | URL ends /package/argo |
| 27 | rdrr-ggplot | rdrr.io | search "ggplot" | 2 | URL /search?q=ggplot |
Notes on the harder ones:
- Several targets are only observable after a scroll (Clojars pagination) or inside a menu that must be opened first (VS Marketplace "Sort By", FRED "Sort by Relevance", Modrinth "Sort by"): the agent must open the menu, then pick.
- letcode-dropdowns / letcode-radio / qaclickjet-form have no URL change at all; they are judged from the observed form state, so they cannot pass by navigating anywhere.
- todomvc-active depends on persisted state, so the suite (and the script runner) clears the site's localStorage first.
Sites rejected and why
Blocked / bot walls for headless Chromium (Cloudflare "Just a moment", Akamai "Access Denied", 403, captcha): congress.gov, bls.gov, catalog.hathitrust.org, dp.la (CloudFront error), philpapers.org, zbmath.org, tvtropes.org, terraria.wiki.gg, mobygames.com, anaconda.org, hansard.parliament.uk, latlong.net, chessgames.com (403), data.humdata.org (403), pkgs.org ("Human Verification" page after search), imslp.org (its search redirects to Google → captcha), hal.science and openwrt.org ("Oh noes!" error page), search.scielo.org (never finishes "Establishing a secure connection").
In the training data / sites lists (LEAK by domain or base domain), so unusable even though they would be good: data.gov, data.gov.uk, data.europa.eu, data.gouv.fr, open.canada.ca, data.gov.au, openlibrary.org, archive.org, gutenberg.org, musicbrainz.org, discogs.com, boardgamegeek.com, pypi/crates/npm/hex/pub.dev/hackage/metacpan, packages.debian.org, packages.ubuntu.com, aur.archlinux.org, repology.org, addons.mozilla.org, readthedocs.org, bugzilla.mozilla.org, bugs.launchpad.net, bugs.debian.org, bugzilla.kernel.org, gitlab.com, codeberg.org, sourceforge.net, stackoverflow.com, wikidata/wiktionary/wikivoyage/wikiquote/commons, wikihow.com, lichess.org, geonames.org, timeanddate.com, zenodo.org, figshare.com, dataverse.harvard.edu, dblp.org, doaj.org, core.ac.uk, biorxiv.org, inspirehep.net, eric.ed.gov, ncbi/pubmed, worldbank.org, who.int, ec.europa.eu, usa.gov/nist/cdc/fda/epa/nps.gov, ocw.mit.edu, plato.stanford.edu, selenium.dev, w3schools.com, demoqa.com, the-internet.herokuapp.com, saucedemo.com, demoblaze.com, parabank, practice.expandtesting.com, automationexercise.com, opencart demo, nopcommerce demo, orangehrm demo, magento softwaretestingboard, juice-shop, webscraper.io, scrapethissite.com, toscrape.com, scrapingcourse.com (suite B), and all the airline/hotel/travel sites in finetune/sites.txt.
Reachable but rejected for the agent's action space or for flakiness:
- bugs.kde.org simple search: the Product
<select>has ~235 options, so the observation is truncated at 250 actions and the "Words" text field is never offered. - gitea.com explore: the custom Sort dropdown's options do not navigate when clicked (URL stays
sort=recentupdate). - rawg.io: Enter in the search box does not navigate (
rawg.io/?); results are only rendered in-page. - weather.gov: neither the "Go" nor the "Get Weather" form submits from a synthetic click (no navigation after 1.5 s).
- data.cityofnewyork.us: the search box is a combobox, so no Enter action is offered, and clicking the search button drops the query when a filter is applied afterwards.
- uniprot.org: the facet sidebar ("Reviewed") is not rendered in the 1120×780 viewport.
- bstackdemo.com: vendor checkboxes and "Add to cart" buttons are custom elements that are not observed as actions.
- cms.demo.katalon.com: search only returns blog posts; the shop's sorting
<select>is a selectWoo widget with no observable options. - rahulshettyacademy autocomplete: jQuery-UI suggestions have no
role=option, so they cannot be clicked (the form task only types into that field and judges its value); its discount checkboxes are mutually exclusive (ticking one clears the others), so the task asks for a single one. - testpages.eviltester.com basic form, demo.guru99.com register form: every field is an unlabeled
textbox, so the goal cannot name the fields; guru99 also needs a password field, which jev hides. - letcode.in/forms: only the Email field remains observable after scrolling (layout overlaps).
- oxylabs sandbox: the platform sub-category chips ("switch", "wii") are not links.
- scryfall.com: search box is an unlabeled textbox; sunrise-sunset.org: the Search button does not navigate.
- pokemondb.net Pokédex filter (Name + Type, client-side): worked 3/4 runs, but on some first loads the whole page is covered by an overlay so no control is observable at all → dropped as flaky.
- issues.jenkins.io: the "Search for issues" link is only present in a collapsed menu on some loads.
- ultimateqa.com forms: only name + message + submit (too small); openalex.org (API-only page); melpa/opencollective: no searchable form on the landing page.
- wikipedia-like wiki farms (miraheze) and mediawiki search are already heavily represented in training; only one MediaWiki task (Gentoo wiki talk page) is kept.
Validation results
Filled in from python apps/suite_c_scripts.py with REPEATS=3 (see the bottom of this file).
Final run of REPEATS=3 python apps/suite_c_scripts.py (2026-09-26, four shards in parallel, final judge code):
27/27 tasks passed 3/3, every start-page check failed (a do-nothing run cannot pass).
| task | scripted runs | steps | task | scripted runs | steps |
|---|---|---|---|---|---|
| met-sunflowers | 3/3 | 4 | nuget-serilog-tool | 3/3 | 4 |
| alpine-curl-filter | 3/3 | 5 | freesound-rain-cc0 | 3/3 | 4 |
| ats-shampoo-haircare | 3/3 | 6 | bnf-hugo-printed-p2 | 3/3 | 4 |
| vsm-python-installs | 3/3 | 4 | wp-cache-commercial-redis | 3/3 | 4 |
| todomvc-active | 3/3 | 5 | setlist-radiohead-uk | 3/3 | 4 |
| jetbrains-rust-free | 3/3 | 4 | luarocks-rapidjson | 3/3 | 5 |
| letcode-dropdowns | 3/3 | 5 | letcode-radio | 3/3 | 4 |
| clojars-ring-page3 | 3/3 | 7 | fred-unemployment-pop | 3/3 | 4 |
| modrinth-sodium | 3/3 | 5 | tvmaze-friends-episodes | 3/3 | 4 |
| qaclickjet-form | 3/3 | 4 | cocktail-margarita | 3/3 | 3 |
| mealdb-arrabiata | 3/3 | 3 | gentoo-openrc-talk | 3/3 | 3 |
| webkit-css-bugs | 3/3 | 3 | govdata-wetter-energie | 3/3 | 3 |
| fedora-vim-common | 3/3 | 3 | racket-argo | 3/3 | 3 |
| rdrr-ggplot | 3/3 | 2 |
19 tasks need 4+ actions (the first 19 in TASKS, also reported separately by the suite as "multi-step tasks").
The agent suite itself (apps/browser_suite_c.py) has not been run yet: it needs the laya systemone server (:8791) and
sglang (:30000), and the GPU belonged to another session while this suite was built.