# Suite C — held-out, multi-step real-website evaluation

`apps/browser_suite_c.py` runs the jev-ultrafast agent against 27 live tasks on domains that appear in **no** training
source (`finetune/out/train_domains.json`, `finetune/sites*.txt`) and not in suites A/B (checked at start-up, both on the
exact host and on the registrable base domain; prints `LEAK` otherwise).  19 tasks need 4+ actions.

    python apps/browser_suite_c.py [name-filter]          REPEATS=3 SUITE_OUT=/tmp/suite_c.json
    python apps/suite_c_scripts.py [name-filter]          model-free scripted solutions; REPEATS=3 SHARD=i/n

Validation protocol (no model, no GPU): every task has a scripted solution in `apps/suite_c_scripts.py`
(observe → pick the action by label → act, through jev's `Browser`).  A task is kept only if the script passed its check
3/3 on separate runs **and** the check failed on the start page (a do-nothing run cannot pass).  Checks read the final
URL/title/text or the observed form state (checked radios/checkboxes, current `<select>` value), never the model.
Because an observation only covers the viewport, the judge (`final_state`) scrolls to the top and then screen by screen,
merging the actions and visible text of the whole page before the check runs, so a check never depends on where the
agent left the scroll position.

## Tasks

Steps = number of browser actions of the scripted solution (a scroll counts as one action).

| # | name | site | goal | scripted steps | check verifies |
|---|------|------|------|:--:|----------------|
| 1 | met-sunflowers | metmuseum.org collection search | search "sunflowers", tick "Has image", sort by date oldest first | 4: fill, Enter, checkbox, select | URL `q=sunflowers&showOnly=withImage&sortBy=Date` (newest gives `DateDesc`) |
| 2 | nuget-serilog-tool | nuget.org | search "serilog", package type ".NET tool", sort by downloads | 4: fill, Enter, radio, select | URL `q=serilog`, `packagetype=dotnettool`, `sortby=totalDownloads-desc` |
| 3 | alpine-curl-filter | pkgs.alpinelinux.org | package name curl, branch v3.20, repo main, arch aarch64 | 5: fill, 3× select, Enter | URL `name=curl&branch=v3.20&repo=main&arch=aarch64` |
| 4 | freesound-rain-cc0 | freesound.org | search "rain", license CC0 facet, category Soundscapes facet | 4: fill, Enter, 2× facet link | URL `q=rain` and `f=license:"Creative Commons 0" category:"Soundscapes"` |
| 5 | ats-shampoo-haircare | automationteststore.com | search "shampoo", category Hair Care, "search in descriptions", Search, sort price high→low | 6: fill, Enter, select, checkbox, button, select | URL `keyword=shampoo`, `category_id` contains 52, `description=1`, `sort=p.price-DESC` |
| 6 | bnf-hugo-printed-p2 | catalogue.bnf.fr | search "victor hugo", facet "Texte imprimé et livre numérique", page 2 | 4: fill, submit, facet, "Page suivante" | URL `motRecherche=victor hugo`, `listeAffinages` has `FacNatDoc_a`, `pageEnCours=2` |
| 7 | vsm-python-installs | marketplace.visualstudio.com | search "python", Sort By menu → Installs | 4: fill, Enter, menu, menuitem | URL `term=python&sortBy=Installs` |
| 8 | wp-cache-commercial-redis | wordpress.org/plugins | search "cache", "Commercial" filter, open Redis Object Cache | 4: fill, Enter, filter, link | URL `/plugins/redis-cache` |
| 9 | todomvc-active | demo.playwright.dev/todomvc | add "buy milk" and "walk the dog", show Active | 5: fill, Enter, fill, Enter, link | URL ends `#/active`, both todos in text, "2 items left"; localStorage is cleared before each run |
| 10 | setlist-radiohead-uk | setlist.fm | search "radiohead", Artist → Radiohead, Country → United Kingdom | 4: fill, Enter, 2× select | URL `query=radiohead&artist=bd6bd12&country=gb` |
| 11 | jetbrains-rust-free | plugins.jetbrains.com | search "rust", "free" filter, open the Rust plugin by JetBrains | 4: fill, Enter, button, link | URL `/plugin/22407-rust` |
| 12 | luarocks-rapidjson | luarocks.org | search "json", tick "Include non-root", Search, open rapidjson | 5: fill, Enter, checkbox, button, link | URL `/modules/xpol/rapidjson` |
| 13 | letcode-dropdowns | letcode.in/dropdowns | four `<select>`s: Apple, Batman, Swift, India (the last one is below the fold) | 5: 3× select, scroll, select | observed `current_value` of the four selects |
| 14 | letcode-radio | letcode.in/radio | radios Foo + Going, tick "I agree…", untick "Remember me" | 4: 2× radio, 2× checkbox | observed `checked` state of the four controls |
| 15 | clojars-ring-page3 | clojars.org | search "ring", go to page 3 (pagination is below the fold) | 7: fill, Enter, 4× scroll, link "3" | URL `q=ring&page=3` |
| 16 | fred-unemployment-pop | fred.stlouisfed.org | search "unemployment rate", sort by popularity | 4: fill, Enter, sort menu, "Popularity" | URL `st=unemployment rate` and `ob=p` |
| 17 | modrinth-sodium | modrinth.com/mods | search "sodium", game version 1.21.11, sort by downloads | 5: fill, Enter, version button, sort menu, option | URL `q=sodium&v=1.21.11&s=downloads` |
| 18 | tvmaze-friends-episodes | tvmaze.com | search "friends", open show, Episodes tab | 4: fill, Enter, link, tab link | URL `/shows/431/friends/episodes` |
| 19 | qaclickjet-form | rahulshettyacademy.com/dropdownsPractise | radio Round Trip, tick Senior Citizen, currency USD, type "India" in the country box | 4: radio, checkbox, select, fill | observed radio/checkbox/select state and the country field's value |
| 20 | cocktail-margarita | thecocktaildb.com | search "margarita", open Margarita | 3 | URL `/drink/11007` |
| 21 | mealdb-arrabiata | themealdb.com | search "arrabiata", open Spicy Arrabiata Penne | 3 | URL `/meal/52771` |
| 22 | gentoo-openrc-talk | wiki.gentoo.org | search "OpenRC" (lands on the article), open Discussion | 3 | URL contains `Talk:OpenRC` |
| 23 | webkit-css-bugs | bugs.webkit.org | Browse → WebKit product → CSS component | 3 | URL `buglist.cgi?...product=WebKit&component=CSS` |
| 24 | govdata-wetter-energie | govdata.de | search "Wetter", category Energie | 3 | URL `q=Wetter&groups=ener` |
| 25 | fedora-vim-common | packages.fedoraproject.org | search "vim", open vim-common | 3 | URL `/pkgs/vim/vim-common` |
| 26 | racket-argo | pkgs.racket-lang.org | search "json", open argo | 3 | URL ends `/package/argo` |
| 27 | rdrr-ggplot | rdrr.io | search "ggplot" | 2 | URL `/search?q=ggplot` |

Notes on the harder ones:
- Several targets are only observable after a scroll (Clojars pagination) or inside a menu that must be opened first
  (VS Marketplace "Sort By", FRED "Sort by Relevance", Modrinth "Sort by"): the agent must open the menu, then pick.
- letcode-dropdowns / letcode-radio / qaclickjet-form have no URL change at all; they are judged from the observed form
  state, so they cannot pass by navigating anywhere.
- todomvc-active depends on persisted state, so the suite (and the script runner) clears the site's localStorage first.

## Sites rejected and why

Blocked / bot walls for headless Chromium (Cloudflare "Just a moment", Akamai "Access Denied", 403, captcha):
congress.gov, bls.gov, catalog.hathitrust.org, dp.la (CloudFront error), philpapers.org, zbmath.org, tvtropes.org,
terraria.wiki.gg, mobygames.com, anaconda.org, hansard.parliament.uk, latlong.net, chessgames.com (403),
data.humdata.org (403), pkgs.org ("Human Verification" page after search), imslp.org (its search redirects to Google →
captcha), hal.science and openwrt.org ("Oh noes!" error page), search.scielo.org (never finishes "Establishing a secure
connection").

In the training data / sites lists (LEAK by domain or base domain), so unusable even though they would be good:
data.gov, data.gov.uk, data.europa.eu, data.gouv.fr, open.canada.ca, data.gov.au, openlibrary.org, archive.org,
gutenberg.org, musicbrainz.org, discogs.com, boardgamegeek.com, pypi/crates/npm/hex/pub.dev/hackage/metacpan,
packages.debian.org, packages.ubuntu.com, aur.archlinux.org, repology.org, addons.mozilla.org, readthedocs.org,
bugzilla.mozilla.org, bugs.launchpad.net, bugs.debian.org, bugzilla.kernel.org, gitlab.com, codeberg.org, sourceforge.net,
stackoverflow.com, wikidata/wiktionary/wikivoyage/wikiquote/commons, wikihow.com, lichess.org, geonames.org, timeanddate.com,
zenodo.org, figshare.com, dataverse.harvard.edu, dblp.org, doaj.org, core.ac.uk, biorxiv.org, inspirehep.net, eric.ed.gov,
ncbi/pubmed, worldbank.org, who.int, ec.europa.eu, usa.gov/nist/cdc/fda/epa/nps.gov, ocw.mit.edu, plato.stanford.edu,
selenium.dev, w3schools.com, demoqa.com, the-internet.herokuapp.com, saucedemo.com, demoblaze.com, parabank,
practice.expandtesting.com, automationexercise.com, opencart demo, nopcommerce demo, orangehrm demo, magento
softwaretestingboard, juice-shop, webscraper.io, scrapethissite.com, toscrape.com, scrapingcourse.com (suite B), and all
the airline/hotel/travel sites in finetune/sites.txt.

Reachable but rejected for the agent's action space or for flakiness:
- bugs.kde.org simple search: the Product `<select>` has ~235 options, so the observation is truncated at 250 actions
  and the "Words" text field is never offered.
- gitea.com explore: the custom Sort dropdown's options do not navigate when clicked (URL stays `sort=recentupdate`).
- rawg.io: Enter in the search box does not navigate (`rawg.io/?`); results are only rendered in-page.
- weather.gov: neither the "Go" nor the "Get Weather" form submits from a synthetic click (no navigation after 1.5 s).
- data.cityofnewyork.us: the search box is a combobox, so no Enter action is offered, and clicking the search button
  drops the query when a filter is applied afterwards.
- uniprot.org: the facet sidebar ("Reviewed") is not rendered in the 1120×780 viewport.
- bstackdemo.com: vendor checkboxes and "Add to cart" buttons are custom elements that are not observed as actions.
- cms.demo.katalon.com: search only returns blog posts; the shop's sorting `<select>` is a selectWoo widget with no
  observable options.
- rahulshettyacademy autocomplete: jQuery-UI suggestions have no `role=option`, so they cannot be clicked (the form task
  only types into that field and judges its value); its discount checkboxes are mutually exclusive (ticking one clears
  the others), so the task asks for a single one.
- testpages.eviltester.com basic form, demo.guru99.com register form: every field is an unlabeled `textbox`, so the
  goal cannot name the fields; guru99 also needs a password field, which jev hides.
- letcode.in/forms: only the Email field remains observable after scrolling (layout overlaps).
- oxylabs sandbox: the platform sub-category chips ("switch", "wii") are not links.
- scryfall.com: search box is an unlabeled textbox; sunrise-sunset.org: the Search button does not navigate.
- pokemondb.net Pokédex filter (Name + Type, client-side): worked 3/4 runs, but on some first loads the whole page is
  covered by an overlay so no control is observable at all → dropped as flaky.
- issues.jenkins.io: the "Search for issues" link is only present in a collapsed menu on some loads.
- ultimateqa.com forms: only name + message + submit (too small); openalex.org (API-only page); melpa/opencollective:
  no searchable form on the landing page.
- wikipedia-like wiki farms (miraheze) and mediawiki search are already heavily represented in training; only one
  MediaWiki task (Gentoo wiki talk page) is kept.

## Validation results

Filled in from `python apps/suite_c_scripts.py` with REPEATS=3 (see the bottom of this file).

Final run of `REPEATS=3 python apps/suite_c_scripts.py` (2026-09-26, four shards in parallel, final judge code):
27/27 tasks passed 3/3, every start-page check failed (a do-nothing run cannot pass).

| task | scripted runs | steps | task | scripted runs | steps |
|------|:---:|:---:|------|:---:|:---:|
| met-sunflowers | 3/3 | 4 | nuget-serilog-tool | 3/3 | 4 |
| alpine-curl-filter | 3/3 | 5 | freesound-rain-cc0 | 3/3 | 4 |
| ats-shampoo-haircare | 3/3 | 6 | bnf-hugo-printed-p2 | 3/3 | 4 |
| vsm-python-installs | 3/3 | 4 | wp-cache-commercial-redis | 3/3 | 4 |
| todomvc-active | 3/3 | 5 | setlist-radiohead-uk | 3/3 | 4 |
| jetbrains-rust-free | 3/3 | 4 | luarocks-rapidjson | 3/3 | 5 |
| letcode-dropdowns | 3/3 | 5 | letcode-radio | 3/3 | 4 |
| clojars-ring-page3 | 3/3 | 7 | fred-unemployment-pop | 3/3 | 4 |
| modrinth-sodium | 3/3 | 5 | tvmaze-friends-episodes | 3/3 | 4 |
| qaclickjet-form | 3/3 | 4 | cocktail-margarita | 3/3 | 3 |
| mealdb-arrabiata | 3/3 | 3 | gentoo-openrc-talk | 3/3 | 3 |
| webkit-css-bugs | 3/3 | 3 | govdata-wetter-energie | 3/3 | 3 |
| fedora-vim-common | 3/3 | 3 | racket-argo | 3/3 | 3 |
| rdrr-ggplot | 3/3 | 2 | | | |

19 tasks need 4+ actions (the first 19 in `TASKS`, also reported separately by the suite as "multi-step tasks").
The agent suite itself (`apps/browser_suite_c.py`) has not been run yet: it needs the laya systemone server (:8791) and
sglang (:30000), and the GPU belonged to another session while this suite was built.