| chrome ok |
| sglang ok |
| sglang probe: {"id":"c42f75d308554851b909fec4820e9d89","object":"chat.completion","created":1789922929,"model":"Qwen/Qwe |
| == goals (421 pages) |
| [CLICK ] Cart <- Take me to the cart page. |
| [DONE ] <- Go to https://www.demoblaze.com/cart.html and stop when it has loaded. |
| [CLICK ] Sequential Art <- I want to explore books in the Sequential Art category. |
| cases: 5244 |
| == real DONE |
| wrote 700 DONE cases |
| == step-2 |
| wrote 1262 |
| == dagger (policy v9) |
| wrote 154 cases -> ../laya/finetune/out/dagger_cases.jsonl |
| == build v10 |
| operation label counts: {'TYPE_TEXT': 3990, 'CLICK': 10417, 'SELECT': 975, 'DONE': 3720} |
| train items 31174 (op 19102, target 12072), eval cases 2734, skipped 0, seq len mean 839 max 1024 |
| == train v10 (4 epochs) |
| === epoch 1/4 avg loss 1.0265 1953s |
| === epoch 2/4 avg loss 1.2231 3940s |
| === epoch 3/4 avg loss 1.3842 5885s |
| === epoch 4/4 avg loss 1.1751 7837s |
| saved finetune/out/laya-browser-v10 |
| == calibrate |
| n=2671 acc=0.758 T=1: nll 2.569 mean conf 0.927 | T=5.03: nll 0.924 mean conf 0.779 |
| wrote temperature 5.025579877567767 -> /home/ckl/projects/S/laya/finetune/out/laya-browser-v10/rl_agent_config.json |
| == eval v10 |
| /home/ckl/projects/S/laya/finetune/out/laya-browser-v10/: cases 2734 operation acc 0.876 target top-1 0.656 (n=2609, mean normalized rank 0.220) 41 ms/case |
| live CLICK n=1050 op acc 0.96 target top-1 0.73 |
| live DONE n= 125 op acc 0.85 |
| live SELECT n= 32 op acc 0.94 target top-1 0.81 |
| live TYPE_TEXT n= 117 op acc 0.81 target top-1 0.93 |
| mind2web CLICK n=1084 op acc 0.91 target top-1 0.53 |
| mind2web SELECT n= 126 op acc 0.33 target top-1 0.60 |
| mind2web TYPE_TEXT n= 200 op acc 0.64 target top-1 0.81 |
| == eval v9 (same eval) |
| /home/ckl/projects/S/laya/finetune/out/laya-browser-v9/: cases 2734 operation acc 0.723 target top-1 0.592 (n=2609, mean normalized rank 0.238) 45 ms/case |
| live CLICK n=1050 op acc 0.67 target top-1 0.58 |
| live DONE n= 125 op acc 0.13 |
| live SELECT n= 32 op acc 0.28 target top-1 0.78 |
| live TYPE_TEXT n= 117 op acc 0.79 target top-1 0.88 |
| mind2web CLICK n=1084 op acc 0.91 target top-1 0.51 |
| mind2web SELECT n= 126 op acc 0.37 target top-1 0.63 |
| mind2web TYPE_TEXT n= 200 op acc 0.64 target top-1 0.85 |
| == suite v10 |
| FAIL wiki-random steps= 1 status=done 3.5s https://en.wikipedia.org/wiki/Main_Page#/media/File:Volcano_Misti,_Per |
| FAIL wiki-search steps= 4 status=blocked 1.6s https://en.wikipedia.org/wiki/Main_Page |
| FAIL hn-new steps= 1 status=done 1.4s https://news.ycombinator.com/login?goto=news |
| PASS hn-login-page steps= 1 status=done 0.9s https://news.ycombinator.com/login?goto=news |
| FAIL gh-issues steps= 1 status=done 4.8s https://github.com/tile-ai/tilelang |
| PASS py-downloads steps= 1 status=done 2.2s https://www.python.org/downloads/ |
| PASS books-travel steps= 1 status=done 0.3s https://books.toscrape.com/catalogue/category/books/travel_2/index.htm |
| PASS books-open-book steps= 1 status=done 0.4s https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h |
| FAIL internet-dropdown steps=20 status=ready 5.1s https://the-internet.herokuapp.com/dropdown |
| PASS internet-checkbox steps= 1 status=done 0.7s https://the-internet.herokuapp.com/checkboxes |
| FAIL books-page2 steps= 1 status=done 0.4s https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h |
| PASS quotes-tag-love steps= 1 status=done 2.3s https://quotes.toscrape.com/tag/love/ |
| PASS hn-past steps= 4 status=blocked 1.7s https://news.ycombinator.com/front |
| PASS ddg-search steps= 3 status=error:ValueError 19.7s https://duckduckgo.com/?ia=web&origin=funnel_home_website_measuresearc |
| FAIL arxiv-search steps= 2 status=error:ValueError 13.6s https://arxiv.org/ |
| FAIL flights steps=20 status=ready 8.6s https://www.google.com/travel/flights?hl=en |
| == 8/16 passed | median wall 2.2s |
| V10_DONE |
|
|
