File size: 5,449 Bytes
adf912b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 | == build v10s (format v3, mmBERT-base tokenizer)
operation label counts: {'TYPE_TEXT': 3990, 'CLICK': 10417, 'SELECT': 975, 'DONE': 3720}
train items 31174 (op 19102, target 12072), eval cases 2734, skipped 0, seq len mean 794 max 1024
== train v10s (4 epochs)
=== epoch 1/4 avg loss 0.9579 923s
=== epoch 2/4 avg loss 1.1214 1845s
=== epoch 3/4 avg loss 1.1182 2773s
=== epoch 4/4 avg loss 0.9524 3701s
saved finetune/out/laya-browser-v10s
== calibrate
n=2671 acc=0.758 T=1: nll 2.487 mean conf 0.904 | T=4.74: nll 1.032 mean conf 0.759
wrote temperature 4.737765564273641 -> /home/ckl/projects/S/laya/finetune/out/laya-browser-v10s/rl_agent_config.json
== eval v10s
/home/ckl/projects/S/laya/finetune/out/laya-browser-v10s/: cases 2734 operation acc 0.887 target top-1 0.631 (n=2609, mean normalized rank 0.229) 43 ms/case
live CLICK n=1050 op acc 0.96 target top-1 0.71
live DONE n= 125 op acc 0.86
live SELECT n= 32 op acc 0.94 target top-1 0.81
live TYPE_TEXT n= 117 op acc 0.84 target top-1 0.94
mind2web CLICK n=1084 op acc 0.93 target top-1 0.51
mind2web SELECT n= 126 op acc 0.38 target top-1 0.36
mind2web TYPE_TEXT n= 200 op acc 0.65 target top-1 0.83
== profile step
[laya] loaded /home/ckl/projects/S/laya/finetune/out/laya-browser-v10s in 8.4s
3 questions / 65 options / 2538 tokens: agent.predict 24.4 ms | fast 22.8 ms (tokenize 2.6 + forward 19.7 + post 0.1)
3 questions / 28 options / 1743 tokens: agent.predict 18.7 ms | fast 16.8 ms (tokenize 1.8 + forward 13.7 + post 0.1)
3 questions / 31 options / 1910 tokens: agent.predict 19.6 ms | fast 19.4 ms (tokenize 2.3 + forward 16.9 + post 0.2)
3 questions / 31 options / 1988 tokens: agent.predict 18.1 ms | fast 17.2 ms (tokenize 1.9 + forward 15.2 + post 0.1)
== suite v10s tau=0
FAIL wiki-einstein steps= 2 status=done 3.6s https://en.wikipedia.org/wiki/Main_Page
FAIL wiki-search steps= 4 status=blocked 1.9s https://en.wikipedia.org/wiki/Main_Page
PASS hn-new steps= 1 status=done 1.3s https://news.ycombinator.com/newest
PASS hn-login-page steps= 1 status=done 14.1s https://news.ycombinator.com/login?goto=news
FAIL gh-issues steps= 1 status=done 5.8s https://github.com/tile-ai/tilelang
PASS py-downloads steps= 1 status=done 15.4s https://www.python.org/downloads/
PASS books-travel steps= 1 status=done 0.3s https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
PASS books-open-book steps= 1 status=done 0.4s https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
FAIL internet-dropdown steps= 1 status=done 28.0s https://the-internet.herokuapp.com/dropdown
PASS internet-checkbox steps= 1 status=done 21.4s https://the-internet.herokuapp.com/checkboxes
FAIL books-page2 steps= 1 status=done 0.2s https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
PASS quotes-tag-love steps= 1 status=done 2.1s https://quotes.toscrape.com/tag/love/
PASS hn-past steps= 1 status=done 1.3s https://news.ycombinator.com/front
PASS ddg-search steps= 3 status=error:ValueError 61.8s https://duckduckgo.com/?ia=web&origin=funnel_home_website_measuresearc
FAIL arxiv-search steps= 2 status=error:ValueError 10.9s https://arxiv.org/
FAIL flights steps=20 status=ready 78.7s https://www.google.com/travel/flights?tfs=CBwQARocagwIAxIIL20vMDg5NjZy
== 9/16 passed (56%, 1 run(s) per task) | median wall 5.8s
server calls=292 escalated=0 (0%)
== suite v10s tau=0.7
FAIL wiki-einstein steps= 2 status=done 3.4s https://en.wikipedia.org/wiki/Main_Page
FAIL wiki-search steps= 4 status=blocked 3.2s https://en.wikipedia.org/wiki/Main_Page
PASS hn-new steps= 1 status=done 2.9s https://news.ycombinator.com/newest
PASS hn-login-page steps= 1 status=done 2.4s https://news.ycombinator.com/login?goto=news
FAIL gh-issues steps= 1 status=done 5.4s https://github.com/tile-ai/tilelang
PASS py-downloads steps= 1 status=done 0.9s https://www.python.org/downloads/
PASS books-travel steps= 1 status=done 0.7s https://books.toscrape.com/catalogue/category/books/travel_2/index.htm
PASS books-open-book steps= 1 status=done 0.9s https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
FAIL internet-dropdown steps= 1 status=done 1.4s https://the-internet.herokuapp.com/dropdown
PASS internet-checkbox steps= 1 status=done 0.7s https://the-internet.herokuapp.com/checkboxes
FAIL books-page2 steps= 1 status=done 0.9s https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.h
PASS quotes-tag-love steps= 1 status=done 2.4s https://quotes.toscrape.com/tag/love/
PASS hn-past steps= 1 status=done 2.3s https://news.ycombinator.com/front
PASS ddg-search steps= 3 status=error:ValueError 23.5s https://duckduckgo.com/?ia=web&origin=funnel_home_website_measuresearc
FAIL arxiv-search steps= 2 status=error:ValueError 30.6s https://arxiv.org/
FAIL flights steps= 5 status=error:ValueError 62.5s https://www.google.com/travel/flights?hl=en
== 9/16 passed (56%, 1 run(s) per task) | median wall 2.4s
server calls=389 escalated=363 (93%)
V10S_DONE
|