File size: 2,776 Bytes
adf912b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | == per-step latency
-- laya-browser-v10
3 questions / 65 options / 2804 tokens: agent.predict 57.4 ms | fast 55.5 ms (tokenize 3.6 + forward 51.0 + post 0.3)
3 questions / 28 options / 1750 tokens: agent.predict 41.8 ms | fast 40.7 ms (tokenize 2.5 + forward 38.4 + post 0.2)
3 questions / 31 options / 1910 tokens: agent.predict 46.2 ms | fast 45.2 ms (tokenize 3.2 + forward 41.2 + post 0.2)
3 questions / 31 options / 1988 tokens: agent.predict 47.4 ms | fast 45.9 ms (tokenize 2.7 + forward 43.2 + post 0.1)
-- laya-browser-v10s
3 questions / 65 options / 2750 tokens: agent.predict 23.9 ms | fast 20.7 ms (tokenize 2.6 + forward 17.9 + post 0.1)
3 questions / 28 options / 1743 tokens: agent.predict 17.0 ms | fast 16.7 ms (tokenize 2.0 + forward 14.6 + post 0.1)
3 questions / 31 options / 1910 tokens: agent.predict 19.1 ms | fast 18.6 ms (tokenize 2.1 + forward 16.7 + post 0.1)
3 questions / 31 options / 1988 tokens: agent.predict 19.8 ms | fast 19.0 ms (tokenize 1.9 + forward 17.1 + post 0.1)
== suite laya-browser-v10 tau=0 x3
== 28/48 passed (58%, 3 run(s) per task) | median wall 1.4s
per task: wiki-einstein=0/3 wiki-search=0/3 hn-new=3/3 hn-login-page=3/3 gh-issues=1/3 py-downloads=3/3 books-travel=3/3 books-open-book=3/3 internet-dropdown=0/3 internet-checkbox=3/3 books-page2=0/3 quotes-tag-love=3/3 hn-past=3/3 ddg-search=3/3 arxiv-search=0/3 flights=0/3
server calls=679 escalated=0 (0%)
== suite laya-browser-v10 tau=0.7 x3
== 20/48 passed (42%, 3 run(s) per task) | median wall 2.4s
per task: wiki-einstein=0/3 wiki-search=0/3 hn-new=0/3 hn-login-page=1/3 gh-issues=0/3 py-downloads=3/3 books-travel=3/3 books-open-book=3/3 internet-dropdown=0/3 internet-checkbox=3/3 books-page2=0/3 quotes-tag-love=3/3 hn-past=3/3 ddg-search=1/3 arxiv-search=0/3 flights=0/3
server calls=531 escalated=416 (78%)
== suite laya-browser-v10s tau=0 x3
== 24/48 passed (50%, 3 run(s) per task) | median wall 1.3s
per task: wiki-einstein=0/3 wiki-search=0/3 hn-new=3/3 hn-login-page=3/3 gh-issues=0/3 py-downloads=3/3 books-travel=3/3 books-open-book=3/3 internet-dropdown=0/3 internet-checkbox=3/3 books-page2=0/3 quotes-tag-love=3/3 hn-past=3/3 ddg-search=0/3 arxiv-search=0/3 flights=0/3
server calls=731 escalated=0 (0%)
== suite laya-browser-v10s tau=0.7 x3
== 25/48 passed (52%, 3 run(s) per task) | median wall 1.5s
per task: wiki-einstein=0/3 wiki-search=0/3 hn-new=3/3 hn-login-page=3/3 gh-issues=0/3 py-downloads=3/3 books-travel=3/3 books-open-book=3/3 internet-dropdown=0/3 internet-checkbox=3/3 books-page2=0/3 quotes-tag-love=3/3 hn-past=3/3 ddg-search=1/3 arxiv-search=0/3 flights=0/3
server calls=487 escalated=412 (85%)
FINAL_DONE
|