Add Frozen80 per-question results
Browse files- README.md +1 -1
- README.zh-CN.md +1 -1
- evaluation/per-question/mlx-8bit-mac.jsonl +80 -0
- evaluation/per-question/mlx-8bit.jsonl +80 -0
- evaluation/per-question/reference.jsonl +80 -0
README.md
CHANGED
|
@@ -41,7 +41,7 @@ Frozen80 with identical prompt tokens, compared with the HF BF16 release (full d
|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 66/80 · 82.50% | 80/80 | 0.1201 |
|
| 42 |
| Apple M5 24 GB (Metal) | mlx 0.32.2 on Darwin arm64 (Device(gpu, 0)) | 66/80 · 82.50% | 80/80 | 0.1480 |
|
| 43 |
|
| 44 |
-
Verified on an Apple M5, 24 GB: peak memory 5.44 GB. The same file on CUDA and Metal gave identical decisions on 80/80 prompts (max Δp 0.054). Frozen80 is a reused development panel, not a blind benchmark.
|
| 45 |
|
| 46 |
## Conversion
|
| 47 |
|
|
|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 66/80 · 82.50% | 80/80 | 0.1201 |
|
| 42 |
| Apple M5 24 GB (Metal) | mlx 0.32.2 on Darwin arm64 (Device(gpu, 0)) | 66/80 · 82.50% | 80/80 | 0.1480 |
|
| 43 |
|
| 44 |
+
Verified on an Apple M5, 24 GB: peak memory 5.44 GB. The same file on CUDA and Metal gave identical decisions on 80/80 prompts (max Δp 0.054). Frozen80 is a reused development panel, not a blind benchmark. Per-question rows (candidate probabilities, choice, correctness; join with Frozen80 by `panel_index`): [evaluation/per-question/](evaluation/per-question/).
|
| 45 |
|
| 46 |
## Conversion
|
| 47 |
|
README.zh-CN.md
CHANGED
|
@@ -41,7 +41,7 @@ Frozen80 使用完全相同的 prompt token，与 HF BF16 发布版（完整深
|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 66/80 · 82.50% | 80/80 | 0.1201 |
|
| 42 |
| Apple M5 24 GB (Metal) | mlx 0.32.2 on Darwin arm64 (Device(gpu, 0)) | 66/80 · 82.50% | 80/80 | 0.1480 |
|
| 43 |
|
| 44 |
-
已在 Apple M5, 24 GB 上实测：峰值内存 5.44 GB。同一份文件在 CUDA 与 Metal 上 80/80 题决策完全相同（最大 Δp 0.054）。Frozen80 是复用的开发面板，不是盲测。
|
| 45 |
|
| 46 |
## 转换说明
|
| 47 |
|
|
|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 66/80 · 82.50% | 80/80 | 0.1201 |
|
| 42 |
| Apple M5 24 GB (Metal) | mlx 0.32.2 on Darwin arm64 (Device(gpu, 0)) | 66/80 · 82.50% | 80/80 | 0.1480 |
|
| 43 |
|
| 44 |
+
已在 Apple M5, 24 GB 上实测：峰值内存 5.44 GB。同一份文件在 CUDA 与 Metal 上 80/80 题决策完全相同（最大 Δp 0.054）。Frozen80 是复用的开发面板，不是盲测。逐题明细（候选概率、选择、是否正确，可按 `panel_index` 与 Frozen80 关联）：[evaluation/per-question/](evaluation/per-question/)。
|
| 45 |
|
| 46 |
## 转换说明
|
| 47 |
|
evaluation/per-question/mlx-8bit-mac.jsonl
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"panel_index": 0, "task_family": "browser", "logits": [12.8125, 12.875, 12.0625, 12.125, 20.625, 12.125, 12.1875, 12.1875, 12.75, 12.75, 11.9375, 11.375, 11.875, 12.25, 11.4375, 11.9375], "probabilities": [0.00040321299456991255, 0.0004292181693017483, 0.0001904644159367308, 0.00020274825510568917, 0.9964608550071716, 0.00020274825510568917, 0.00021582446061074734, 0.00021582446061074734, 0.0003787837049458176, 0.0003787837049458176, 0.0001680842397036031, 9.577153105055913e-05, 0.00015790048928465694, 0.00022974387684371322, 0.00010194822971243411, 0.0001680842397036031], "choice": "candidate-c59cf6b3cdd1ee8dfe92", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00017052888870239258}
|
| 2 |
+
{"panel_index": 1, "task_family": "browser", "logits": [13.375, 13.4375, 12.8125, 12.8125, 12.875, 12.75, 12.5, 13.3125, 20.625, 16.375, 13.0625, 13.3125, 13.125, 13.5625, 12.875, 13.0625], "probabilities": [0.0006949820090085268, 0.0007398043526336551, 0.00039598860894329846, 0.00039598860894329846, 0.0004215278313495219, 0.00037199698272161186, 0.00028971151914447546, 0.0006528749945573509, 0.978607177734375, 0.013959080912172794, 0.0005084595177322626, 0.0006528749945573509, 0.000541252491530031, 0.0008383081876672804, 0.0004215278313495219, 0.0005084595177322626], "choice": "candidate-d2449a13158601200ec0", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0018326649442315102}
|
| 3 |
+
{"panel_index": 2, "task_family": "browser", "logits": [12.75, 13.375, 12.5, 12.8125, 13.6875, 20.25, 12.625, 12.9375, 12.9375, 11.5625, 12.0625, 11.9375, 12.6875, 12.5, 11.1875, 12.4375], "probabilities": [0.0005486929439939559, 0.0010250930208712816, 0.00042732246220111847, 0.0005840804078616202, 0.0014011366292834282, 0.9920598268508911, 0.00048421983956359327, 0.0006618499173782766, 0.0006618499173782766, 0.00016734187374822795, 0.00027590017998591065, 0.00024348103033844382, 0.0005154491518624127, 0.00042732246220111847, 0.00011501226254040375, 0.0004014321602880955], "choice": "candidate-9409b72a36db8d4a0a79", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 8.492660708725452e-05}
|
| 4 |
+
{"panel_index": 3, "task_family": "browser", "logits": [17.5, 14.5, 13.6875, 13.75, 14.0625, 14.0625, 20.5, 14.6875, 14.4375, 14.0625, 17.625, 13.0, 13.625, 14.9375, 12.9375, 13.75], "probabilities": [0.044130101799964905, 0.002197108929976821, 0.0009749610908329487, 0.0010378403821960092, 0.0014185601612553, 0.0014185601612553, 0.886376678943634, 0.00265021831728518, 0.002063991967588663, 0.0014185601612553, 0.0500059612095356, 0.0004902412183582783, 0.0009158909088000655, 0.0034029479138553143, 0.00046053886762820184, 0.0010378403821960092], "choice": "candidate-3b25ae8d064a298ef021", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.018641650676727295}
|
| 5 |
+
{"panel_index": 4, "task_family": "browser", "logits": [20.25, 13.75, 12.8125, 13.5, 12.3125, 12.125, 11.5, 11.8125, 12.125, 10.8125, 11.75, 11.4375, 12.1875, 12.25, 11.25, 12.0625], "probabilities": [0.993965208530426, 0.0014943663263693452, 0.0005852022441104054, 0.0011638135183602571, 0.0003549430111888796, 0.0002942581777460873, 0.00015750502643641084, 0.00021528392971958965, 0.0002942581777460873, 7.919850031612441e-05, 0.0002022404660237953, 0.00014796231698710471, 0.00031323631992563605, 0.00033343824907206, 0.00012266510748304427, 0.0002764300734270364], "choice": "candidate-34e89f28b090ce5327b0", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00041282176971435547}
|
| 6 |
+
{"panel_index": 5, "task_family": "browser", "logits": [20.25, 13.3125, 12.75, 12.3125, 15.125, 12.375, 11.9375, 15.5625, 12.5, 11.25, 11.875, 11.5625, 12.0625, 12.6875, 11.875, 11.9375], "probabilities": [0.9804983735084534, 0.0009517634171061218, 0.000542298483196646, 0.0003501340397633612, 0.0058302562683820724, 0.0003727159055415541, 0.00024064349418040365, 0.00903007946908474, 0.00042234244756400585, 0.00012100316962460056, 0.0002260635665152222, 0.0001653916697250679, 0.00027268484700471163, 0.0005094421212561429, 0.0002260635665152222, 0.00024064349418040365], "choice": "candidate-fb3025ec54721504f9b5", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.002537531778216362}
|
| 7 |
+
{"panel_index": 6, "task_family": "browser", "logits": [13.8125, 13.5625, 12.5625, 12.4375, 14.375, 13.6875, 15.375, 14.875, 13.875, 16.25, 19.75, 12.75, 13.3125, 13.5625, 11.875, 14.5625], "probabilities": [0.002450336702167988, 0.0019083237275481224, 0.0007020332850515842, 0.0006195421447046101, 0.004300475586205721, 0.0021624143701046705, 0.011689901351928711, 0.007090284489095211, 0.002608370501548052, 0.028042616322636604, 0.9286439418792725, 0.0008468135492876172, 0.001486204913817346, 0.0019083237275481224, 0.0003530045214574784, 0.005187362432479858], "choice": "candidate-888bb5997d1aba7b25f7", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0027889013290405273}
|
| 8 |
+
{"panel_index": 7, "task_family": "browser", "logits": [20.125, 12.9375, 12.0625, 11.875, 12.25, 11.5625, 11.5, 11.625, 11.9375, 10.5625, 11.4375, 11.125, 12.1875, 11.9375, 10.6875, 11.625], "probabilities": [0.9961690306663513, 0.0007530807051807642, 0.0003139308246318251, 0.0002602576860226691, 0.0003786727611441165, 0.00019040863844566047, 0.00017887230205815285, 0.0002026888687396422, 0.00027704297099262476, 7.004739745752886e-05, 0.00016803502512630075, 0.00012293708277866244, 0.00035573000786826015, 0.00027704297099262476, 7.937409827718511e-05, 0.0002026888687396422], "choice": "candidate-41c5af16d7bf373cc74c", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 9.02414321899414e-05}
|
| 9 |
+
{"panel_index": 8, "task_family": "browser", "logits": [12.5, 12.625, 20.375, 12.875, 12.5625, 12.0, 13.375, 12.625, 12.1875, 11.0625, 11.4375, 11.375, 12.0625, 11.6875, 11.0, 11.8125], "probabilities": [0.0003783677238970995, 0.000428746803663671, 0.9953665733337402, 0.0005505218869075179, 0.0004027702088933438, 0.0002294915757374838, 0.0009076567366719246, 0.000428746803663671, 0.0002768198028206825, 8.987019100459293e-05, 0.0001307603670284152, 0.00012283804244361818, 0.00024429260520264506, 0.00016789966321084648, 8.442525722784922e-05, 0.00019025524670723826], "choice": "candidate-a1a3ad1a8495aa678cbe", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00027632713317871094}
|
| 10 |
+
{"panel_index": 9, "task_family": "browser", "logits": [20.0, 15.3125, 13.1875, 12.75, 12.9375, 13.5, 12.1875, 12.8125, 12.8125, 11.75, 12.3125, 12.0625, 12.625, 12.8125, 11.5, 12.4375], "probabilities": [0.9818562865257263, 0.009042585268616676, 0.0010799829615280032, 0.0006972894771024585, 0.0008410915033891797, 0.0014761612983420491, 0.00039730334538035095, 0.0007422605995088816, 0.0007422605995088816, 0.000256518367677927, 0.00045020371908321977, 0.00035061893868260086, 0.0006153557915240526, 0.0007422605995088816, 0.00019977668125648052, 0.000510147656314075], "choice": "candidate-976c2731e7518a1f1b2b", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.002420365810394287}
|
| 11 |
+
{"panel_index": 10, "task_family": "browser", "logits": [14.4375, 14.8125, 12.8125, 13.375, 15.8125, 14.4375, 13.75, 15.1875, 14.0, 14.5, 14.25, 19.125, 14.25, 15.1875, 12.5, 13.75], "probabilities": [0.007981746457517147, 0.011613370850682259, 0.0015716985799372196, 0.0027584186755120754, 0.031568415462970734, 0.007981746457517147, 0.004013474564999342, 0.016897358000278473, 0.005153402220457792, 0.008496524766087532, 0.006617099978029728, 0.8666689991950989, 0.006617099978029728, 0.016897358000278473, 0.0011498794192448258, 0.004013474564999342], "choice": "candidate-016db9380de74cbcab5d", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.003324151039123535}
|
| 12 |
+
{"panel_index": 11, "task_family": "browser", "logits": [14.625, 13.5, 13.5625, 13.1875, 12.75, 12.5625, 12.8125, 13.125, 13.6875, 16.625, 13.5, 13.125, 14.6875, 19.625, 12.25, 13.125], "probabilities": [0.006222897209227085, 0.002020279411226511, 0.002150575630366802, 0.0014780682977288961, 0.0009543122141622007, 0.0007911527645774186, 0.001015860354527831, 0.001388516160659492, 0.002436921698972583, 0.04598134383559227, 0.002020279411226511, 0.001388516160659492, 0.0066242399625480175, 0.923559844493866, 0.0005788198322989047, 0.001388516160659492], "choice": "candidate-2dc16f49fe3acb051b39", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0007507205009460449}
|
| 13 |
+
{"panel_index": 12, "task_family": "browser", "logits": [13.6875, 13.3125, 13.8125, 13.4375, 12.75, 13.0, 12.5, 12.8125, 14.3125, 12.875, 12.6875, 12.6875, 13.75, 18.125, 12.0625, 20.25], "probabilities": [0.001246684929355979, 0.000856833066791296, 0.001412679092027247, 0.0009709190926514566, 0.00048820878146216273, 0.0006268725846894085, 0.0003802173596341163, 0.0005196954007260501, 0.002329113194718957, 0.0005532131181098521, 0.00045862956903874874, 0.00045862956903874874, 0.001327088801190257, 0.1054237112402916, 0.0002454868226777762, 0.8827019333839417], "choice": "candidate-bb1945d8d438379625ee", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.01216500997543335}
|
| 14 |
+
{"panel_index": 13, "task_family": "browser", "logits": [13.5, 13.3125, 12.3125, 12.3125, 12.25, 12.375, 13.0, 13.4375, 12.875, 14.0625, 18.0, 12.375, 13.875, 14.25, 16.625, 13.375], "probabilities": [0.008035835810005665, 0.006661941763013601, 0.0024507911875844, 0.0024507911875844, 0.002302306005731225, 0.0026088545564562082, 0.004873979836702347, 0.007548969238996506, 0.004301271867007017, 0.014103330671787262, 0.7233627438545227, 0.0026088545564562082, 0.011692069470882416, 0.017011864110827446, 0.182894766330719, 0.007091599982231855], "choice": "candidate-94dddd06eb43ed2921b0", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0016222596168518066}
|
| 15 |
+
{"panel_index": 14, "task_family": "browser", "logits": [13.0, 20.0, 12.625, 12.25, 12.1875, 11.875, 12.0625, 12.3125, 12.5, 11.75, 12.0625, 15.5, 12.875, 12.0, 11.5, 12.1875], "probabilities": [0.0008961857529357076, 0.9827871322631836, 0.0006159392069093883, 0.00042332831071689725, 0.000397680007154122, 0.00029094895580783486, 0.00035095136263407767, 0.00045063052675686777, 0.0005435643834061921, 0.0002567615592852235, 0.00035095136263407767, 0.010917781852185726, 0.0007908811094239354, 0.0003296883951406926, 0.00019996608898509294, 0.000397680007154122], "choice": "candidate-b56ad12ceea4c52feb3f", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.001398622989654541}
|
| 16 |
+
{"panel_index": 15, "task_family": "browser", "logits": [16.625, 15.875, 14.75, 19.375, 18.25, 15.4375, 14.3125, 17.25, 14.5, 14.25, 15.4375, 14.0, 16.5, 14.875, 12.75, 13.9375], "probabilities": [0.037944961339235306, 0.017923928797245026, 0.00581904873251915, 0.5935590267181396, 0.19270041584968567, 0.011572560295462608, 0.0037570595741271973, 0.07089051604270935, 0.00453187944367528, 0.0035294308327138424, 0.011572560295462608, 0.0027487240731716156, 0.03348631411790848, 0.006593846715986729, 0.0007875224691815674, 0.002582187531515956], "choice": "candidate-da095abe0a63c03f3114", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.07998168468475342}
|
| 17 |
+
{"panel_index": 16, "task_family": "hs3", "logits": [15.5, 18.625], "probabilities": [0.04208773374557495, 0.957912266254425], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 7.450580596923828e-09}
|
| 18 |
+
{"panel_index": 17, "task_family": "hs3", "logits": [17.25, 18.5], "probabilities": [0.22270014882087708, 0.7772998809814453], "choice": "no", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.09812116622924805}
|
| 19 |
+
{"panel_index": 18, "task_family": "hs3", "logits": [19.625, 16.375], "probabilities": [0.9626730680465698, 0.037326887249946594], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.725290298461914e-09}
|
| 20 |
+
{"panel_index": 19, "task_family": "hs3", "logits": [18.875, 12.8125], "probabilities": [0.9976769089698792, 0.0023231625091284513], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 21 |
+
{"panel_index": 20, "task_family": "hs3", "logits": [19.125, 13.0], "probabilities": [0.9978172779083252, 0.0021827169694006443], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00014044553972780704}
|
| 22 |
+
{"panel_index": 21, "task_family": "hs3", "logits": [11.6875, 18.5], "probabilities": [0.0010987314162775874, 0.9989012479782104], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.4924596548080444e-10}
|
| 23 |
+
{"panel_index": 22, "task_family": "hs3", "logits": [19.375, 12.6875], "probabilities": [0.9987550973892212, 0.0012448437046259642], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 7.539987564086914e-05}
|
| 24 |
+
{"panel_index": 23, "task_family": "hs3", "logits": [19.0, 15.5625], "probabilities": [0.9688562154769897, 0.031143829226493835], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.005186472088098526}
|
| 25 |
+
{"panel_index": 24, "task_family": "hs3", "logits": [19.0, 12.875], "probabilities": [0.9978172779083252, 0.0021827169694006443], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.656612873077393e-10}
|
| 26 |
+
{"panel_index": 25, "task_family": "hs3", "logits": [13.125, 19.0], "probabilities": [0.0028009277302771807, 0.9971990585327148], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 6.984919309616089e-10}
|
| 27 |
+
{"panel_index": 26, "task_family": "hs3", "logits": [18.875, 15.875], "probabilities": [0.9525741338729858, 0.047425881028175354], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.005977451801300049}
|
| 28 |
+
{"panel_index": 27, "task_family": "hs3", "logits": [19.25, 13.0], "probabilities": [0.9980732202529907, 0.001926734927110374], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00022608041763305664}
|
| 29 |
+
{"panel_index": 28, "task_family": "hs3", "logits": [15.6875, 18.5], "probabilities": [0.05665243789553642, 0.9433475732803345], "choice": "no", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 1.1175870895385742e-08}
|
| 30 |
+
{"panel_index": 29, "task_family": "hs3", "logits": [12.0625, 18.5], "probabilities": [0.0015978461597114801, 0.9984021782875061], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.820766091346741e-10}
|
| 31 |
+
{"panel_index": 30, "task_family": "hs3", "logits": [18.75, 14.125], "probabilities": [0.9902915358543396, 0.00970847811549902], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005828738212585449}
|
| 32 |
+
{"panel_index": 31, "task_family": "hs3", "logits": [19.25, 13.0625], "probabilities": [0.9979493021965027, 0.0020507434383034706], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001240087440237403}
|
| 33 |
+
{"panel_index": 32, "task_family": "boolq", "logits": [13.125, 18.5], "probabilities": [0.004609572701156139, 0.9953904151916504], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 34 |
+
{"panel_index": 33, "task_family": "boolq", "logits": [12.375, 19.0], "probabilities": [0.0013250223128125072, 0.9986749291419983], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00017615989781916142}
|
| 35 |
+
{"panel_index": 34, "task_family": "boolq", "logits": [12.8125, 18.625], "probabilities": [0.0029810331761837006, 0.9970190525054932], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 36 |
+
{"panel_index": 35, "task_family": "boolq", "logits": [19.5, 13.625], "probabilities": [0.9971990585327148, 0.0028009277302771807], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00032836198806762695}
|
| 37 |
+
{"panel_index": 36, "task_family": "boolq", "logits": [19.875, 12.1875], "probabilities": [0.9995417594909668, 0.0004583129193633795], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.775582834146917e-05}
|
| 38 |
+
{"panel_index": 37, "task_family": "boolq", "logits": [20.0, 12.3125], "probabilities": [0.9995417594909668, 0.0004583129193633795], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 39 |
+
{"panel_index": 38, "task_family": "boolq", "logits": [19.625, 12.25], "probabilities": [0.999373733997345, 0.0006263344548642635], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.9103830456733704e-10}
|
| 40 |
+
{"panel_index": 39, "task_family": "boolq", "logits": [13.125, 19.125], "probabilities": [0.002472624182701111, 0.9975274205207825], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00032836198806762695}
|
| 41 |
+
{"panel_index": 40, "task_family": "boolq", "logits": [19.75, 12.0625], "probabilities": [0.9995417594909668, 0.0004583129193633795], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.775582834146917e-05}
|
| 42 |
+
{"panel_index": 41, "task_family": "boolq", "logits": [13.25, 18.75], "probabilities": [0.004070138558745384, 0.9959298968315125], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.656612873077393e-10}
|
| 43 |
+
{"panel_index": 42, "task_family": "boolq", "logits": [19.5, 12.25], "probabilities": [0.9992903470993042, 0.0007096707122400403], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.9103830456733704e-10}
|
| 44 |
+
{"panel_index": 43, "task_family": "boolq", "logits": [14.75, 18.375], "probabilities": [0.025957360863685608, 0.9740425944328308], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.001627923920750618}
|
| 45 |
+
{"panel_index": 44, "task_family": "boolq", "logits": [19.125, 12.8125], "probabilities": [0.9981897473335266, 0.0018102105241268873], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00010955333709716797}
|
| 46 |
+
{"panel_index": 45, "task_family": "boolq", "logits": [20.5, 13.75], "probabilities": [0.9988304972648621, 0.0011695101857185364], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 47 |
+
{"panel_index": 46, "task_family": "boolq", "logits": [13.8125, 18.625], "probabilities": [0.00806199200451374, 0.9919379353523254], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 48 |
+
{"panel_index": 47, "task_family": "boolq", "logits": [12.75, 18.625], "probabilities": [0.0028009277302771807, 0.9971990585327148], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00016927719116210938}
|
| 49 |
+
{"panel_index": 48, "task_family": "mnli", "logits": [16.375, 19.375, 14.3125], "probabilities": [0.04714163765311241, 0.9468649625778198, 0.005993385799229145], "choice": "neutral", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.010328948497772217}
|
| 50 |
+
{"panel_index": 49, "task_family": "mnli", "logits": [13.125, 20.5, 12.3125], "probabilities": [0.0006261604139581323, 0.9990960359573364, 0.0002778570051304996], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.800060272216797e-05}
|
| 51 |
+
{"panel_index": 50, "task_family": "mnli", "logits": [20.25, 12.4375, 11.4375], "probabilities": [0.9994468092918396, 0.0004044212691951543, 0.00014877831563353539], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 9.593830327503383e-06}
|
| 52 |
+
{"panel_index": 51, "task_family": "mnli", "logits": [13.5, 20.25, 12.5], "probabilities": [0.001169007271528244, 0.9984009861946106, 0.00043005385668948293], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 8.731149137020111e-11}
|
| 53 |
+
{"panel_index": 52, "task_family": "mnli", "logits": [11.25, 12.0, 19.875], "probabilities": [0.0001794597483240068, 0.0003799163969233632, 0.999440610408783], "choice": "contradiction", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.83393669128418e-05}
|
| 54 |
+
{"panel_index": 53, "task_family": "mnli", "logits": [20.125, 12.0625, 11.25], "probabilities": [0.999545156955719, 0.0003149947733618319, 0.00013977808703202754], "choice": "entailment", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 1.919269561767578e-05}
|
| 55 |
+
{"panel_index": 54, "task_family": "mnli", "logits": [16.75, 19.75, 13.5], "probabilities": [0.0473388247191906, 0.9508256316184998, 0.0018355256179347634], "choice": "neutral", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.012840867042541504}
|
| 56 |
+
{"panel_index": 55, "task_family": "mnli", "logits": [20.125, 12.375, 11.375], "probabilities": [0.9994112253189087, 0.000430489017162472, 0.0001583680132171139], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.777576446533203e-05}
|
| 57 |
+
{"panel_index": 56, "task_family": "mnli", "logits": [19.875, 12.125, 11.375], "probabilities": [0.9993662238121033, 0.0004304696340113878, 0.00020333939755801111], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.4776668548583984e-05}
|
| 58 |
+
{"panel_index": 57, "task_family": "mnli", "logits": [14.5, 17.625, 19.625], "probabilities": [0.005210123024880886, 0.1185818687081337, 0.8762080073356628], "choice": "contradiction", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.043473899364471436}
|
| 59 |
+
{"panel_index": 58, "task_family": "mnli", "logits": [13.875, 19.75, 15.25], "probabilities": [0.0027702394872903824, 0.9862732291221619, 0.010956508107483387], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 9.313225746154785e-10}
|
| 60 |
+
{"panel_index": 59, "task_family": "mnli", "logits": [19.375, 14.375, 16.125], "probabilities": [0.9564689993858337, 0.006444637198001146, 0.037086330354213715], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.06959861516952515}
|
| 61 |
+
{"panel_index": 60, "task_family": "mnli", "logits": [14.1875, 14.9375, 19.875], "probabilities": [0.0033526471816003323, 0.007097553927451372, 0.9895498752593994], "choice": "contradiction", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0008159875869750977}
|
| 62 |
+
{"panel_index": 61, "task_family": "mnli", "logits": [19.5, 16.75, 14.0625], "probabilities": [0.9360857605934143, 0.05984196439385414, 0.004072297364473343], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 7.450580596923828e-09}
|
| 63 |
+
{"panel_index": 62, "task_family": "mnli", "logits": [19.875, 12.5625, 11.625], "probabilities": [0.9990724325180054, 0.0006665283581241965, 0.00026101621915586293], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.6829435480758548e-05}
|
| 64 |
+
{"panel_index": 63, "task_family": "mnli", "logits": [20.0, 12.1875, 11.375], "probabilities": [0.9994162321090698, 0.0004044089000672102, 0.00017945536819752306], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.9103830456733704e-11}
|
| 65 |
+
{"panel_index": 64, "task_family": "score", "logits": [18.75, 18.375], "probabilities": [0.5926666259765625, 0.4073334336280823], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 66 |
+
{"panel_index": 65, "task_family": "score", "logits": [12.125, 19.75], "probabilities": [0.00048785729450173676, 0.9995121955871582], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.1641532182693481e-10}
|
| 67 |
+
{"panel_index": 66, "task_family": "score", "logits": [12.0625, 19.75], "probabilities": [0.0004583129193633795, 0.9995417594909668], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 68 |
+
{"panel_index": 67, "task_family": "score", "logits": [13.5625, 19.375], "probabilities": [0.0029810331761837006, 0.9970190525054932], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00019174814224243164}
|
| 69 |
+
{"panel_index": 68, "task_family": "score", "logits": [12.5625, 19.5], "probabilities": [0.0009697522036731243, 0.9990302324295044], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.8770179748535156e-05}
|
| 70 |
+
{"panel_index": 69, "task_family": "score", "logits": [19.75, 12.5625], "probabilities": [0.9992445707321167, 0.0007554057519882917], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 4.57763671875e-05}
|
| 71 |
+
{"panel_index": 70, "task_family": "score", "logits": [12.75, 19.75], "probabilities": [0.0009110511164180934, 0.9990890026092529], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.8770179748535156e-05}
|
| 72 |
+
{"panel_index": 71, "task_family": "score", "logits": [12.625, 19.75], "probabilities": [0.0008040857501327991, 0.9991958737373352], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.7462298274040222e-10}
|
| 73 |
+
{"panel_index": 72, "task_family": "score", "logits": [19.125, 18.375], "probabilities": [0.6791787147521973, 0.3208213150501251], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.1479693353176117}
|
| 74 |
+
{"panel_index": 73, "task_family": "score", "logits": [11.9375, 19.5], "probabilities": [0.0005193048273213208, 0.999480664730072], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 75 |
+
{"panel_index": 74, "task_family": "score", "logits": [12.1875, 19.75], "probabilities": [0.0005193048273213208, 0.999480664730072], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 76 |
+
{"panel_index": 75, "task_family": "score", "logits": [12.6875, 19.375], "probabilities": [0.0012448437046259642, 0.9987550973892212], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.4924596548080444e-10}
|
| 77 |
+
{"panel_index": 76, "task_family": "score", "logits": [15.25, 19.375], "probabilities": [0.01590639166533947, 0.9840936064720154], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0010085217654705048}
|
| 78 |
+
{"panel_index": 77, "task_family": "score", "logits": [15.375, 19.125], "probabilities": [0.022977374494075775, 0.977022647857666], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0013620853424072266}
|
| 79 |
+
{"panel_index": 78, "task_family": "score", "logits": [13.75, 19.625], "probabilities": [0.0028009277302771807, 0.9971990585327148], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00016927719116210938}
|
| 80 |
+
{"panel_index": 79, "task_family": "score", "logits": [13.375, 19.375], "probabilities": [0.002472624182701111, 0.9975274205207825], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00028990767896175385}
|
evaluation/per-question/mlx-8bit.jsonl
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"panel_index": 0, "task_family": "browser", "logits": [12.8125, 12.9375, 12.0625, 12.1875, 20.625, 12.1875, 12.25, 12.25, 12.8125, 12.8125, 12.0, 11.4375, 11.9375, 12.3125, 11.5, 12.0], "probabilities": [0.0004031364223919809, 0.00045681343181058764, 0.00019042821077164263, 0.00021578346786554903, 0.9962716102600098, 0.00021578346786554903, 0.00022970023564994335, 0.00022970023564994335, 0.0004031364223919809, 0.0004031364223919809, 0.0001788906956790015, 0.00010192886111326516, 0.00016805231280159205, 0.00024451472563669086, 0.0001085027470253408, 0.0001788906956790015], "choice": "candidate-c59cf6b3cdd1ee8dfe92", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.4417618988081813e-05}
|
| 2 |
+
{"panel_index": 1, "task_family": "browser", "logits": [13.375, 13.4375, 12.8125, 12.8125, 12.875, 12.75, 12.5625, 13.3125, 20.625, 16.375, 13.0, 13.3125, 13.1875, 13.5625, 12.875, 13.0625], "probabilities": [0.0006949661765247583, 0.0007397874142043293, 0.00039597961585968733, 0.00039597961585968733, 0.00042151822708547115, 0.00037198851350694895, 0.0003083893971052021, 0.000652860093396157, 0.9785849452018738, 0.01395876333117485, 0.0004776427522301674, 0.000652860093396157, 0.0005761469365097582, 0.0008382890955545008, 0.00042151822708547115, 0.0005084479344077408], "choice": "candidate-d2449a13158601200ec0", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.001832982525229454}
|
| 3 |
+
{"panel_index": 2, "task_family": "browser", "logits": [12.6875, 13.3125, 12.5, 12.75, 13.6875, 20.25, 12.625, 12.9375, 12.9375, 11.5625, 12.0625, 11.9375, 12.6875, 12.4375, 11.125, 12.4375], "probabilities": [0.0005155335529707372, 0.0009631435968913138, 0.000427392398705706, 0.0005487827002070844, 0.0014013658510521054, 0.992222249507904, 0.0004842990601900965, 0.0006619581836275756, 0.0006619581836275756, 0.00016736926045268774, 0.00027594532002694905, 0.0002435208880342543, 0.0005155335529707372, 0.0004014978767372668, 0.00010806173668242991, 0.0004014978767372668], "choice": "candidate-9409b72a36db8d4a0a79", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00018918514251708984}
|
| 4 |
+
{"panel_index": 3, "task_family": "browser", "logits": [17.5, 14.5625, 13.6875, 13.75, 14.125, 14.125, 20.5, 14.625, 14.4375, 14.0625, 17.75, 13.0, 13.625, 15.0, 12.9375, 13.8125], "probabilities": [0.043818600475788116, 0.0023223005700856447, 0.0009680791408754885, 0.0010305145988240838, 0.0014993901131674647, 0.0014993901131674647, 0.8801200985908508, 0.0024720770306885242, 0.002049422822892666, 0.0014085470465943217, 0.05626419186592102, 0.00048678077291697264, 0.0009094259003177285, 0.0035968502052128315, 0.00045728811528533697, 0.001096977386623621], "choice": "candidate-3b25ae8d064a298ef021", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.012385070323944092}
|
| 5 |
+
{"panel_index": 4, "task_family": "browser", "logits": [20.25, 13.75, 12.8125, 13.5625, 12.3125, 12.1875, 11.5625, 11.8125, 12.125, 10.8125, 11.75, 11.4375, 12.1875, 12.25, 11.25, 12.125], "probabilities": [0.9938439130783081, 0.0014941839035600424, 0.0005851307651028037, 0.0012387222377583385, 0.00035489967558532953, 0.00031319810659624636, 0.00016764280735515058, 0.00021525764896068722, 0.0002942222636193037, 7.918883056845516e-05, 0.00020221577142365277, 0.0001479442580603063, 0.00031319810659624636, 0.00033339750370942056, 0.00012265011901035905, 0.0002942222636193037], "choice": "candidate-34e89f28b090ce5327b0", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005341172218322754}
|
| 6 |
+
{"panel_index": 5, "task_family": "browser", "logits": [20.25, 13.3125, 12.8125, 12.3125, 15.0625, 12.375, 11.9375, 15.6875, 12.5, 11.1875, 11.875, 11.5625, 12.0625, 12.6875, 11.9375, 11.9375], "probabilities": [0.9796252250671387, 0.000950915738940239, 0.0005767594557255507, 0.00034982222132384777, 0.0054721408523619175, 0.0003723839472513646, 0.00024042920267675072, 0.010223307646811008, 0.000421966309659183, 0.00011357066978234798, 0.00022586224076803774, 0.00016524437523912638, 0.0002724419755395502, 0.0005089884507469833, 0.00024042920267675072, 0.00024042920267675072], "choice": "candidate-fb3025ec54721504f9b5", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0014300942420959473}
|
| 7 |
+
{"panel_index": 6, "task_family": "browser", "logits": [13.75, 13.5625, 12.5, 12.375, 14.375, 13.6875, 15.375, 14.875, 13.875, 16.125, 19.875, 12.75, 13.3125, 13.5, 11.875, 14.5625], "probabilities": [0.002055233344435692, 0.0017038476653397083, 0.0005888343439437449, 0.0005196444108150899, 0.003839681390672922, 0.001930712372995913, 0.01043733675032854, 0.006330563221126795, 0.002328884555026889, 0.02209584042429924, 0.9395389556884766, 0.0007560778176411986, 0.0013269586488604546, 0.0016006173100322485, 0.0003151802229695022, 0.004631540272384882], "choice": "candidate-888bb5997d1aba7b25f7", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.008106112480163574}
|
| 8 |
+
{"panel_index": 7, "task_family": "browser", "logits": [20.125, 12.9375, 12.0625, 11.875, 12.1875, 11.625, 11.5, 11.625, 11.9375, 10.5625, 11.5, 11.1875, 12.1875, 11.9375, 10.75, 11.6875], "probabilities": [0.9961429238319397, 0.0007530608563683927, 0.00031392258824780583, 0.00026025084662251174, 0.0003557206364348531, 0.00020268354273866862, 0.00017886758723761886, 0.00020268354273866862, 0.0002770356950350106, 7.004554936429486e-05, 0.00017886758723761886, 0.00013086236140225083, 0.0003557206364348531, 0.0002770356950350106, 8.449109736829996e-05, 0.00021575558639597148], "choice": "candidate-41c5af16d7bf373cc74c", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 6.413459777832031e-05}
|
| 9 |
+
{"panel_index": 8, "task_family": "browser", "logits": [12.4375, 12.625, 20.375, 12.875, 12.5, 12.0, 13.4375, 12.5625, 12.1875, 11.0625, 11.4375, 11.375, 12.0625, 11.625, 11.0, 11.8125], "probabilities": [0.0003554522409103811, 0.0004287574265617877, 0.9953912496566772, 0.0005505354492925107, 0.0003783770662266761, 0.00022949725098442286, 0.0009662197553552687, 0.00040278019150719047, 0.00027682664222083986, 8.987241017166525e-05, 0.00013076361210551113, 0.0001228410837939009, 0.00024429865879938006, 0.00015773098857607692, 8.442733815172687e-05, 0.00019025994697585702], "choice": "candidate-a1a3ad1a8495aa678cbe", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0002516508102416992}
|
| 10 |
+
{"panel_index": 9, "task_family": "browser", "logits": [20.0, 15.3125, 13.3125, 12.8125, 13.0, 13.6875, 12.3125, 12.9375, 12.875, 11.875, 12.375, 12.1875, 12.6875, 12.875, 11.5625, 12.5], "probabilities": [0.9808847904205322, 0.00903363712131977, 0.0012225700775161386, 0.0007415260770358145, 0.0008944509318098426, 0.0017788279801607132, 0.00044975822675041854, 0.000840259192045778, 0.0007893501315265894, 0.00029038579668849707, 0.0004787652869708836, 0.00039691023994237185, 0.000654394447337836, 0.0007893501315265894, 0.0002124508173437789, 0.0005425121635198593], "choice": "candidate-976c2731e7518a1f1b2b", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0014488697052001953}
|
| 11 |
+
{"panel_index": 10, "task_family": "browser", "logits": [14.375, 14.8125, 12.75, 13.3125, 15.75, 14.375, 13.6875, 15.125, 13.9375, 14.5, 14.1875, 19.125, 14.125, 15.25, 12.4375, 13.625], "probabilities": [0.007538495119661093, 0.011675848625600338, 0.0014844178222119808, 0.0026052340399473906, 0.029815323650836945, 0.007538495119661093, 0.0037905939389020205, 0.0159589946269989, 0.004867217503488064, 0.008542235009372234, 0.006249631755053997, 0.8713315725326538, 0.005870985332876444, 0.018083911389112473, 0.0010860235197469592, 0.003560933517292142], "choice": "candidate-016db9380de74cbcab5d", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0013384222984313965}
|
| 12 |
+
{"panel_index": 11, "task_family": "browser", "logits": [14.625, 13.5, 13.5625, 13.125, 12.75, 12.5625, 12.875, 13.125, 13.6875, 16.75, 13.5, 13.125, 14.6875, 19.625, 12.3125, 13.125], "probabilities": [0.006184947676956654, 0.002007958944886923, 0.0021374605130404234, 0.0013800484593957663, 0.0009484923793934286, 0.0007863279897719622, 0.001074782689101994, 0.0013800484593957663, 0.0024220603518188, 0.051785942167043686, 0.002007958944886923, 0.0013800484593957663, 0.006583842448890209, 0.9179276823997498, 0.0006123927887529135, 0.0013800484593957663], "choice": "candidate-2dc16f49fe3acb051b39", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.006382882595062256}
|
| 13 |
+
{"panel_index": 12, "task_family": "browser", "logits": [13.625, 13.3125, 13.8125, 13.4375, 12.75, 12.9375, 12.5, 12.75, 14.3125, 12.8125, 12.6875, 12.6875, 13.75, 18.125, 12.0625, 20.25], "probabilities": [0.0011713607236742973, 0.0008569859783165157, 0.001412931364029646, 0.000971092376857996, 0.0004882959183305502, 0.0005889972089789808, 0.0003802852297667414, 0.0004882959183305502, 0.0023295290302485228, 0.0005197881837375462, 0.00045871149632148445, 0.00045871149632148445, 0.0013273257063701749, 0.10544253885746002, 0.00024553065304644406, 0.8828595876693726], "choice": "candidate-bb1945d8d438379625ee", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.012007355690002441}
|
| 14 |
+
{"panel_index": 13, "task_family": "browser", "logits": [13.5625, 13.375, 12.375, 12.4375, 12.3125, 12.4375, 13.0625, 13.5, 12.9375, 14.0625, 17.875, 12.375, 13.9375, 14.3125, 16.75, 13.375], "probabilities": [0.009061223827302456, 0.007512019947171211, 0.002763517666608095, 0.0029417492914944887, 0.0025960844941437244, 0.0029417492914944887, 0.005495910998433828, 0.008512231521308422, 0.004850123543292284, 0.014939434826374054, 0.6762104034423828, 0.002763517666608095, 0.013184004463255405, 0.019182609394192696, 0.21953336894512177, 0.007512019947171211], "choice": "candidate-94dddd06eb43ed2921b0", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.045530080795288086}
|
| 15 |
+
{"panel_index": 14, "task_family": "browser", "logits": [12.9375, 20.125, 12.625, 12.25, 12.125, 11.875, 12.0625, 12.3125, 12.4375, 11.75, 12.0, 15.4375, 12.8125, 11.9375, 11.4375, 12.1875], "probabilities": [0.0007450456032529473, 0.9855404496192932, 0.0005450871540233493, 0.00037463250919245183, 0.00033061200520023704, 0.00025748086045496166, 0.000310581352096051, 0.0003987941308878362, 0.0004518929636105895, 0.0002272260608151555, 0.0002917640667874366, 0.009076514281332493, 0.0006575004663318396, 0.0002740870404522866, 0.00016624215641058981, 0.0003519345191307366], "choice": "candidate-b56ad12ceea4c52feb3f", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0013546943664550781}
|
| 16 |
+
{"panel_index": 15, "task_family": "browser", "logits": [16.5, 15.875, 14.75, 19.5, 18.125, 15.375, 14.25, 17.125, 14.5, 14.1875, 15.4375, 14.0, 16.375, 14.875, 12.6875, 13.875], "probabilities": [0.03225051611661911, 0.01726245880126953, 0.005604299250990152, 0.6477689146995544, 0.16378162801265717, 0.010470209643244743, 0.0033991800155490637, 0.06025189161300659, 0.004364632535725832, 0.0031932340934872627, 0.011145479045808315, 0.0026472839526832104, 0.028460979461669922, 0.006350503768771887, 0.0007125068223103881, 0.0023362203501164913], "choice": "candidate-da095abe0a63c03f3114", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.025771796703338623}
|
| 17 |
+
{"panel_index": 16, "task_family": "hs3", "logits": [15.4375, 18.625], "probabilities": [0.03963884338736534, 0.9603611826896667], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0024489164352416992}
|
| 18 |
+
{"panel_index": 17, "task_family": "hs3", "logits": [17.375, 18.5], "probabilities": [0.24508501589298248, 0.7549149990081787], "choice": "no", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.07573629915714264}
|
| 19 |
+
{"panel_index": 18, "task_family": "hs3", "logits": [19.625, 16.5], "probabilities": [0.957912266254425, 0.04208773002028465], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.004760846495628357}
|
| 20 |
+
{"panel_index": 19, "task_family": "hs3", "logits": [18.875, 12.8125], "probabilities": [0.9976769089698792, 0.0023231622762978077], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 21 |
+
{"panel_index": 20, "task_family": "hs3", "logits": [19.125, 13.0], "probabilities": [0.9978172779083252, 0.0021827167365700006], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001404457725584507}
|
| 22 |
+
{"panel_index": 21, "task_family": "hs3", "logits": [11.625, 18.5], "probabilities": [0.0010322309099137783, 0.9989677667617798], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 6.651878356933594e-05}
|
| 23 |
+
{"panel_index": 22, "task_family": "hs3", "logits": [19.375, 12.6875], "probabilities": [0.9987550973892212, 0.0012448435882106423], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 7.539987564086914e-05}
|
| 24 |
+
{"panel_index": 23, "task_family": "hs3", "logits": [19.0, 15.4375], "probabilities": [0.9724147915840149, 0.027585279196500778], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0016279220581054688}
|
| 25 |
+
{"panel_index": 24, "task_family": "hs3", "logits": [19.0, 12.875], "probabilities": [0.9978172779083252, 0.0021827167365700006], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 26 |
+
{"panel_index": 25, "task_family": "hs3", "logits": [13.125, 19.0], "probabilities": [0.0028009277302771807, 0.9971990585327148], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 6.984919309616089e-10}
|
| 27 |
+
{"panel_index": 26, "task_family": "hs3", "logits": [18.875, 16.125], "probabilities": [0.9399133324623108, 0.06008664891123772], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.006683349609375}
|
| 28 |
+
{"panel_index": 27, "task_family": "hs3", "logits": [19.25, 13.0], "probabilities": [0.9980732202529907, 0.0019267346942797303], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00022608041763305664}
|
| 29 |
+
{"panel_index": 28, "task_family": "hs3", "logits": [15.75, 18.5], "probabilities": [0.06008664891123772, 0.9399133324623108], "choice": "no", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0034342408180236816}
|
| 30 |
+
{"panel_index": 29, "task_family": "hs3", "logits": [12.0625, 18.5], "probabilities": [0.0015978460432961583, 0.9984021782875061], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.656612873077393e-10}
|
| 31 |
+
{"panel_index": 30, "task_family": "hs3", "logits": [18.75, 14.125], "probabilities": [0.9902915358543396, 0.00970847811549902], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005828738212585449}
|
| 32 |
+
{"panel_index": 31, "task_family": "hs3", "logits": [19.25, 13.0625], "probabilities": [0.9979493021965027, 0.002050743205472827], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00012400851119309664}
|
| 33 |
+
{"panel_index": 32, "task_family": "boolq", "logits": [13.0625, 18.5], "probabilities": [0.004331502597779036, 0.9956685304641724], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00027811527252197266}
|
| 34 |
+
{"panel_index": 33, "task_family": "boolq", "logits": [12.375, 18.875], "probabilities": [0.0015011822106316686, 0.998498797416687], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 35 |
+
{"panel_index": 34, "task_family": "boolq", "logits": [12.8125, 18.625], "probabilities": [0.002981032943353057, 0.9970190525054932], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 36 |
+
{"panel_index": 35, "task_family": "boolq", "logits": [19.5, 13.5625], "probabilities": [0.997368335723877, 0.0026316738221794367], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00015908479690551758}
|
| 37 |
+
{"panel_index": 36, "task_family": "boolq", "logits": [19.875, 12.1875], "probabilities": [0.9995417594909668, 0.000458312890259549], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.7755799237638712e-05}
|
| 38 |
+
{"panel_index": 37, "task_family": "boolq", "logits": [20.0, 12.3125], "probabilities": [0.9995417594909668, 0.000458312890259549], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.9103830456733704e-11}
|
| 39 |
+
{"panel_index": 38, "task_family": "boolq", "logits": [19.625, 12.25], "probabilities": [0.999373733997345, 0.0006263343966566026], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 40 |
+
{"panel_index": 39, "task_family": "boolq", "logits": [13.125, 19.125], "probabilities": [0.0024726237170398235, 0.9975274205207825], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00032836198806762695}
|
| 41 |
+
{"panel_index": 40, "task_family": "boolq", "logits": [19.75, 12.0625], "probabilities": [0.9995417594909668, 0.000458312890259549], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.7755799237638712e-05}
|
| 42 |
+
{"panel_index": 41, "task_family": "boolq", "logits": [13.1875, 18.75], "probabilities": [0.003824483836069703, 0.9961755275726318], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0002456542570143938}
|
| 43 |
+
{"panel_index": 42, "task_family": "boolq", "logits": [19.625, 12.3125], "probabilities": [0.9993333220481873, 0.0006667023408226669], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.297494888305664e-05}
|
| 44 |
+
{"panel_index": 43, "task_family": "boolq", "logits": [14.75, 18.375], "probabilities": [0.02595735900104046, 0.9740425944328308], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0016279257833957672}
|
| 45 |
+
{"panel_index": 44, "task_family": "boolq", "logits": [19.125, 12.8125], "probabilities": [0.9981897473335266, 0.0018102104077115655], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00010955333709716797}
|
| 46 |
+
{"panel_index": 45, "task_family": "boolq", "logits": [20.5, 13.75], "probabilities": [0.9988304972648621, 0.0011695100693032146], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.1641532182693481e-10}
|
| 47 |
+
{"panel_index": 46, "task_family": "boolq", "logits": [13.875, 18.625], "probabilities": [0.008577484637498856, 0.9914224743843079], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.000515492632985115}
|
| 48 |
+
{"panel_index": 47, "task_family": "boolq", "logits": [12.8125, 18.625], "probabilities": [0.002981032943353057, 0.9970190525054932], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00034935888834297657}
|
| 49 |
+
{"panel_index": 48, "task_family": "mnli", "logits": [16.25, 19.375, 14.25], "probabilities": [0.04184935986995697, 0.9524869918823242, 0.005663693882524967], "choice": "neutral", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.004734925925731659}
|
| 50 |
+
{"panel_index": 49, "task_family": "mnli", "logits": [13.125, 20.375, 12.3125], "probabilities": [0.0007094471948221326, 0.9989757537841797, 0.0003148153191432357], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00010228157043457031}
|
| 51 |
+
{"panel_index": 50, "task_family": "mnli", "logits": [20.375, 12.375, 11.4375], "probabilities": [0.9995334148406982, 0.00033530613291077316, 0.00013130776642356068], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 9.614229202270508e-05}
|
| 52 |
+
{"panel_index": 51, "task_family": "mnli", "logits": [13.5, 20.25, 12.5], "probabilities": [0.0011690071551129222, 0.9984009861946106, 0.00043005382758565247], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.1641532182693481e-10}
|
| 53 |
+
{"panel_index": 52, "task_family": "mnli", "logits": [11.25, 12.0, 19.875], "probabilities": [0.00017945973377209157, 0.00037991636781953275, 0.999440610408783], "choice": "contradiction", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.83393669128418e-05}
|
| 54 |
+
{"panel_index": 53, "task_family": "mnli", "logits": [20.125, 12.0625, 11.1875], "probabilities": [0.9995536208152771, 0.0003149974509142339, 0.00013131041487213224], "choice": "entailment", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 1.9081664504483342e-05}
|
| 55 |
+
{"panel_index": 54, "task_family": "mnli", "logits": [16.5, 19.75, 13.5], "probabilities": [0.03725764527916908, 0.9608873724937439, 0.001854949165135622], "choice": "neutral", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.022902607917785645}
|
| 56 |
+
{"panel_index": 55, "task_family": "mnli", "logits": [20.125, 12.375, 11.375], "probabilities": [0.9994112253189087, 0.00043048898805864155, 0.00015836799866519868], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.777576446533203e-05}
|
| 57 |
+
{"panel_index": 56, "task_family": "mnli", "logits": [19.875, 12.125, 11.375], "probabilities": [0.9993662238121033, 0.00043046960490755737, 0.00020333938300609589], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.4776668548583984e-05}
|
| 58 |
+
{"panel_index": 57, "task_family": "mnli", "logits": [14.375, 17.125, 19.625], "probabilities": [0.004826048389077187, 0.07549208402633667, 0.9196819067001343], "choice": "contradiction", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 9.313225746154785e-10}
|
| 59 |
+
{"panel_index": 58, "task_family": "mnli", "logits": [13.875, 19.75, 15.3125], "probabilities": [0.0027682832442224026, 0.9855768084526062, 0.01165490597486496], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0006983987987041473}
|
| 60 |
+
{"panel_index": 59, "task_family": "mnli", "logits": [19.25, 14.4375, 16.625], "probabilities": [0.9254397749900818, 0.007521526888012886, 0.0670386329293251], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.03888293355703354}
|
| 61 |
+
{"panel_index": 60, "task_family": "mnli", "logits": [14.1875, 14.875, 19.875], "probabilities": [0.0033540891017764807, 0.006670401897281408, 0.9899755120277405], "choice": "contradiction", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0003929487429559231}
|
| 62 |
+
{"panel_index": 61, "task_family": "mnli", "logits": [19.5, 16.375, 14.0], "probabilities": [0.9541769027709961, 0.041923608630895615, 0.0038995034992694855], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.018091142177581787}
|
| 63 |
+
{"panel_index": 62, "task_family": "mnli", "logits": [19.75, 12.5625, 11.625], "probabilities": [0.9989491105079651, 0.000755182234570384, 0.0002957336255349219], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00010657310485839844}
|
| 64 |
+
{"panel_index": 63, "task_family": "mnli", "logits": [20.0, 12.1875, 11.375], "probabilities": [0.9994162321090698, 0.0004044089000672102, 0.00017945535364560783], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.9103830456733704e-11}
|
| 65 |
+
{"panel_index": 64, "task_family": "score", "logits": [18.75, 18.375], "probabilities": [0.5926666259765625, 0.4073334038257599], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 2.9802322387695312e-08}
|
| 66 |
+
{"panel_index": 65, "task_family": "score", "logits": [12.1875, 19.75], "probabilities": [0.0005193048273213208, 0.999480664730072], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.153085708618164e-05}
|
| 67 |
+
{"panel_index": 66, "task_family": "score", "logits": [12.0625, 19.75], "probabilities": [0.000458312890259549, 0.9995417594909668], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.9103830456733704e-11}
|
| 68 |
+
{"panel_index": 67, "task_family": "score", "logits": [13.5625, 19.375], "probabilities": [0.002981032943353057, 0.9970190525054932], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00019174814224243164}
|
| 69 |
+
{"panel_index": 68, "task_family": "score", "logits": [12.5625, 19.5], "probabilities": [0.0009697520872578025, 0.9990302324295044], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.8770179748535156e-05}
|
| 70 |
+
{"panel_index": 69, "task_family": "score", "logits": [19.75, 12.5625], "probabilities": [0.9992445707321167, 0.0007554056355729699], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 4.57763671875e-05}
|
| 71 |
+
{"panel_index": 70, "task_family": "score", "logits": [12.75, 19.75], "probabilities": [0.0009110510000027716, 0.9990890026092529], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.8770179748535156e-05}
|
| 72 |
+
{"panel_index": 71, "task_family": "score", "logits": [12.625, 19.75], "probabilities": [0.0008040856919251382, 0.9991958737373352], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 73 |
+
{"panel_index": 72, "task_family": "score", "logits": [19.125, 18.5], "probabilities": [0.6513549089431763, 0.3486451208591461], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.1201455295085907}
|
| 74 |
+
{"panel_index": 73, "task_family": "score", "logits": [11.9375, 19.5], "probabilities": [0.0005193048273213208, 0.999480664730072], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 75 |
+
{"panel_index": 74, "task_family": "score", "logits": [12.1875, 19.75], "probabilities": [0.0005193048273213208, 0.999480664730072], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 76 |
+
{"panel_index": 75, "task_family": "score", "logits": [12.6875, 19.375], "probabilities": [0.0012448435882106423, 0.9987550973892212], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 77 |
+
{"panel_index": 76, "task_family": "score", "logits": [15.1875, 19.375], "probabilities": [0.014957085251808167, 0.985042929649353], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.001957833766937256}
|
| 78 |
+
{"panel_index": 77, "task_family": "score", "logits": [15.375, 19.125], "probabilities": [0.022977372631430626, 0.977022647857666], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0013620853424072266}
|
| 79 |
+
{"panel_index": 78, "task_family": "score", "logits": [13.75, 19.625], "probabilities": [0.0028009277302771807, 0.9971990585327148], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00016927719116210938}
|
| 80 |
+
{"panel_index": 79, "task_family": "score", "logits": [13.375, 19.375], "probabilities": [0.0024726237170398235, 0.9975274205207825], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00028990721330046654}
|
evaluation/per-question/reference.jsonl
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"panel_index": 0, "id": "mind2web:bef473f1-82a1-4359-a2c0-59b6dc2f6abb:22ad3562-e0f4-42c3-b096-8c173a47673c", "task_family": "browser", "gold": "candidate-c59cf6b3cdd1ee8dfe92", "candidate_ids": ["candidate-f1094edca3d95c3ab877", "candidate-365bc85de0d8df852b74", "candidate-528c0da02e102bf4290b", "candidate-68a3dbfbe50374b33385", "candidate-c59cf6b3cdd1ee8dfe92", "candidate-6a5d2547e9b7961af9f2", "candidate-a44d590fb75558028c8a", "candidate-519332b584cbfc17ffca", "candidate-beb9d47a1825f26d1023", "candidate-d27c1c551c69db04ab36", "candidate-ed6c918d8bc2facd9fb5", "candidate-7111d2a0c81d7ac049be", "candidate-7dae2a8703bdc9f5d043", "candidate-7d02e4befb688b244764", "candidate-7a89b1b7a87361ea291e", "candidate-089cbe46c0766ba6701e"], "prompt_sha256": "b3fd1770027f0a5409a9d04f32bde5d632ef802787a60f4932e72ecc89167216", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [12.8125, 12.9375, 12.125, 12.1875, 20.625, 12.1875, 12.25, 12.25, 12.8125, 12.75, 12.0, 11.4375, 11.9375, 12.3125, 11.4375, 12.0], "probabilities": [0.00040314404759556055, 0.0004568220756482333, 0.00020271356333978474, 0.0002157874550903216, 0.9962903261184692, 0.0002157874550903216, 0.00022970455756876618, 0.00022970455756876618, 0.00040314404759556055, 0.0003787188034038991, 0.00017889408627524972, 0.00010193077469011769, 0.00016805544146336615, 0.0002445192658342421, 0.00010193077469011769, 0.00017889408627524972], "choice": "candidate-c59cf6b3cdd1ee8dfe92"}
|
| 2 |
+
{"panel_index": 1, "id": "mind2web:5f9182dc-d35d-4c0e-9abe-cd913c136528:8450e2c6-f8aa-40bc-876e-21cf29a8cb77", "task_family": "browser", "gold": "candidate-d2449a13158601200ec0", "candidate_ids": ["candidate-0ac56ef6c8e3a6864ae3", "candidate-33ff2e219977f0035115", "candidate-ae85d05843eef166dc37", "candidate-1491fc48d5480406272d", "candidate-458964bfea8f39410a8c", "candidate-6a3568401638b21de4c7", "candidate-5554637f601e90894848", "candidate-fb424fe48f87098b50ab", "candidate-d2449a13158601200ec0", "candidate-d26fca5ac7bd53d243e4", "candidate-59a124a5de2d7629d231", "candidate-37c56ce11f36b7ace139", "candidate-bfd1999ae1c4a3ad89f3", "candidate-77988599c59cbb77c809", "candidate-eb1b6dd49112ec23d746", "candidate-a8a56f7aec962e7413b8"], "prompt_sha256": "9aca18e5ab33a35ca47a65fe691068ba467052dea071279b1751dbfdc95fb224", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [13.375, 13.375, 12.8125, 12.8125, 12.8125, 12.75, 12.5, 13.3125, 20.625, 16.5, 13.0, 13.3125, 13.0625, 13.5625, 12.75, 13.0], "probabilities": [0.0006938408478163183, 0.0006938408478163183, 0.00039533860399387777, 0.00039533860399387777, 0.00039533860399387777, 0.00037138626794330776, 0.000289235933450982, 0.0006518031586892903, 0.9770007133483887, 0.015791745856404305, 0.00047686940524727106, 0.0006518031586892903, 0.0005076248198747635, 0.0008369318093173206, 0.00037138626794330776, 0.00047686940524727106], "choice": "candidate-d2449a13158601200ec0"}
|
| 3 |
+
{"panel_index": 2, "id": "mind2web:0b59dd33-7f6a-48df-aa1e-9cc67177287f:b99c5581-8a56-4bd7-bbe4-782795ebf93c", "task_family": "browser", "gold": "candidate-9409b72a36db8d4a0a79", "candidate_ids": ["candidate-0120149ac9a61a819265", "candidate-4312931ba215cd0b2310", "candidate-dd54dfdd68be2163e2b6", "candidate-d8ba462bbea45dc4eb77", "candidate-cb3001c9ecbf0dfd250c", "candidate-9409b72a36db8d4a0a79", "candidate-f6ba1cc847604be6bc73", "candidate-2f2099d2f07ae88921ce", "candidate-77f1025025b0e2d0d1fd", "candidate-e743f54599422c4ee674", "candidate-cd31d3ac291210a0a5e6", "candidate-991d9567f58fb994a717", "candidate-d65d92a915149075d6ad", "candidate-98c01750f337ab587383", "candidate-3939d47e8ecda2b48209", "candidate-a7c78ea57e9f0bf1b5b6"], "prompt_sha256": "d960c318e8c88d1c79f12958cf002b9e2b0b9f76a03321dac1dd1d86d3d1c031", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [12.75, 13.375, 12.5625, 12.8125, 13.625, 20.25, 12.625, 13.0, 12.9375, 11.5625, 12.0625, 12.0, 12.6875, 12.5, 11.1875, 12.5], "probabilities": [0.0005486779846251011, 0.0010250654304400086, 0.00045487002353183925, 0.0005840646917931736, 0.0013162100221961737, 0.992033064365387, 0.00048420665552839637, 0.0007045164820738137, 0.0006618319894187152, 0.0001673373335506767, 0.0002758926129899919, 0.00025917711900547147, 0.0005154352984391153, 0.00042731084977276623, 0.00011500916298246011, 0.00042731084977276623], "choice": "candidate-9409b72a36db8d4a0a79"}
|
| 4 |
+
{"panel_index": 3, "id": "mind2web:274571ea-fc2f-4353-86ba-00ecb112d6d2:e6345ea9-a5a4-4b88-95b5-4efececed261", "task_family": "browser", "gold": "candidate-3b25ae8d064a298ef021", "candidate_ids": ["candidate-6c861878e1df45af7cb0", "candidate-5dc0c98aa1601abb7255", "candidate-df37b5f34129ec35c5e2", "candidate-7fc3aa839f6eddc4f5e0", "candidate-a105394e5c10cc41573b", "candidate-99243ece7e7fc3575645", "candidate-3b25ae8d064a298ef021", "candidate-c9f0a04f58f5031fe4e6", "candidate-b039454179929264efa9", "candidate-5dc632768de9b7f12a36", "candidate-0b63de0d609d3e1730f1", "candidate-ed1ed5d91fa057eb68be", "candidate-9827a0d3bbbb4c3f2ef3", "candidate-720f59bcd44dc13ecfd0", "candidate-520f72cca9eaf711108b", "candidate-d45432c0ee25d5e74fe8"], "prompt_sha256": "be5d6ca2e7f4e736ac3a410e66cdb9fb2dce3de6f3cb2d58e0425d11bd3e5264", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [17.625, 14.5625, 13.75, 13.875, 14.1875, 14.1875, 20.5, 14.6875, 14.4375, 14.3125, 17.875, 13.0625, 13.625, 15.0, 13.0, 13.75], "probabilities": [0.04895426332950592, 0.0022896211594343185, 0.0010160132078453898, 0.001151293865405023, 0.0015736321220174432, 0.0015736321220174432, 0.8677350282669067, 0.0025944807566702366, 0.002020583488047123, 0.001783158746547997, 0.06285851448774338, 0.000510883517563343, 0.0008966285386122763, 0.003546234918758273, 0.00047993066255003214, 0.0010160132078453898], "choice": "candidate-3b25ae8d064a298ef021"}
|
| 5 |
+
{"panel_index": 4, "id": "mind2web:d637c171-dc6e-4a4e-a162-9c230e822932:9053f0e2-da05-4721-87b5-13edf923052b", "task_family": "browser", "gold": "candidate-34e89f28b090ce5327b0", "candidate_ids": ["candidate-34e89f28b090ce5327b0", "candidate-08b4435b3120f48b358c", "candidate-55a0ff963ae8b79b2ac8", "candidate-f95a96c1f7da545920f5", "candidate-98f2cb734f98ff10ff0a", "candidate-d82967df7ccb99323f3d", "candidate-244abe86bc95c904fd2d", "candidate-c71ff755e4fd0c874dfe", "candidate-8d25c242ac6c2febeb64", "candidate-ad36e87635c46e6e4d49", "candidate-88d796f046707b090dc5", "candidate-c16587c8a38af93f0938", "candidate-a96e4a183f184af811a8", "candidate-6d24b42f26f361f2c6c0", "candidate-227d6d1ece1c8d3e341b", "candidate-7a2474fe7e75e66c0d24"], "prompt_sha256": "e3450db0dcc97dbea2e0858fda6b8120ec2573e8c374f5f799c7ed8142a80f80", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [20.25, 13.6875, 12.75, 13.375, 12.25, 12.0625, 11.4375, 11.75, 12.0625, 10.75, 11.6875, 11.375, 12.125, 12.1875, 11.1875, 12.0625], "probabilities": [0.9943780303001404, 0.0014044102281332016, 0.000549974967725575, 0.0010274884989485145, 0.0003335766668897122, 0.0002765447716228664, 0.00014802375517319888, 0.00020232447423040867, 0.0002765447716228664, 7.44310236768797e-05, 0.00019006624643225223, 0.00013905545347370207, 0.00029438038473017514, 0.0003133662976324558, 0.00011528102186275646, 0.0002765447716228664], "choice": "candidate-34e89f28b090ce5327b0"}
|
| 6 |
+
{"panel_index": 5, "id": "mind2web:1efdcd9d-ebc6-4bb7-8823-e54dfe25f409:f6e611c9-ad21-49ca-a841-7ad529b56c95", "task_family": "browser", "gold": "candidate-fb3025ec54721504f9b5", "candidate_ids": ["candidate-fb3025ec54721504f9b5", "candidate-cfc771a1bb7562ebe782", "candidate-68c510f13da5956437bc", "candidate-ac1438ee133b5015c383", "candidate-58473582dc6c65ae4b7e", "candidate-30cbf6942451aa0f1470", "candidate-7ec55a659d963a2cc94d", "candidate-49df5ddbae6f1748c0de", "candidate-b5be7aabde3c609cc40f", "candidate-0e3fd6a816e692376632", "candidate-a3e570f7c516b4577418", "candidate-8a6b58c9cb19ccc7e1fa", "candidate-d8437c7bdf7236744ac4", "candidate-cbf6c1cc621042e7aff0", "candidate-b6291f89245ae394921e", "candidate-c5a94f8fac1dcdad627a"], "prompt_sha256": "36792836eaeed2b15b8a762fc57a57b4b104a27be92bff8c84681bb90d68476b", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [20.25, 13.3125, 12.8125, 12.375, 15.0625, 12.375, 11.9375, 15.8125, 12.5625, 11.25, 11.9375, 11.625, 12.125, 12.6875, 11.875, 12.0], "probabilities": [0.9781951308250427, 0.0009495273698121309, 0.0005759174819104373, 0.0003718402876984328, 0.005464153364300728, 0.0003718402876984328, 0.0002400781522737816, 0.011567611247301102, 0.00044852501014247537, 0.00012071886158082634, 0.0002400781522737816, 0.0001756449055392295, 0.00028958951588720083, 0.0005082454299554229, 0.0002255325234727934, 0.00025556181208230555], "choice": "candidate-fb3025ec54721504f9b5"}
|
| 7 |
+
{"panel_index": 6, "id": "mind2web:d1de3d1a-3df1-4421-98f3-f8d078752893:e1f18ee3-1577-44fb-a283-1be215e5ae52", "task_family": "browser", "gold": "candidate-888bb5997d1aba7b25f7", "candidate_ids": ["candidate-8d8171f50795c09388a0", "candidate-8d26ef6bd8e9f762c73d", "candidate-bb2ff00206221aa2b02d", "candidate-0196e00c3795624bcc41", "candidate-a9f93c25b378cb9448f1", "candidate-0b7a0a178194ef69867b", "candidate-880053bf9e617437e7cb", "candidate-cf42d1a8a48eeea86134", "candidate-67a5886245e859456378", "candidate-b7c7a409d7192fcbe70f", "candidate-888bb5997d1aba7b25f7", "candidate-dd07004fe07f84bf771d", "candidate-a8b562a5018f0346093a", "candidate-ba9bcd4bb262d6387c1e", "candidate-d4dbe03106eb96d6c9dc", "candidate-b3ddfc4f75cd756d4767"], "prompt_sha256": "873952066cc2833adc3b2b3af9b8f8261f195c13cd41118b830744f22683e25f", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [13.75, 13.5625, 12.5625, 12.4375, 14.25, 13.5625, 15.3125, 14.75, 13.8125, 16.25, 19.75, 12.6875, 13.25, 13.4375, 11.8125, 14.5625], "probabilities": [0.00230879127047956, 0.001914055086672306, 0.0007041415083222091, 0.0006214026943780482, 0.0038065530825406313, 0.001914055086672306, 0.011014626361429691, 0.0062759448774158955, 0.002457695547491312, 0.028126833960413933, 0.931432843208313, 0.0007978968787938356, 0.0014003525720909238, 0.0016891476698219776, 0.0003326128935441375, 0.005202941596508026], "choice": "candidate-888bb5997d1aba7b25f7"}
|
| 8 |
+
{"panel_index": 7, "id": "mind2web:80e12375-19ad-400f-9e35-2a3853173bed:e89bb795-2d24-4e2c-bcae-1294e3501dfa", "task_family": "browser", "gold": "candidate-41c5af16d7bf373cc74c", "candidate_ids": ["candidate-41c5af16d7bf373cc74c", "candidate-03bd5f2aaa71b912d85f", "candidate-39aafe4d05bcbbf1a9d8", "candidate-7db455c43ca404d644b3", "candidate-8bdbe188e49bad6ea2d9", "candidate-92648d446426dec0e918", "candidate-36e6b14fcf2f5e8e9c23", "candidate-fe113518f51162e48db7", "candidate-f6b0a3559a94799eebe8", "candidate-853b6e262fd75054c736", "candidate-a77385eabd707033f244", "candidate-b7024479880013db2adc", "candidate-b5cc7d0d8e9ea791112e", "candidate-1b8a962d51e9c3897102", "candidate-2b4df4c4f3a6e5c34716", "candidate-c3a3c2242e59afcbb3fb"], "prompt_sha256": "33badf184ca6146904e42f3db06fbd31f551b1ce3082595b5db35494ee245a4b", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [20.125, 13.0, 12.125, 11.875, 12.1875, 11.5625, 11.5, 11.6875, 11.9375, 10.5625, 11.5, 11.1875, 12.1875, 11.9375, 10.6875, 11.6875], "probabilities": [0.9960787892341614, 0.000801577465608716, 0.000334147218381986, 0.00026023408281616867, 0.00035569784813560545, 0.00019039132166653872, 0.00017885610577650368, 0.0002157416456611827, 0.00027701776707544923, 7.00410600984469e-05, 0.00017885610577650368, 0.0001308539358433336, 0.00035569784813560545, 0.00027701776707544923, 7.936691690701991e-05, 0.0002157416456611827], "choice": "candidate-41c5af16d7bf373cc74c"}
|
| 9 |
+
{"panel_index": 8, "id": "mind2web:0245df99-2909-465a-861e-7fbca948e82f:dc1847f7-919b-4a2f-b778-2ee33edacc46", "task_family": "browser", "gold": "candidate-a1a3ad1a8495aa678cbe", "candidate_ids": ["candidate-80e6a190025076e4821d", "candidate-cd39817d36a3b7f8086c", "candidate-a1a3ad1a8495aa678cbe", "candidate-74eaca906c939e439348", "candidate-1c996fce61cea008cd97", "candidate-b86e5504c07b50684120", "candidate-ca5385f6ff0f2b510b2a", "candidate-27ecbf7b79508deea5b5", "candidate-335d71d7afcd7c39e60c", "candidate-3fdeb3ac89f0ecf87cff", "candidate-d594855a68f3d65bda29", "candidate-ff28229b986139c46fad", "candidate-4d0fd233b1f9ec995708", "candidate-491e3298e78304983a55", "candidate-438c3b7b4da6155b0b79", "candidate-66345b239aef37d2f31c"], "prompt_sha256": "19bd775a219dfb7da24c37ac1e5319666e84c1584357f8e31021b2c8e9fe3952", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [12.375, 12.5625, 20.375, 12.8125, 12.5, 11.9375, 13.3125, 12.5625, 12.1875, 11.0, 11.4375, 11.3125, 12.0, 11.625, 10.9375, 11.75], "probabilities": [0.00033400097163394094, 0.0004028820840176195, 0.995642900466919, 0.0005173108074814081, 0.00037847270141355693, 0.00021564723283518106, 0.0008529013721272349, 0.0004028820840176195, 0.00027689654962159693, 8.444867125945166e-05, 0.00013079667405690998, 0.0001154276542365551, 0.0002295552840223536, 0.0001577708899276331, 7.93321814853698e-05, 0.0001787778310244903], "choice": "candidate-a1a3ad1a8495aa678cbe"}
|
| 10 |
+
{"panel_index": 9, "id": "mind2web:8eae88ef-9641-43c6-be6d-f8abc96d99fa:282c09d0-c9e0-4007-b88a-27887fe1e388", "task_family": "browser", "gold": "candidate-976c2731e7518a1f1b2b", "candidate_ids": ["candidate-976c2731e7518a1f1b2b", "candidate-55380cc26f4800be25ce", "candidate-495e2ce778804116afed", "candidate-6a3ce8bdfda5c7e09a24", "candidate-7e020b58dbe771cceb6b", "candidate-ca7f03f7b7139457a468", "candidate-4556165f722234b67398", "candidate-0232a0c57f2c0a004a53", "candidate-892461b7e0e4bf188cd7", "candidate-98a8e1da12737fa9372c", "candidate-f6c802447f1e19395568", "candidate-e95807766432f0e50c72", "candidate-955408afe8906a0bb10d", "candidate-cd1be178e6146e0fd80a", "candidate-38a919ada71ae1df352a", "candidate-38c746333273edc02a2e"], "prompt_sha256": "05bf3a03b2c0d0a936c4e37c690299142bb6148b2eda9be40aa00c6a13a66811", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [20.0, 15.4375, 13.375, 12.875, 13.0, 13.625, 12.3125, 13.0, 12.875, 11.9375, 12.4375, 12.25, 12.75, 12.9375, 11.5625, 12.5625], "probabilities": [0.979435920715332, 0.010221331380307674, 0.0012994963908568025, 0.0007881844067014754, 0.0008931299671530724, 0.0016685865120962262, 0.0004490939318202436, 0.0008931299671530724, 0.0007881844067014754, 0.00030865747248753905, 0.0005088900798000395, 0.00042188470251858234, 0.0006955703138373792, 0.0008390179136767983, 0.00021213694708421826, 0.0005766480462625623], "choice": "candidate-976c2731e7518a1f1b2b"}
|
| 11 |
+
{"panel_index": 10, "id": "mind2web:3f0988e0-e3a4-4dd7-a89f-482175a474a8:a684c98a-c238-4bef-b2ad-f476f07d73f8", "task_family": "browser", "gold": "candidate-016db9380de74cbcab5d", "candidate_ids": ["candidate-3f6330dc6b89ca9febbe", "candidate-0acffd69c24f4be0a17e", "candidate-ae8922f024c12c266278", "candidate-c12d0dfb7e58d7f22bc7", "candidate-86f75c1fc5227d6081b8", "candidate-6ff913a423628d92b636", "candidate-87e0dada00c5e19e9dce", "candidate-2c14d16a67e5d74c7a66", "candidate-f6a74c2240fb95cf9da0", "candidate-ac52e1e2c9fab34583a1", "candidate-0c2b330dc3030639c697", "candidate-016db9380de74cbcab5d", "candidate-cab20f7657710f02c34a", "candidate-d48a5d89e157a64dd7b4", "candidate-b4b12517818528229992", "candidate-09c561e352fda884d806"], "prompt_sha256": "8c9d16ca40b336c54eab472ed9a7d490065e655ff833efc53466acbdaf85024c", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.25, 14.6875, 12.6875, 13.25, 15.625, 14.125, 13.5, 14.9375, 13.875, 14.4375, 14.125, 19.0, 14.0625, 15.1875, 12.375, 13.625], "probabilities": [0.007526915520429611, 0.011657915078103542, 0.0015777271473780274, 0.0027689975686371326, 0.029769528657197952, 0.006642479449510574, 0.0035554631613194942, 0.014969058334827423, 0.005173168145120144, 0.009079193696379662, 0.006642479449510574, 0.8699931502342224, 0.006240031681954861, 0.01922065205872059, 0.001154289930127561, 0.004028867930173874], "choice": "candidate-016db9380de74cbcab5d"}
|
| 12 |
+
{"panel_index": 11, "id": "mind2web:5b56d5b8-1f41-43ca-9f21-369d849f1aa0:edb09eb7-6a8c-4aeb-9b52-796762ca821d", "task_family": "browser", "gold": "candidate-2dc16f49fe3acb051b39", "candidate_ids": ["candidate-021004f4c9756e9f14bf", "candidate-9393f92278ae33837cfc", "candidate-d53f82cb73ef67bc0ce8", "candidate-332f924943c28c247430", "candidate-afd6b2a84985b30c66cd", "candidate-21f93b804332aae201c9", "candidate-bdba9a0649055da71f94", "candidate-0cf8d953323f4b29b10c", "candidate-dea9b4fc8a207df523df", "candidate-fa70b5bc02a04fa0a18a", "candidate-1932fde20b275a9b3081", "candidate-5c9c9ac05919970737c1", "candidate-544fea69d385caac3601", "candidate-2dc16f49fe3acb051b39", "candidate-ca60dc8da2fba3f06111", "candidate-ea45663120f48fa54b88"], "prompt_sha256": "fc3a0ccb7f4fba5d2ba07c7ec21def24cb6fe447d1c9b124d867a6654c6c09b4", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.5625, 13.5, 13.5625, 13.1875, 12.6875, 12.5625, 12.75, 13.0625, 13.625, 16.625, 13.5, 13.125, 14.6875, 19.625, 12.25, 13.0625], "probabilities": [0.005850622430443764, 0.0020219211000949144, 0.0021523237228393555, 0.001479269121773541, 0.00089722202392295, 0.0007917956681922078, 0.0009550878894515336, 0.001305450452491641, 0.0022911368869245052, 0.04601871222257614, 0.0020219211000949144, 0.0013896446907892823, 0.006629623472690582, 0.924310564994812, 0.0005792900337837636, 0.001305450452491641], "choice": "candidate-2dc16f49fe3acb051b39"}
|
| 13 |
+
{"panel_index": 12, "id": "mind2web:6e565708-43e2-492b-9f1d-25d51387dcf7:bee75aa3-6f7d-4626-be6f-1b217ac16733", "task_family": "browser", "gold": "candidate-bb1945d8d438379625ee", "candidate_ids": ["candidate-69abfa80e562043e1fa3", "candidate-c4afb9a40ecda9a03c50", "candidate-ad4a56ee767e7be58ceb", "candidate-16c8c33fbf8679f935d6", "candidate-777d646249275ea3bc88", "candidate-ce64cb6d800f5f6f7886", "candidate-cb8d5f96e26185552cf9", "candidate-8203c66243ac00ab8c67", "candidate-32a97c97039dcc468732", "candidate-6fbd3f273e3c48d30d0c", "candidate-97141ccdf3d685ac91be", "candidate-d5d5ef09f20a3d2ceade", "candidate-04470be9c3bbc28919e6", "candidate-0fb9a6cde602f212d85e", "candidate-4ca405ae57be628d94ee", "candidate-bb1945d8d438379625ee"], "prompt_sha256": "a7f850a48b635b1a855ab90b47855fd8269980f52d324bdf635009f9344a5fa4", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [13.5625, 13.25, 13.75, 13.375, 12.625, 12.875, 12.4375, 12.6875, 14.1875, 12.625, 12.625, 12.5625, 13.625, 18.0, 12.0, 20.25], "probabilities": [0.0011153576197102666, 0.0008160130237229168, 0.001345377997495234, 0.0009246639092452824, 0.0004367803339846432, 0.0005608369829133153, 0.0003621035721153021, 0.0004649501934181899, 0.0020837620832026005, 0.0004367803339846432, 0.0004367803339846432, 0.0004103171231690794, 0.0011872919276356697, 0.09431827813386917, 0.00023379162303172052, 0.894866943359375], "choice": "candidate-bb1945d8d438379625ee"}
|
| 14 |
+
{"panel_index": 13, "id": "mind2web:44a12ff5-0172-444a-b979-f224162c1aa8:5b8da6f5-c53c-4b69-bfad-7bdfd2e6ce20", "task_family": "browser", "gold": "candidate-3ef3bb624316b6eb3508", "candidate_ids": ["candidate-3ef3bb624316b6eb3508", "candidate-5b52b8bc7f6397531f34", "candidate-6a5a6a33fbc65504059f", "candidate-bde4765810f968ad4833", "candidate-1f66a27bf372d26b14b3", "candidate-8033052a6976a4b01205", "candidate-45ec5aa3dbee10fc9a75", "candidate-ea5b9b527b94b21e6535", "candidate-56683febea1548c570fd", "candidate-bd10584aedf8afa78746", "candidate-94dddd06eb43ed2921b0", "candidate-c9197b3d3e5e97a0f6ff", "candidate-331b8cbdf48fd7f61188", "candidate-c8041a4dd7e8b00e705c", "candidate-f5c1b8b97d7add30b043", "candidate-7161a3b5bba5920fcb33"], "prompt_sha256": "c8d922fd930198c507227984857f26500162ceac030c038e92206981e9747cc3", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [13.5625, 13.375, 12.375, 12.375, 12.25, 12.4375, 13.0, 13.4375, 13.0, 14.0, 18.0, 12.375, 13.875, 14.3125, 16.625, 13.375], "probabilities": [0.008534916676580906, 0.0070756953209638596, 0.0026030028238892555, 0.0026030028238892555, 0.002297141822054982, 0.002770882099866867, 0.004863049369305372, 0.007532038260251284, 0.004863049369305372, 0.013219138607382774, 0.7217404842376709, 0.0026030028238892555, 0.011665849015116692, 0.018068421632051468, 0.18248458206653595, 0.0070756953209638596], "choice": "candidate-94dddd06eb43ed2921b0"}
|
| 15 |
+
{"panel_index": 14, "id": "mind2web:caafd610-202e-49d2-85d1-3f167f3ab443:d7bfb473-8c73-4808-96bc-187d00be5ad7", "task_family": "browser", "gold": "candidate-b56ad12ceea4c52feb3f", "candidate_ids": ["candidate-a78c74bd163f786d9539", "candidate-b56ad12ceea4c52feb3f", "candidate-b4ae8d879b43b183d54a", "candidate-9ebb992dd5a329ac8366", "candidate-f2c9f9f313e700cc877d", "candidate-af82db0cdef64a4a6cf2", "candidate-5fdd288dfbfc0a932f78", "candidate-f0729506779d73c72c2f", "candidate-4ffcfa3737d12e4f223c", "candidate-efc032a84043e09d128a", "candidate-4a068ffbc0d398300aaa", "candidate-7f042d5315889e8f72e0", "candidate-137b03b260a683ac0cd6", "candidate-874b3d025779abce748c", "candidate-5de2a5273572d4bc9c39", "candidate-49b9b04e59bbfb197a63"], "prompt_sha256": "d94ca8165f95e231260aacbaa5bc8e99ca44241878d872928f7c96298839ae96", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [12.9375, 20.0, 12.625, 12.25, 12.0625, 11.875, 12.0, 12.375, 12.4375, 11.75, 12.0625, 15.375, 12.875, 12.0, 11.4375, 12.1875], "probabilities": [0.0008430868037976325, 0.9841857552528381, 0.0006168154650367796, 0.00042393067269586027, 0.0003514508716762066, 0.0002913630160037428, 0.00033015754888765514, 0.00048037641681730747, 0.0005113580264151096, 0.0002571269287727773, 0.0003514508716762066, 0.009648618288338184, 0.000792006787378341, 0.00033015754888765514, 0.0001881180942291394, 0.00039824601844884455], "choice": "candidate-b56ad12ceea4c52feb3f"}
|
| 16 |
+
{"panel_index": 15, "id": "mind2web:8368b990-c6ca-4cfe-a7ab-c2a88697639d:bf14f1d4-470f-4110-b3f4-019a9f7d0aed", "task_family": "browser", "gold": "candidate-8b6280aa9682e1a79ce0", "candidate_ids": ["candidate-cc60994b6c36f4d19840", "candidate-f8c1512ffc92c70eafe7", "candidate-53d2672be9bfe9447865", "candidate-da095abe0a63c03f3114", "candidate-e926e62f231edca6157e", "candidate-3eccf28da059e9811f3b", "candidate-c8e6f0df1300d65debaa", "candidate-e76caeae9d53c8942ed1", "candidate-de3ea77a36017ef9c156", "candidate-1ab38af3801bba39e08c", "candidate-8b6280aa9682e1a79ce0", "candidate-451ffc245026adbc3d68", "candidate-77aac247bed375f821ae", "candidate-274f32a25dc920013c6f", "candidate-e97a589066e18a2db4b7", "candidate-c6136bbe715f050ff989"], "prompt_sha256": "52d0c2eeb247ef1c21d83eaac84a9f88d22eb50e65489a0dfb02c1c86b73c5e7", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [16.25, 15.625, 14.5, 19.5, 18.0, 15.5625, 14.1875, 17.0, 14.4375, 14.125, 15.4375, 13.875, 16.375, 14.75, 12.625, 13.8125], "probabilities": [0.026116006076335907, 0.01397889107465744, 0.004538281355053186, 0.6735407114028931, 0.15028725564479828, 0.013131953775882721, 0.0033202776685357094, 0.05528758838772774, 0.004263320937752724, 0.0031191124580800533, 0.011588907800614834, 0.002429167041555047, 0.029593313112854958, 0.005827269051223993, 0.00069596798857674, 0.0022819912992417812], "choice": "candidate-da095abe0a63c03f3114"}
|
| 17 |
+
{"panel_index": 16, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0010788", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "43d8aa81e0401b3731c05f98e2bd2b3dbf59139695ae6b700948ef63b4d14e30", "candidate_token_ids": [32, 33], "logits": [15.375, 18.5], "probabilities": [0.042087726294994354, 0.957912266254425], "choice": "no"}
|
| 18 |
+
{"panel_index": 17, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0001994", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "06ef7447367570160457d3b2f434528fe7391c356d258e3934f257f80f5c01f5", "candidate_token_ids": [32, 33], "logits": [17.625, 18.375], "probabilities": [0.3208213150501251, 0.6791787147521973], "choice": "no"}
|
| 19 |
+
{"panel_index": 18, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0000105", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "9fea3b00dd4a71a0363c8bd87b99ae4334c242011eba5648c197e999aa174c64", "candidate_token_ids": [32, 33], "logits": [19.625, 16.375], "probabilities": [0.9626730680465698, 0.037326883524656296], "choice": "yes"}
|
| 20 |
+
{"panel_index": 19, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0001238", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "8dace247ec58eab0973f7a1087fe3af7c8695a2992be4d300ba004babaf8cbbe", "candidate_token_ids": [32, 33], "logits": [18.875, 12.8125], "probabilities": [0.9976769089698792, 0.0023231625091284513], "choice": "yes"}
|
| 21 |
+
{"panel_index": 20, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0011197", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "b3fa91e805df134cb80d0ae82eabc86c63266d874b7fb6cdd584d222f7cd9426", "candidate_token_ids": [32, 33], "logits": [19.0, 12.9375], "probabilities": [0.9976769089698792, 0.0023231625091284513], "choice": "yes"}
|
| 22 |
+
{"panel_index": 21, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0017850", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "8e283dc3e113374111c25df9df8ed38d3a0991314388f80ace18540c352235d0", "candidate_token_ids": [32, 33], "logits": [11.6875, 18.5], "probabilities": [0.001098731067031622, 0.9989012479782104], "choice": "no"}
|
| 23 |
+
{"panel_index": 22, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0019640", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "475cb8ff904634ace670760f71e582174addfbcf71d28985c0e1c7c1f9b3dbcc", "candidate_token_ids": [32, 33], "logits": [19.375, 12.625], "probabilities": [0.9988304972648621, 0.0011695101857185364], "choice": "yes"}
|
| 24 |
+
{"panel_index": 23, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0010645", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "62ea4bbe1886885ce73d0c1abf72e927fd23117d83868d32d50405bd0a692191", "candidate_token_ids": [32, 33], "logits": [19.0, 15.375], "probabilities": [0.9740425944328308, 0.02595735713839531], "choice": "yes"}
|
| 25 |
+
{"panel_index": 24, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0020714", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "43596c9cb034207f4c188aa25416d7d290143d2f1850853e180cbee8e1ffbe81", "candidate_token_ids": [32, 33], "logits": [19.0, 12.875], "probabilities": [0.9978172779083252, 0.002182716503739357], "choice": "yes"}
|
| 26 |
+
{"panel_index": 25, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0008261", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "bc7fe561ed0c1b12e7a21d23aa6bc9a8fff8e502d0ec7328c0c3500042da1225", "candidate_token_ids": [32, 33], "logits": [13.125, 19.0], "probabilities": [0.0028009270317852497, 0.9971990585327148], "choice": "no"}
|
| 27 |
+
{"panel_index": 26, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0015735", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "cba3fb62985350d60d64f9988e06a26577a2e643809593877cd7eddaf1e3e1b9", "candidate_token_ids": [32, 33], "logits": [18.875, 16.0], "probabilities": [0.9465966820716858, 0.0534033328294754], "choice": "yes"}
|
| 28 |
+
{"panel_index": 27, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0014712", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "fc5d5293343cdb9a2c4d587a01f6d4b5c2adf98a0ee775d07a1601ae06342dea", "candidate_token_ids": [32, 33], "logits": [19.375, 13.0], "probabilities": [0.9982993006706238, 0.001700722612440586], "choice": "yes"}
|
| 29 |
+
{"panel_index": 28, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0024669", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "05bc8e81c80ffdc18ed69091c6d4c21decc0b2cd30b5164c34d24f95c17126ff", "candidate_token_ids": [32, 33], "logits": [15.6875, 18.5], "probabilities": [0.05665242671966553, 0.9433475732803345], "choice": "no"}
|
| 30 |
+
{"panel_index": 29, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0032390", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "1a9cc68b19fe2ec0158f8bbeab86b096a1496bbb755f023f66f4e3715b53da46", "candidate_token_ids": [32, 33], "logits": [12.0625, 18.5], "probabilities": [0.001597845577634871, 0.9984021782875061], "choice": "no"}
|
| 31 |
+
{"panel_index": 30, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0018442", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "f68722d712d424bdeddd2531a73b47948597a47e37bd59132b14df56fa98e9b8", "candidate_token_ids": [32, 33], "logits": [18.75, 14.0625], "probabilities": [0.9908744096755981, 0.00912563782185316], "choice": "yes"}
|
| 32 |
+
{"panel_index": 31, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0014165", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "b2c1d04fc65e345807d5cf3dc93324826502c3131fbb3eca59bf166ef773b131", "candidate_token_ids": [32, 33], "logits": [19.25, 13.0], "probabilities": [0.9980732202529907, 0.0019267346942797303], "choice": "yes"}
|
| 33 |
+
{"panel_index": 32, "id": "boolq/8703574dc0db27742840afd3c1429829ef2f81c557670018e6ad5456fc53b150", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "938a388a0ea3db943c165f47c21a399dcbc2040c6f61340bd280786a9f5704f3", "candidate_token_ids": [32, 33], "logits": [13.125, 18.5], "probabilities": [0.004609572701156139, 0.9953904151916504], "choice": "no"}
|
| 34 |
+
{"panel_index": 33, "id": "boolq/42a6042685586d3796f9fe4460dd6e9cec208f39de59f765dcbef491fe576361", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "0d97ae562208ecc24ae9df8ab3678c7ae4fbe3aa22b17fffbf97fb6b2cfc2fec", "candidate_token_ids": [32, 33], "logits": [12.375, 18.875], "probabilities": [0.0015011822106316686, 0.998498797416687], "choice": "no"}
|
| 35 |
+
{"panel_index": 34, "id": "boolq/5ba6d06215b4f7b269d0b30369fea59d748f02f41d1977f14c46c9153cd45cde", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "a3a3f58378880f9c73e1dc63d51ea8d3951c8c90e5b4b593a8c7a1656898903e", "candidate_token_ids": [32, 33], "logits": [12.8125, 18.625], "probabilities": [0.002981032943353057, 0.9970190525054932], "choice": "no"}
|
| 36 |
+
{"panel_index": 35, "id": "boolq/4c460ed013de1b5d82c629bbf4e3bdbdaf4441a81341404d48d8e02ead5b9814", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "e7363021faeb0475c8d4a53d1e355b57cd7c2336433356f455b0dd025084fcdd", "candidate_token_ids": [32, 33], "logits": [19.5, 13.5], "probabilities": [0.9975274205207825, 0.0024726232513785362], "choice": "yes"}
|
| 37 |
+
{"panel_index": 36, "id": "boolq/46e903566fe8817ff37dce61d1ef85b757aa465dadb873c6356b6fc23f58476b", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "c59ba2cb157be77ed4543c9c8e414ce9d572e94a4f7af562105af456ca0993e3", "candidate_token_ids": [32, 33], "logits": [19.875, 12.125], "probabilities": [0.9995694756507874, 0.0004305570910219103], "choice": "yes"}
|
| 38 |
+
{"panel_index": 37, "id": "boolq/3f87d9cb171dfe78b389d22677512f92cb48f7eedeefd35f2db1a2444cf08af2", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "f4a7827700e4744e20ca7891fc537579e56467de67e1c9e0218eb74d1f850913", "candidate_token_ids": [32, 33], "logits": [20.0, 12.3125], "probabilities": [0.9995417594909668, 0.0004583129193633795], "choice": "yes"}
|
| 39 |
+
{"panel_index": 38, "id": "boolq/dbadfab033f2646b1a933fad5e132dc65cd8998c337d0332763f1df07680d7e7", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "349dd93162e0859e82c150bd521f171b1edd2cc2e98f05ed41fb16defcb80cb5", "candidate_token_ids": [32, 33], "logits": [19.625, 12.25], "probabilities": [0.999373733997345, 0.000626334163825959], "choice": "yes"}
|
| 40 |
+
{"panel_index": 39, "id": "boolq/6909b3bd9078683a97a30ee70c6f3e91a9b8848a34efc0784eddf12cfa305cd4", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "f5291ad0f6c72b83b6fd627aca39178a0202dd9a6674a6fca53a3a2dd9dfe3f7", "candidate_token_ids": [32, 33], "logits": [13.125, 19.0], "probabilities": [0.0028009270317852497, 0.9971990585327148], "choice": "no"}
|
| 41 |
+
{"panel_index": 40, "id": "boolq/791b5a767a61e438fe080d99068ef294b361fead41dbfc61b61021e33f2fff9f", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "44bb0f449bae5d6f5c4037e0f69fc24b29c530637f5f59fd6436f33027b12bc8", "candidate_token_ids": [32, 33], "logits": [19.75, 12.0], "probabilities": [0.9995694756507874, 0.0004305570910219103], "choice": "yes"}
|
| 42 |
+
{"panel_index": 41, "id": "boolq/cca90e3be9b421447fe9ae0594193ab92a92adb0083f27a46bc224ad0e67c46c", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "1f4adf18c45e5ad59ac100c1e54416032c380206c3420bf39071a4ac2010d52b", "candidate_token_ids": [32, 33], "logits": [13.25, 18.75], "probabilities": [0.004070138093084097, 0.9959298968315125], "choice": "no"}
|
| 43 |
+
{"panel_index": 42, "id": "boolq/179184609853d9f8f1f862bd5c3113e363f622606fcf5c188cf0da1d3ed61e86", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "e88d50a2527287c73f777f23fcb092662ed71ed0c5d7fe0168ec8994e8fb19a0", "candidate_token_ids": [32, 33], "logits": [19.5, 12.25], "probabilities": [0.9992903470993042, 0.0007096704212017357], "choice": "yes"}
|
| 44 |
+
{"panel_index": 43, "id": "boolq/58b070021c1d992130604f309a01b95166a281de0499349346694570a6c6ac57", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "8446057016a93c99345544b2afe9eaac75f9ffb0acc99cca9f255e73cb837e34", "candidate_token_ids": [32, 33], "logits": [14.8125, 18.375], "probabilities": [0.027585284784436226, 0.9724147915840149], "choice": "no"}
|
| 45 |
+
{"panel_index": 44, "id": "boolq/11b00c8dcb82dfbdaf02aa139e8c70603fb8c432e0fc51873ca00b20815a90f5", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "ecf3d435d5e2e523fc05ac04d40a235e7808770163f14f11214c899c8738f1b7", "candidate_token_ids": [32, 33], "logits": [19.125, 12.75], "probabilities": [0.9982993006706238, 0.001700722612440586], "choice": "yes"}
|
| 46 |
+
{"panel_index": 45, "id": "boolq/155bc4f5246c5b35ec14d7a2f697381de04c2cefbbc293b387bfdf83f2fea269", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "3c4715c57e5dd569ad4eb1b9d4a725fefaf222f34bd9739091e97cbe8bb6945d", "candidate_token_ids": [32, 33], "logits": [20.5, 13.75], "probabilities": [0.9988304972648621, 0.0011695101857185364], "choice": "yes"}
|
| 47 |
+
{"panel_index": 46, "id": "boolq/31fa2124dc5769d9e6fae53be4190290131f8fb025a089f8372d6f5d61504af5", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "9a87a3789d0d031052a1165e22df7f13784f74d7d02fffac973d290d993657f7", "candidate_token_ids": [32, 33], "logits": [13.8125, 18.625], "probabilities": [0.00806199200451374, 0.9919379353523254], "choice": "no"}
|
| 48 |
+
{"panel_index": 47, "id": "boolq/a1ea17311c3aea8a751a1cd170483c439ce1338b451927c7db07692878ff0d82", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "cebe40ca184a82f8468175b02fdafcbbd829564cf847b3fc0135fd98f67083ea", "candidate_token_ids": [32, 33], "logits": [12.8125, 18.75], "probabilities": [0.0026316740550100803, 0.997368335723877], "choice": "no"}
|
| 49 |
+
{"panel_index": 48, "id": "mnli/ab466125416d6bc3f307f7fd25b78e63a8d90f8f54713027fdc7d75fffd8558d", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "5834f0baf5838f80f70d57a54ea54fdf72bab1fffb7ed8abd7b6b83fec52396e", "candidate_token_ids": [32, 33, 34], "logits": [16.125, 19.375, 14.25], "probabilities": [0.03711443394422531, 0.957193911075592, 0.005691682919859886], "choice": "neutral"}
|
| 50 |
+
{"panel_index": 49, "id": "mnli/8f4dbf064e95ffddc782d0d50ab5e6e0fb83aab546d48b00bff5f03706183bf9", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "52a795fb9d86048dfec129052857ec13bfa6dca7b7314081a62143a83055313a", "candidate_token_ids": [32, 33, 34], "logits": [13.125, 20.5, 12.375], "probabilities": [0.0006261488306336105, 0.9990780353546143, 0.0002957718097604811], "choice": "neutral"}
|
| 51 |
+
{"panel_index": 50, "id": "mnli/53ac0a380403b0ccd3ae48ea7763a52664f1d1744f6244c25d5980d69fcf7f0f", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "f6adf5004089ef9a0cc851a183a3e4c72d4b8a84fd54803ef1789646635f01dd", "candidate_token_ids": [32, 33, 34], "logits": [20.25, 12.4375, 11.5], "probabilities": [0.9994372725486755, 0.0004044174565933645, 0.00015837214596103877], "choice": "entailment"}
|
| 52 |
+
{"panel_index": 51, "id": "mnli/e6cf83f056ba7587936d8026ff9709bc50de62e0dfb282f6b8d6aa85649e688d", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "9ddcac65f7c20b39c2eafa8d124294ab44516209a07832ffa4575b35c2ea2396", "candidate_token_ids": [32, 33, 34], "logits": [13.5, 20.25, 12.5], "probabilities": [0.001169007271528244, 0.9984009861946106, 0.00043005376937799156], "choice": "neutral"}
|
| 53 |
+
{"panel_index": 52, "id": "mnli/834677de1c970abc266e34006f2e681c7301ff5d043796d92da66364b792f235", "task_family": "mnli", "gold": "contradiction", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "c1a73d0727accd049baa352165c01aebbbc68ef10ab728e698ef44f9efbc7b6c", "candidate_token_ids": [32, 33, 34], "logits": [11.25, 11.9375, 19.75], "probabilities": [0.0002033447235589847, 0.000404399266699329, 0.9993922710418701], "choice": "contradiction"}
|
| 54 |
+
{"panel_index": 53, "id": "mnli/13f1e11494916138e91023e5bf4d2483ab5d93f00e2e1e558c5f595ca0e8e22d", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "5aedfe5d0e0bcfe168031cd031b0355c0ed3ce9f22948073417e36f0682394e8", "candidate_token_ids": [32, 33, 34], "logits": [20.125, 12.0, 11.25], "probabilities": [0.9995643496513367, 0.0002959157864097506, 0.00013978072092868388], "choice": "entailment"}
|
| 55 |
+
{"panel_index": 54, "id": "mnli/9cf51bc8717b59b904d198884ba5b832b53cdc96a5b1aafee7203dca9bfc824d", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "37b5f12a82b74d56402bb3c33b63e27e4d08218a0ec1ec3a2087df90fa35ea3f", "candidate_token_ids": [32, 33, 34], "logits": [16.875, 19.625, 13.5], "probabilities": [0.05996336042881012, 0.9379847645759583, 0.0020518333185464144], "choice": "neutral"}
|
| 56 |
+
{"panel_index": 55, "id": "mnli/3d8bd065ebff13c30fa3aa4943c5f2851ca8a5860bae22e25f212be97790ab28", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "0a2b268fa0d617735fea967a5896b3d205b101d83b018ec9fca7d988f11d10db", "candidate_token_ids": [32, 33, 34], "logits": [20.125, 12.4375, 11.375], "probabilities": [0.9993834495544434, 0.00045824036351405084, 0.00015836361853871495], "choice": "entailment"}
|
| 57 |
+
{"panel_index": 56, "id": "mnli/4dc433ef7c48f1158e61aa4f1b568bac7ee2ede91001c9b8d933fd5c54e71820", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "28ea8cef7c3cefbc4e52117dbbd91fe6ba0185a3067aff8e0a364e78d7918819", "candidate_token_ids": [32, 33, 34], "logits": [19.75, 12.0625, 11.375], "probabilities": [0.9993114471435547, 0.00045820733066648245, 0.0002304011140950024], "choice": "entailment"}
|
| 58 |
+
{"panel_index": 57, "id": "mnli/533107d0478441289d303edb55e3769ea75e77ee657738576b1e86f9f523ff09", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "f5983d410d1ffb47caf1d2c160be923e0ed6839419930f59eadfb89cfb2b4c1c", "candidate_token_ids": [32, 33, 34], "logits": [14.375, 17.125, 19.625], "probabilities": [0.004826047457754612, 0.07549208402633667, 0.9196819067001343], "choice": "contradiction"}
|
| 59 |
+
{"panel_index": 58, "id": "mnli/e6856d10a89f46cbfc7fa77bf9266c482f130090a961f5bcb10be540e7c5fc2c", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "b0414aaab93e025d307d733858e0be03f543479f804f1dc12a6324b94a1a28c2", "candidate_token_ids": [32, 33, 34], "logits": [13.875, 19.75, 15.25], "probabilities": [0.0027702387887984514, 0.9862732291221619, 0.010956507176160812], "choice": "neutral"}
|
| 60 |
+
{"panel_index": 59, "id": "mnli/4bd800d8bc83435b531bb472e9078bb054a7bb68bf31036bc12502e177dea19b", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "a2314ed00eeb6299fbe1343e769272e96da353429390c352cf49d59d0c2967e8", "candidate_token_ids": [32, 33, 34], "logits": [19.25, 14.4375, 17.125], "probabilities": [0.8868703842163086, 0.0072080534882843494, 0.10592156648635864], "choice": "entailment"}
|
| 61 |
+
{"panel_index": 60, "id": "mnli/b0220f9d099b621bde205ea6efa72624cc1948cceb2502cdc38d19c3762c2ba0", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "5e649a151d402888b8c9e9b8ba693d39efad2f95912eb424363518e92fb26dee", "candidate_token_ids": [32, 33, 34], "logits": [14.1875, 15.0, 20.0], "probabilities": [0.0029611403588205576, 0.006673032883554697, 0.9903658628463745], "choice": "contradiction"}
|
| 62 |
+
{"panel_index": 61, "id": "mnli/5c5d3ecddeb08dada33aaa9e323aa10bdfa371aa2f535b2cd5f4f653b0e25e9c", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "808dae944e65ee87bcaaa32c78f64633ec95e1e202c2e49fc0a34ffc831c7327", "candidate_token_ids": [32, 33, 34], "logits": [19.5, 16.75, 14.0625], "probabilities": [0.9360857605934143, 0.059841956943273544, 0.004072296433150768], "choice": "entailment"}
|
| 63 |
+
{"panel_index": 62, "id": "mnli/dba1b74655afea3f9676c7a8687e80c0da196b11f2324d6e3a41c3870b9d8591", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "1e22807eca9d351d7041b7c3a7d466c369566a06caf7559c424c855f9dc8de40", "candidate_token_ids": [32, 33, 34], "logits": [19.75, 12.4375, 11.5625], "probabilities": [0.9990556836128235, 0.0006665171240456402, 0.0002778456546366215], "choice": "entailment"}
|
| 64 |
+
{"panel_index": 63, "id": "mnli/c8c695b22a0a846dde5f0ae3c17127f27c53f23219de2f4c97f2a9bebe1e0f81", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "b2d7992028f3524744fa167194e920d2d2f96e049d9baa0c16a60567a2506e01", "candidate_token_ids": [32, 33, 34], "logits": [20.0, 12.1875, 11.375], "probabilities": [0.9994162321090698, 0.00040440892917104065, 0.00017945538274943829], "choice": "entailment"}
|
| 65 |
+
{"panel_index": 64, "id": "goemotions:0d6f68c6aab93172edb30824a77a2f20e857a379845cfc16d8315af53d05bc1f:approval", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "62e2f3ce6d8db5d5399c3dbeaa448fa75306f4d790a05d15745663f6ceabe85f", "candidate_token_ids": [32, 33], "logits": [18.75, 18.375], "probabilities": [0.5926666259765625, 0.4073334336280823], "choice": "yes"}
|
| 66 |
+
{"panel_index": 65, "id": "goemotions:5bb2f0355125b6ea32d14bc489651edeb09f6b4ef9f51b90b8ed330b8fd14cef:sadness", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "5cd89e6970749f8b4a0f8526f39a21035bd2a2d2dab17c9b22cb526e792d40bf", "candidate_token_ids": [32, 33], "logits": [12.125, 19.75], "probabilities": [0.00048785717808641493, 0.9995121955871582], "choice": "no"}
|
| 67 |
+
{"panel_index": 66, "id": "goemotions:5695bd603bf8ce60300298693b29b4295f2d34c083c7e52bff826db80a6ea996:grief", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "4a701cb3026d72bdc664138b3258e27154414aef18bc5d9658a22ca9b16431d3", "candidate_token_ids": [32, 33], "logits": [12.0625, 19.75], "probabilities": [0.0004583129193633795, 0.9995417594909668], "choice": "no"}
|
| 68 |
+
{"panel_index": 67, "id": "goemotions:e97ef69f22521e136e92eee4bbc1e46d8571bf91280b2bc7ae24f2e1ec5d0745:desire", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "74cd0d0acfe64e81b3b02d676d3543c54fe2736d05f9d6964e9bf589a8fa604e", "candidate_token_ids": [32, 33], "logits": [13.625, 19.375], "probabilities": [0.0031726830638945103, 0.9968273043632507], "choice": "no"}
|
| 69 |
+
{"panel_index": 68, "id": "goemotions:0c48f870b5edcb7a8dcfeb58087c7e3e56a9a2d89a0687e66d9974557c74556a:love", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "74095411b7d11586abb60a651ae44840355675c465df21605302ced8aa7460e9", "candidate_token_ids": [32, 33], "logits": [12.5, 19.5], "probabilities": [0.0009110512328334153, 0.9990890026092529], "choice": "no"}
|
| 70 |
+
{"panel_index": 69, "id": "goemotions:30e748ace2950567f2e1dcc3c638f6d5fc1e2df8078792928acf9550168933c7:approval", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "a8e377d44a31f2539125c844e84396f69387ac0177550bfe52378a2461e31ae7", "candidate_token_ids": [32, 33], "logits": [19.75, 12.5], "probabilities": [0.9992903470993042, 0.0007096704212017357], "choice": "yes"}
|
| 71 |
+
{"panel_index": 70, "id": "goemotions:3b0e3210dddfd4dbb4b0e76cf62299246fd5a912229994a9e07de7dfc64c2262:embarrassment", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "897f9ee99d20643b8ec00e25182dded61636e60cfd09ee330ca3b25540069833", "candidate_token_ids": [32, 33], "logits": [12.6875, 19.625], "probabilities": [0.0009697519126348197, 0.9990302324295044], "choice": "no"}
|
| 72 |
+
{"panel_index": 71, "id": "goemotions:4308985a124eb5964a14e35ae0d7db83da5f33c019222497141a9a455cb5500c:joy", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "51b9806351f409e9959d99b25249e1561bcb1d2da032317b72b84a429a28d713", "candidate_token_ids": [32, 33], "logits": [12.625, 19.75], "probabilities": [0.0008040859247557819, 0.9991958737373352], "choice": "no"}
|
| 73 |
+
{"panel_index": 72, "id": "goemotions:1285f8257d6a85acd0b9b01bfbebadff45ec085608b755eb415c74c624f8f5eb:optimism", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "5af8f3b74ad252af8af13b9bbe1c5b4585c431868cfaa0c4f99abdb42e654a36", "candidate_token_ids": [32, 33], "logits": [18.875, 18.75], "probabilities": [0.531209409236908, 0.4687906503677368], "choice": "yes"}
|
| 74 |
+
{"panel_index": 73, "id": "goemotions:ee392e883c467546e7e603f5e6b94702211b3b4af3519a0bb7bd3b21bf4cbfd9:joy", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "266326a7e5d742f0cc33238676458ba8635f29efc4d3c95b662feaa590c2b38e", "candidate_token_ids": [32, 33], "logits": [11.9375, 19.5], "probabilities": [0.0005193048273213208, 0.999480664730072], "choice": "no"}
|
| 75 |
+
{"panel_index": 74, "id": "goemotions:43ef087b074a832ce71850c4c8d1672ca1d68b8b03830f65be8a52da990b454c:surprise", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "50981dcbdce4641c8cadc2eb3b78a20e23fca85a04376f095c1419f77d976694", "candidate_token_ids": [32, 33], "logits": [12.1875, 19.75], "probabilities": [0.0005193048273213208, 0.999480664730072], "choice": "no"}
|
| 76 |
+
{"panel_index": 75, "id": "goemotions:8285d89a0201a60100cf7540b9e92fbf0564e50afc4e58afb812f91b6fc64cba:desire", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "538fef417332ce98ebd74b27c7c5b59a41fc722574e13987db181517abe649bb", "candidate_token_ids": [32, 33], "logits": [12.6875, 19.375], "probabilities": [0.0012448433553799987, 0.9987550973892212], "choice": "no"}
|
| 77 |
+
{"panel_index": 76, "id": "goemotions:cfa49c3d3a46c50e494455f5f1aa94c401a83a9450181bcf5d1f64ae75e30e64:annoyance", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "22d93f0279f9bacc380b1a29a2debaca5cae59cd9a7ff8e82c8f42ec97d1aa8d", "candidate_token_ids": [32, 33], "logits": [15.1875, 19.25], "probabilities": [0.016914913430809975, 0.9830850958824158], "choice": "no"}
|
| 78 |
+
{"panel_index": 77, "id": "goemotions:e97ef69f22521e136e92eee4bbc1e46d8571bf91280b2bc7ae24f2e1ec5d0745:confusion", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "b7bc2566159ea8e56d02da9bdc7d94820e6e704c057d22624565cf84c3d8ccc8", "candidate_token_ids": [32, 33], "logits": [15.3125, 19.125], "probabilities": [0.02161533385515213, 0.9783847332000732], "choice": "no"}
|
| 79 |
+
{"panel_index": 78, "id": "goemotions:a312dda34fe450feab6e22e6d2aa7493d77c757d4800e2bfd9904549ff4c3756:realization", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "53472c74825a9353b6efae1df7e1106ab8c2aeb0f510b2923fd2754893391868", "candidate_token_ids": [32, 33], "logits": [13.6875, 19.625], "probabilities": [0.0026316740550100803, 0.997368335723877], "choice": "no"}
|
| 80 |
+
{"panel_index": 79, "id": "goemotions:4c893be6bdf7ad879db4d2299867417db381418ebeed081d85cbd13e6b3371a6:anger", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "6a4428c6d807234388965c34cc8785054f982294a2b8b91ab744deaa27db2c6c", "candidate_token_ids": [32, 33], "logits": [13.375, 19.5], "probabilities": [0.002182716503739357, 0.9978172779083252], "choice": "no"}
|