Add Frozen80 per-question results
Browse files- README.md +1 -1
- README.zh-CN.md +1 -1
- evaluation/per-question/mlx-4bit.jsonl +80 -0
- evaluation/per-question/reference.jsonl +80 -0
README.md
CHANGED
|
@@ -40,7 +40,7 @@ Frozen80 with identical prompt tokens, compared with the HF BF16 release (full d
|
|
| 40 |
|---|---|---:|---:|---:|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 70/80 · 87.50% | 78/80 | 0.8700 |
|
| 42 |
|
| 43 |
-
Converted and scored with MLX on Linux (CUDA); the files are platform-independent and load unchanged on Apple Silicon (for 4B, the same kind of file gave identical decisions on CUDA and Metal). Peak memory on Frozen80 was 23.17 GB; plan for a Mac with at least 32 GB of unified memory. Frozen80 is a reused development panel, not a blind benchmark.
|
| 44 |
|
| 45 |
## Conversion
|
| 46 |
|
|
|
|
| 40 |
|---|---|---:|---:|---:|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 70/80 · 87.50% | 78/80 | 0.8700 |
|
| 42 |
|
| 43 |
+
Converted and scored with MLX on Linux (CUDA); the files are platform-independent and load unchanged on Apple Silicon (for 4B, the same kind of file gave identical decisions on CUDA and Metal). Peak memory on Frozen80 was 23.17 GB; plan for a Mac with at least 32 GB of unified memory. Frozen80 is a reused development panel, not a blind benchmark. Per-question rows (candidate probabilities, choice, correctness; join with Frozen80 by `panel_index`): [evaluation/per-question/](evaluation/per-question/).
|
| 44 |
|
| 45 |
## Conversion
|
| 46 |
|
README.zh-CN.md
CHANGED
|
@@ -40,7 +40,7 @@ Frozen80 使用完全相同的 prompt token，与 HF BF16 发布版（完整深
|
|
| 40 |
|---|---|---:|---:|---:|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 70/80 · 87.50% | 78/80 | 0.8700 |
|
| 42 |
|
| 43 |
-
在 Linux（CUDA）上用 MLX 转换并评测；MLX 文件与平台无关，可直接在 Apple Silicon 上加载（4B 的同类文件在 CUDA 与 Metal 上决策完全一致）。Frozen80 评测峰值内存 23.17 GB，建议使用统一内存不少于 32 GB 的 Mac。Frozen80 是复用的开发面板，不是盲测。
|
| 44 |
|
| 45 |
## 转换说明
|
| 46 |
|
|
|
|
| 40 |
|---|---|---:|---:|---:|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 70/80 · 87.50% | 78/80 | 0.8700 |
|
| 42 |
|
| 43 |
+
在 Linux（CUDA）上用 MLX 转换并评测；MLX 文件与平台无关，可直接在 Apple Silicon 上加载（4B 的同类文件在 CUDA 与 Metal 上决策完全一致）。Frozen80 评测峰值内存 23.17 GB，建议使用统一内存不少于 32 GB 的 Mac。Frozen80 是复用的开发面板，不是盲测。逐题明细（候选概率、选择、是否正确，可按 `panel_index` 与 Frozen80 关联）：[evaluation/per-question/](evaluation/per-question/)。
|
| 44 |
|
| 45 |
## 转换说明
|
| 46 |
|
evaluation/per-question/mlx-4bit.jsonl
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"panel_index": 0, "task_family": "browser", "logits": [15.75, 15.5625, 14.0625, 13.75, 23.5, 14.4375, 13.4375, 13.375, 13.875, 15.5, 13.4375, 13.5, 12.8125, 12.6875, 13.625, 13.625], "probabilities": [0.0004299858119338751, 0.00035647061304189265, 7.953935710247606e-05, 5.819224315928295e-05, 0.9982430934906006, 0.0001157290898845531, 4.257436376065016e-05, 3.999490218120627e-05, 6.594046135433018e-05, 0.00033487327164039016, 4.257436376065016e-05, 4.53201646450907e-05, 2.2788410205976106e-05, 2.0110699551878497e-05, 5.135447281645611e-05, 5.135447281645611e-05], "choice": "candidate-c59cf6b3cdd1ee8dfe92", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0008731484413146973}
|
| 2 |
+
{"panel_index": 1, "task_family": "browser", "logits": [14.125, 17.375, 14.5, 14.0625, 14.4375, 13.4375, 13.375, 14.0625, 22.75, 16.875, 14.8125, 14.3125, 14.9375, 14.625, 17.0, 15.0625], "probabilities": [0.00017715891590341926, 0.00456899031996727, 0.00025776473921723664, 0.00016642545233480632, 0.0002421476528979838, 8.908110612537712e-05, 8.36839753901586e-05, 0.00016642545233480632, 0.9866270422935486, 0.002771233208477497, 0.0003523225605022162, 0.00021369454043451697, 0.0003992338024545461, 0.0002920857514254749, 0.003140218323096633, 0.00045239119208417833], "choice": "candidate-d2449a13158601200ec0", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.006388723850250244}
|
| 3 |
+
{"panel_index": 2, "task_family": "browser", "logits": [17.0, 15.5, 14.8125, 14.5625, 15.625, 22.0, 13.3125, 12.6875, 12.5625, 12.1875, 12.0625, 12.5, 11.8125, 12.125, 12.375, 12.8125], "probabilities": [0.006657630670815706, 0.0014855182962492108, 0.0007469655247405171, 0.0005817372584715486, 0.0016833129338920116, 0.9880800843238831, 0.0001666705502429977, 8.921229891711846e-05, 7.87295721238479e-05, 5.411001984612085e-05, 4.7751920646987855e-05, 7.395960710709915e-05, 3.718923107953742e-05, 5.0831640692194924e-05, 6.52691232971847e-05, 0.00010109077993547544], "choice": "candidate-9409b72a36db8d4a0a79", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00444948673248291}
|
| 4 |
+
{"panel_index": 3, "task_family": "browser", "logits": [17.125, 15.3125, 15.3125, 16.625, 15.5, 14.1875, 21.25, 14.6875, 15.0625, 14.875, 16.75, 13.875, 14.25, 17.125, 16.125, 14.75], "probabilities": [0.01501185167580843, 0.0024506172630935907, 0.0024506172630935907, 0.009105149656534195, 0.002956009702757001, 0.0007955991313792765, 0.9287503957748413, 0.0013117213966324925, 0.0019085423555225134, 0.001582237659022212, 0.010317486710846424, 0.0005820728256367147, 0.000846910523250699, 0.01501185167580843, 0.005522551015019417, 0.0013963197125121951], "choice": "candidate-3b25ae8d064a298ef021", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.05750685930252075}
|
| 5 |
+
{"panel_index": 4, "task_family": "browser", "logits": [22.0, 16.5, 15.4375, 16.875, 14.375, 12.8125, 12.1875, 12.25, 11.6875, 10.875, 11.125, 11.375, 12.4375, 12.375, 12.0, 13.0], "probabilities": [0.9876103401184082, 0.00403613829985261, 0.0013948522973805666, 0.0058725448325276375, 0.00048204799531958997, 0.00010104272223543376, 5.40842920599971e-05, 5.757241160608828e-05, 3.280377859482542e-05, 1.455658730264986e-05, 1.869103107310366e-05, 2.3999758923309855e-05, 6.944556662347168e-05, 6.523809861391783e-05, 4.483743759919889e-05, 0.00012188083928776905], "choice": "candidate-34e89f28b090ce5327b0", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.003576219081878662}
|
| 6 |
+
{"panel_index": 5, "task_family": "browser", "logits": [21.75, 14.0625, 15.5, 14.0625, 15.3125, 13.0, 12.5, 15.5, 13.625, 12.625, 12.1875, 12.125, 12.5625, 13.375, 14.6875, 13.125], "probabilities": [0.991529107093811, 0.00045463888091035187, 0.0019141016528010368, 0.00045463888091035187, 0.0015868464251980186, 0.0001571189786773175, 9.529752423986793e-05, 0.0019141016528010368, 0.00029353698482736945, 0.00010798624862218276, 6.972112169023603e-05, 6.549695535795763e-05, 0.0001014436493278481, 0.00022860680473968387, 0.0008493774803355336, 0.0001780391321517527], "choice": "candidate-fb3025ec54721504f9b5", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.000830233097076416}
|
| 7 |
+
{"panel_index": 6, "task_family": "browser", "logits": [13.125, 12.5, 13.75, 13.75, 15.125, 12.6875, 15.125, 12.5625, 13.5625, 15.0, 21.125, 13.6875, 12.625, 14.1875, 13.5, 14.75], "probabilities": [0.0003308977757114917, 0.00017711675900500268, 0.0006181985372677445, 0.0006181985372677445, 0.0024450223427265882, 0.00021364368149079382, 0.0024450223427265882, 0.00018853986694011837, 0.0005125044262968004, 0.0021577246952801943, 0.9863922595977783, 0.0005807435372844338, 0.000200699592824094, 0.0009574844734743237, 0.0004814534622710198, 0.0016804375918582082], "choice": "candidate-888bb5997d1aba7b25f7", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00975346565246582}
|
| 8 |
+
{"panel_index": 7, "task_family": "browser", "logits": [22.125, 13.625, 13.5625, 13.5, 13.625, 12.125, 11.4375, 12.0, 12.0, 11.25, 11.3125, 11.375, 12.875, 12.5, 11.875, 12.625], "probabilities": [0.9987427592277527, 0.0002032125339610502, 0.00019090055138804018, 0.0001793344272300601, 0.0002032125339610502, 4.5342847442952916e-05, 2.279981526953634e-05, 4.00149219785817e-05, 4.00149219785817e-05, 1.890171733975876e-05, 2.0120767658227123e-05, 2.141845106962137e-05, 9.599083568900824e-05, 6.597346509806812e-05, 3.531304173520766e-05, 7.475772872567177e-05], "choice": "candidate-41c5af16d7bf373cc74c", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001258254051208496}
|
| 9 |
+
{"panel_index": 8, "task_family": "browser", "logits": [14.75, 14.8125, 22.625, 14.8125, 15.5, 13.75, 15.625, 14.1875, 13.9375, 12.6875, 12.5, 12.75, 12.0625, 12.5625, 12.75, 13.25], "probabilities": [0.0003786991292145103, 0.0004031230346299708, 0.9962384700775146, 0.0004031230346299708, 0.0008017057552933693, 0.0001393156562699005, 0.0009084516786970198, 0.00021577627921942621, 0.00016804672486614436, 4.814620115212165e-05, 3.991458652308211e-05, 5.125134703121148e-05, 2.577079794718884e-05, 4.2488867620704696e-05, 5.125134703121148e-05, 8.449920278508216e-05], "choice": "candidate-a1a3ad1a8495aa678cbe", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00028961896896362305}
|
| 10 |
+
{"panel_index": 9, "task_family": "browser", "logits": [21.375, 14.625, 13.3125, 13.3125, 13.8125, 15.5625, 12.5, 13.5625, 13.5, 13.1875, 13.0, 12.875, 13.25, 13.3125, 12.3125, 13.8125], "probabilities": [0.9918723702430725, 0.0011613629758358002, 0.000312576798023656, 0.000312576798023656, 0.0005153517704457045, 0.0029656444676220417, 0.0001387051015626639, 0.00040135628660209477, 0.0003770394541788846, 0.0002758480259217322, 0.00022868593805469573, 0.0002018146333284676, 0.00029363861540332437, 0.000312576798023656, 0.00011499051470309496, 0.0005153517704457045], "choice": "candidate-976c2731e7518a1f1b2b", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0013207793235778809}
|
| 11 |
+
{"panel_index": 10, "task_family": "browser", "logits": [15.5625, 16.75, 15.125, 14.5, 15.4375, 15.25, 14.375, 14.625, 16.25, 14.4375, 15.5, 20.75, 14.625, 19.0, 15.0, 14.9375], "probabilities": [0.004495193716138601, 0.014739175327122211, 0.002902316628023982, 0.0015534977428615093, 0.003966995514929295, 0.003288754727691412, 0.00137095688842237, 0.0017603435553610325, 0.008939762599766254, 0.0014593754895031452, 0.004222845192998648, 0.8047316670417786, 0.0017603435553610325, 0.13984137773513794, 0.0025612851604819298, 0.002406104002147913], "choice": "candidate-016db9380de74cbcab5d", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.15753722190856934}
|
| 12 |
+
{"panel_index": 11, "task_family": "browser", "logits": [16.75, 15.1875, 14.6875, 13.625, 15.8125, 14.1875, 13.9375, 14.5625, 14.125, 16.5, 13.3125, 16.75, 19.375, 19.375, 14.1875, 13.8125], "probabilities": [0.03160127252340317, 0.00662398524582386, 0.004017651081085205, 0.0013884633081033826, 0.012375232763588428, 0.0024368278682231903, 0.0018978039734065533, 0.0035455645993351936, 0.0022891887929290533, 0.02461109682917595, 0.0010158210061490536, 0.03160127252340317, 0.43624207377433777, 0.43624207377433777, 0.0024368278682231903, 0.0016748060006648302], "choice": "candidate-544fea69d385caac3601", "correct": false, "agrees_with_reference": false, "max_abs_probability_delta": 0.43152138590812683}
|
| 13 |
+
{"panel_index": 12, "task_family": "browser", "logits": [14.0, 13.0, 17.375, 13.625, 14.125, 13.1875, 12.625, 12.4375, 14.25, 12.6875, 12.25, 13.1875, 13.4375, 22.875, 14.0625, 19.375], "probabilities": [0.00013506198592949659, 4.968651046510786e-05, 0.003947088960558176, 9.28266454138793e-05, 0.00015304519911296666, 5.993339073029347e-05, 3.4149001294281334e-05, 2.8310523703112267e-05, 0.0001734229299472645, 3.635143366409466e-05, 2.347025292692706e-05, 5.993339073029347e-05, 7.695596286794171e-05, 0.9658207297325134, 0.00014377270417753607, 0.029165254905819893], "choice": "candidate-0fb9a6cde602f212d85e", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0033386945724487305}
|
| 14 |
+
{"panel_index": 13, "task_family": "browser", "logits": [13.5625, 12.875, 13.6875, 13.6875, 13.6875, 12.4375, 13.625, 13.0, 13.0625, 14.5, 17.875, 13.875, 14.0625, 14.4375, 20.875, 14.1875], "probabilities": [0.0006285944837145507, 0.0003160771739203483, 0.0007122908718883991, 0.0007122908718883991, 0.0007122908718883991, 0.00020407479314599186, 0.0006691355374641716, 0.00035816236049868166, 0.0003812617505900562, 0.0016051724087446928, 0.046910010278224945, 0.0008591865771450102, 0.0010363772744312882, 0.0015079202130436897, 0.942212700843811, 0.0011743693612515926], "choice": "candidate-f5c1b8b97d7add30b043", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.031216368079185486}
|
| 15 |
+
{"panel_index": 14, "task_family": "browser", "logits": [13.5625, 21.0, 13.75, 13.25, 13.3125, 12.9375, 12.625, 13.0, 13.1875, 13.3125, 12.75, 18.0, 14.5625, 14.1875, 13.1875, 13.125], "probabilities": [0.0005567604093812406, 0.945656955242157, 0.0006715815397910774, 0.0004073347372468561, 0.00043360545532777905, 0.0002980125427711755, 0.00021803051640745252, 0.0003172325959894806, 0.0003826554457191378, 0.00043360545532777905, 0.0002470609615556896, 0.047081489115953445, 0.001513432478532195, 0.0010401657782495022, 0.0003826554457191378, 0.0003594716254156083], "choice": "candidate-b56ad12ceea4c52feb3f", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0478680320084095}
|
| 16 |
+
{"panel_index": 15, "task_family": "browser", "logits": [17.5, 16.25, 16.0, 20.875, 18.5, 15.375, 15.375, 18.5, 14.5, 16.5, 14.75, 15.0, 19.375, 15.9375, 15.25, 15.9375], "probabilities": [0.022717734798789024, 0.006508741993457079, 0.00506901228800416, 0.6639096140861511, 0.06175321340560913, 0.002713247202336788, 0.002713247202336788, 0.06175321340560913, 0.0011310497066006064, 0.008357387967407703, 0.0014522966230288148, 0.0018647860269993544, 0.14813826978206635, 0.004761896096169949, 0.002394432667642832, 0.004761896096169949], "choice": "candidate-da095abe0a63c03f3114", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.10046396031975746}
|
| 17 |
+
{"panel_index": 16, "task_family": "hs3", "logits": [17.75, 21.75], "probabilities": [0.01798621006309986, 0.9820137619972229], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 18 |
+
{"panel_index": 17, "task_family": "hs3", "logits": [16.25, 22.5], "probabilities": [0.0019267346942797303, 0.9980732202529907], "choice": "no", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 19 |
+
{"panel_index": 18, "task_family": "hs3", "logits": [21.5, 17.5], "probabilities": [0.9820137619972229, 0.01798621006309986], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.011293411254882812}
|
| 20 |
+
{"panel_index": 19, "task_family": "hs3", "logits": [22.625, 16.875], "probabilities": [0.9968273043632507, 0.0031726835295557976], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.656612873077393e-10}
|
| 21 |
+
{"panel_index": 20, "task_family": "hs3", "logits": [22.625, 16.375], "probabilities": [0.9980732202529907, 0.0019267346942797303], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0008741923375055194}
|
| 22 |
+
{"panel_index": 21, "task_family": "hs3", "logits": [15.0, 21.125], "probabilities": [0.0021827167365700006, 0.9978172779083252], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 23 |
+
{"panel_index": 22, "task_family": "hs3", "logits": [22.875, 16.125], "probabilities": [0.9988304972648621, 0.0011695100693032146], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00015556812286376953}
|
| 24 |
+
{"panel_index": 23, "task_family": "hs3", "logits": [21.25, 17.75], "probabilities": [0.970687747001648, 0.02931222692131996], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.02409110590815544}
|
| 25 |
+
{"panel_index": 24, "task_family": "hs3", "logits": [23.125, 17.375], "probabilities": [0.9968273043632507, 0.0031726835295557976], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00037175649777054787}
|
| 26 |
+
{"panel_index": 25, "task_family": "hs3", "logits": [16.375, 21.75], "probabilities": [0.004609572701156139, 0.9953904151916504], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0006105531938374043}
|
| 27 |
+
{"panel_index": 26, "task_family": "hs3", "logits": [18.875, 21.5], "probabilities": [0.06754668802022934, 0.9324532747268677], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.06347662210464478}
|
| 28 |
+
{"panel_index": 27, "task_family": "hs3", "logits": [22.25, 16.75], "probabilities": [0.9959298968315125, 0.004070138558745384], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005394816398620605}
|
| 29 |
+
{"panel_index": 28, "task_family": "hs3", "logits": [20.75, 18.75], "probabilities": [0.8807970285415649, 0.11920291185379028], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.14973852038383484}
|
| 30 |
+
{"panel_index": 29, "task_family": "hs3", "logits": [14.125, 21.75], "probabilities": [0.00048785723629407585, 0.9995121955871582], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00013088664854876697}
|
| 31 |
+
{"panel_index": 30, "task_family": "hs3", "logits": [21.625, 17.875], "probabilities": [0.977022647857666, 0.022977372631430626], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.862645149230957e-09}
|
| 32 |
+
{"panel_index": 31, "task_family": "hs3", "logits": [22.125, 17.0], "probabilities": [0.9940890073776245, 0.0059110685251653194], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00130149582400918}
|
| 33 |
+
{"panel_index": 32, "task_family": "boolq", "logits": [15.9375, 21.75], "probabilities": [0.002981032943353057, 0.9970190525054932], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0016286373138427734}
|
| 34 |
+
{"panel_index": 33, "task_family": "boolq", "logits": [14.25, 21.625], "probabilities": [0.0006263343966566026, 0.999373733997345], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 35 |
+
{"panel_index": 34, "task_family": "boolq", "logits": [14.25, 21.25], "probabilities": [0.0009110510000027716, 0.9990890026092529], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.5150361731648445e-05}
|
| 36 |
+
{"panel_index": 35, "task_family": "boolq", "logits": [20.625, 16.875], "probabilities": [0.977022647857666, 0.022977372631430626], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.02092665433883667}
|
| 37 |
+
{"panel_index": 36, "task_family": "boolq", "logits": [22.0, 14.8125], "probabilities": [0.9992445707321167, 0.0007554056355729699], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.869699478149414e-05}
|
| 38 |
+
{"panel_index": 37, "task_family": "boolq", "logits": [22.0, 14.1875], "probabilities": [0.999595582485199, 0.00040448145591653883], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.6106834411621094e-05}
|
| 39 |
+
{"panel_index": 38, "task_family": "boolq", "logits": [22.125, 14.9375], "probabilities": [0.9992445707321167, 0.0007554056355729699], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0003249049186706543}
|
| 40 |
+
{"panel_index": 39, "task_family": "boolq", "logits": [15.625, 21.25], "probabilities": [0.003593603614717722, 0.9964063763618469], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0010159690864384174}
|
| 41 |
+
{"panel_index": 40, "task_family": "boolq", "logits": [21.875, 13.6875], "probabilities": [0.9997219443321228, 0.00027803104603663087], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.792609691619873e-05}
|
| 42 |
+
{"panel_index": 41, "task_family": "boolq", "logits": [15.125, 21.25], "probabilities": [0.0021827167365700006, 0.9978172779083252], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0009899735450744629}
|
| 43 |
+
{"panel_index": 42, "task_family": "boolq", "logits": [21.75, 14.0625], "probabilities": [0.9995417594909668, 0.000458312890259549], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.7755799237638712e-05}
|
| 44 |
+
{"panel_index": 43, "task_family": "boolq", "logits": [15.3125, 21.25], "probabilities": [0.0026316738221794367, 0.997368335723877], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0022737320978194475}
|
| 45 |
+
{"panel_index": 44, "task_family": "boolq", "logits": [22.125, 14.4375], "probabilities": [0.9995417594909668, 0.000458312890259549], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.3831376135349274e-05}
|
| 46 |
+
{"panel_index": 45, "task_family": "boolq", "logits": [21.375, 14.5], "probabilities": [0.9989677667617798, 0.0010322309099137783], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00017633027164265513}
|
| 47 |
+
{"panel_index": 46, "task_family": "boolq", "logits": [16.625, 21.25], "probabilities": [0.00970847811549902, 0.9902915358543396], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.006197929382324219}
|
| 48 |
+
{"panel_index": 47, "task_family": "boolq", "logits": [15.9375, 21.5], "probabilities": [0.003824483836069703, 0.9961755275726318], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0007851123809814453}
|
| 49 |
+
{"panel_index": 48, "task_family": "mnli", "logits": [15.0625, 22.25, 15.5625], "probabilities": [0.0007544659310951829, 0.9980015158653259, 0.0012439042329788208], "choice": "neutral", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0011983513832092285}
|
| 50 |
+
{"panel_index": 49, "task_family": "mnli", "logits": [13.6875, 21.875, 14.5], "probabilities": [0.00027785697602666914, 0.9990960359573364, 0.0006261603557504714], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00016799179138615727}
|
| 51 |
+
{"panel_index": 50, "task_family": "mnli", "logits": [21.0, 17.125, 15.6875], "probabilities": [0.9749592542648315, 0.02023463509976864, 0.004806147422641516], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.018943727016448975}
|
| 52 |
+
{"panel_index": 51, "task_family": "mnli", "logits": [15.25, 21.875, 16.375], "probabilities": [0.001319636357948184, 0.9946156144142151, 0.0040647671557962894], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0011199116706848145}
|
| 53 |
+
{"panel_index": 52, "task_family": "mnli", "logits": [12.875, 13.375, 21.875], "probabilities": [0.00012336952204350382, 0.00020340185437817127, 0.9996732473373413], "choice": "contradiction", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00011962652206420898}
|
| 54 |
+
{"panel_index": 53, "task_family": "mnli", "logits": [21.625, 16.25, 14.9375], "probabilities": [0.9941569566726685, 0.004603860899806023, 0.0012391124619171023], "choice": "entailment", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.001871347427368164}
|
| 55 |
+
{"panel_index": 54, "task_family": "mnli", "logits": [21.375, 16.875, 15.3125], "probabilities": [0.9867406487464905, 0.010961700230836868, 0.0022976964246481657], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0040070414543151855}
|
| 56 |
+
{"panel_index": 55, "task_family": "mnli", "logits": [22.125, 15.5, 14.625], "probabilities": [0.9981235861778259, 0.0013242907589301467, 0.0005520466365851462], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.000704646110534668}
|
| 57 |
+
{"panel_index": 56, "task_family": "mnli", "logits": [22.25, 14.6875, 13.9375], "probabilities": [0.9992355108261108, 0.0005191774689592421, 0.00024524214677512646], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0003471970558166504}
|
| 58 |
+
{"panel_index": 57, "task_family": "mnli", "logits": [13.25, 21.875, 14.3125], "probabilities": [0.000179434719029814, 0.9993013143539429, 0.0005192116368561983], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00033664703369140625}
|
| 59 |
+
{"panel_index": 58, "task_family": "mnli", "logits": [13.375, 22.5, 14.1875], "probabilities": [0.00010887021926464513, 0.9996457099914551, 0.0002453428169246763], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001429915428161621}
|
| 60 |
+
{"panel_index": 59, "task_family": "mnli", "logits": [17.625, 18.25, 21.375], "probabilities": [0.022031612694263458, 0.0411604680120945, 0.9368079304695129], "choice": "contradiction", "correct": false, "agrees_with_reference": false, "max_abs_probability_delta": 0.8700051009654999}
|
| 61 |
+
{"panel_index": 60, "task_family": "mnli", "logits": [14.5, 22.375, 15.8125], "probabilities": [0.0003794488147832453, 0.9982106685638428, 0.001409823657013476], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.024762272834777832}
|
| 62 |
+
{"panel_index": 61, "task_family": "mnli", "logits": [21.25, 16.625, 15.25], "probabilities": [0.9878666400909424, 0.009684705175459385, 0.0024486768525093794], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.03855389356613159}
|
| 63 |
+
{"panel_index": 62, "task_family": "mnli", "logits": [22.75, 15.5, 14.5625], "probabilities": [0.9990127086639404, 0.0007094734464772046, 0.0002778338093776256], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00032824277877807617}
|
| 64 |
+
{"panel_index": 63, "task_family": "mnli", "logits": [22.5, 15.0625, 14.375], "probabilities": [0.9991160035133362, 0.0005882347468286753, 0.0002957830438390374], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.000424802303314209}
|
| 65 |
+
{"panel_index": 64, "task_family": "score", "logits": [20.375, 20.625], "probabilities": [0.4378235340118408, 0.562176525592804], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.41983732394874096}
|
| 66 |
+
{"panel_index": 65, "task_family": "score", "logits": [14.8125, 22.125], "probabilities": [0.0006667023408226669, 0.9993333220481873], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 8.875131607055664e-05}
|
| 67 |
+
{"panel_index": 66, "task_family": "score", "logits": [14.1875, 22.125], "probabilities": [0.0003569705004338175, 0.9996429681777954], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.751101369038224e-05}
|
| 68 |
+
{"panel_index": 67, "task_family": "score", "logits": [17.125, 22.25], "probabilities": [0.0059110685251653194, 0.9940890073776245], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.001666247844696045}
|
| 69 |
+
{"panel_index": 68, "task_family": "score", "logits": [14.0625, 22.0], "probabilities": [0.0003569705004338175, 0.9996429681777954], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0003984350769314915}
|
| 70 |
+
{"panel_index": 69, "task_family": "score", "logits": [21.875, 14.75], "probabilities": [0.9991958737373352, 0.0008040856919251382], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001656662207096815}
|
| 71 |
+
{"panel_index": 70, "task_family": "score", "logits": [15.625, 22.375], "probabilities": [0.0011695100693032146, 0.9988304972648621], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0002409219741821289}
|
| 72 |
+
{"panel_index": 71, "task_family": "score", "logits": [15.0625, 22.25], "probabilities": [0.0007554056355729699, 0.9992445707321167], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.869699478149414e-05}
|
| 73 |
+
{"panel_index": 72, "task_family": "score", "logits": [18.0, 22.375], "probabilities": [0.012431648559868336, 0.9875683188438416], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.7939677238464355e-09}
|
| 74 |
+
{"panel_index": 73, "task_family": "score", "logits": [14.5625, 22.25], "probabilities": [0.000458312890259549, 0.9995417594909668], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0003458857536315918}
|
| 75 |
+
{"panel_index": 74, "task_family": "score", "logits": [14.375, 21.5], "probabilities": [0.0008040856919251382, 0.9991958737373352], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005209445953369141}
|
| 76 |
+
{"panel_index": 75, "task_family": "score", "logits": [17.375, 22.125], "probabilities": [0.008577484637498856, 0.9914224743843079], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0024094590917229652}
|
| 77 |
+
{"panel_index": 76, "task_family": "score", "logits": [17.375, 22.75], "probabilities": [0.004609572701156139, 0.9953904151916504], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.003967940807342529}
|
| 78 |
+
{"panel_index": 77, "task_family": "score", "logits": [15.3125, 22.375], "probabilities": [0.000855900754686445, 0.9991440773010254], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0006452814559452236}
|
| 79 |
+
{"panel_index": 78, "task_family": "score", "logits": [15.9375, 22.5], "probabilities": [0.0014103588182479143, 0.9985895752906799], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0003998520551249385}
|
| 80 |
+
{"panel_index": 79, "task_family": "score", "logits": [17.75, 22.375], "probabilities": [0.00970847811549902, 0.9902915358543396], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0012784656137228012}
|
evaluation/per-question/reference.jsonl
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"panel_index": 0, "id": "mind2web:bef473f1-82a1-4359-a2c0-59b6dc2f6abb:22ad3562-e0f4-42c3-b096-8c173a47673c", "task_family": "browser", "gold": "candidate-c59cf6b3cdd1ee8dfe92", "candidate_ids": ["candidate-f1094edca3d95c3ab877", "candidate-365bc85de0d8df852b74", "candidate-528c0da02e102bf4290b", "candidate-68a3dbfbe50374b33385", "candidate-c59cf6b3cdd1ee8dfe92", "candidate-6a5d2547e9b7961af9f2", "candidate-a44d590fb75558028c8a", "candidate-519332b584cbfc17ffca", "candidate-beb9d47a1825f26d1023", "candidate-d27c1c551c69db04ab36", "candidate-ed6c918d8bc2facd9fb5", "candidate-7111d2a0c81d7ac049be", "candidate-7dae2a8703bdc9f5d043", "candidate-7d02e4befb688b244764", "candidate-7a89b1b7a87361ea291e", "candidate-089cbe46c0766ba6701e"], "prompt_sha256": "b3fd1770027f0a5409a9d04f32bde5d632ef802787a60f4932e72ecc89167216", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [15.375, 14.8125, 13.625, 13.875, 24.125, 14.4375, 13.6875, 13.75, 14.625, 15.9375, 13.6875, 13.0625, 12.5, 12.5, 13.1875, 13.125], "probabilities": [0.0001583212724654004, 9.02087485883385e-05, 2.7512114684213884e-05, 3.532625487423502e-05, 0.9991162419319153, 6.199950439622626e-05, 2.928649337263778e-05, 3.117531014140695e-05, 7.478568295482546e-05, 0.0002778625057544559, 2.928649337263778e-05, 1.5675930626457557e-05, 8.931875527196098e-06, 8.931875527196098e-06, 1.7763157302397303e-05, 1.6686941307852976e-05], "choice": "candidate-c59cf6b3cdd1ee8dfe92"}
|
| 2 |
+
{"panel_index": 1, "id": "mind2web:5f9182dc-d35d-4c0e-9abe-cd913c136528:8450e2c6-f8aa-40bc-876e-21cf29a8cb77", "task_family": "browser", "gold": "candidate-d2449a13158601200ec0", "candidate_ids": ["candidate-0ac56ef6c8e3a6864ae3", "candidate-33ff2e219977f0035115", "candidate-ae85d05843eef166dc37", "candidate-1491fc48d5480406272d", "candidate-458964bfea8f39410a8c", "candidate-6a3568401638b21de4c7", "candidate-5554637f601e90894848", "candidate-fb424fe48f87098b50ab", "candidate-d2449a13158601200ec0", "candidate-d26fca5ac7bd53d243e4", "candidate-59a124a5de2d7629d231", "candidate-37c56ce11f36b7ace139", "candidate-bfd1999ae1c4a3ad89f3", "candidate-77988599c59cbb77c809", "candidate-eb1b6dd49112ec23d746", "candidate-a8a56f7aec962e7413b8"], "prompt_sha256": "9aca18e5ab33a35ca47a65fe691068ba467052dea071279b1751dbfdc95fb224", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.1875, 17.375, 13.8125, 13.9375, 14.0625, 13.5, 13.125, 13.8125, 23.375, 17.375, 14.625, 13.875, 14.875, 14.4375, 16.375, 14.125], "probabilities": [0.00010159576049773023, 0.002461440162733197, 6.982568447710946e-05, 7.912285218480974e-05, 8.96579513209872e-05, 5.1085560698993504e-05, 3.5110555472783744e-05, 6.982568447710946e-05, 0.9930157661437988, 0.002461440162733197, 0.00015735458873678, 7.432905113091692e-05, 0.00020204730390105397, 0.0001304515462834388, 0.0009055131813511252, 9.544038766762242e-05], "choice": "candidate-d2449a13158601200ec0"}
|
| 3 |
+
{"panel_index": 2, "id": "mind2web:0b59dd33-7f6a-48df-aa1e-9cc67177287f:b99c5581-8a56-4bd7-bbe4-782795ebf93c", "task_family": "browser", "gold": "candidate-9409b72a36db8d4a0a79", "candidate_ids": ["candidate-0120149ac9a61a819265", "candidate-4312931ba215cd0b2310", "candidate-dd54dfdd68be2163e2b6", "candidate-d8ba462bbea45dc4eb77", "candidate-cb3001c9ecbf0dfd250c", "candidate-9409b72a36db8d4a0a79", "candidate-f6ba1cc847604be6bc73", "candidate-2f2099d2f07ae88921ce", "candidate-77f1025025b0e2d0d1fd", "candidate-e743f54599422c4ee674", "candidate-cd31d3ac291210a0a5e6", "candidate-991d9567f58fb994a717", "candidate-d65d92a915149075d6ad", "candidate-98c01750f337ab587383", "candidate-3939d47e8ecda2b48209", "candidate-a7c78ea57e9f0bf1b5b6"], "prompt_sha256": "d960c318e8c88d1c79f12958cf002b9e2b0b9f76a03321dac1dd1d86d3d1c031", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [17.375, 15.3125, 15.0, 15.0625, 16.25, 22.875, 14.375, 14.25, 13.4375, 13.5625, 13.0, 13.125, 12.5, 12.5625, 12.5625, 12.9375], "probabilities": [0.004056241828948259, 0.0005156932165846229, 0.00037728925235569477, 0.0004016222956124693, 0.0013168688165023923, 0.992529571056366, 0.00020194837998133153, 0.00017821881920099258, 7.908411498647183e-05, 8.961404819274321e-05, 5.106054959469475e-05, 5.785917892353609e-05, 3.0969786166679114e-05, 3.296716386103071e-05, 3.296716386103071e-05, 4.796694702235982e-05], "choice": "candidate-9409b72a36db8d4a0a79"}
|
| 4 |
+
{"panel_index": 3, "id": "mind2web:274571ea-fc2f-4353-86ba-00ecb112d6d2:e6345ea9-a5a4-4b88-95b5-4efececed261", "task_family": "browser", "gold": "candidate-3b25ae8d064a298ef021", "candidate_ids": ["candidate-6c861878e1df45af7cb0", "candidate-5dc0c98aa1601abb7255", "candidate-df37b5f34129ec35c5e2", "candidate-7fc3aa839f6eddc4f5e0", "candidate-a105394e5c10cc41573b", "candidate-99243ece7e7fc3575645", "candidate-3b25ae8d064a298ef021", "candidate-c9f0a04f58f5031fe4e6", "candidate-b039454179929264efa9", "candidate-5dc632768de9b7f12a36", "candidate-0b63de0d609d3e1730f1", "candidate-ed1ed5d91fa057eb68be", "candidate-9827a0d3bbbb4c3f2ef3", "candidate-720f59bcd44dc13ecfd0", "candidate-520f72cca9eaf711108b", "candidate-d45432c0ee25d5e74fe8"], "prompt_sha256": "be5d6ca2e7f4e736ac3a410e66cdb9fb2dce3de6f3cb2d58e0425d11bd3e5264", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [16.5, 14.6875, 14.5625, 15.1875, 15.4375, 14.375, 20.875, 14.75, 14.8125, 14.5, 16.875, 13.0, 14.25, 18.375, 16.5, 14.125], "probabilities": [0.010967337526381016, 0.0017903685802593827, 0.0015799946850165725, 0.002951818984001875, 0.0037902104668319225, 0.0013098616618663073, 0.8712435364723206, 0.0019058375619351864, 0.0020287532825022936, 0.0014842678792774677, 0.015957381576299667, 0.00033118491410277784, 0.001155948848463595, 0.0715160220861435, 0.010967337526381016, 0.0010201212717220187], "choice": "candidate-3b25ae8d064a298ef021"}
|
| 5 |
+
{"panel_index": 4, "id": "mind2web:d637c171-dc6e-4a4e-a162-9c230e822932:9053f0e2-da05-4721-87b5-13edf923052b", "task_family": "browser", "gold": "candidate-34e89f28b090ce5327b0", "candidate_ids": ["candidate-34e89f28b090ce5327b0", "candidate-08b4435b3120f48b358c", "candidate-55a0ff963ae8b79b2ac8", "candidate-f95a96c1f7da545920f5", "candidate-98f2cb734f98ff10ff0a", "candidate-d82967df7ccb99323f3d", "candidate-244abe86bc95c904fd2d", "candidate-c71ff755e4fd0c874dfe", "candidate-8d25c242ac6c2febeb64", "candidate-ad36e87635c46e6e4d49", "candidate-88d796f046707b090dc5", "candidate-c16587c8a38af93f0938", "candidate-a96e4a183f184af811a8", "candidate-6d24b42f26f361f2c6c0", "candidate-227d6d1ece1c8d3e341b", "candidate-7a2474fe7e75e66c0d24"], "prompt_sha256": "e3450db0dcc97dbea2e0858fda6b8120ec2573e8c374f5f799c7ed8142a80f80", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [22.5, 16.625, 15.3125, 17.125, 14.5, 12.625, 12.0, 12.4375, 12.1875, 11.1875, 11.3125, 11.1875, 12.625, 12.375, 12.0, 12.5625], "probabilities": [0.9911865592002869, 0.0027840391267091036, 0.0007493139710277319, 0.004590104799717665, 0.0003325060533825308, 5.0991453463211656e-05, 2.7293759558233432e-05, 4.227339741191827e-05, 3.2922554964898154e-05, 1.2111531759728678e-05, 1.372416409139987e-05, 1.2111531759728678e-05, 5.0991453463211656e-05, 3.9712180296191946e-05, 2.7293759558233432e-05, 4.7902041842462495e-05], "choice": "candidate-34e89f28b090ce5327b0"}
|
| 6 |
+
{"panel_index": 5, "id": "mind2web:1efdcd9d-ebc6-4bb7-8823-e54dfe25f409:f6e611c9-ad21-49ca-a841-7ad529b56c95", "task_family": "browser", "gold": "candidate-fb3025ec54721504f9b5", "candidate_ids": ["candidate-fb3025ec54721504f9b5", "candidate-cfc771a1bb7562ebe782", "candidate-68c510f13da5956437bc", "candidate-ac1438ee133b5015c383", "candidate-58473582dc6c65ae4b7e", "candidate-30cbf6942451aa0f1470", "candidate-7ec55a659d963a2cc94d", "candidate-49df5ddbae6f1748c0de", "candidate-b5be7aabde3c609cc40f", "candidate-0e3fd6a816e692376632", "candidate-a3e570f7c516b4577418", "candidate-8a6b58c9cb19ccc7e1fa", "candidate-d8437c7bdf7236744ac4", "candidate-cbf6c1cc621042e7aff0", "candidate-b6291f89245ae394921e", "candidate-c5a94f8fac1dcdad627a"], "prompt_sha256": "36792836eaeed2b15b8a762fc57a57b4b104a27be92bff8c84681bb90d68476b", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [21.75, 14.375, 15.625, 14.375, 15.375, 13.125, 12.625, 15.6875, 13.75, 12.4375, 12.125, 11.8125, 12.3125, 12.9375, 14.5625, 12.5625], "probabilities": [0.9906988739967346, 0.0006208973936736584, 0.00216714502312243, 0.0006208973936736584, 0.001687774434685707, 0.00017789009143598378, 0.00010789580119308084, 0.002306913724169135, 0.00033234243164770305, 8.944876026362181e-05, 6.544210918946192e-05, 4.787847137777135e-05, 7.893824658822268e-05, 0.00014747606473974884, 0.000748945283703506, 0.00010135871707461774], "choice": "candidate-fb3025ec54721504f9b5"}
|
| 7 |
+
{"panel_index": 6, "id": "mind2web:d1de3d1a-3df1-4421-98f3-f8d078752893:e1f18ee3-1577-44fb-a283-1be215e5ae52", "task_family": "browser", "gold": "candidate-888bb5997d1aba7b25f7", "candidate_ids": ["candidate-8d8171f50795c09388a0", "candidate-8d26ef6bd8e9f762c73d", "candidate-bb2ff00206221aa2b02d", "candidate-0196e00c3795624bcc41", "candidate-a9f93c25b378cb9448f1", "candidate-0b7a0a178194ef69867b", "candidate-880053bf9e617437e7cb", "candidate-cf42d1a8a48eeea86134", "candidate-67a5886245e859456378", "candidate-b7c7a409d7192fcbe70f", "candidate-888bb5997d1aba7b25f7", "candidate-dd07004fe07f84bf771d", "candidate-a8b562a5018f0346093a", "candidate-ba9bcd4bb262d6387c1e", "candidate-d4dbe03106eb96d6c9dc", "candidate-b3ddfc4f75cd756d4767"], "prompt_sha256": "873952066cc2833adc3b2b3af9b8f8261f195c13cd41118b830744f22683e25f", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [13.5, 12.5625, 13.0625, 13.5625, 14.3125, 12.625, 13.75, 12.75, 13.8125, 15.125, 22.0, 13.5, 12.875, 14.0625, 13.125, 13.5625], "probabilities": [0.0002026841539191082, 7.937225018395111e-05, 0.00013086272520013154, 0.00021575613936875015, 0.00045675577712245286, 8.4491322922986e-05, 0.00026025157421827316, 9.574121213518083e-05, 0.00027703639352694154, 0.001029315055347979, 0.9961457252502441, 0.0002026841539191082, 0.00010848901001736522, 0.00035572174238041043, 0.00013930264685768634, 0.00021575613936875015], "choice": "candidate-888bb5997d1aba7b25f7"}
|
| 8 |
+
{"panel_index": 7, "id": "mind2web:80e12375-19ad-400f-9e35-2a3853173bed:e89bb795-2d24-4e2c-bcae-1294e3501dfa", "task_family": "browser", "gold": "candidate-41c5af16d7bf373cc74c", "candidate_ids": ["candidate-41c5af16d7bf373cc74c", "candidate-03bd5f2aaa71b912d85f", "candidate-39aafe4d05bcbbf1a9d8", "candidate-7db455c43ca404d644b3", "candidate-8bdbe188e49bad6ea2d9", "candidate-92648d446426dec0e918", "candidate-36e6b14fcf2f5e8e9c23", "candidate-fe113518f51162e48db7", "candidate-f6b0a3559a94799eebe8", "candidate-853b6e262fd75054c736", "candidate-a77385eabd707033f244", "candidate-b7024479880013db2adc", "candidate-b5cc7d0d8e9ea791112e", "candidate-1b8a962d51e9c3897102", "candidate-2b4df4c4f3a6e5c34716", "candidate-c3a3c2242e59afcbb3fb"], "prompt_sha256": "33badf184ca6146904e42f3db06fbd31f551b1ce3082595b5db35494ee245a4b", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [22.5, 14.25, 13.5625, 13.9375, 13.625, 12.625, 11.9375, 13.0625, 13.4375, 12.0625, 12.3125, 12.0625, 13.5625, 12.75, 11.8125, 13.0625], "probabilities": [0.9986169338226318, 0.00026089721359312534, 0.00013118736387696117, 0.00019087646796833724, 0.0001396482257405296, 5.137371044838801e-05, 2.5832323444774374e-05, 7.956915214890614e-05, 0.00011577244731597602, 2.9271855964907445e-05, 3.758580714929849e-05, 2.9271855964907445e-05, 0.00013118736387696117, 5.821403829031624e-05, 2.2796944904257543e-05, 7.956915214890614e-05], "choice": "candidate-41c5af16d7bf373cc74c"}
|
| 9 |
+
{"panel_index": 8, "id": "mind2web:0245df99-2909-465a-861e-7fbca948e82f:dc1847f7-919b-4a2f-b778-2ee33edacc46", "task_family": "browser", "gold": "candidate-a1a3ad1a8495aa678cbe", "candidate_ids": ["candidate-80e6a190025076e4821d", "candidate-cd39817d36a3b7f8086c", "candidate-a1a3ad1a8495aa678cbe", "candidate-74eaca906c939e439348", "candidate-1c996fce61cea008cd97", "candidate-b86e5504c07b50684120", "candidate-ca5385f6ff0f2b510b2a", "candidate-27ecbf7b79508deea5b5", "candidate-335d71d7afcd7c39e60c", "candidate-3fdeb3ac89f0ecf87cff", "candidate-d594855a68f3d65bda29", "candidate-ff28229b986139c46fad", "candidate-4d0fd233b1f9ec995708", "candidate-491e3298e78304983a55", "candidate-438c3b7b4da6155b0b79", "candidate-66345b239aef37d2f31c"], "prompt_sha256": "19bd775a219dfb7da24c37ac1e5319666e84c1584357f8e31021b2c8e9fe3952", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.9375, 14.9375, 22.75, 14.5, 15.1875, 13.875, 15.625, 14.6875, 14.6875, 13.3125, 12.625, 12.3125, 12.3125, 12.9375, 12.8125, 12.5625], "probabilities": [0.0004032402648590505, 0.0004032402648590505, 0.9965280890464783, 0.0002603514585644007, 0.0005177707644179463, 0.00013935611059423536, 0.0008019390515983105, 0.00031404386390931904, 0.00031404386390931904, 7.94027146184817e-05, 3.992619167547673e-05, 2.9210626962594688e-05, 2.9210626962594688e-05, 5.457263614516705e-05, 4.816018554265611e-05, 3.750718678929843e-05], "choice": "candidate-a1a3ad1a8495aa678cbe"}
|
| 10 |
+
{"panel_index": 9, "id": "mind2web:8eae88ef-9641-43c6-be6d-f8abc96d99fa:282c09d0-c9e0-4007-b88a-27887fe1e388", "task_family": "browser", "gold": "candidate-976c2731e7518a1f1b2b", "candidate_ids": ["candidate-976c2731e7518a1f1b2b", "candidate-55380cc26f4800be25ce", "candidate-495e2ce778804116afed", "candidate-6a3ce8bdfda5c7e09a24", "candidate-7e020b58dbe771cceb6b", "candidate-ca7f03f7b7139457a468", "candidate-4556165f722234b67398", "candidate-0232a0c57f2c0a004a53", "candidate-892461b7e0e4bf188cd7", "candidate-98a8e1da12737fa9372c", "candidate-f6c802447f1e19395568", "candidate-e95807766432f0e50c72", "candidate-955408afe8906a0bb10d", "candidate-cd1be178e6146e0fd80a", "candidate-38a919ada71ae1df352a", "candidate-38c746333273edc02a2e"], "prompt_sha256": "05bf3a03b2c0d0a936c4e37c690299142bb6148b2eda9be40aa00c6a13a66811", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [21.75, 15.0, 13.9375, 13.5625, 14.0625, 16.0, 13.1875, 14.125, 14.625, 13.75, 14.0, 13.375, 13.8125, 14.3125, 13.0, 14.0625], "probabilities": [0.9905515909194946, 0.001159816631115973, 0.0004008219111710787, 0.0002754806191660464, 0.0004541907401289791, 0.0031527087558060884, 0.00018933485262095928, 0.00048348354175686836, 0.0007971295854076743, 0.000332293042447418, 0.0004266727191861719, 0.00022838143922854215, 0.0003537240845616907, 0.0005831924499943852, 0.00015696411719545722, 0.0004541907401289791], "choice": "candidate-976c2731e7518a1f1b2b"}
|
| 11 |
+
{"panel_index": 10, "id": "mind2web:3f0988e0-e3a4-4dd7-a89f-482175a474a8:a684c98a-c238-4bef-b2ad-f476f07d73f8", "task_family": "browser", "gold": "candidate-016db9380de74cbcab5d", "candidate_ids": ["candidate-3f6330dc6b89ca9febbe", "candidate-0acffd69c24f4be0a17e", "candidate-ae8922f024c12c266278", "candidate-c12d0dfb7e58d7f22bc7", "candidate-86f75c1fc5227d6081b8", "candidate-6ff913a423628d92b636", "candidate-87e0dada00c5e19e9dce", "candidate-2c14d16a67e5d74c7a66", "candidate-f6a74c2240fb95cf9da0", "candidate-ac52e1e2c9fab34583a1", "candidate-0c2b330dc3030639c697", "candidate-016db9380de74cbcab5d", "candidate-cab20f7657710f02c34a", "candidate-d48a5d89e157a64dd7b4", "candidate-b4b12517818528229992", "candidate-09c561e352fda884d806"], "prompt_sha256": "8c9d16ca40b336c54eab472ed9a7d490065e655ff833efc53466acbdaf85024c", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [15.1875, 16.25, 14.375, 14.375, 14.5, 14.75, 13.875, 14.0625, 16.5, 14.375, 15.5625, 21.5, 14.5, 17.25, 15.0, 13.9375], "probabilities": [0.0017450687009841204, 0.005049523897469044, 0.0007743695750832558, 0.0007743695750832558, 0.0008774756570346653, 0.0011267010122537613, 0.0004696788964793086, 0.0005665408680215478, 0.006483716890215874, 0.0007743695750832558, 0.002539060078561306, 0.9622688889503479, 0.0008774756570346653, 0.013726028613746166, 0.0014467127621173859, 0.0004999705706723034], "choice": "candidate-016db9380de74cbcab5d"}
|
| 12 |
+
{"panel_index": 11, "id": "mind2web:5b56d5b8-1f41-43ca-9f21-369d849f1aa0:edb09eb7-6a8c-4aeb-9b52-796762ca821d", "task_family": "browser", "gold": "candidate-2dc16f49fe3acb051b39", "candidate_ids": ["candidate-021004f4c9756e9f14bf", "candidate-9393f92278ae33837cfc", "candidate-d53f82cb73ef67bc0ce8", "candidate-332f924943c28c247430", "candidate-afd6b2a84985b30c66cd", "candidate-21f93b804332aae201c9", "candidate-bdba9a0649055da71f94", "candidate-0cf8d953323f4b29b10c", "candidate-dea9b4fc8a207df523df", "candidate-fa70b5bc02a04fa0a18a", "candidate-1932fde20b275a9b3081", "candidate-5c9c9ac05919970737c1", "candidate-544fea69d385caac3601", "candidate-2dc16f49fe3acb051b39", "candidate-ca60dc8da2fba3f06111", "candidate-ea45663120f48fa54b88"], "prompt_sha256": "fc3a0ccb7f4fba5d2ba07c7ec21def24cb6fe447d1c9b124d867a6654c6c09b4", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [17.75, 15.5625, 14.125, 13.9375, 15.625, 14.125, 13.5625, 14.5, 14.3125, 17.125, 13.375, 16.125, 17.125, 20.625, 13.625, 13.3125], "probabilities": [0.04895586892962456, 0.005492695607244968, 0.0013046296080574393, 0.001081575988791883, 0.00584694417193532, 0.0013046296080574393, 0.0007433555438183248, 0.0018982249312102795, 0.0015736836940050125, 0.0262041874229908, 0.0006162634235806763, 0.009639982134103775, 0.0262041874229908, 0.8677634596824646, 0.0007912978762760758, 0.000578925886657089], "choice": "candidate-2dc16f49fe3acb051b39"}
|
| 13 |
+
{"panel_index": 12, "id": "mind2web:6e565708-43e2-492b-9f1d-25d51387dcf7:bee75aa3-6f7d-4626-be6f-1b217ac16733", "task_family": "browser", "gold": "candidate-bb1945d8d438379625ee", "candidate_ids": ["candidate-69abfa80e562043e1fa3", "candidate-c4afb9a40ecda9a03c50", "candidate-ad4a56ee767e7be58ceb", "candidate-16c8c33fbf8679f935d6", "candidate-777d646249275ea3bc88", "candidate-ce64cb6d800f5f6f7886", "candidate-cb8d5f96e26185552cf9", "candidate-8203c66243ac00ab8c67", "candidate-32a97c97039dcc468732", "candidate-6fbd3f273e3c48d30d0c", "candidate-97141ccdf3d685ac91be", "candidate-d5d5ef09f20a3d2ceade", "candidate-04470be9c3bbc28919e6", "candidate-0fb9a6cde602f212d85e", "candidate-4ca405ae57be628d94ee", "candidate-bb1945d8d438379625ee"], "prompt_sha256": "a7f850a48b635b1a855ab90b47855fd8269980f52d324bdf635009f9344a5fa4", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [13.3125, 13.0, 17.125, 13.0625, 13.3125, 12.875, 12.0, 12.5, 14.1875, 12.5, 11.75, 12.625, 13.5, 22.625, 14.1875, 19.0], "probabilities": [8.750399138079956e-05, 6.401928840205073e-05, 0.00396073330193758, 6.814818334532902e-05, 8.750399138079956e-05, 5.649682498187758e-05, 2.3551379854325205e-05, 3.8829657569294795e-05, 0.0002099111588904634, 3.8829657569294795e-05, 1.8341832401347347e-05, 4.399977115099318e-05, 0.00010554996697464958, 0.9691594243049622, 0.0002099111588904634, 0.02582722343504429], "choice": "candidate-0fb9a6cde602f212d85e"}
|
| 14 |
+
{"panel_index": 13, "id": "mind2web:44a12ff5-0172-444a-b979-f224162c1aa8:5b8da6f5-c53c-4b69-bfad-7bdfd2e6ce20", "task_family": "browser", "gold": "candidate-3ef3bb624316b6eb3508", "candidate_ids": ["candidate-3ef3bb624316b6eb3508", "candidate-5b52b8bc7f6397531f34", "candidate-6a5a6a33fbc65504059f", "candidate-bde4765810f968ad4833", "candidate-1f66a27bf372d26b14b3", "candidate-8033052a6976a4b01205", "candidate-45ec5aa3dbee10fc9a75", "candidate-ea5b9b527b94b21e6535", "candidate-56683febea1548c570fd", "candidate-bd10584aedf8afa78746", "candidate-94dddd06eb43ed2921b0", "candidate-c9197b3d3e5e97a0f6ff", "candidate-331b8cbdf48fd7f61188", "candidate-c8041a4dd7e8b00e705c", "candidate-f5c1b8b97d7add30b043", "candidate-7161a3b5bba5920fcb33"], "prompt_sha256": "c8d922fd930198c507227984857f26500162ceac030c038e92206981e9747cc3", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.1875, 13.25, 13.6875, 14.3125, 13.6875, 12.8125, 13.75, 13.625, 13.9375, 15.5625, 17.125, 14.1875, 14.8125, 14.875, 21.25, 14.4375], "probabilities": [0.0008317325264215469, 0.00032571115298196673, 0.0005044713034294546, 0.000942476384807378, 0.0005044713034294546, 0.00021029490744695067, 0.0005370068829506636, 0.0004739069554489106, 0.0006477539427578449, 0.0032895661424845457, 0.01569364219903946, 0.0008317325264215469, 0.0015538809821009636, 0.0016540978103876114, 0.9709312319755554, 0.0010679656406864524], "choice": "candidate-f5c1b8b97d7add30b043"}
|
| 15 |
+
{"panel_index": 14, "id": "mind2web:caafd610-202e-49d2-85d1-3f167f3ab443:d7bfb473-8c73-4808-96bc-187d00be5ad7", "task_family": "browser", "gold": "candidate-b56ad12ceea4c52feb3f", "candidate_ids": ["candidate-a78c74bd163f786d9539", "candidate-b56ad12ceea4c52feb3f", "candidate-b4ae8d879b43b183d54a", "candidate-9ebb992dd5a329ac8366", "candidate-f2c9f9f313e700cc877d", "candidate-af82db0cdef64a4a6cf2", "candidate-5fdd288dfbfc0a932f78", "candidate-f0729506779d73c72c2f", "candidate-4ffcfa3737d12e4f223c", "candidate-efc032a84043e09d128a", "candidate-4a068ffbc0d398300aaa", "candidate-7f042d5315889e8f72e0", "candidate-137b03b260a683ac0cd6", "candidate-874b3d025779abce748c", "candidate-5de2a5273572d4bc9c39", "candidate-49b9b04e59bbfb197a63"], "prompt_sha256": "d94ca8165f95e231260aacbaa5bc8e99ca44241878d872928f7c96298839ae96", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.0, 21.625, 13.6875, 13.3125, 13.1875, 13.0625, 12.625, 12.8125, 13.5625, 13.125, 12.5, 19.375, 15.0625, 13.9375, 13.4375, 12.5], "probabilities": [0.000439703551819548, 0.9008560180664062, 0.0003216939512640238, 0.00022109683777671307, 0.00019511724531184882, 0.0001721903681755066, 0.00011117445683339611, 0.00013410199608188123, 0.00028389395447447896, 0.00018329570593778044, 9.811112249735743e-05, 0.09494952112436295, 0.001272324356250465, 0.000413063244195655, 0.0002505355223547667, 9.811112249735743e-05], "choice": "candidate-b56ad12ceea4c52feb3f"}
|
| 16 |
+
{"panel_index": 15, "id": "mind2web:8368b990-c6ca-4cfe-a7ab-c2a88697639d:bf14f1d4-470f-4110-b3f4-019a9f7d0aed", "task_family": "browser", "gold": "candidate-8b6280aa9682e1a79ce0", "candidate_ids": ["candidate-cc60994b6c36f4d19840", "candidate-f8c1512ffc92c70eafe7", "candidate-53d2672be9bfe9447865", "candidate-da095abe0a63c03f3114", "candidate-e926e62f231edca6157e", "candidate-3eccf28da059e9811f3b", "candidate-c8e6f0df1300d65debaa", "candidate-e76caeae9d53c8942ed1", "candidate-de3ea77a36017ef9c156", "candidate-1ab38af3801bba39e08c", "candidate-8b6280aa9682e1a79ce0", "candidate-451ffc245026adbc3d68", "candidate-77aac247bed375f821ae", "candidate-274f32a25dc920013c6f", "candidate-e97a589066e18a2db4b7", "candidate-c6136bbe715f050ff989"], "prompt_sha256": "52d0c2eeb247ef1c21d83eaac84a9f88d22eb50e65489a0dfb02c1c86b73c5e7", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [18.875, 17.0, 15.6875, 21.0, 17.875, 14.9375, 14.625, 18.375, 15.0625, 15.3125, 14.125, 13.9375, 18.25, 15.4375, 14.0, 14.3125], "probabilities": [0.08906733989715576, 0.013658919371664524, 0.0036762480158358812, 0.745751678943634, 0.03276604413986206, 0.0017365366220474243, 0.0012704774271696806, 0.05402207374572754, 0.0019677537493407726, 0.002526646014302969, 0.0007705834577791393, 0.0006388361216522753, 0.047674309462308884, 0.0028630648739635944, 0.0006800375413149595, 0.0009295011404901743], "choice": "candidate-da095abe0a63c03f3114"}
|
| 17 |
+
{"panel_index": 16, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0010788", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "43d8aa81e0401b3731c05f98e2bd2b3dbf59139695ae6b700948ef63b4d14e30", "candidate_token_ids": [32, 33], "logits": [17.75, 21.75], "probabilities": [0.01798621006309986, 0.9820137619972229], "choice": "no"}
|
| 18 |
+
{"panel_index": 17, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0001994", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "06ef7447367570160457d3b2f434528fe7391c356d258e3934f257f80f5c01f5", "candidate_token_ids": [32, 33], "logits": [16.25, 22.5], "probabilities": [0.0019267346942797303, 0.9980732202529907], "choice": "no"}
|
| 19 |
+
{"panel_index": 18, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0000105", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "9fea3b00dd4a71a0363c8bd87b99ae4334c242011eba5648c197e999aa174c64", "candidate_token_ids": [32, 33], "logits": [22.25, 17.25], "probabilities": [0.9933071732521057, 0.006692850962281227], "choice": "yes"}
|
| 20 |
+
{"panel_index": 19, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0001238", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "8dace247ec58eab0973f7a1087fe3af7c8695a2992be4d300ba004babaf8cbbe", "candidate_token_ids": [32, 33], "logits": [22.875, 17.125], "probabilities": [0.9968273043632507, 0.0031726830638945103], "choice": "yes"}
|
| 21 |
+
{"panel_index": 20, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0011197", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "b3fa91e805df134cb80d0ae82eabc86c63266d874b7fb6cdd584d222f7cd9426", "candidate_token_ids": [32, 33], "logits": [22.75, 16.875], "probabilities": [0.9971990585327148, 0.0028009270317852497], "choice": "yes"}
|
| 22 |
+
{"panel_index": 21, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0017850", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "8e283dc3e113374111c25df9df8ed38d3a0991314388f80ace18540c352235d0", "candidate_token_ids": [32, 33], "logits": [15.125, 21.25], "probabilities": [0.002182716503739357, 0.9978172779083252], "choice": "no"}
|
| 23 |
+
{"panel_index": 22, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0019640", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "475cb8ff904634ace670760f71e582174addfbcf71d28985c0e1c7c1f9b3dbcc", "candidate_token_ids": [32, 33], "logits": [23.375, 16.75], "probabilities": [0.9986749291419983, 0.001325022429227829], "choice": "yes"}
|
| 24 |
+
{"panel_index": 23, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0010645", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "62ea4bbe1886885ce73d0c1abf72e927fd23117d83868d32d50405bd0a692191", "candidate_token_ids": [32, 33], "logits": [21.25, 18.375], "probabilities": [0.9465966820716858, 0.0534033328294754], "choice": "yes"}
|
| 25 |
+
{"panel_index": 24, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0020714", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "43596c9cb034207f4c188aa25416d7d290143d2f1850853e180cbee8e1ffbe81", "candidate_token_ids": [32, 33], "logits": [23.5, 17.625], "probabilities": [0.9971990585327148, 0.0028009270317852497], "choice": "yes"}
|
| 26 |
+
{"panel_index": 25, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0008261", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "bc7fe561ed0c1b12e7a21d23aa6bc9a8fff8e502d0ec7328c0c3500042da1225", "candidate_token_ids": [32, 33], "logits": [16.75, 22.0], "probabilities": [0.005220125894993544, 0.9947799444198608], "choice": "no"}
|
| 27 |
+
{"panel_index": 26, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0015735", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "cba3fb62985350d60d64f9988e06a26577a2e643809593877cd7eddaf1e3e1b9", "candidate_token_ids": [32, 33], "logits": [16.75, 22.25], "probabilities": [0.004070138093084097, 0.9959298968315125], "choice": "no"}
|
| 28 |
+
{"panel_index": 27, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0014712", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "fc5d5293343cdb9a2c4d587a01f6d4b5c2adf98a0ee775d07a1601ae06342dea", "candidate_token_ids": [32, 33], "logits": [22.375, 17.0], "probabilities": [0.9953904151916504, 0.004609572701156139], "choice": "yes"}
|
| 29 |
+
{"panel_index": 28, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0024669", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "05bc8e81c80ffdc18ed69091c6d4c21decc0b2cd30b5164c34d24f95c17126ff", "candidate_token_ids": [32, 33], "logits": [20.375, 19.375], "probabilities": [0.7310585379600525, 0.2689414322376251], "choice": "yes"}
|
| 30 |
+
{"panel_index": 29, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0032390", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "1a9cc68b19fe2ec0158f8bbeab86b096a1496bbb755f023f66f4e3715b53da46", "candidate_token_ids": [32, 33], "logits": [13.8125, 21.75], "probabilities": [0.0003569705877453089, 0.9996429681777954], "choice": "no"}
|
| 31 |
+
{"panel_index": 30, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0018442", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "f68722d712d424bdeddd2531a73b47948597a47e37bd59132b14df56fa98e9b8", "candidate_token_ids": [32, 33], "logits": [21.625, 17.875], "probabilities": [0.977022647857666, 0.022977370768785477], "choice": "yes"}
|
| 32 |
+
{"panel_index": 31, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0014165", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "b2c1d04fc65e345807d5cf3dc93324826502c3131fbb3eca59bf166ef773b131", "candidate_token_ids": [32, 33], "logits": [22.625, 17.25], "probabilities": [0.9953904151916504, 0.004609572701156139], "choice": "yes"}
|
| 33 |
+
{"panel_index": 32, "id": "boolq/8703574dc0db27742840afd3c1429829ef2f81c557670018e6ad5456fc53b150", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "938a388a0ea3db943c165f47c21a399dcbc2040c6f61340bd280786a9f5704f3", "candidate_token_ids": [32, 33], "logits": [16.125, 21.5], "probabilities": [0.004609572701156139, 0.9953904151916504], "choice": "no"}
|
| 34 |
+
{"panel_index": 33, "id": "boolq/42a6042685586d3796f9fe4460dd6e9cec208f39de59f765dcbef491fe576361", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "0d97ae562208ecc24ae9df8ab3678c7ae4fbe3aa22b17fffbf97fb6b2cfc2fec", "candidate_token_ids": [32, 33], "logits": [14.375, 21.75], "probabilities": [0.000626334163825959, 0.999373733997345], "choice": "no"}
|
| 35 |
+
{"panel_index": 34, "id": "boolq/5ba6d06215b4f7b269d0b30369fea59d748f02f41d1977f14c46c9153cd45cde", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "a3a3f58378880f9c73e1dc63d51ea8d3951c8c90e5b4b593a8c7a1656898903e", "candidate_token_ids": [32, 33], "logits": [14.0625, 21.125], "probabilities": [0.0008559006382711232, 0.9991440773010254], "choice": "no"}
|
| 36 |
+
{"panel_index": 35, "id": "boolq/4c460ed013de1b5d82c629bbf4e3bdbdaf4441a81341404d48d8e02ead5b9814", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "e7363021faeb0475c8d4a53d1e355b57cd7c2336433356f455b0dd025084fcdd", "candidate_token_ids": [32, 33], "logits": [21.625, 15.4375], "probabilities": [0.9979493021965027, 0.0020507434383034706], "choice": "yes"}
|
| 37 |
+
{"panel_index": 36, "id": "boolq/46e903566fe8817ff37dce61d1ef85b757aa465dadb873c6356b6fc23f58476b", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "c59ba2cb157be77ed4543c9c8e414ce9d572e94a4f7af562105af456ca0993e3", "candidate_token_ids": [32, 33], "logits": [22.5, 15.375], "probabilities": [0.9991958737373352, 0.0008040859247557819], "choice": "yes"}
|
| 38 |
+
{"panel_index": 37, "id": "boolq/3f87d9cb171dfe78b389d22677512f92cb48f7eedeefd35f2db1a2444cf08af2", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "f4a7827700e4744e20ca7891fc537579e56467de67e1c9e0218eb74d1f850913", "candidate_token_ids": [32, 33], "logits": [22.5, 14.75], "probabilities": [0.9995694756507874, 0.0004305570910219103], "choice": "yes"}
|
| 39 |
+
{"panel_index": 38, "id": "boolq/dbadfab033f2646b1a933fad5e132dc65cd8998c337d0332763f1df07680d7e7", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "349dd93162e0859e82c150bd521f171b1edd2cc2e98f05ed41fb16defcb80cb5", "candidate_token_ids": [32, 33], "logits": [23.25, 15.5], "probabilities": [0.9995694756507874, 0.0004305570910219103], "choice": "yes"}
|
| 40 |
+
{"panel_index": 39, "id": "boolq/6909b3bd9078683a97a30ee70c6f3e91a9b8848a34efc0784eddf12cfa305cd4", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "f5291ad0f6c72b83b6fd627aca39178a0202dd9a6674a6fca53a3a2dd9dfe3f7", "candidate_token_ids": [32, 33], "logits": [16.125, 21.5], "probabilities": [0.004609572701156139, 0.9953904151916504], "choice": "no"}
|
| 41 |
+
{"panel_index": 40, "id": "boolq/791b5a767a61e438fe080d99068ef294b361fead41dbfc61b61021e33f2fff9f", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "44bb0f449bae5d6f5c4037e0f69fc24b29c530637f5f59fd6436f33027b12bc8", "candidate_token_ids": [32, 33], "logits": [22.0, 13.875], "probabilities": [0.9997040629386902, 0.0002959571429528296], "choice": "yes"}
|
| 42 |
+
{"panel_index": 41, "id": "boolq/cca90e3be9b421447fe9ae0594193ab92a92adb0083f27a46bc224ad0e67c46c", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "1f4adf18c45e5ad59ac100c1e54416032c380206c3420bf39071a4ac2010d52b", "candidate_token_ids": [32, 33], "logits": [15.375, 21.125], "probabilities": [0.0031726830638945103, 0.9968273043632507], "choice": "no"}
|
| 43 |
+
{"panel_index": 42, "id": "boolq/179184609853d9f8f1f862bd5c3113e363f622606fcf5c188cf0da1d3ed61e86", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "e88d50a2527287c73f777f23fcb092662ed71ed0c5d7fe0168ec8994e8fb19a0", "candidate_token_ids": [32, 33], "logits": [22.5, 14.75], "probabilities": [0.9995694756507874, 0.0004305570910219103], "choice": "yes"}
|
| 44 |
+
{"panel_index": 43, "id": "boolq/58b070021c1d992130604f309a01b95166a281de0499349346694570a6c6ac57", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "8446057016a93c99345544b2afe9eaac75f9ffb0acc99cca9f255e73cb837e34", "candidate_token_ids": [32, 33], "logits": [15.9375, 21.25], "probabilities": [0.004905405919998884, 0.9950946569442749], "choice": "no"}
|
| 45 |
+
{"panel_index": 44, "id": "boolq/11b00c8dcb82dfbdaf02aa139e8c70603fb8c432e0fc51873ca00b20815a90f5", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "ecf3d435d5e2e523fc05ac04d40a235e7808770163f14f11214c899c8738f1b7", "candidate_token_ids": [32, 33], "logits": [22.5, 14.6875], "probabilities": [0.999595582485199, 0.00040448151412419975], "choice": "yes"}
|
| 46 |
+
{"panel_index": 45, "id": "boolq/155bc4f5246c5b35ec14d7a2f697381de04c2cefbbc293b387bfdf83f2fea269", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "3c4715c57e5dd569ad4eb1b9d4a725fefaf222f34bd9739091e97cbe8bb6945d", "candidate_token_ids": [32, 33], "logits": [22.0, 14.9375], "probabilities": [0.9991440773010254, 0.0008559006382711232], "choice": "yes"}
|
| 47 |
+
{"panel_index": 46, "id": "boolq/31fa2124dc5769d9e6fae53be4190290131f8fb025a089f8372d6f5d61504af5", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "9a87a3789d0d031052a1165e22df7f13784f74d7d02fffac973d290d993657f7", "candidate_token_ids": [32, 33], "logits": [16.5, 20.625], "probabilities": [0.01590639166533947, 0.9840936064720154], "choice": "no"}
|
| 48 |
+
{"panel_index": 47, "id": "boolq/a1ea17311c3aea8a751a1cd170483c439ce1338b451927c7db07692878ff0d82", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "cebe40ca184a82f8468175b02fdafcbbd829564cf847b3fc0135fd98f67083ea", "candidate_token_ids": [32, 33], "logits": [16.375, 21.75], "probabilities": [0.004609572701156139, 0.9953904151916504], "choice": "no"}
|
| 49 |
+
{"panel_index": 48, "id": "mnli/ab466125416d6bc3f307f7fd25b78e63a8d90f8f54713027fdc7d75fffd8558d", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "5834f0baf5838f80f70d57a54ea54fdf72bab1fffb7ed8abd7b6b83fec52396e", "candidate_token_ids": [32, 33, 34], "logits": [15.25, 21.75, 15.375], "probabilities": [0.0014986329479143023, 0.9968031644821167, 0.0016981738153845072], "choice": "neutral"}
|
| 50 |
+
{"panel_index": 49, "id": "mnli/8f4dbf064e95ffddc782d0d50ab5e6e0fb83aab546d48b00bff5f03706183bf9", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "52a795fb9d86048dfec129052857ec13bfa6dca7b7314081a62143a83055313a", "candidate_token_ids": [32, 33, 34], "logits": [14.1875, 22.25, 14.5625], "probabilities": [0.00031489436514675617, 0.9992269277572632, 0.0004581685643643141], "choice": "neutral"}
|
| 51 |
+
{"panel_index": 50, "id": "mnli/53ac0a380403b0ccd3ae48ea7763a52664f1d1744f6244c25d5980d69fcf7f0f", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "f6adf5004089ef9a0cc851a183a3e4c72d4b8a84fd54803ef1789646635f01dd", "candidate_token_ids": [32, 33, 34], "logits": [21.875, 16.5, 15.375], "probabilities": [0.9939029812812805, 0.004602684639394283, 0.0014942727284505963], "choice": "entailment"}
|
| 52 |
+
{"panel_index": 51, "id": "mnli/e6cf83f056ba7587936d8026ff9709bc50de62e0dfb282f6b8d6aa85649e688d", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "9ddcac65f7c20b39c2eafa8d124294ab44516209a07832ffa4575b35c2ea2396", "candidate_token_ids": [32, 33, 34], "logits": [15.3125, 22.125, 16.375], "probabilities": [0.0010952489683404565, 0.9957355260848999, 0.0031692080665379763], "choice": "neutral"}
|
| 53 |
+
{"panel_index": 52, "id": "mnli/834677de1c970abc266e34006f2e681c7301ff5d043796d92da66364b792f235", "task_family": "mnli", "gold": "contradiction", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "c1a73d0727accd049baa352165c01aebbbc68ef10ab728e698ef44f9efbc7b6c", "candidate_token_ids": [32, 33, 34], "logits": [12.75, 13.6875, 22.5], "probabilities": [5.828258872497827e-05, 0.00014882980030961335, 0.9997928738594055], "choice": "contradiction"}
|
| 54 |
+
{"panel_index": 53, "id": "mnli/13f1e11494916138e91023e5bf4d2483ab5d93f00e2e1e558c5f595ca0e8e22d", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "5aedfe5d0e0bcfe168031cd031b0355c0ed3ce9f22948073417e36f0682394e8", "candidate_token_ids": [32, 33, 34], "logits": [22.125, 16.375, 15.0], "probabilities": [0.9960283041000366, 0.003170139854773879, 0.0008015368366613984], "choice": "entailment"}
|
| 55 |
+
{"panel_index": 54, "id": "mnli/9cf51bc8717b59b904d198884ba5b832b53cdc96a5b1aafee7203dca9bfc824d", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "37b5f12a82b74d56402bb3c33b63e27e4d08218a0ec1ec3a2087df90fa35ea3f", "candidate_token_ids": [32, 33, 34], "logits": [21.875, 17.0, 15.5], "probabilities": [0.9907476902008057, 0.0075644515454769135, 0.0016878575552254915], "choice": "entailment"}
|
| 56 |
+
{"panel_index": 55, "id": "mnli/3d8bd065ebff13c30fa3aa4943c5f2851ca8a5860bae22e25f212be97790ab28", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "0a2b268fa0d617735fea967a5896b3d205b101d83b018ec9fca7d988f11d10db", "candidate_token_ids": [32, 33, 34], "logits": [23.125, 16.125, 14.875], "probabilities": [0.9988282322883606, 0.0009108134545385838, 0.0002609523944556713], "choice": "entailment"}
|
| 57 |
+
{"panel_index": 56, "id": "mnli/4dc433ef7c48f1158e61aa4f1b568bac7ee2ede91001c9b8d933fd5c54e71820", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "28ea8cef7c3cefbc4e52117dbbd91fe6ba0185a3067aff8e0a364e78d7918819", "candidate_token_ids": [32, 33, 34], "logits": [23.125, 15.0625, 13.9375], "probabilities": [0.9995827078819275, 0.0003150064731016755, 0.00010226762242382392], "choice": "entailment"}
|
| 58 |
+
{"panel_index": 57, "id": "mnli/533107d0478441289d303edb55e3769ea75e77ee657738576b1e86f9f523ff09", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "f5983d410d1ffb47caf1d2c160be923e0ed6839419930f59eadfb89cfb2b4c1c", "candidate_token_ids": [32, 33, 34], "logits": [12.625, 22.25, 14.125], "probabilities": [6.603258952964097e-05, 0.9996379613876343, 0.0002959375560749322], "choice": "neutral"}
|
| 59 |
+
{"panel_index": 58, "id": "mnli/e6856d10a89f46cbfc7fa77bf9266c482f130090a961f5bcb10be540e7c5fc2c", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "b0414aaab93e025d307d733858e0be03f543479f804f1dc12a6324b94a1a28c2", "candidate_token_ids": [32, 33, 34], "logits": [12.8125, 22.875, 14.1875], "probabilities": [4.264027302269824e-05, 0.9997887015342712, 0.0001686455652816221], "choice": "neutral"}
|
| 60 |
+
{"panel_index": 59, "id": "mnli/4bd800d8bc83435b531bb472e9078bb054a7bb68bf31036bc12502e177dea19b", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "a2314ed00eeb6299fbe1343e769272e96da353429390c352cf49d59d0c2967e8", "candidate_token_ids": [32, 33, 34], "logits": [19.25, 20.0, 17.75], "probabilities": [0.2993895411491394, 0.6338075995445251, 0.06680282950401306], "choice": "neutral"}
|
| 61 |
+
{"panel_index": 60, "id": "mnli/b0220f9d099b621bde205ea6efa72624cc1948cceb2502cdc38d19c3762c2ba0", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "5e649a151d402888b8c9e9b8ba693d39efad2f95912eb424363518e92fb26dee", "candidate_token_ids": [32, 33, 34], "logits": [14.625, 22.0, 18.375], "probabilities": [0.0006100860773585737, 0.9734483957290649, 0.025941520929336548], "choice": "neutral"}
|
| 62 |
+
{"panel_index": 61, "id": "mnli/5c5d3ecddeb08dada33aaa9e323aa10bdfa371aa2f535b2cd5f4f653b0e25e9c", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "808dae944e65ee87bcaaa32c78f64633ec95e1e202c2e49fc0a34ffc831c7327", "candidate_token_ids": [32, 33, 34], "logits": [21.375, 18.375, 15.75], "probabilities": [0.9493127465248108, 0.0472634993493557, 0.003423756454139948], "choice": "entailment"}
|
| 63 |
+
{"panel_index": 62, "id": "mnli/dba1b74655afea3f9676c7a8687e80c0da196b11f2324d6e3a41c3870b9d8591", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "1e22807eca9d351d7041b7c3a7d466c369566a06caf7559c424c855f9dc8de40", "candidate_token_ids": [32, 33, 34], "logits": [23.125, 15.5625, 14.25], "probabilities": [0.9993409514427185, 0.0005192322423681617, 0.00013974947796668857], "choice": "entailment"}
|
| 64 |
+
{"panel_index": 63, "id": "mnli/c8c695b22a0a846dde5f0ae3c17127f27c53f23219de2f4c97f2a9bebe1e0f81", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "b2d7992028f3524744fa167194e920d2d2f96e049d9baa0c16a60567a2506e01", "candidate_token_ids": [32, 33, 34], "logits": [23.0, 15.0625, 13.8125], "probabilities": [0.9995408058166504, 0.00035693409154191613, 0.00010226333688478917], "choice": "entailment"}
|
| 65 |
+
{"panel_index": 64, "id": "goemotions:0d6f68c6aab93172edb30824a77a2f20e857a379845cfc16d8315af53d05bc1f:approval", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "62e2f3ce6d8db5d5399c3dbeaa448fa75306f4d790a05d15745663f6ceabe85f", "candidate_token_ids": [32, 33], "logits": [18.0, 22.0], "probabilities": [0.01798621006309986, 0.9820137619972229], "choice": "no"}
|
| 66 |
+
{"panel_index": 65, "id": "goemotions:5bb2f0355125b6ea32d14bc489651edeb09f6b4ef9f51b90b8ed330b8fd14cef:sadness", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "5cd89e6970749f8b4a0f8526f39a21035bd2a2d2dab17c9b22cb526e792d40bf", "candidate_token_ids": [32, 33], "logits": [14.8125, 22.0], "probabilities": [0.000755405577365309, 0.9992445707321167], "choice": "no"}
|
| 67 |
+
{"panel_index": 66, "id": "goemotions:5695bd603bf8ce60300298693b29b4295f2d34c083c7e52bff826db80a6ea996:grief", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "4a701cb3026d72bdc664138b3258e27154414aef18bc5d9658a22ca9b16431d3", "candidate_token_ids": [32, 33], "logits": [14.4375, 22.25], "probabilities": [0.00040448151412419975, 0.999595582485199], "choice": "no"}
|
| 68 |
+
{"panel_index": 67, "id": "goemotions:e97ef69f22521e136e92eee4bbc1e46d8571bf91280b2bc7ae24f2e1ec5d0745:desire", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "74cd0d0acfe64e81b3b02d676d3543c54fe2736d05f9d6964e9bf589a8fa604e", "candidate_token_ids": [32, 33], "logits": [17.0, 21.875], "probabilities": [0.007577240467071533, 0.9924227595329285], "choice": "no"}
|
| 69 |
+
{"panel_index": 68, "id": "goemotions:0c48f870b5edcb7a8dcfeb58087c7e3e56a9a2d89a0687e66d9974557c74556a:love", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "74095411b7d11586abb60a651ae44840355675c465df21605302ced8aa7460e9", "candidate_token_ids": [32, 33], "logits": [14.4375, 21.625], "probabilities": [0.000755405577365309, 0.9992445707321167], "choice": "no"}
|
| 70 |
+
{"panel_index": 69, "id": "goemotions:30e748ace2950567f2e1dcc3c638f6d5fc1e2df8078792928acf9550168933c7:approval", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "a8e377d44a31f2539125c844e84396f69387ac0177550bfe52378a2461e31ae7", "candidate_token_ids": [32, 33], "logits": [22.25, 15.3125], "probabilities": [0.9990302324295044, 0.0009697519126348197], "choice": "yes"}
|
| 71 |
+
{"panel_index": 70, "id": "goemotions:3b0e3210dddfd4dbb4b0e76cf62299246fd5a912229994a9e07de7dfc64c2262:embarrassment", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "897f9ee99d20643b8ec00e25182dded61636e60cfd09ee330ca3b25540069833", "candidate_token_ids": [32, 33], "logits": [15.6875, 22.25], "probabilities": [0.0014103584690019488, 0.9985895752906799], "choice": "no"}
|
| 72 |
+
{"panel_index": 71, "id": "goemotions:4308985a124eb5964a14e35ae0d7db83da5f33c019222497141a9a455cb5500c:joy", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "51b9806351f409e9959d99b25249e1561bcb1d2da032317b72b84a429a28d713", "candidate_token_ids": [32, 33], "logits": [15.0, 22.125], "probabilities": [0.0008040859247557819, 0.9991958737373352], "choice": "no"}
|
| 73 |
+
{"panel_index": 72, "id": "goemotions:1285f8257d6a85acd0b9b01bfbebadff45ec085608b755eb415c74c624f8f5eb:optimism", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "5af8f3b74ad252af8af13b9bbe1c5b4585c431868cfaa0c4f99abdb42e654a36", "candidate_token_ids": [32, 33], "logits": [18.125, 22.5], "probabilities": [0.01243165135383606, 0.9875683188438416], "choice": "no"}
|
| 74 |
+
{"panel_index": 73, "id": "goemotions:ee392e883c467546e7e603f5e6b94702211b3b4af3519a0bb7bd3b21bf4cbfd9:joy", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "266326a7e5d742f0cc33238676458ba8635f29efc4d3c95b662feaa590c2b38e", "candidate_token_ids": [32, 33], "logits": [14.625, 21.75], "probabilities": [0.0008040859247557819, 0.9991958737373352], "choice": "no"}
|
| 75 |
+
{"panel_index": 74, "id": "goemotions:43ef087b074a832ce71850c4c8d1672ca1d68b8b03830f65be8a52da990b454c:surprise", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "50981dcbdce4641c8cadc2eb3b78a20e23fca85a04376f095c1419f77d976694", "candidate_token_ids": [32, 33], "logits": [14.875, 21.5], "probabilities": [0.001325022429227829, 0.9986749291419983], "choice": "no"}
|
| 76 |
+
{"panel_index": 75, "id": "goemotions:8285d89a0201a60100cf7540b9e92fbf0564e50afc4e58afb812f91b6fc64cba:desire", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "538fef417332ce98ebd74b27c7c5b59a41fc722574e13987db181517abe649bb", "candidate_token_ids": [32, 33], "logits": [18.0, 22.5], "probabilities": [0.01098694372922182, 0.9890130758285522], "choice": "no"}
|
| 77 |
+
{"panel_index": 76, "id": "goemotions:cfa49c3d3a46c50e494455f5f1aa94c401a83a9450181bcf5d1f64ae75e30e64:annoyance", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "22d93f0279f9bacc380b1a29a2debaca5cae59cd9a7ff8e82c8f42ec97d1aa8d", "candidate_token_ids": [32, 33], "logits": [17.75, 22.5], "probabilities": [0.008577484637498856, 0.9914224743843079], "choice": "no"}
|
| 78 |
+
{"panel_index": 77, "id": "goemotions:e97ef69f22521e136e92eee4bbc1e46d8571bf91280b2bc7ae24f2e1ec5d0745:confusion", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "b7bc2566159ea8e56d02da9bdc7d94820e6e704c057d22624565cf84c3d8ccc8", "candidate_token_ids": [32, 33], "logits": [15.625, 22.125], "probabilities": [0.0015011822106316686, 0.998498797416687], "choice": "no"}
|
| 79 |
+
{"panel_index": 78, "id": "goemotions:a312dda34fe450feab6e22e6d2aa7493d77c757d4800e2bfd9904549ff4c3756:realization", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "53472c74825a9353b6efae1df7e1106ab8c2aeb0f510b2923fd2754893391868", "candidate_token_ids": [32, 33], "logits": [15.8125, 22.125], "probabilities": [0.0018102108733728528, 0.9981897473335266], "choice": "no"}
|
| 80 |
+
{"panel_index": 79, "id": "goemotions:4c893be6bdf7ad879db4d2299867417db381418ebeed081d85cbd13e6b3371a6:anger", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "6a4428c6d807234388965c34cc8785054f982294a2b8b91ab744deaa27db2c6c", "candidate_token_ids": [32, 33], "logits": [17.875, 22.375], "probabilities": [0.01098694372922182, 0.9890130758285522], "choice": "no"}
|