Add Frozen80 per-question results
Browse files- README.md +1 -1
- README.zh-CN.md +1 -1
- evaluation/per-question/mlx-8bit.jsonl +80 -0
- evaluation/per-question/reference.jsonl +80 -0
README.md
CHANGED
|
@@ -40,7 +40,7 @@ Frozen80 with identical prompt tokens, compared with the HF BF16 release (full d
|
|
| 40 |
|---|---|---:|---:|---:|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 69/80 · 86.25% | 79/80 | 0.0622 |
|
| 42 |
|
| 43 |
-
Converted and scored with MLX on Linux (CUDA); the files are platform-independent and load unchanged on Apple Silicon (for 4B, the same kind of file gave identical decisions on CUDA and Metal). Peak memory on Frozen80 was 11.28 GB; plan for a Mac with at least 16 GB of unified memory. Frozen80 is a reused development panel, not a blind benchmark.
|
| 44 |
|
| 45 |
## Conversion
|
| 46 |
|
|
|
|
| 40 |
|---|---|---:|---:|---:|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 69/80 · 86.25% | 79/80 | 0.0622 |
|
| 42 |
|
| 43 |
+
Converted and scored with MLX on Linux (CUDA); the files are platform-independent and load unchanged on Apple Silicon (for 4B, the same kind of file gave identical decisions on CUDA and Metal). Peak memory on Frozen80 was 11.28 GB; plan for a Mac with at least 16 GB of unified memory. Frozen80 is a reused development panel, not a blind benchmark. Per-question rows (candidate probabilities, choice, correctness; join with Frozen80 by `panel_index`): [evaluation/per-question/](evaluation/per-question/).
|
| 44 |
|
| 45 |
## Conversion
|
| 46 |
|
README.zh-CN.md
CHANGED
|
@@ -40,7 +40,7 @@ Frozen80 使用完全相同的 prompt token，与 HF BF16 发布版（完整深
|
|
| 40 |
|---|---|---:|---:|---:|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 69/80 · 86.25% | 79/80 | 0.0622 |
|
| 42 |
|
| 43 |
-
在 Linux（CUDA）上用 MLX 转换并评测；MLX 文件与平台无关，可直接在 Apple Silicon 上加载（4B 的同类文件在 CUDA 与 Metal 上决策完全一致）。Frozen80 评测峰值内存 11.28 GB，建议使用统一内存不少于 16 GB 的 Mac。Frozen80 是复用的开发面板，不是盲测。
|
| 44 |
|
| 45 |
## 转换说明
|
| 46 |
|
|
|
|
| 40 |
|---|---|---:|---:|---:|
|
| 41 |
| NVIDIA RTX PRO 6000 (CUDA) | mlx 0.32.2 on Linux x86_64 (Device(gpu, 0)) | 69/80 · 86.25% | 79/80 | 0.0622 |
|
| 42 |
|
| 43 |
+
在 Linux（CUDA）上用 MLX 转换并评测；MLX 文件与平台无关，可直接在 Apple Silicon 上加载（4B 的同类文件在 CUDA 与 Metal 上决策完全一致）。Frozen80 评测峰值内存 11.28 GB，建议使用统一内存不少于 16 GB 的 Mac。Frozen80 是复用的开发面板，不是盲测。逐题明细（候选概率、选择、是否正确，可按 `panel_index` 与 Frozen80 关联）：[evaluation/per-question/](evaluation/per-question/)。
|
| 44 |
|
| 45 |
## 转换说明
|
| 46 |
|
evaluation/per-question/mlx-8bit.jsonl
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"panel_index": 0, "task_family": "browser", "logits": [13.8125, 13.75, 12.5625, 12.75, 21.25, 12.875, 11.3125, 11.75, 12.625, 10.5, 10.9375, 11.125, 10.5625, 11.4375, 10.9375, 10.6875], "probabilities": [0.0005874166381545365, 0.0005518270190805197, 0.00016829771630000323, 0.00020300575124565512, 0.9977265000343323, 0.00023003567184787244, 4.821811307920143e-05, 7.468165858881548e-05, 0.00017915194621309638, 2.1396657757577486e-05, 3.313978959340602e-05, 3.9974205719772726e-05, 2.2776615878683515e-05, 5.463828347274102e-05, 3.313978959340602e-05, 2.5809289581957273e-05], "choice": "candidate-c59cf6b3cdd1ee8dfe92", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.787799505516887e-05}
|
| 2 |
+
{"panel_index": 1, "task_family": "browser", "logits": [12.4375, 12.1875, 11.9375, 12.375, 11.8125, 11.9375, 11.75, 11.875, 20.75, 17.375, 12.5625, 12.5625, 11.4375, 12.0, 11.875, 10.875], "probabilities": [0.00023677019635215402, 0.00018439677660353482, 0.0001436083548469469, 0.0002224249328719452, 0.00012673392484430224, 0.0001436083548469469, 0.00011905553401447833, 0.00013490760466083884, 0.9647166728973389, 0.03301078453660011, 0.000268295785645023, 0.000268295785645023, 8.71028532856144e-05, 0.00015287025598809123, 0.00013490760466083884, 4.962971433997154e-05], "choice": "candidate-d2449a13158601200ec0", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.2928924560546875e-05}
|
| 3 |
+
{"panel_index": 2, "task_family": "browser", "logits": [14.0625, 13.5625, 13.9375, 14.125, 15.25, 21.125, 12.875, 13.3125, 12.5, 11.1875, 12.25, 12.0625, 12.5, 12.3125, 11.8125, 11.9375], "probabilities": [0.0008502371492795646, 0.0005156948463991284, 0.0007503315573558211, 0.0009050724329426885, 0.0027878209948539734, 0.9925326704978943, 0.00025930762058123946, 0.00040162348886951804, 0.0001782193430699408, 4.796710709342733e-05, 0.0001387974334647879, 0.00011506706505315378, 0.0001782193430699408, 0.000147749058669433, 8.961431740317494e-05, 0.00010154632764169946], "choice": "candidate-9409b72a36db8d4a0a79", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0006356239318847656}
|
| 4 |
+
{"panel_index": 3, "task_family": "browser", "logits": [16.375, 13.9375, 14.4375, 14.3125, 15.8125, 15.375, 21.5, 13.875, 14.0625, 12.9375, 15.5625, 12.75, 13.0625, 17.75, 13.875, 13.4375], "probabilities": [0.005705137737095356, 0.0004985094419680536, 0.0008219032315537333, 0.0007253269432112575, 0.003250689944252372, 0.0020988034084439278, 0.959456741809845, 0.00046830635983496904, 0.0005648851511068642, 0.00018339138478040695, 0.0025316395331174135, 0.00015203675138764083, 0.0002078096877085045, 0.02256426215171814, 0.00046830635983496904, 0.00030236138263717294], "choice": "candidate-3b25ae8d064a298ef021", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.005692839622497559}
|
| 5 |
+
{"panel_index": 4, "task_family": "browser", "logits": [21.0, 13.8125, 11.9375, 14.25, 11.1875, 10.6875, 9.3125, 9.75, 10.5625, 8.5, 9.9375, 10.25, 10.1875, 10.9375, 12.0625, 10.375], "probabilities": [0.9975650310516357, 0.0007541359518654644, 0.00011565048043848947, 0.0011680284515023232, 5.462943954626098e-05, 3.313442357466556e-05, 8.377694939554203e-06, 1.297563449043082e-05, 2.9241024094517343e-05, 3.7175798297539586e-06, 1.5651588910259306e-05, 2.1393194401753135e-05, 2.009704076044727e-05, 4.2545445467112586e-05, 0.0001310491788899526, 2.424166632408742e-05], "choice": "candidate-34e89f28b090ce5327b0", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.000125885009765625}
|
| 6 |
+
{"panel_index": 5, "task_family": "browser", "logits": [20.875, 12.3125, 13.125, 12.1875, 15.0625, 11.625, 10.125, 11.4375, 11.25, 9.125, 9.8125, 10.625, 10.3125, 14.1875, 14.8125, 10.6875], "probabilities": [0.9923182725906372, 0.00018967257346957922, 0.000427433755248785, 0.0001673854567343369, 0.0029669776558876038, 9.537336882203817e-05, 2.1280675355228595e-05, 7.9067271144595e-05, 6.554908759426326e-05, 7.828725756553467e-06, 1.5569268725812435e-05, 3.5085886338492855e-05, 2.5669389287941158e-05, 0.0012368207098916173, 0.0023106844164431095, 3.734874553629197e-05], "choice": "candidate-fb3025ec54721504f9b5", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005102079594507813}
|
| 7 |
+
{"panel_index": 6, "task_family": "browser", "logits": [14.375, 13.9375, 14.6875, 14.5, 14.1875, 14.0, 15.875, 13.875, 15.125, 15.8125, 21.25, 13.0625, 13.375, 14.8125, 12.625, 16.0], "probabilities": [0.0010077691404148936, 0.0006506647332571447, 0.0013774576364085078, 0.0011419520014896989, 0.0008354701567441225, 0.0006926291971467435, 0.004516508895903826, 0.0006112431292422116, 0.002133447676897049, 0.0042428672313690186, 0.9752942323684692, 0.0002712374844122678, 0.00037073763087391853, 0.0015608640387654305, 0.0001751240051817149, 0.005117875523865223], "choice": "candidate-888bb5997d1aba7b25f7", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005990914069116116}
|
| 8 |
+
{"panel_index": 7, "task_family": "browser", "logits": [20.875, 12.0625, 11.5625, 11.3125, 11.1875, 10.9375, 9.125, 9.6875, 10.5625, 8.1875, 9.375, 9.9375, 12.0, 10.8125, 9.5, 10.125], "probabilities": [0.9992793202400208, 0.00014875337365083396, 9.022346057463437e-05, 7.026609091553837e-05, 6.200964708114043e-05, 4.829315730603412e-05, 7.883643775130622e-06, 1.3836222933605313e-05, 3.319136521895416e-05, 3.0872770366840996e-06, 1.0122799722012132e-05, 1.7766053133527748e-05, 0.00013974089233670384, 4.261855792719871e-05, 1.1470627214293927e-05, 2.14299579965882e-05], "choice": "candidate-41c5af16d7bf373cc74c", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.6419915482401848e-05}
|
| 9 |
+
{"panel_index": 8, "task_family": "browser", "logits": [13.625, 15.125, 21.125, 14.875, 13.1875, 12.125, 14.3125, 14.1875, 13.3125, 11.0, 11.0625, 11.375, 11.6875, 12.8125, 11.6875, 11.625], "probabilities": [0.0005484018474817276, 0.002457766328006983, 0.9915335774421692, 0.001914110267534852, 0.0003540746693033725, 0.00012236500333528966, 0.0010906272800639272, 0.0009624751401133835, 0.00040121920756064355, 3.972608465119265e-05, 4.228820762364194e-05, 5.780111678177491e-05, 7.900474884081632e-05, 0.00024335186753887683, 7.900474884081632e-05, 7.421810732921585e-05], "choice": "candidate-a1a3ad1a8495aa678cbe", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0009151101112365723}
|
| 10 |
+
{"panel_index": 9, "task_family": "browser", "logits": [20.875, 13.1875, 13.0, 12.25, 12.5625, 15.1875, 11.0625, 11.0625, 11.3125, 9.5625, 10.0625, 10.6875, 10.75, 10.9375, 10.1875, 12.5], "probabilities": [0.9947840571403503, 0.00045613135444000363, 0.0003781462728511542, 0.00017862359527498484, 0.00024414961808361113, 0.0033703807275742292, 5.447714647743851e-05, 5.447714647743851e-05, 6.99500014889054e-05, 1.2155487638665363e-05, 2.0041014067828655e-05, 3.744155401363969e-05, 3.985631337855011e-05, 4.807591176358983e-05, 2.2709444237989374e-05, 0.00022935726155992597], "choice": "candidate-976c2731e7518a1f1b2b", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.3577852668240666e-05}
|
| 11 |
+
{"panel_index": 10, "task_family": "browser", "logits": [14.3125, 18.25, 14.0, 14.0625, 16.625, 13.9375, 13.0, 13.6875, 14.6875, 12.3125, 13.1875, 20.375, 13.625, 16.125, 11.4375, 12.5625], "probabilities": [0.0019856495782732964, 0.10184440016746521, 0.0014527325984090567, 0.0015464251628145576, 0.020054355263710022, 0.0013647163286805153, 0.0005344305536709726, 0.0010628419695422053, 0.002889103488996625, 0.0002687285887077451, 0.0006446461193263531, 0.85273277759552, 0.0009984472999349236, 0.012163578532636166, 0.00011202270252397284, 0.0003450541407801211], "choice": "candidate-016db9380de74cbcab5d", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.011358372867107391}
|
| 12 |
+
{"panel_index": 11, "task_family": "browser", "logits": [15.375, 13.0, 15.0625, 13.375, 13.75, 12.5625, 12.625, 13.0, 13.25, 16.625, 12.0, 13.0625, 18.5, 20.5, 11.8125, 11.75], "probabilities": [0.005073413252830505, 0.00047190109035000205, 0.003711789147928357, 0.0006866116891615093, 0.0009990142425522208, 0.0003046820929739624, 0.0003243325336370617, 0.00047190109035000205, 0.0006059330771677196, 0.017707955092191696, 0.00017360264610033482, 0.000502335955388844, 0.11547036468982697, 0.8532170057296753, 0.0001439216866856441, 0.00013520187349058688], "choice": "candidate-2dc16f49fe3acb051b39", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.028387129306793213}
|
| 13 |
+
{"panel_index": 12, "task_family": "browser", "logits": [12.5625, 11.4375, 13.1875, 12.8125, 11.6875, 12.125, 11.6875, 11.8125, 12.5, 10.5625, 10.375, 11.125, 12.1875, 20.625, 11.25, 18.875], "probabilities": [0.0002678492455743253, 8.695787255419418e-05, 0.0005004080012440681, 0.0003439250576775521, 0.00011165613977937028, 0.00017293638666160405, 0.00011165613977937028, 0.00012652298028115183, 0.0002516210079193115, 3.624944656621665e-05, 3.00518349831691e-05, 6.36197510175407e-05, 0.00018408987671136856, 0.8499424457550049, 7.209062459878623e-05, 0.1476978361606598], "choice": "candidate-0fb9a6cde602f212d85e", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.028841495513916016}
|
| 14 |
+
{"panel_index": 13, "task_family": "browser", "logits": [14.75, 14.0625, 15.625, 14.375, 13.25, 14.8125, 18.25, 13.6875, 13.4375, 15.1875, 18.25, 13.0, 13.5625, 15.125, 19.25, 13.1875], "probabilities": [0.006008476484566927, 0.0030212507117539644, 0.01441358495503664, 0.00412956066429615, 0.0013406722573563457, 0.006395989563316107, 0.1989733874797821, 0.0020764735527336597, 0.0016171591123566031, 0.009306108579039574, 0.1989733874797821, 0.0010441165650263429, 0.001832481473684311, 0.008742280304431915, 0.5408657193183899, 0.001259444747120142], "choice": "candidate-f5c1b8b97d7add30b043", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.04167109727859497}
|
| 15 |
+
{"panel_index": 14, "task_family": "browser", "logits": [12.5, 21.625, 12.0625, 12.5625, 11.5625, 11.5, 10.125, 11.0625, 11.4375, 9.5625, 10.3125, 12.3125, 12.375, 13.25, 13.5, 14.625], "probabilities": [0.0001086811680579558, 0.9979098439216614, 7.016979361651465e-05, 0.00011569045454962179, 4.256015017745085e-05, 3.9981550798984244e-05, 1.010892719932599e-05, 2.581403350632172e-05, 3.755920261028223e-05, 5.759893156209728e-06, 1.2193681868666317e-05, 9.009981295093894e-05, 9.591078560333699e-05, 0.00023007794516161084, 0.00029542596894316375, 0.000909975788090378], "choice": "candidate-b56ad12ceea4c52feb3f", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0004163384437561035}
|
| 16 |
+
{"panel_index": 15, "task_family": "browser", "logits": [14.6875, 14.25, 14.9375, 16.625, 17.25, 17.0, 16.125, 19.375, 14.25, 15.6875, 19.375, 14.0625, 14.625, 14.875, 11.8125, 12.25], "probabilities": [0.003838765434920788, 0.0024784926790744066, 0.004929072689265013, 0.026646310463547707, 0.049781862646341324, 0.0387701541185379, 0.016161805018782616, 0.41681843996047974, 0.0024784926790744066, 0.010434846393764019, 0.41681843996047974, 0.0020547432359308004, 0.0036061860155314207, 0.004630435258150101, 0.00021656831086147577, 0.0003354274667799473], "choice": "candidate-e76caeae9d53c8942ed1", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.005025841295719147}
|
| 17 |
+
{"panel_index": 16, "task_family": "hs3", "logits": [17.875, 20.25], "probabilities": [0.08509904146194458, 0.9149009585380554], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 18 |
+
{"panel_index": 17, "task_family": "hs3", "logits": [17.5, 20.375], "probabilities": [0.0534033365547657, 0.9465966820716858], "choice": "no", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.014143407344818115}
|
| 19 |
+
{"panel_index": 18, "task_family": "hs3", "logits": [20.875, 17.75], "probabilities": [0.957912266254425, 0.04208773002028465], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.004760846495628357}
|
| 20 |
+
{"panel_index": 19, "task_family": "hs3", "logits": [21.125, 14.625], "probabilities": [0.998498797416687, 0.0015011822106316686], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 9.666336700320244e-05}
|
| 21 |
+
{"panel_index": 20, "task_family": "hs3", "logits": [21.0, 14.5], "probabilities": [0.998498797416687, 0.0015011822106316686], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 9.082374162971973e-05}
|
| 22 |
+
{"panel_index": 21, "task_family": "hs3", "logits": [14.5, 21.5], "probabilities": [0.0009110510000027716, 0.9990890026092529], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 23 |
+
{"panel_index": 22, "task_family": "hs3", "logits": [21.25, 14.3125], "probabilities": [0.9990302324295044, 0.0009697520872578025], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 6.247893907129765e-05}
|
| 24 |
+
{"panel_index": 23, "task_family": "hs3", "logits": [18.25, 20.0], "probabilities": [0.14804719388484955, 0.851952850818634], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 25 |
+
{"panel_index": 24, "task_family": "hs3", "logits": [21.25, 14.8125], "probabilities": [0.9984021782875061, 0.0015978460432961583], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.656612873077393e-10}
|
| 26 |
+
{"panel_index": 25, "task_family": "hs3", "logits": [15.75, 21.5], "probabilities": [0.0031726835295557976, 0.9968273043632507], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00037175649777054787}
|
| 27 |
+
{"panel_index": 26, "task_family": "hs3", "logits": [19.625, 19.375], "probabilities": [0.562176525592804, 0.4378235340118408], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.062176525592803955}
|
| 28 |
+
{"panel_index": 27, "task_family": "hs3", "logits": [21.125, 15.3125], "probabilities": [0.9970190525054932, 0.002981032943353057], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 29 |
+
{"panel_index": 28, "task_family": "hs3", "logits": [19.75, 19.75], "probabilities": [0.5, 0.5], "choice": "yes", "correct": true, "agrees_with_reference": false, "max_abs_probability_delta": 0.03120940923690796}
|
| 30 |
+
{"panel_index": 29, "task_family": "hs3", "logits": [14.1875, 21.125], "probabilities": [0.0009697520872578025, 0.9990302324295044], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.7462298274040222e-10}
|
| 31 |
+
{"panel_index": 30, "task_family": "hs3", "logits": [20.625, 17.5], "probabilities": [0.957912266254425, 0.04208773002028465], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.005338143557310104}
|
| 32 |
+
{"panel_index": 31, "task_family": "hs3", "logits": [21.25, 15.25], "probabilities": [0.9975274205207825, 0.0024726237170398235], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001494884490966797}
|
| 33 |
+
{"panel_index": 32, "task_family": "boolq", "logits": [16.375, 20.875], "probabilities": [0.010986944660544395, 0.9890130758285522], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 9.313225746154785e-10}
|
| 34 |
+
{"panel_index": 33, "task_family": "boolq", "logits": [14.125, 21.0], "probabilities": [0.0010322309099137783, 0.9989677667617798], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 6.651878356933594e-05}
|
| 35 |
+
{"panel_index": 34, "task_family": "boolq", "logits": [14.625, 20.875], "probabilities": [0.0019267346942797303, 0.9980732202529907], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 36 |
+
{"panel_index": 35, "task_family": "boolq", "logits": [21.0, 15.3125], "probabilities": [0.9966233968734741, 0.003376612439751625], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0002039293758571148}
|
| 37 |
+
{"panel_index": 36, "task_family": "boolq", "logits": [21.125, 14.8125], "probabilities": [0.9981897473335266, 0.0018102104077115655], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00010955333709716797}
|
| 38 |
+
{"panel_index": 37, "task_family": "boolq", "logits": [21.0, 13.4375], "probabilities": [0.999480664730072, 0.0005193048273213208], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.349781036376953e-05}
|
| 39 |
+
{"panel_index": 38, "task_family": "boolq", "logits": [20.875, 12.875], "probabilities": [0.9996646642684937, 0.0003353501670062542], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.9103830456733704e-11}
|
| 40 |
+
{"panel_index": 39, "task_family": "boolq", "logits": [14.875, 20.75], "probabilities": [0.0028009277302771807, 0.9971990585327148], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00018010521307587624}
|
| 41 |
+
{"panel_index": 40, "task_family": "boolq", "logits": [20.875, 13.625], "probabilities": [0.9992903470993042, 0.0007096706540323794], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.57763671875e-05}
|
| 42 |
+
{"panel_index": 41, "task_family": "boolq", "logits": [14.1875, 20.875], "probabilities": [0.0012448435882106423, 0.9987550973892212], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00025633862242102623}
|
| 43 |
+
{"panel_index": 42, "task_family": "boolq", "logits": [21.125, 13.625], "probabilities": [0.9994471669197083, 0.0005527787143364549], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.1641532182693481e-10}
|
| 44 |
+
{"panel_index": 43, "task_family": "boolq", "logits": [14.0625, 21.0], "probabilities": [0.0009697520872578025, 0.9990302324295044], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 6.247893907129765e-05}
|
| 45 |
+
{"panel_index": 44, "task_family": "boolq", "logits": [20.875, 13.875], "probabilities": [0.9990890026092529, 0.0009110510000027716], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.8770179748535156e-05}
|
| 46 |
+
{"panel_index": 45, "task_family": "boolq", "logits": [21.125, 13.6875], "probabilities": [0.9994115829467773, 0.0005884087295271456], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.563013160601258e-05}
|
| 47 |
+
{"panel_index": 46, "task_family": "boolq", "logits": [15.3125, 21.0], "probabilities": [0.003376612439751625, 0.9966233968734741], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00021702051162719727}
|
| 48 |
+
{"panel_index": 47, "task_family": "boolq", "logits": [15.1875, 21.0], "probabilities": [0.002981032943353057, 0.9970190525054932], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00019174814224243164}
|
| 49 |
+
{"panel_index": 48, "task_family": "mnli", "logits": [14.375, 21.375, 14.0625], "probabilities": [0.0009104442433454096, 0.9984235763549805, 0.0006660954095423222], "choice": "neutral", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.00014722347259521484}
|
| 50 |
+
{"panel_index": 49, "task_family": "mnli", "logits": [14.6875, 21.25, 13.0], "probabilities": [0.0014099909458309412, 0.9983291029930115, 0.00026082200929522514], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00010752677917480469}
|
| 51 |
+
{"panel_index": 50, "task_family": "mnli", "logits": [21.0, 12.3125, 11.3125], "probabilities": [0.9997692704200745, 0.00016864229110069573, 6.204004603205249e-05], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.1827872842550278e-11}
|
| 52 |
+
{"panel_index": 51, "task_family": "mnli", "logits": [14.0, 21.25, 12.625], "probabilities": [0.0007095433538779616, 0.9991111159324646, 0.00017940056568477303], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.1076775738038123e-05}
|
| 53 |
+
{"panel_index": 52, "task_family": "mnli", "logits": [13.4375, 13.1875, 20.375], "probabilities": [0.0009690203005447984, 0.0007546736742369831, 0.9982763528823853], "choice": "contradiction", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00011104345321655273}
|
| 54 |
+
{"panel_index": 53, "task_family": "mnli", "logits": [21.125, 12.5625, 12.0], "probabilities": [0.9997000694274902, 0.00019108352717012167, 0.00010887614189414307], "choice": "entailment", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 7.033348083496094e-06}
|
| 55 |
+
{"panel_index": 54, "task_family": "mnli", "logits": [15.8125, 21.125, 13.1875], "probabilities": [0.004903663881123066, 0.9947410821914673, 0.00035522005055099726], "choice": "neutral", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0002957293763756752}
|
| 56 |
+
{"panel_index": 55, "task_family": "mnli", "logits": [21.375, 13.6875, 12.25], "probabilities": [0.9994328618049622, 0.00045826294808648527, 0.00010884703806368634], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.392862319946289e-05}
|
| 57 |
+
{"panel_index": 56, "task_family": "mnli", "logits": [21.0, 12.5625, 11.8125], "probabilities": [0.9996812343597412, 0.0002165219484595582, 0.00010227769962511957], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.74913215637207e-05}
|
| 58 |
+
{"panel_index": 57, "task_family": "mnli", "logits": [14.6875, 21.5, 14.9375], "probabilities": [0.0010971833253279328, 0.9974939823150635, 0.0014088114257901907], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005152225494384766}
|
| 59 |
+
{"panel_index": 58, "task_family": "mnli", "logits": [12.75, 21.0, 12.4375], "probabilities": [0.0002611404052004218, 0.9995477795600891, 0.00019105442333966494], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 1.1573341907933354e-05}
|
| 60 |
+
{"panel_index": 59, "task_family": "mnli", "logits": [21.375, 15.4375, 14.75], "probabilities": [0.9960502982139587, 0.002628196030855179, 0.0013215398648753762], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0002537369728088379}
|
| 61 |
+
{"panel_index": 60, "task_family": "mnli", "logits": [14.6875, 21.0, 16.5], "probabilities": [0.0017903574043884873, 0.9872423410415649, 0.010967273265123367], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0050147175788879395}
|
| 62 |
+
{"panel_index": 61, "task_family": "mnli", "logits": [20.125, 20.0, 14.375], "probabilities": [0.5303127765655518, 0.4679993689060211, 0.0016878697788342834], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.03115832805633545}
|
| 63 |
+
{"panel_index": 62, "task_family": "mnli", "logits": [21.25, 13.5, 12.6875], "probabilities": [0.9993785619735718, 0.00043047493090853095, 0.00019102207443211228], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.820766091346741e-11}
|
| 64 |
+
{"panel_index": 63, "task_family": "mnli", "logits": [21.0, 12.0, 11.375], "probabilities": [0.9998106360435486, 0.00012338647502474487, 6.604400550713763e-05], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.0040740966796875e-05}
|
| 65 |
+
{"panel_index": 64, "task_family": "score", "logits": [15.5625, 21.125], "probabilities": [0.003824483836069703, 0.9961755275726318], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0002456542570143938}
|
| 66 |
+
{"panel_index": 65, "task_family": "score", "logits": [14.5625, 20.875], "probabilities": [0.0018102104077115655, 0.9981897473335266], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0002124309539794922}
|
| 67 |
+
{"panel_index": 66, "task_family": "score", "logits": [13.4375, 20.875], "probabilities": [0.0005884087295271456, 0.9994115829467773], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.792543429881334e-05}
|
| 68 |
+
{"panel_index": 67, "task_family": "score", "logits": [21.125, 16.75], "probabilities": [0.9875683188438416, 0.012431648559868336], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0016319779679179192}
|
| 69 |
+
{"panel_index": 68, "task_family": "score", "logits": [13.9375, 20.875], "probabilities": [0.0009697520872578025, 0.9990302324295044], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001289844512939453}
|
| 70 |
+
{"panel_index": 69, "task_family": "score", "logits": [21.25, 14.1875], "probabilities": [0.9991440773010254, 0.000855900754686445], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.000100495177321136}
|
| 71 |
+
{"panel_index": 70, "task_family": "score", "logits": [15.375, 21.0], "probabilities": [0.003593603614717722, 0.9964063763618469], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00021702051162719727}
|
| 72 |
+
{"panel_index": 71, "task_family": "score", "logits": [13.8125, 21.125], "probabilities": [0.0006667023408226669, 0.9993333220481873], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.297494888305664e-05}
|
| 73 |
+
{"panel_index": 72, "task_family": "score", "logits": [21.125, 15.0625], "probabilities": [0.9976769089698792, 0.0023231622762978077], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001404457725584507}
|
| 74 |
+
{"panel_index": 73, "task_family": "score", "logits": [13.8125, 21.0], "probabilities": [0.0007554056355729699, 0.9992445707321167], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00010049500269815326}
|
| 75 |
+
{"panel_index": 74, "task_family": "score", "logits": [14.0, 21.0], "probabilities": [0.0009110510000027716, 0.9990890026092529], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 76 |
+
{"panel_index": 75, "task_family": "score", "logits": [14.1875, 21.125], "probabilities": [0.0009697520872578025, 0.9990302324295044], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001289844512939453}
|
| 77 |
+
{"panel_index": 76, "task_family": "score", "logits": [15.4375, 21.125], "probabilities": [0.003376612439751625, 0.9966233968734741], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0}
|
| 78 |
+
{"panel_index": 77, "task_family": "score", "logits": [14.9375, 21.125], "probabilities": [0.002050743205472827, 0.9979493021965027], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 79 |
+
{"panel_index": 78, "task_family": "score", "logits": [15.125, 21.25], "probabilities": [0.0021827167365700006, 0.9978172779083252], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.3283064365386963e-10}
|
| 80 |
+
{"panel_index": 79, "task_family": "score", "logits": [15.75, 21.0], "probabilities": [0.005220126826316118, 0.9947799444198608], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00033479928970336914}
|
evaluation/per-question/reference.jsonl
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"panel_index": 0, "id": "mind2web:bef473f1-82a1-4359-a2c0-59b6dc2f6abb:22ad3562-e0f4-42c3-b096-8c173a47673c", "task_family": "browser", "gold": "candidate-c59cf6b3cdd1ee8dfe92", "candidate_ids": ["candidate-f1094edca3d95c3ab877", "candidate-365bc85de0d8df852b74", "candidate-528c0da02e102bf4290b", "candidate-68a3dbfbe50374b33385", "candidate-c59cf6b3cdd1ee8dfe92", "candidate-6a5d2547e9b7961af9f2", "candidate-a44d590fb75558028c8a", "candidate-519332b584cbfc17ffca", "candidate-beb9d47a1825f26d1023", "candidate-d27c1c551c69db04ab36", "candidate-ed6c918d8bc2facd9fb5", "candidate-7111d2a0c81d7ac049be", "candidate-7dae2a8703bdc9f5d043", "candidate-7d02e4befb688b244764", "candidate-7a89b1b7a87361ea291e", "candidate-089cbe46c0766ba6701e"], "prompt_sha256": "b3fd1770027f0a5409a9d04f32bde5d632ef802787a60f4932e72ecc89167216", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [13.875, 13.75, 12.625, 12.6875, 21.25, 12.875, 11.3125, 11.6875, 12.5625, 10.4375, 10.875, 11.125, 10.5, 11.375, 10.9375, 10.625], "probabilities": [0.0006252946332097054, 0.0005518206162378192, 0.00017914992349687964, 0.00019070410053245723, 0.9977151155471802, 0.00023003305250313133, 4.821755283046514e-05, 7.01561220921576e-05, 0.00016829578089527786, 2.0100065739825368e-05, 3.1131588912103325e-05, 3.997374733444303e-05, 2.13964085560292e-05, 5.132731530466117e-05, 3.313940396765247e-05, 2.4245306121883914e-05], "choice": "candidate-c59cf6b3cdd1ee8dfe92"}
|
| 2 |
+
{"panel_index": 1, "id": "mind2web:5f9182dc-d35d-4c0e-9abe-cd913c136528:8450e2c6-f8aa-40bc-876e-21cf29a8cb77", "task_family": "browser", "gold": "candidate-d2449a13158601200ec0", "candidate_ids": ["candidate-0ac56ef6c8e3a6864ae3", "candidate-33ff2e219977f0035115", "candidate-ae85d05843eef166dc37", "candidate-1491fc48d5480406272d", "candidate-458964bfea8f39410a8c", "candidate-6a3568401638b21de4c7", "candidate-5554637f601e90894848", "candidate-fb424fe48f87098b50ab", "candidate-d2449a13158601200ec0", "candidate-d26fca5ac7bd53d243e4", "candidate-59a124a5de2d7629d231", "candidate-37c56ce11f36b7ace139", "candidate-bfd1999ae1c4a3ad89f3", "candidate-77988599c59cbb77c809", "candidate-eb1b6dd49112ec23d746", "candidate-a8a56f7aec962e7413b8"], "prompt_sha256": "9aca18e5ab33a35ca47a65fe691068ba467052dea071279b1751dbfdc95fb224", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [12.4375, 12.1875, 11.9375, 12.375, 11.75, 11.9375, 11.75, 11.75, 20.75, 17.375, 12.5625, 12.5625, 11.375, 12.0, 11.6875, 10.8125], "probabilities": [0.00023678313300479203, 0.00018440686108078808, 0.0001436162128811702, 0.0002224371419288218, 0.00011906202416867018, 0.0001436162128811702, 0.00011906202416867018, 0.00011906202416867018, 0.9647696018218994, 0.033012598752975464, 0.0002683104248717427, 0.0002683104248717427, 8.183005411410704e-05, 0.00015287866699509323, 0.00011184842878719792, 4.662536230171099e-05], "choice": "candidate-d2449a13158601200ec0"}
|
| 3 |
+
{"panel_index": 2, "id": "mind2web:0b59dd33-7f6a-48df-aa1e-9cc67177287f:b99c5581-8a56-4bd7-bbe4-782795ebf93c", "task_family": "browser", "gold": "candidate-9409b72a36db8d4a0a79", "candidate_ids": ["candidate-0120149ac9a61a819265", "candidate-4312931ba215cd0b2310", "candidate-dd54dfdd68be2163e2b6", "candidate-d8ba462bbea45dc4eb77", "candidate-cb3001c9ecbf0dfd250c", "candidate-9409b72a36db8d4a0a79", "candidate-f6ba1cc847604be6bc73", "candidate-2f2099d2f07ae88921ce", "candidate-77f1025025b0e2d0d1fd", "candidate-e743f54599422c4ee674", "candidate-cd31d3ac291210a0a5e6", "candidate-991d9567f58fb994a717", "candidate-d65d92a915149075d6ad", "candidate-98c01750f337ab587383", "candidate-3939d47e8ecda2b48209", "candidate-a7c78ea57e9f0bf1b5b6"], "prompt_sha256": "d960c318e8c88d1c79f12958cf002b9e2b0b9f76a03321dac1dd1d86d3d1c031", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.25, 13.625, 14.0, 14.125, 15.3125, 21.125, 13.0, 13.375, 12.5625, 11.3125, 12.4375, 12.1875, 12.6875, 12.4375, 11.9375, 12.0625], "probabilities": [0.0010249248007312417, 0.00054860272211954, 0.0007982123061083257, 0.0009044930338859558, 0.0029657185077667236, 0.9918970465660095, 0.00029364589136093855, 0.00042725223465822637, 0.0001895920286187902, 5.4319025366567075e-05, 0.00016731438518036157, 0.0001303045719396323, 0.0002148359053535387, 0.00016731438518036157, 0.0001014813024085015, 0.00011499338870635256], "choice": "candidate-9409b72a36db8d4a0a79"}
|
| 4 |
+
{"panel_index": 3, "id": "mind2web:274571ea-fc2f-4353-86ba-00ecb112d6d2:e6345ea9-a5a4-4b88-95b5-4efececed261", "task_family": "browser", "gold": "candidate-3b25ae8d064a298ef021", "candidate_ids": ["candidate-6c861878e1df45af7cb0", "candidate-5dc0c98aa1601abb7255", "candidate-df37b5f34129ec35c5e2", "candidate-7fc3aa839f6eddc4f5e0", "candidate-a105394e5c10cc41573b", "candidate-99243ece7e7fc3575645", "candidate-3b25ae8d064a298ef021", "candidate-c9f0a04f58f5031fe4e6", "candidate-b039454179929264efa9", "candidate-5dc632768de9b7f12a36", "candidate-0b63de0d609d3e1730f1", "candidate-ed1ed5d91fa057eb68be", "candidate-9827a0d3bbbb4c3f2ef3", "candidate-720f59bcd44dc13ecfd0", "candidate-520f72cca9eaf711108b", "candidate-d45432c0ee25d5e74fe8"], "prompt_sha256": "be5d6ca2e7f4e736ac3a410e66cdb9fb2dce3de6f3cb2d58e0425d11bd3e5264", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [16.375, 13.875, 14.375, 14.125, 15.6875, 15.5, 21.375, 13.8125, 13.9375, 12.75, 15.875, 12.75, 13.0625, 17.75, 13.75, 13.625], "probabilities": [0.006426410283893347, 0.000527511874679476, 0.000869720068294555, 0.0006773386849090457, 0.0032314022537320852, 0.002678926568478346, 0.9537639021873474, 0.0004955515614710748, 0.0005615334957838058, 0.00017125802696682513, 0.0038978150114417076, 0.00017125802696682513, 0.00023408200650010258, 0.025416946038603783, 0.00046552764251828194, 0.0004108266730327159], "choice": "candidate-3b25ae8d064a298ef021"}
|
| 5 |
+
{"panel_index": 4, "id": "mind2web:d637c171-dc6e-4a4e-a162-9c230e822932:9053f0e2-da05-4721-87b5-13edf923052b", "task_family": "browser", "gold": "candidate-34e89f28b090ce5327b0", "candidate_ids": ["candidate-34e89f28b090ce5327b0", "candidate-08b4435b3120f48b358c", "candidate-55a0ff963ae8b79b2ac8", "candidate-f95a96c1f7da545920f5", "candidate-98f2cb734f98ff10ff0a", "candidate-d82967df7ccb99323f3d", "candidate-244abe86bc95c904fd2d", "candidate-c71ff755e4fd0c874dfe", "candidate-8d25c242ac6c2febeb64", "candidate-ad36e87635c46e6e4d49", "candidate-88d796f046707b090dc5", "candidate-c16587c8a38af93f0938", "candidate-a96e4a183f184af811a8", "candidate-6d24b42f26f361f2c6c0", "candidate-227d6d1ece1c8d3e341b", "candidate-7a2474fe7e75e66c0d24"], "prompt_sha256": "e3450db0dcc97dbea2e0858fda6b8120ec2573e8c374f5f799c7ed8142a80f80", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [21.0, 13.8125, 11.875, 14.1875, 11.1875, 10.625, 9.25, 9.6875, 10.4375, 8.4375, 9.8125, 10.1875, 10.1875, 10.8125, 11.8125, 10.1875], "probabilities": [0.9976909160614014, 0.0007542310631833971, 0.00010865729564102367, 0.001097399741411209, 5.4636315326206386e-05, 3.113083585049026e-05, 7.871107300161384e-06, 1.2191009773232508e-05, 2.580836917331908e-05, 3.4927829801745247e-06, 1.381422407575883e-05, 2.0099578250665218e-05, 2.0099578250665218e-05, 3.7550951674347743e-05, 0.00010207407467532903, 2.0099578250665218e-05], "choice": "candidate-34e89f28b090ce5327b0"}
|
| 6 |
+
{"panel_index": 5, "id": "mind2web:1efdcd9d-ebc6-4bb7-8823-e54dfe25f409:f6e611c9-ad21-49ca-a841-7ad529b56c95", "task_family": "browser", "gold": "candidate-fb3025ec54721504f9b5", "candidate_ids": ["candidate-fb3025ec54721504f9b5", "candidate-cfc771a1bb7562ebe782", "candidate-68c510f13da5956437bc", "candidate-ac1438ee133b5015c383", "candidate-58473582dc6c65ae4b7e", "candidate-30cbf6942451aa0f1470", "candidate-7ec55a659d963a2cc94d", "candidate-49df5ddbae6f1748c0de", "candidate-b5be7aabde3c609cc40f", "candidate-0e3fd6a816e692376632", "candidate-a3e570f7c516b4577418", "candidate-8a6b58c9cb19ccc7e1fa", "candidate-d8437c7bdf7236744ac4", "candidate-cbf6c1cc621042e7aff0", "candidate-b6291f89245ae394921e", "candidate-c5a94f8fac1dcdad627a"], "prompt_sha256": "36792836eaeed2b15b8a762fc57a57b4b104a27be92bff8c84681bb90d68476b", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [21.0, 12.3125, 13.1875, 12.25, 15.125, 11.6875, 10.1875, 11.625, 11.3125, 9.1875, 9.875, 10.6875, 10.375, 14.5, 14.6875, 10.75], "probabilities": [0.992821991443634, 0.00016747041081544012, 0.00040174060268327594, 0.00015732388419564813, 0.0027886328753083944, 8.964045264292508e-05, 2.0001487428089604e-05, 8.420940139330924e-05, 6.160891643958166e-05, 7.3581363722041715e-06, 1.4633399587182794e-05, 3.297687726444565e-05, 2.412639878457412e-05, 0.0014926474541425705, 0.0018004764569923282, 3.510370515868999e-05], "choice": "candidate-fb3025ec54721504f9b5"}
|
| 7 |
+
{"panel_index": 6, "id": "mind2web:d1de3d1a-3df1-4421-98f3-f8d078752893:e1f18ee3-1577-44fb-a283-1be215e5ae52", "task_family": "browser", "gold": "candidate-888bb5997d1aba7b25f7", "candidate_ids": ["candidate-8d8171f50795c09388a0", "candidate-8d26ef6bd8e9f762c73d", "candidate-bb2ff00206221aa2b02d", "candidate-0196e00c3795624bcc41", "candidate-a9f93c25b378cb9448f1", "candidate-0b7a0a178194ef69867b", "candidate-880053bf9e617437e7cb", "candidate-cf42d1a8a48eeea86134", "candidate-67a5886245e859456378", "candidate-b7c7a409d7192fcbe70f", "candidate-888bb5997d1aba7b25f7", "candidate-dd07004fe07f84bf771d", "candidate-a8b562a5018f0346093a", "candidate-ba9bcd4bb262d6387c1e", "candidate-d4dbe03106eb96d6c9dc", "candidate-b3ddfc4f75cd756d4767"], "prompt_sha256": "873952066cc2833adc3b2b3af9b8f8261f195c13cd41118b830744f22683e25f", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.375, 13.9375, 14.8125, 14.4375, 14.125, 13.9375, 15.8125, 13.8125, 15.3125, 15.8125, 21.25, 13.125, 13.4375, 14.75, 12.625, 15.875], "probabilities": [0.0010082769440487027, 0.0006509925005957484, 0.001561649958603084, 0.0010733051458373666, 0.0007852468406781554, 0.0006509925005957484, 0.004245004616677761, 0.0005744988447986543, 0.0025747253093868494, 0.004245004616677761, 0.9757855534553528, 0.0002888761810027063, 0.0003948469238821417, 0.0014670342206954956, 0.0001752122479956597, 0.004518784116953611], "choice": "candidate-888bb5997d1aba7b25f7"}
|
| 8 |
+
{"panel_index": 7, "id": "mind2web:80e12375-19ad-400f-9e35-2a3853173bed:e89bb795-2d24-4e2c-bcae-1294e3501dfa", "task_family": "browser", "gold": "candidate-41c5af16d7bf373cc74c", "candidate_ids": ["candidate-41c5af16d7bf373cc74c", "candidate-03bd5f2aaa71b912d85f", "candidate-39aafe4d05bcbbf1a9d8", "candidate-7db455c43ca404d644b3", "candidate-8bdbe188e49bad6ea2d9", "candidate-92648d446426dec0e918", "candidate-36e6b14fcf2f5e8e9c23", "candidate-fe113518f51162e48db7", "candidate-f6b0a3559a94799eebe8", "candidate-853b6e262fd75054c736", "candidate-a77385eabd707033f244", "candidate-b7024479880013db2adc", "candidate-b5cc7d0d8e9ea791112e", "candidate-1b8a962d51e9c3897102", "candidate-2b4df4c4f3a6e5c34716", "candidate-c3a3c2242e59afcbb3fb"], "prompt_sha256": "33badf184ca6146904e42f3db06fbd31f551b1ce3082595b5db35494ee245a4b", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [20.875, 12.0625, 11.625, 11.3125, 11.25, 11.0, 9.1875, 9.75, 10.5625, 8.1875, 9.375, 10.0, 11.875, 10.8125, 9.5, 10.125], "probabilities": [0.9992802739143372, 0.0001487534900661558, 9.604248043615371e-05, 7.026617822702974e-05, 6.600896449526772e-05, 5.140783468959853e-05, 8.39209769765148e-06, 1.4728590031154454e-05, 3.319139068480581e-05, 3.087280219915556e-06, 1.0122801541001536e-05, 1.891188367153518e-05, 0.000123320976854302, 4.2618587031029165e-05, 1.1470638128230348e-05, 2.142997254850343e-05], "choice": "candidate-41c5af16d7bf373cc74c"}
|
| 9 |
+
{"panel_index": 8, "id": "mind2web:0245df99-2909-465a-861e-7fbca948e82f:dc1847f7-919b-4a2f-b778-2ee33edacc46", "task_family": "browser", "gold": "candidate-a1a3ad1a8495aa678cbe", "candidate_ids": ["candidate-80e6a190025076e4821d", "candidate-cd39817d36a3b7f8086c", "candidate-a1a3ad1a8495aa678cbe", "candidate-74eaca906c939e439348", "candidate-1c996fce61cea008cd97", "candidate-b86e5504c07b50684120", "candidate-ca5385f6ff0f2b510b2a", "candidate-27ecbf7b79508deea5b5", "candidate-335d71d7afcd7c39e60c", "candidate-3fdeb3ac89f0ecf87cff", "candidate-d594855a68f3d65bda29", "candidate-ff28229b986139c46fad", "candidate-4d0fd233b1f9ec995708", "candidate-491e3298e78304983a55", "candidate-438c3b7b4da6155b0b79", "candidate-66345b239aef37d2f31c"], "prompt_sha256": "19bd775a219dfb7da24c37ac1e5319666e84c1584357f8e31021b2c8e9fe3952", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [13.625, 15.125, 21.25, 14.8125, 13.125, 12.125, 14.375, 14.3125, 13.375, 11.0, 11.0625, 11.375, 11.6875, 12.8125, 11.6875, 11.5625], "probabilities": [0.0004844095092266798, 0.0021709727589040995, 0.9924486875534058, 0.0015883175656199455, 0.00029380922205746174, 0.00010808637307491153, 0.0010254948865622282, 0.0009633632143959403, 0.000377258489606902, 3.509050657157786e-05, 3.7353645893745124e-05, 5.105638410896063e-05, 6.978580495342612e-05, 0.0002149553911294788, 6.978580495342612e-05, 6.15857497905381e-05], "choice": "candidate-a1a3ad1a8495aa678cbe"}
|
| 10 |
+
{"panel_index": 9, "id": "mind2web:8eae88ef-9641-43c6-be6d-f8abc96d99fa:282c09d0-c9e0-4007-b88a-27887fe1e388", "task_family": "browser", "gold": "candidate-976c2731e7518a1f1b2b", "candidate_ids": ["candidate-976c2731e7518a1f1b2b", "candidate-55380cc26f4800be25ce", "candidate-495e2ce778804116afed", "candidate-6a3ce8bdfda5c7e09a24", "candidate-7e020b58dbe771cceb6b", "candidate-ca7f03f7b7139457a468", "candidate-4556165f722234b67398", "candidate-0232a0c57f2c0a004a53", "candidate-892461b7e0e4bf188cd7", "candidate-98a8e1da12737fa9372c", "candidate-f6c802447f1e19395568", "candidate-e95807766432f0e50c72", "candidate-955408afe8906a0bb10d", "candidate-cd1be178e6146e0fd80a", "candidate-38a919ada71ae1df352a", "candidate-38c746333273edc02a2e"], "prompt_sha256": "05bf3a03b2c0d0a936c4e37c690299142bb6148b2eda9be40aa00c6a13a66811", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [20.875, 13.0625, 13.0, 12.25, 12.5625, 15.1875, 11.0625, 11.0, 11.25, 9.5, 10.125, 10.6875, 10.75, 10.9375, 10.125, 12.5625], "probabilities": [0.9948309063911438, 0.00040255350177176297, 0.0003781640261877328, 0.0001786320353858173, 0.0002441610558889806, 0.0033705392852425575, 5.4479693062603474e-05, 5.117893670103513e-05, 6.571505218744278e-05, 1.1419564543757588e-05, 2.1334553821361624e-05, 3.744330751942471e-05, 3.985819057561457e-05, 4.807816367247142e-05, 2.1334553821361624e-05, 0.0002441610558889806], "choice": "candidate-976c2731e7518a1f1b2b"}
|
| 11 |
+
{"panel_index": 10, "id": "mind2web:3f0988e0-e3a4-4dd7-a89f-482175a474a8:a684c98a-c238-4bef-b2ad-f476f07d73f8", "task_family": "browser", "gold": "candidate-016db9380de74cbcab5d", "candidate_ids": ["candidate-3f6330dc6b89ca9febbe", "candidate-0acffd69c24f4be0a17e", "candidate-ae8922f024c12c266278", "candidate-c12d0dfb7e58d7f22bc7", "candidate-86f75c1fc5227d6081b8", "candidate-6ff913a423628d92b636", "candidate-87e0dada00c5e19e9dce", "candidate-2c14d16a67e5d74c7a66", "candidate-f6a74c2240fb95cf9da0", "candidate-ac52e1e2c9fab34583a1", "candidate-0c2b330dc3030639c697", "candidate-016db9380de74cbcab5d", "candidate-cab20f7657710f02c34a", "candidate-d48a5d89e157a64dd7b4", "candidate-b4b12517818528229992", "candidate-09c561e352fda884d806"], "prompt_sha256": "8c9d16ca40b336c54eab472ed9a7d490065e655ff833efc53466acbdaf85024c", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.1875, 18.125, 14.0, 14.0625, 16.625, 13.8125, 12.875, 13.5625, 14.75, 12.1875, 13.1875, 20.375, 13.75, 16.5, 11.4375, 12.5625], "probabilities": [0.0017641966696828604, 0.09048602730035782, 0.0014625706244260073, 0.0015568981179967523, 0.0201901625841856, 0.0012125135399401188, 0.0004748271021526307, 0.0009443064336664975, 0.0030962617602199316, 0.0002387580752838403, 0.000649011752102524, 0.8585075736045837, 0.0011390510480850935, 0.017817756161093712, 0.00011278133024461567, 0.00034739094553515315], "choice": "candidate-016db9380de74cbcab5d"}
|
| 12 |
+
{"panel_index": 11, "id": "mind2web:5b56d5b8-1f41-43ca-9f21-369d849f1aa0:edb09eb7-6a8c-4aeb-9b52-796762ca821d", "task_family": "browser", "gold": "candidate-2dc16f49fe3acb051b39", "candidate_ids": ["candidate-021004f4c9756e9f14bf", "candidate-9393f92278ae33837cfc", "candidate-d53f82cb73ef67bc0ce8", "candidate-332f924943c28c247430", "candidate-afd6b2a84985b30c66cd", "candidate-21f93b804332aae201c9", "candidate-bdba9a0649055da71f94", "candidate-0cf8d953323f4b29b10c", "candidate-dea9b4fc8a207df523df", "candidate-fa70b5bc02a04fa0a18a", "candidate-1932fde20b275a9b3081", "candidate-5c9c9ac05919970737c1", "candidate-544fea69d385caac3601", "candidate-2dc16f49fe3acb051b39", "candidate-ca60dc8da2fba3f06111", "candidate-ea45663120f48fa54b88"], "prompt_sha256": "fc3a0ccb7f4fba5d2ba07c7ec21def24cb6fe447d1c9b124d867a6654c6c09b4", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [15.6875, 13.125, 15.0, 13.5625, 13.8125, 12.625, 12.6875, 13.125, 13.3125, 16.25, 12.0625, 13.1875, 18.375, 20.625, 11.9375, 11.875], "probabilities": [0.006323311477899551, 0.00048760147183202207, 0.0031795611139386892, 0.0007552119204774499, 0.0009697112836875021, 0.0002957452379632741, 0.00031481916084885597, 0.00048760147183202207, 0.0005881596589460969, 0.011097758077085018, 0.00016851055261213332, 0.000519049062859267, 0.09292039275169373, 0.8816041350364685, 0.00014871005259919912, 0.00013970016152597964], "choice": "candidate-2dc16f49fe3acb051b39"}
|
| 13 |
+
{"panel_index": 12, "id": "mind2web:6e565708-43e2-492b-9f1d-25d51387dcf7:bee75aa3-6f7d-4626-be6f-1b217ac16733", "task_family": "browser", "gold": "candidate-bb1945d8d438379625ee", "candidate_ids": ["candidate-69abfa80e562043e1fa3", "candidate-c4afb9a40ecda9a03c50", "candidate-ad4a56ee767e7be58ceb", "candidate-16c8c33fbf8679f935d6", "candidate-777d646249275ea3bc88", "candidate-ce64cb6d800f5f6f7886", "candidate-cb8d5f96e26185552cf9", "candidate-8203c66243ac00ab8c67", "candidate-32a97c97039dcc468732", "candidate-6fbd3f273e3c48d30d0c", "candidate-97141ccdf3d685ac91be", "candidate-d5d5ef09f20a3d2ceade", "candidate-04470be9c3bbc28919e6", "candidate-0fb9a6cde602f212d85e", "candidate-4ca405ae57be628d94ee", "candidate-bb1945d8d438379625ee"], "prompt_sha256": "a7f850a48b635b1a855ab90b47855fd8269980f52d324bdf635009f9344a5fa4", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [12.625, 11.5, 13.1875, 12.875, 11.75, 12.1875, 11.6875, 12.0, 12.5625, 10.625, 10.5, 11.1875, 12.3125, 20.75, 11.3125, 18.75], "probabilities": [0.00026015937328338623, 8.446137508144602e-05, 0.0004565939016174525, 0.0003340512339491397, 0.00010845055658137426, 0.0001679714914644137, 0.00010187987209064886, 0.0001392532722093165, 0.00024439708795398474, 3.5208737244829535e-05, 3.1071605917532e-05, 6.179326737765223e-05, 0.0001903366355691105, 0.8787839412689209, 7.002093479968607e-05, 0.11893047392368317], "choice": "candidate-0fb9a6cde602f212d85e"}
|
| 14 |
+
{"panel_index": 13, "id": "mind2web:44a12ff5-0172-444a-b979-f224162c1aa8:5b8da6f5-c53c-4b69-bfad-7bdfd2e6ce20", "task_family": "browser", "gold": "candidate-3ef3bb624316b6eb3508", "candidate_ids": ["candidate-3ef3bb624316b6eb3508", "candidate-5b52b8bc7f6397531f34", "candidate-6a5a6a33fbc65504059f", "candidate-bde4765810f968ad4833", "candidate-1f66a27bf372d26b14b3", "candidate-8033052a6976a4b01205", "candidate-45ec5aa3dbee10fc9a75", "candidate-ea5b9b527b94b21e6535", "candidate-56683febea1548c570fd", "candidate-bd10584aedf8afa78746", "candidate-94dddd06eb43ed2921b0", "candidate-c9197b3d3e5e97a0f6ff", "candidate-331b8cbdf48fd7f61188", "candidate-c8041a4dd7e8b00e705c", "candidate-f5c1b8b97d7add30b043", "candidate-7161a3b5bba5920fcb33"], "prompt_sha256": "c8d922fd930198c507227984857f26500162ceac030c038e92206981e9747cc3", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.875, 14.25, 15.5625, 14.4375, 13.3125, 15.0, 18.25, 13.8125, 13.4375, 15.1875, 18.125, 13.0625, 13.6875, 15.1875, 19.375, 13.25], "probabilities": [0.006471399683505297, 0.00346389040350914, 0.012869914062321186, 0.004178249277174473, 0.0013564788969233632, 0.007333056069910526, 0.1891220062971115, 0.002236455911770463, 0.0015370920300483704, 0.008845353499054909, 0.16689959168434143, 0.001056426903232932, 0.0019736653193831444, 0.008845353499054909, 0.5825368165969849, 0.0012742941034957767], "choice": "candidate-f5c1b8b97d7add30b043"}
|
| 15 |
+
{"panel_index": 14, "id": "mind2web:caafd610-202e-49d2-85d1-3f167f3ab443:d7bfb473-8c73-4808-96bc-187d00be5ad7", "task_family": "browser", "gold": "candidate-b56ad12ceea4c52feb3f", "candidate_ids": ["candidate-a78c74bd163f786d9539", "candidate-b56ad12ceea4c52feb3f", "candidate-b4ae8d879b43b183d54a", "candidate-9ebb992dd5a329ac8366", "candidate-f2c9f9f313e700cc877d", "candidate-af82db0cdef64a4a6cf2", "candidate-5fdd288dfbfc0a932f78", "candidate-f0729506779d73c72c2f", "candidate-4ffcfa3737d12e4f223c", "candidate-efc032a84043e09d128a", "candidate-4a068ffbc0d398300aaa", "candidate-7f042d5315889e8f72e0", "candidate-137b03b260a683ac0cd6", "candidate-874b3d025779abce748c", "candidate-5de2a5273572d4bc9c39", "candidate-49b9b04e59bbfb197a63"], "prompt_sha256": "d94ca8165f95e231260aacbaa5bc8e99ca44241878d872928f7c96298839ae96", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [12.5, 21.625, 12.0625, 12.625, 11.625, 11.5, 10.1875, 11.125, 11.4375, 9.625, 10.375, 12.3125, 12.3125, 13.3125, 13.6875, 14.9375], "probabilities": [0.00010863579518627375, 0.9974935054779053, 7.014053699094802e-05, 0.00012310047168284655, 4.528613499132916e-05, 3.996487066615373e-05, 1.0756400115496945e-05, 2.746743120951578e-05, 3.754352292162366e-05, 6.1288119468372315e-06, 1.297469498240389e-05, 9.006223262986168e-05, 9.006223262986168e-05, 0.0002448145532980561, 0.00035620303242467344, 0.0012432708172127604], "choice": "candidate-b56ad12ceea4c52feb3f"}
|
| 16 |
+
{"panel_index": 15, "id": "mind2web:8368b990-c6ca-4cfe-a7ab-c2a88697639d:bf14f1d4-470f-4110-b3f4-019a9f7d0aed", "task_family": "browser", "gold": "candidate-8b6280aa9682e1a79ce0", "candidate_ids": ["candidate-cc60994b6c36f4d19840", "candidate-f8c1512ffc92c70eafe7", "candidate-53d2672be9bfe9447865", "candidate-da095abe0a63c03f3114", "candidate-e926e62f231edca6157e", "candidate-3eccf28da059e9811f3b", "candidate-c8e6f0df1300d65debaa", "candidate-e76caeae9d53c8942ed1", "candidate-de3ea77a36017ef9c156", "candidate-1ab38af3801bba39e08c", "candidate-8b6280aa9682e1a79ce0", "candidate-451ffc245026adbc3d68", "candidate-77aac247bed375f821ae", "candidate-274f32a25dc920013c6f", "candidate-e97a589066e18a2db4b7", "candidate-c6136bbe715f050ff989"], "prompt_sha256": "52d0c2eeb247ef1c21d83eaac84a9f88d22eb50e65489a0dfb02c1c86b73c5e7", "candidate_token_ids": [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "logits": [14.625, 14.1875, 14.875, 16.5, 17.25, 17.125, 16.25, 19.375, 14.3125, 15.5625, 19.375, 14.125, 14.6875, 14.9375, 11.875, 12.25], "probabilities": [0.0035949943121522665, 0.002321102889254689, 0.004616064485162497, 0.023442309349775314, 0.04962736740708351, 0.04379599541425705, 0.018256887793540955, 0.41552484035491943, 0.0026301539037376642, 0.009180138818919659, 0.41552484035491943, 0.0021804743446409702, 0.0038268512580543756, 0.004913774318993092, 0.00022982028895057738, 0.0003343865682836622], "choice": "candidate-e76caeae9d53c8942ed1"}
|
| 17 |
+
{"panel_index": 16, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0010788", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "43d8aa81e0401b3731c05f98e2bd2b3dbf59139695ae6b700948ef63b4d14e30", "candidate_token_ids": [32, 33], "logits": [17.875, 20.25], "probabilities": [0.08509904146194458, 0.9149009585380554], "choice": "no"}
|
| 18 |
+
{"panel_index": 17, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0001994", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "06ef7447367570160457d3b2f434528fe7391c356d258e3934f257f80f5c01f5", "candidate_token_ids": [32, 33], "logits": [17.75, 20.375], "probabilities": [0.06754669547080994, 0.9324532747268677], "choice": "no"}
|
| 19 |
+
{"panel_index": 18, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0000105", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "9fea3b00dd4a71a0363c8bd87b99ae4334c242011eba5648c197e999aa174c64", "candidate_token_ids": [32, 33], "logits": [20.875, 17.625], "probabilities": [0.9626730680465698, 0.037326883524656296], "choice": "yes"}
|
| 20 |
+
{"panel_index": 19, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0001238", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "8dace247ec58eab0973f7a1087fe3af7c8695a2992be4d300ba004babaf8cbbe", "candidate_token_ids": [32, 33], "logits": [21.0, 14.5625], "probabilities": [0.9984021782875061, 0.001597845577634871], "choice": "yes"}
|
| 21 |
+
{"panel_index": 20, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0011197", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "b3fa91e805df134cb80d0ae82eabc86c63266d874b7fb6cdd584d222f7cd9426", "candidate_token_ids": [32, 33], "logits": [21.125, 14.5625], "probabilities": [0.9985895752906799, 0.0014103584690019488], "choice": "yes"}
|
| 22 |
+
{"panel_index": 21, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0017850", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "8e283dc3e113374111c25df9df8ed38d3a0991314388f80ace18540c352235d0", "candidate_token_ids": [32, 33], "logits": [14.5, 21.5], "probabilities": [0.0009110512328334153, 0.9990890026092529], "choice": "no"}
|
| 23 |
+
{"panel_index": 22, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0019640", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "475cb8ff904634ace670760f71e582174addfbcf71d28985c0e1c7c1f9b3dbcc", "candidate_token_ids": [32, 33], "logits": [21.125, 14.25], "probabilities": [0.9989677667617798, 0.0010322310263291001], "choice": "yes"}
|
| 24 |
+
{"panel_index": 23, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0010645", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "62ea4bbe1886885ce73d0c1abf72e927fd23117d83868d32d50405bd0a692191", "candidate_token_ids": [32, 33], "logits": [18.25, 20.0], "probabilities": [0.14804719388484955, 0.851952850818634], "choice": "no"}
|
| 25 |
+
{"panel_index": 24, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0020714", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "43596c9cb034207f4c188aa25416d7d290143d2f1850853e180cbee8e1ffbe81", "candidate_token_ids": [32, 33], "logits": [21.25, 14.8125], "probabilities": [0.9984021782875061, 0.001597845577634871], "choice": "yes"}
|
| 26 |
+
{"panel_index": 25, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0008261", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "bc7fe561ed0c1b12e7a21d23aa6bc9a8fff8e502d0ec7328c0c3500042da1225", "candidate_token_ids": [32, 33], "logits": [15.625, 21.5], "probabilities": [0.0028009270317852497, 0.9971990585327148], "choice": "no"}
|
| 27 |
+
{"panel_index": 26, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0015735", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "cba3fb62985350d60d64f9988e06a26577a2e643809593877cd7eddaf1e3e1b9", "candidate_token_ids": [32, 33], "logits": [19.5, 19.5], "probabilities": [0.5, 0.5], "choice": "yes"}
|
| 28 |
+
{"panel_index": 27, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0014712", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "fc5d5293343cdb9a2c4d587a01f6d4b5c2adf98a0ee775d07a1601ae06342dea", "candidate_token_ids": [32, 33], "logits": [21.125, 15.3125], "probabilities": [0.9970190525054932, 0.002981032943353057], "choice": "yes"}
|
| 29 |
+
{"panel_index": 28, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0024669", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "05bc8e81c80ffdc18ed69091c6d4c21decc0b2cd30b5164c34d24f95c17126ff", "candidate_token_ids": [32, 33], "logits": [19.625, 19.75], "probabilities": [0.4687906503677368, 0.531209409236908], "choice": "no"}
|
| 30 |
+
{"panel_index": 29, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0032390", "task_family": "hs3", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "1a9cc68b19fe2ec0158f8bbeab86b096a1496bbb755f023f66f4e3715b53da46", "candidate_token_ids": [32, 33], "logits": [14.1875, 21.125], "probabilities": [0.0009697519126348197, 0.9990302324295044], "choice": "no"}
|
| 31 |
+
{"panel_index": 30, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0018442", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "f68722d712d424bdeddd2531a73b47948597a47e37bd59132b14df56fa98e9b8", "candidate_token_ids": [32, 33], "logits": [20.625, 17.625], "probabilities": [0.9525741338729858, 0.04742587357759476], "choice": "yes"}
|
| 32 |
+
{"panel_index": 31, "id": "hs3-f6d145777bcbde96137596340fab89793acd1031-train-0014165", "task_family": "hs3", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "b2c1d04fc65e345807d5cf3dc93324826502c3131fbb3eca59bf166ef773b131", "candidate_token_ids": [32, 33], "logits": [21.375, 15.3125], "probabilities": [0.9976769089698792, 0.0023231625091284513], "choice": "yes"}
|
| 33 |
+
{"panel_index": 32, "id": "boolq/8703574dc0db27742840afd3c1429829ef2f81c557670018e6ad5456fc53b150", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "938a388a0ea3db943c165f47c21a399dcbc2040c6f61340bd280786a9f5704f3", "candidate_token_ids": [32, 33], "logits": [16.375, 20.875], "probabilities": [0.01098694372922182, 0.9890130758285522], "choice": "no"}
|
| 34 |
+
{"panel_index": 33, "id": "boolq/42a6042685586d3796f9fe4460dd6e9cec208f39de59f765dcbef491fe576361", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "0d97ae562208ecc24ae9df8ab3678c7ae4fbe3aa22b17fffbf97fb6b2cfc2fec", "candidate_token_ids": [32, 33], "logits": [14.1875, 21.0], "probabilities": [0.001098731067031622, 0.9989012479782104], "choice": "no"}
|
| 35 |
+
{"panel_index": 34, "id": "boolq/5ba6d06215b4f7b269d0b30369fea59d748f02f41d1977f14c46c9153cd45cde", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "a3a3f58378880f9c73e1dc63d51ea8d3951c8c90e5b4b593a8c7a1656898903e", "candidate_token_ids": [32, 33], "logits": [14.625, 20.875], "probabilities": [0.0019267346942797303, 0.9980732202529907], "choice": "no"}
|
| 36 |
+
{"panel_index": 35, "id": "boolq/4c460ed013de1b5d82c629bbf4e3bdbdaf4441a81341404d48d8e02ead5b9814", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "e7363021faeb0475c8d4a53d1e355b57cd7c2336433356f455b0dd025084fcdd", "candidate_token_ids": [32, 33], "logits": [21.0, 15.25], "probabilities": [0.9968273043632507, 0.0031726830638945103], "choice": "yes"}
|
| 37 |
+
{"panel_index": 36, "id": "boolq/46e903566fe8817ff37dce61d1ef85b757aa465dadb873c6356b6fc23f58476b", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "c59ba2cb157be77ed4543c9c8e414ce9d572e94a4f7af562105af456ca0993e3", "candidate_token_ids": [32, 33], "logits": [21.125, 14.75], "probabilities": [0.9982993006706238, 0.001700722612440586], "choice": "yes"}
|
| 38 |
+
{"panel_index": 37, "id": "boolq/3f87d9cb171dfe78b389d22677512f92cb48f7eedeefd35f2db1a2444cf08af2", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "f4a7827700e4744e20ca7891fc537579e56467de67e1c9e0218eb74d1f850913", "candidate_token_ids": [32, 33], "logits": [21.0, 13.5], "probabilities": [0.9994471669197083, 0.000552778597921133], "choice": "yes"}
|
| 39 |
+
{"panel_index": 38, "id": "boolq/dbadfab033f2646b1a933fad5e132dc65cd8998c337d0332763f1df07680d7e7", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "349dd93162e0859e82c150bd521f171b1edd2cc2e98f05ed41fb16defcb80cb5", "candidate_token_ids": [32, 33], "logits": [20.875, 12.875], "probabilities": [0.9996646642684937, 0.00033535013790242374], "choice": "yes"}
|
| 40 |
+
{"panel_index": 39, "id": "boolq/6909b3bd9078683a97a30ee70c6f3e91a9b8848a34efc0784eddf12cfa305cd4", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "f5291ad0f6c72b83b6fd627aca39178a0202dd9a6674a6fca53a3a2dd9dfe3f7", "candidate_token_ids": [32, 33], "logits": [14.9375, 20.75], "probabilities": [0.002981032943353057, 0.9970190525054932], "choice": "no"}
|
| 41 |
+
{"panel_index": 40, "id": "boolq/791b5a767a61e438fe080d99068ef294b361fead41dbfc61b61021e33f2fff9f", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "44bb0f449bae5d6f5c4037e0f69fc24b29c530637f5f59fd6436f33027b12bc8", "candidate_token_ids": [32, 33], "logits": [20.875, 13.6875], "probabilities": [0.9992445707321167, 0.000755405577365309], "choice": "yes"}
|
| 42 |
+
{"panel_index": 41, "id": "boolq/cca90e3be9b421447fe9ae0594193ab92a92adb0083f27a46bc224ad0e67c46c", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "1f4adf18c45e5ad59ac100c1e54416032c380206c3420bf39071a4ac2010d52b", "candidate_token_ids": [32, 33], "logits": [14.25, 20.75], "probabilities": [0.0015011822106316686, 0.998498797416687], "choice": "no"}
|
| 43 |
+
{"panel_index": 42, "id": "boolq/179184609853d9f8f1f862bd5c3113e363f622606fcf5c188cf0da1d3ed61e86", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "e88d50a2527287c73f777f23fcb092662ed71ed0c5d7fe0168ec8994e8fb19a0", "candidate_token_ids": [32, 33], "logits": [21.125, 13.625], "probabilities": [0.9994471669197083, 0.000552778597921133], "choice": "yes"}
|
| 44 |
+
{"panel_index": 43, "id": "boolq/58b070021c1d992130604f309a01b95166a281de0499349346694570a6c6ac57", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "8446057016a93c99345544b2afe9eaac75f9ffb0acc99cca9f255e73cb837e34", "candidate_token_ids": [32, 33], "logits": [14.125, 21.0], "probabilities": [0.0010322310263291001, 0.9989677667617798], "choice": "no"}
|
| 45 |
+
{"panel_index": 44, "id": "boolq/11b00c8dcb82dfbdaf02aa139e8c70603fb8c432e0fc51873ca00b20815a90f5", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "ecf3d435d5e2e523fc05ac04d40a235e7808770163f14f11214c899c8738f1b7", "candidate_token_ids": [32, 33], "logits": [20.75, 13.8125], "probabilities": [0.9990302324295044, 0.0009697519126348197], "choice": "yes"}
|
| 46 |
+
{"panel_index": 45, "id": "boolq/155bc4f5246c5b35ec14d7a2f697381de04c2cefbbc293b387bfdf83f2fea269", "task_family": "boolq", "gold": "yes", "candidate_ids": ["yes", "no"], "prompt_sha256": "3c4715c57e5dd569ad4eb1b9d4a725fefaf222f34bd9739091e97cbe8bb6945d", "candidate_token_ids": [32, 33], "logits": [21.125, 13.625], "probabilities": [0.9994471669197083, 0.000552778597921133], "choice": "yes"}
|
| 47 |
+
{"panel_index": 46, "id": "boolq/31fa2124dc5769d9e6fae53be4190290131f8fb025a089f8372d6f5d61504af5", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "9a87a3789d0d031052a1165e22df7f13784f74d7d02fffac973d290d993657f7", "candidate_token_ids": [32, 33], "logits": [15.375, 21.0], "probabilities": [0.0035936026833951473, 0.9964063763618469], "choice": "no"}
|
| 48 |
+
{"panel_index": 47, "id": "boolq/a1ea17311c3aea8a751a1cd170483c439ce1338b451927c7db07692878ff0d82", "task_family": "boolq", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "cebe40ca184a82f8468175b02fdafcbbd829564cf847b3fc0135fd98f67083ea", "candidate_token_ids": [32, 33], "logits": [15.25, 21.0], "probabilities": [0.0031726830638945103, 0.9968273043632507], "choice": "no"}
|
| 49 |
+
{"panel_index": 48, "id": "mnli/ab466125416d6bc3f307f7fd25b78e63a8d90f8f54713027fdc7d75fffd8558d", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "5834f0baf5838f80f70d57a54ea54fdf72bab1fffb7ed8abd7b6b83fec52396e", "candidate_token_ids": [32, 33, 34], "logits": [14.4375, 21.375, 14.1875], "probabilities": [0.0009690201259218156, 0.9982763528823853, 0.0007546736742369831], "choice": "neutral"}
|
| 50 |
+
{"panel_index": 49, "id": "mnli/8f4dbf064e95ffddc782d0d50ab5e6e0fb83aab546d48b00bff5f03706183bf9", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "52a795fb9d86048dfec129052857ec13bfa6dca7b7314081a62143a83055313a", "candidate_token_ids": [32, 33, 34], "logits": [14.75, 21.25, 13.0625], "probabilities": [0.0015007654437795281, 0.9982215762138367, 0.0002776136971078813], "choice": "neutral"}
|
| 51 |
+
{"panel_index": 50, "id": "mnli/53ac0a380403b0ccd3ae48ea7763a52664f1d1744f6244c25d5980d69fcf7f0f", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "f6adf5004089ef9a0cc851a183a3e4c72d4b8a84fd54803ef1789646635f01dd", "candidate_token_ids": [32, 33, 34], "logits": [21.0, 12.3125, 11.3125], "probabilities": [0.9997692704200745, 0.0001686422765487805, 6.204002420417964e-05], "choice": "entailment"}
|
| 52 |
+
{"panel_index": 51, "id": "mnli/e6cf83f056ba7587936d8026ff9709bc50de62e0dfb282f6b8d6aa85649e688d", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "9ddcac65f7c20b39c2eafa8d124294ab44516209a07832ffa4575b35c2ea2396", "candidate_token_ids": [32, 33, 34], "logits": [14.0, 21.25, 12.5], "probabilities": [0.0007095580222085118, 0.9991320967674255, 0.0001583237899467349], "choice": "neutral"}
|
| 53 |
+
{"panel_index": 52, "id": "mnli/834677de1c970abc266e34006f2e681c7301ff5d043796d92da66364b792f235", "task_family": "mnli", "gold": "contradiction", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "c1a73d0727accd049baa352165c01aebbbc68ef10ab728e698ef44f9efbc7b6c", "candidate_token_ids": [32, 33, 34], "logits": [13.5, 13.25, 20.375], "probabilities": [0.0010314017999917269, 0.0008032565820030868, 0.9981653094291687], "choice": "contradiction"}
|
| 54 |
+
{"panel_index": 53, "id": "mnli/13f1e11494916138e91023e5bf4d2483ab5d93f00e2e1e558c5f595ca0e8e22d", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "5aedfe5d0e0bcfe168031cd031b0355c0ed3ce9f22948073417e36f0682394e8", "candidate_token_ids": [32, 33, 34], "logits": [21.125, 12.5625, 12.0625], "probabilities": [0.9996930360794067, 0.0001910821592900902, 0.00011589720088522881], "choice": "entailment"}
|
| 55 |
+
{"panel_index": 54, "id": "mnli/9cf51bc8717b59b904d198884ba5b832b53cdc96a5b1aafee7203dca9bfc824d", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "37b5f12a82b74d56402bb3c33b63e27e4d08218a0ec1ec3a2087df90fa35ea3f", "candidate_token_ids": [32, 33, 34], "logits": [15.75, 21.125, 13.1875], "probabilities": [0.004607934504747391, 0.9950366616249084, 0.0003553256974555552], "choice": "neutral"}
|
| 56 |
+
{"panel_index": 55, "id": "mnli/3d8bd065ebff13c30fa3aa4943c5f2851ca8a5860bae22e25f212be97790ab28", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "0a2b268fa0d617735fea967a5896b3d205b101d83b018ec9fca7d988f11d10db", "candidate_token_ids": [32, 33, 34], "logits": [21.375, 13.75, 12.375], "probabilities": [0.9993889331817627, 0.00048779702046886086, 0.00012333439372014254], "choice": "entailment"}
|
| 57 |
+
{"panel_index": 56, "id": "mnli/4dc433ef7c48f1158e61aa4f1b568bac7ee2ede91001c9b8d933fd5c54e71820", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "28ea8cef7c3cefbc4e52117dbbd91fe6ba0185a3067aff8e0a364e78d7918819", "candidate_token_ids": [32, 33, 34], "logits": [21.125, 12.5625, 11.8125], "probabilities": [0.9997187256813049, 0.0001910870778374374, 9.026315092341974e-05], "choice": "entailment"}
|
| 58 |
+
{"panel_index": 57, "id": "mnli/533107d0478441289d303edb55e3769ea75e77ee657738576b1e86f9f523ff09", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "f5983d410d1ffb47caf1d2c160be923e0ed6839419930f59eadfb89cfb2b4c1c", "candidate_token_ids": [32, 33, 34], "logits": [14.75, 21.375, 15.0], "probabilities": [0.0013227718882262707, 0.996978759765625, 0.0016984728863462806], "choice": "neutral"}
|
| 59 |
+
{"panel_index": 58, "id": "mnli/e6856d10a89f46cbfc7fa77bf9266c482f130090a961f5bcb10be540e7c5fc2c", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "b0414aaab93e025d307d733858e0be03f543479f804f1dc12a6324b94a1a28c2", "candidate_token_ids": [32, 33, 34], "logits": [12.75, 21.0, 12.375], "probabilities": [0.00026114340289495885, 0.9995593428611755, 0.00017948108143173158], "choice": "neutral"}
|
| 60 |
+
{"panel_index": 59, "id": "mnli/4bd800d8bc83435b531bb472e9078bb054a7bb68bf31036bc12502e177dea19b", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "a2314ed00eeb6299fbe1343e769272e96da353429390c352cf49d59d0c2967e8", "candidate_token_ids": [32, 33, 34], "logits": [21.375, 15.5, 14.8125], "probabilities": [0.9957965612411499, 0.00279698777012527, 0.0014064137358218431], "choice": "entailment"}
|
| 61 |
+
{"panel_index": 60, "id": "mnli/b0220f9d099b621bde205ea6efa72624cc1948cceb2502cdc38d19c3762c2ba0", "task_family": "mnli", "gold": "neutral", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "5e649a151d402888b8c9e9b8ba693d39efad2f95912eb424363518e92fb26dee", "candidate_token_ids": [32, 33, 34], "logits": [14.75, 21.0, 16.875], "probabilities": [0.001896145404316485, 0.982227623462677, 0.015876231715083122], "choice": "neutral"}
|
| 62 |
+
{"panel_index": 61, "id": "mnli/5c5d3ecddeb08dada33aaa9e323aa10bdfa371aa2f535b2cd5f4f653b0e25e9c", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "808dae944e65ee87bcaaa32c78f64633ec95e1e202c2e49fc0a34ffc831c7327", "candidate_token_ids": [32, 33, 34], "logits": [20.125, 20.125, 14.4375], "probabilities": [0.4991544485092163, 0.4991544485092163, 0.0016911615384742618], "choice": "entailment"}
|
| 63 |
+
{"panel_index": 62, "id": "mnli/dba1b74655afea3f9676c7a8687e80c0da196b11f2324d6e3a41c3870b9d8591", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "1e22807eca9d351d7041b7c3a7d466c369566a06caf7559c424c855f9dc8de40", "candidate_token_ids": [32, 33, 34], "logits": [21.25, 13.5, 12.6875], "probabilities": [0.9993785619735718, 0.00043047487270087004, 0.00019102205988019705], "choice": "entailment"}
|
| 64 |
+
{"panel_index": 63, "id": "mnli/c8c695b22a0a846dde5f0ae3c17127f27c53f23219de2f4c97f2a9bebe1e0f81", "task_family": "mnli", "gold": "entailment", "candidate_ids": ["entailment", "neutral", "contradiction"], "prompt_sha256": "b2d7992028f3524744fa167194e920d2d2f96e049d9baa0c16a60567a2506e01", "candidate_token_ids": [32, 33, 34], "logits": [20.875, 12.0, 11.4375], "probabilities": [0.9997805953025818, 0.0001398109452566132, 7.966186967678368e-05], "choice": "entailment"}
|
| 65 |
+
{"panel_index": 64, "id": "goemotions:0d6f68c6aab93172edb30824a77a2f20e857a379845cfc16d8315af53d05bc1f:approval", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "62e2f3ce6d8db5d5399c3dbeaa448fa75306f4d790a05d15745663f6ceabe85f", "candidate_token_ids": [32, 33], "logits": [15.625, 21.125], "probabilities": [0.004070138093084097, 0.9959298968315125], "choice": "no"}
|
| 66 |
+
{"panel_index": 65, "id": "goemotions:5bb2f0355125b6ea32d14bc489651edeb09f6b4ef9f51b90b8ed330b8fd14cef:sadness", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "5cd89e6970749f8b4a0f8526f39a21035bd2a2d2dab17c9b22cb526e792d40bf", "candidate_token_ids": [32, 33], "logits": [14.5625, 21.0], "probabilities": [0.001597845577634871, 0.9984021782875061], "choice": "no"}
|
| 67 |
+
{"panel_index": 66, "id": "goemotions:5695bd603bf8ce60300298693b29b4295f2d34c083c7e52bff826db80a6ea996:grief", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "4a701cb3026d72bdc664138b3258e27154414aef18bc5d9658a22ca9b16431d3", "candidate_token_ids": [32, 33], "logits": [13.5, 20.875], "probabilities": [0.000626334163825959, 0.999373733997345], "choice": "no"}
|
| 68 |
+
{"panel_index": 67, "id": "goemotions:e97ef69f22521e136e92eee4bbc1e46d8571bf91280b2bc7ae24f2e1ec5d0745:desire", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "74cd0d0acfe64e81b3b02d676d3543c54fe2736d05f9d6964e9bf589a8fa604e", "candidate_token_ids": [32, 33], "logits": [21.125, 16.875], "probabilities": [0.9859364032745361, 0.014063626527786255], "choice": "yes"}
|
| 69 |
+
{"panel_index": 68, "id": "goemotions:0c48f870b5edcb7a8dcfeb58087c7e3e56a9a2d89a0687e66d9974557c74556a:love", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "74095411b7d11586abb60a651ae44840355675c465df21605302ced8aa7460e9", "candidate_token_ids": [32, 33], "logits": [13.9375, 20.75], "probabilities": [0.001098731067031622, 0.9989012479782104], "choice": "no"}
|
| 70 |
+
{"panel_index": 69, "id": "goemotions:30e748ace2950567f2e1dcc3c638f6d5fc1e2df8078792928acf9550168933c7:approval", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "a8e377d44a31f2539125c844e84396f69387ac0177550bfe52378a2461e31ae7", "candidate_token_ids": [32, 33], "logits": [21.375, 14.1875], "probabilities": [0.9992445707321167, 0.000755405577365309], "choice": "yes"}
|
| 71 |
+
{"panel_index": 70, "id": "goemotions:3b0e3210dddfd4dbb4b0e76cf62299246fd5a912229994a9e07de7dfc64c2262:embarrassment", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "897f9ee99d20643b8ec00e25182dded61636e60cfd09ee330ca3b25540069833", "candidate_token_ids": [32, 33], "logits": [15.4375, 21.125], "probabilities": [0.003376612439751625, 0.9966233968734741], "choice": "no"}
|
| 72 |
+
{"panel_index": 71, "id": "goemotions:4308985a124eb5964a14e35ae0d7db83da5f33c019222497141a9a455cb5500c:joy", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "51b9806351f409e9959d99b25249e1561bcb1d2da032317b72b84a429a28d713", "candidate_token_ids": [32, 33], "logits": [13.875, 21.125], "probabilities": [0.0007096704212017357, 0.9992903470993042], "choice": "no"}
|
| 73 |
+
{"panel_index": 72, "id": "goemotions:1285f8257d6a85acd0b9b01bfbebadff45ec085608b755eb415c74c624f8f5eb:optimism", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "5af8f3b74ad252af8af13b9bbe1c5b4585c431868cfaa0c4f99abdb42e654a36", "candidate_token_ids": [32, 33], "logits": [21.125, 15.0], "probabilities": [0.9978172779083252, 0.002182716503739357], "choice": "yes"}
|
| 74 |
+
{"panel_index": 73, "id": "goemotions:ee392e883c467546e7e603f5e6b94702211b3b4af3519a0bb7bd3b21bf4cbfd9:joy", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "266326a7e5d742f0cc33238676458ba8635f29efc4d3c95b662feaa590c2b38e", "candidate_token_ids": [32, 33], "logits": [13.8125, 20.875], "probabilities": [0.0008559006382711232, 0.9991440773010254], "choice": "no"}
|
| 75 |
+
{"panel_index": 74, "id": "goemotions:43ef087b074a832ce71850c4c8d1672ca1d68b8b03830f65be8a52da990b454c:surprise", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "50981dcbdce4641c8cadc2eb3b78a20e23fca85a04376f095c1419f77d976694", "candidate_token_ids": [32, 33], "logits": [14.0, 21.0], "probabilities": [0.0009110512328334153, 0.9990890026092529], "choice": "no"}
|
| 76 |
+
{"panel_index": 75, "id": "goemotions:8285d89a0201a60100cf7540b9e92fbf0564e50afc4e58afb812f91b6fc64cba:desire", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "538fef417332ce98ebd74b27c7c5b59a41fc722574e13987db181517abe649bb", "candidate_token_ids": [32, 33], "logits": [14.3125, 21.125], "probabilities": [0.001098731067031622, 0.9989012479782104], "choice": "no"}
|
| 77 |
+
{"panel_index": 76, "id": "goemotions:cfa49c3d3a46c50e494455f5f1aa94c401a83a9450181bcf5d1f64ae75e30e64:annoyance", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "22d93f0279f9bacc380b1a29a2debaca5cae59cd9a7ff8e82c8f42ec97d1aa8d", "candidate_token_ids": [32, 33], "logits": [15.4375, 21.125], "probabilities": [0.003376612439751625, 0.9966233968734741], "choice": "no"}
|
| 78 |
+
{"panel_index": 77, "id": "goemotions:e97ef69f22521e136e92eee4bbc1e46d8571bf91280b2bc7ae24f2e1ec5d0745:confusion", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "b7bc2566159ea8e56d02da9bdc7d94820e6e704c057d22624565cf84c3d8ccc8", "candidate_token_ids": [32, 33], "logits": [15.0625, 21.25], "probabilities": [0.0020507434383034706, 0.9979493021965027], "choice": "no"}
|
| 79 |
+
{"panel_index": 78, "id": "goemotions:a312dda34fe450feab6e22e6d2aa7493d77c757d4800e2bfd9904549ff4c3756:realization", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "53472c74825a9353b6efae1df7e1106ab8c2aeb0f510b2923fd2754893391868", "candidate_token_ids": [32, 33], "logits": [15.25, 21.375], "probabilities": [0.002182716503739357, 0.9978172779083252], "choice": "no"}
|
| 80 |
+
{"panel_index": 79, "id": "goemotions:4c893be6bdf7ad879db4d2299867417db381418ebeed081d85cbd13e6b3371a6:anger", "task_family": "score", "gold": "no", "candidate_ids": ["yes", "no"], "prompt_sha256": "6a4428c6d807234388965c34cc8785054f982294a2b8b91ab744deaa27db2c6c", "candidate_token_ids": [32, 33], "logits": [15.9375, 21.125], "probabilities": [0.005554925184696913, 0.9944451451301575], "choice": "no"}
|