Training and evaluation records

training/bench_bf16.json

@@ -0,0 +1,10 @@
+{
+ "fp8": false,
+ "fp8_layers": 0,
+ "requests": 300,
+ "decisions": 432,
+ "latency_ms_p50": 23.5,
+ "latency_ms_p90": 83.2,
+ "decisions_per_s": 4.0,
+ "gpu": "NVIDIA GeForce RTX 5090"
+}

training/bench_fp8.json

@@ -0,0 +1,10 @@
+{
+ "fp8": true,
+ "fp8_layers": 200,
+ "requests": 300,
+ "decisions": 432,
+ "latency_ms_p50": 14.7,
+ "latency_ms_p90": 90.2,
+ "decisions_per_s": 5.3,
+ "gpu": "NVIDIA GeForce RTX 5090"
+}

training/decider_eval_base.log

@@ -0,0 +1,55 @@
+decider-4b-v2.1 on 77,016 rows (101,716 decisions)
+[transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel.
+2,000 / 77,016 rows  228s  accuracy so far 0.543
+4,000 / 77,016 rows  384s  accuracy so far 0.546
+6,000 / 77,016 rows  597s  accuracy so far 0.537
+8,000 / 77,016 rows  832s  accuracy so far 0.543
+10,000 / 77,016 rows  884s  accuracy so far 0.575
+12,000 / 77,016 rows  942s  accuracy so far 0.592
+14,000 / 77,016 rows  1072s  accuracy so far 0.632
+16,000 / 77,016 rows  1194s  accuracy so far 0.654
+18,000 / 77,016 rows  1254s  accuracy so far 0.662
+20,000 / 77,016 rows  1418s  accuracy so far 0.666
+22,000 / 77,016 rows  1634s  accuracy so far 0.667
+24,000 / 77,016 rows  1847s  accuracy so far 0.668
+26,000 / 77,016 rows  2001s  accuracy so far 0.671
+28,000 / 77,016 rows  2085s  accuracy so far 0.676
+30,000 / 77,016 rows  2170s  accuracy so far 0.682
+32,000 / 77,016 rows  2194s  accuracy so far 0.680
+34,000 / 77,016 rows  2216s  accuracy so far 0.689
+36,000 / 77,016 rows  2334s  accuracy so far 0.696
+38,000 / 77,016 rows  2576s  accuracy so far 0.688
+40,000 / 77,016 rows  2852s  accuracy so far 0.688
+42,000 / 77,016 rows  2926s  accuracy so far 0.687
+44,000 / 77,016 rows  2950s  accuracy so far 0.688
+46,000 / 77,016 rows  2970s  accuracy so far 0.690
+48,000 / 77,016 rows  3007s  accuracy so far 0.688
+50,000 / 77,016 rows  3132s  accuracy so far 0.685
+52,000 / 77,016 rows  3218s  accuracy so far 0.687
+54,000 / 77,016 rows  3311s  accuracy so far 0.691
+56,000 / 77,016 rows  3375s  accuracy so far 0.694
+58,000 / 77,016 rows  3396s  accuracy so far 0.697
+60,000 / 77,016 rows  3418s  accuracy so far 0.700
+62,000 / 77,016 rows  3535s  accuracy so far 0.696
+64,000 / 77,016 rows  3657s  accuracy so far 0.692
+66,000 / 77,016 rows  3754s  accuracy so far 0.689
+68,000 / 77,016 rows  3810s  accuracy so far 0.689
+70,000 / 77,016 rows  3876s  accuracy so far 0.689
+72,000 / 77,016 rows  3935s  accuracy so far 0.689
+74,000 / 77,016 rows  3995s  accuracy so far 0.689
+76,000 / 77,016 rows  4053s  accuracy so far 0.689
+77,016 / 77,016 rows  4082s  accuracy so far 0.689
+all {'n': 101716, 'accuracy': 0.689, 'brier': 0.3876, 'nll': 0.7511, 'ece': 0.0282}
+source/bev_tr {'n': 46320, 'accuracy': 0.6687, 'brier': 0.416, 'nll': 0.7659, 'ece': 0.0272}
+source/bev_tr_en {'n': 5766, 'accuracy': 0.7197, 'brier': 0.352, 'nll': 0.653, 'ece': 0.0206}
+source/beyazperde_tr {'n': 5000, 'accuracy': 0.5204, 'brier': 0.5888, 'nll': 1.0806, 'ece': 0.0503}
+source/jev_bench_tr {'n': 22773, 'accuracy': 0.71, 'brier': 0.3406, 'nll': 0.8272, 'ece': 0.0308}
+source/jev_bench_tr_en {'n': 2000, 'accuracy': 0.768, 'brier': 0.2677, 'nll': 0.6802, 'ece': 0.0338}
+source/massive_tr {'n': 2974, 'accuracy': 0.8285, 'brier': 0.2533, 'nll': 0.7069, 'ece': 0.0362}
+source/squad_tr {'n': 11873, 'accuracy': 0.6832, 'brier': 0.4016, 'nll': 0.585, 'ece': 0.0488}
+source/xnli_tr {'n': 5010, 'accuracy': 0.8128, 'brier': 0.2741, 'nll': 0.4991, 'ece': 0.0356}
+type/choice {'n': 39616, 'accuracy': 0.717, 'brier': 0.3579, 'nll': 0.8249, 'ece': 0.0563}
+type/noul {'n': 39371, 'accuracy': 0.7828, 'brier': 0.2902, 'nll': 0.4514, 'ece': 0.0073}
+type/score {'n': 22729, 'accuracy': 0.4776, 'brier': 0.6083, 'nll': 1.1413, 'ece': 0.0321}
+squad_tr/answerability {'auroc': 0.8093, 'acc_answerable': 0.9178, 'acc_unanswerable': 0.439}
+skipped decisions: {}

training/decider_lora.log

@@ -0,0 +1,380 @@
+19:57:17   bev_tr: 27,000 of 102,782 rows
+19:57:17   bev_tr_en: 6,000 of 34,261 rows
+19:57:17   beyazperde_tr: 5,000 of 5,000 rows
+19:57:17   massive_tr: 5,000 of 5,000 rows
+19:57:17   squad_tr: 20,000 of 20,000 rows
+19:57:17   xnli_tr: 2,490 of 2,490 rows
+19:58:20 items train: 63,990 rows -> 169,657 prompt rows, 56,858,516 tokens
+19:58:21 items holdout: 1,500 rows -> 3,826 prompt rows, 1,277,818 tokens
+19:58:29 LoRA r=64: 121.9M trainable parameters
+[transformers] `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
+20:05:10 train epoch=1 step=20 of=786 ce=0.4711 kl=0.0000 acc=0.802 lr=5.13e-05 tok_per_s=3963 eta_h=4.26
+20:11:12 train epoch=1 step=40 of=786 ce=0.3948 kl=0.0000 acc=0.815 lr=1.00e-04 tok_per_s=4132 eta_h=3.95
+20:17:17 train epoch=1 step=60 of=786 ce=0.4701 kl=0.0000 acc=0.765 lr=9.98e-05 tok_per_s=4186 eta_h=3.79
+20:23:11 train epoch=1 step=80 of=786 ce=0.4196 kl=0.4652 acc=0.841 lr=9.93e-05 tok_per_s=4252 eta_h=3.63
+20:28:30 checkpoint: step 98 / 786
+20:29:06 train epoch=1 step=100 of=786 ce=0.3161 kl=0.4193 acc=0.860 lr=9.84e-05 tok_per_s=4298 eta_h=3.50
+20:35:04 train epoch=1 step=120 of=786 ce=0.3690 kl=0.0000 acc=0.832 lr=9.71e-05 tok_per_s=4325 eta_h=3.38
+20:41:01 train epoch=1 step=140 of=786 ce=0.3986 kl=0.0000 acc=0.822 lr=9.56e-05 tok_per_s=4339 eta_h=3.27
+20:47:04 train epoch=1 step=160 of=786 ce=0.4116 kl=0.0000 acc=0.825 lr=9.37e-05 tok_per_s=4347 eta_h=3.17
+20:53:03 train epoch=1 step=180 of=786 ce=0.3470 kl=0.0000 acc=0.876 lr=9.15e-05 tok_per_s=4354 eta_h=3.06
+20:58:40 checkpoint: step 199 / 786
+20:58:57 train epoch=1 step=200 of=786 ce=0.3483 kl=0.0000 acc=0.883 lr=8.90e-05 tok_per_s=4365 eta_h=2.95
+21:04:53 train epoch=1 step=220 of=786 ce=0.2993 kl=0.0000 acc=0.867 lr=8.62e-05 tok_per_s=4376 eta_h=2.85
+21:10:45 train epoch=1 step=240 of=786 ce=0.3812 kl=0.4214 acc=0.862 lr=8.32e-05 tok_per_s=4386 eta_h=2.74
+21:16:41 train epoch=1 step=260 of=786 ce=0.3078 kl=0.4537 acc=0.873 lr=7.99e-05 tok_per_s=4392 eta_h=2.64
+21:22:42 train epoch=1 step=280 of=786 ce=0.2801 kl=0.4366 acc=0.916 lr=7.64e-05 tok_per_s=4396 eta_h=2.54
+21:28:38 train epoch=1 step=300 of=786 ce=0.3187 kl=0.0000 acc=0.853 lr=7.28e-05 tok_per_s=4399 eta_h=2.43
+21:28:56 checkpoint: step 301 / 786
+21:34:27 train epoch=1 step=320 of=786 ce=0.2944 kl=0.0000 acc=0.862 lr=6.90e-05 tok_per_s=4401 eta_h=2.33
+21:40:18 train epoch=1 step=340 of=786 ce=0.2463 kl=0.0000 acc=0.893 lr=6.50e-05 tok_per_s=4402 eta_h=2.23
+21:46:17 train epoch=1 step=360 of=786 ce=0.2853 kl=0.3493 acc=0.894 lr=6.10e-05 tok_per_s=4403 eta_h=2.13
+21:52:15 train epoch=1 step=380 of=786 ce=0.2339 kl=0.0000 acc=0.931 lr=5.68e-05 tok_per_s=4402 eta_h=2.03
+21:58:08 train epoch=1 step=400 of=786 ce=0.2437 kl=0.0000 acc=0.896 lr=5.26e-05 tok_per_s=4403 eta_h=1.92
+21:59:04 checkpoint: step 403 / 786
+22:04:11 train epoch=1 step=420 of=786 ce=0.2569 kl=0.0000 acc=0.891 lr=4.84e-05 tok_per_s=4404 eta_h=1.83
+22:10:03 train epoch=1 step=440 of=786 ce=0.2504 kl=0.5673 acc=0.905 lr=4.42e-05 tok_per_s=4406 eta_h=1.72
+22:15:59 train epoch=1 step=460 of=786 ce=0.3125 kl=0.3594 acc=0.908 lr=4.01e-05 tok_per_s=4407 eta_h=1.62
+22:21:57 train epoch=1 step=480 of=786 ce=0.2940 kl=0.4133 acc=0.874 lr=3.60e-05 tok_per_s=4409 eta_h=1.52
+22:27:57 train epoch=1 step=500 of=786 ce=0.2922 kl=0.0000 acc=0.866 lr=3.20e-05 tok_per_s=4409 eta_h=1.42
+22:29:13 checkpoint: step 504 / 786
+22:33:59 train epoch=1 step=520 of=786 ce=0.2071 kl=0.0000 acc=0.912 lr=2.82e-05 tok_per_s=4409 eta_h=1.33
+22:39:51 train epoch=1 step=540 of=786 ce=0.2321 kl=0.0000 acc=0.892 lr=2.45e-05 tok_per_s=4413 eta_h=1.23
+22:45:46 train epoch=1 step=560 of=786 ce=0.3239 kl=0.0000 acc=0.862 lr=2.09e-05 tok_per_s=4415 eta_h=1.13
+22:51:46 train epoch=1 step=580 of=786 ce=0.2457 kl=0.4150 acc=0.906 lr=1.76e-05 tok_per_s=4414 eta_h=1.03
+22:57:44 train epoch=1 step=600 of=786 ce=0.2926 kl=0.0000 acc=0.872 lr=1.45e-05 tok_per_s=4416 eta_h=0.93
+22:59:14 checkpoint: step 605 / 786
+23:03:43 train epoch=1 step=620 of=786 ce=0.5139 kl=0.3765 acc=0.883 lr=1.17e-05 tok_per_s=4416 eta_h=0.83
+23:09:47 train epoch=1 step=640 of=786 ce=0.2457 kl=0.3053 acc=0.908 lr=9.13e-06 tok_per_s=4416 eta_h=0.73
+23:15:53 train epoch=1 step=660 of=786 ce=0.2090 kl=0.3768 acc=0.905 lr=6.86e-06 tok_per_s=4416 eta_h=0.63
+23:21:58 train epoch=1 step=680 of=786 ce=0.2614 kl=0.4038 acc=0.910 lr=4.89e-06 tok_per_s=4414 eta_h=0.53
+23:28:00 train epoch=1 step=700 of=786 ce=0.1429 kl=0.0000 acc=0.962 lr=3.23e-06 tok_per_s=4415 eta_h=0.43
+23:29:30 checkpoint: step 705 / 786
+23:33:53 train epoch=1 step=720 of=786 ce=0.2879 kl=0.0000 acc=0.862 lr=1.91e-06 tok_per_s=4416 eta_h=0.33
+23:39:50 train epoch=1 step=740 of=786 ce=0.2532 kl=0.3678 acc=0.896 lr=9.33e-07 tok_per_s=4418 eta_h=0.23
+23:45:45 train epoch=1 step=760 of=786 ce=0.3217 kl=0.0000 acc=0.850 lr=2.99e-07 tok_per_s=4419 eta_h=0.13
+23:51:39 train epoch=1 step=780 of=786 ce=0.3324 kl=0.0928 acc=0.844 lr=1.59e-08 tok_per_s=4421 eta_h=0.03
+23:54:46 validation epoch_end=1 val_rows=3826 val_slot_accuracy=0.8907 val_ce=0.2603
+23:54:46 checkpoint: step 786 / 786, training complete
+23:56:39 temperatures on 2,118 held-out answers: {'choice': 0.936, 'noul': 1.146, 'score': 1.093} (pooled 1.051)
+23:56:39 exported: decider_run/model (load with Decider('decider_run/model'))
+decider-4b-v2.1-tr-lora on 77,016 rows (101,716 decisions)
+256 / 77,016 rows  19s  accuracy so far 0.750
+512 / 77,016 rows  40s  accuracy so far 0.746
+768 / 77,016 rows  73s  accuracy so far 0.744
+1,024 / 77,016 rows  96s  accuracy so far 0.732
+1,280 / 77,016 rows  111s  accuracy so far 0.732
+1,536 / 77,016 rows  138s  accuracy so far 0.730
+1,792 / 77,016 rows  164s  accuracy so far 0.730
+2,048 / 77,016 rows  185s  accuracy so far 0.731
+2,304 / 77,016 rows  200s  accuracy so far 0.729
+2,560 / 77,016 rows  224s  accuracy so far 0.732
+2,816 / 77,016 rows  243s  accuracy so far 0.730
+3,072 / 77,016 rows  261s  accuracy so far 0.731
+3,328 / 77,016 rows  276s  accuracy so far 0.733
+3,584 / 77,016 rows  290s  accuracy so far 0.737
+3,840 / 77,016 rows  309s  accuracy so far 0.737
+4,096 / 77,016 rows  334s  accuracy so far 0.738
+4,352 / 77,016 rows  359s  accuracy so far 0.737
+4,608 / 77,016 rows  385s  accuracy so far 0.739
+4,864 / 77,016 rows  409s  accuracy so far 0.738
+5,120 / 77,016 rows  438s  accuracy so far 0.738
+5,376 / 77,016 rows  455s  accuracy so far 0.738
+5,632 / 77,016 rows  478s  accuracy so far 0.740
+5,888 / 77,016 rows  507s  accuracy so far 0.739
+6,144 / 77,016 rows  525s  accuracy so far 0.739
+6,400 / 77,016 rows  545s  accuracy so far 0.739
+6,656 / 77,016 rows  565s  accuracy so far 0.739
+6,912 / 77,016 rows  581s  accuracy so far 0.738
+7,168 / 77,016 rows  601s  accuracy so far 0.739
+7,424 / 77,016 rows  621s  accuracy so far 0.740
+7,680 / 77,016 rows  643s  accuracy so far 0.739
+7,936 / 77,016 rows  656s  accuracy so far 0.741
+8,192 / 77,016 rows  660s  accuracy so far 0.745
+8,448 / 77,016 rows  665s  accuracy so far 0.750
+8,704 / 77,016 rows  669s  accuracy so far 0.754
+8,960 / 77,016 rows  673s  accuracy so far 0.756
+9,216 / 77,016 rows  678s  accuracy so far 0.758
+9,472 / 77,016 rows  691s  accuracy so far 0.755
+9,728 / 77,016 rows  703s  accuracy so far 0.755
+9,984 / 77,016 rows  706s  accuracy so far 0.757
+10,240 / 77,016 rows  712s  accuracy so far 0.757
+10,496 / 77,016 rows  714s  accuracy so far 0.758
+10,752 / 77,016 rows  717s  accuracy so far 0.761
+11,008 / 77,016 rows  726s  accuracy so far 0.759
+11,264 / 77,016 rows  733s  accuracy so far 0.759
+11,520 / 77,016 rows  735s  accuracy so far 0.760
+11,776 / 77,016 rows  747s  accuracy so far 0.765
+12,032 / 77,016 rows  762s  accuracy so far 0.772
+12,288 / 77,016 rows  778s  accuracy so far 0.778
+12,544 / 77,016 rows  793s  accuracy so far 0.784
+12,800 / 77,016 rows  810s  accuracy so far 0.790
+13,056 / 77,016 rows  825s  accuracy so far 0.795
+13,312 / 77,016 rows  840s  accuracy so far 0.800
+13,568 / 77,016 rows  857s  accuracy so far 0.804
+13,824 / 77,016 rows  872s  accuracy so far 0.809
+14,080 / 77,016 rows  887s  accuracy so far 0.813
+14,336 / 77,016 rows  902s  accuracy so far 0.816
+14,592 / 77,016 rows  917s  accuracy so far 0.819
+14,848 / 77,016 rows  932s  accuracy so far 0.822
+15,104 / 77,016 rows  948s  accuracy so far 0.825
+15,360 / 77,016 rows  963s  accuracy so far 0.828
+15,616 / 77,016 rows  980s  accuracy so far 0.830
+15,872 / 77,016 rows  989s  accuracy so far 0.832
+16,128 / 77,016 rows  1000s  accuracy so far 0.833
+16,384 / 77,016 rows  1011s  accuracy so far 0.835
+16,640 / 77,016 rows  1020s  accuracy so far 0.837
+16,896 / 77,016 rows  1026s  accuracy so far 0.838
+17,152 / 77,016 rows  1031s  accuracy so far 0.839
+17,408 / 77,016 rows  1038s  accuracy so far 0.840
+17,664 / 77,016 rows  1044s  accuracy so far 0.840
+17,920 / 77,016 rows  1050s  accuracy so far 0.841
+18,176 / 77,016 rows  1057s  accuracy so far 0.842
+18,432 / 77,016 rows  1064s  accuracy so far 0.842
+18,688 / 77,016 rows  1071s  accuracy so far 0.842
+18,944 / 77,016 rows  1078s  accuracy so far 0.843
+19,200 / 77,016 rows  1087s  accuracy so far 0.843
+19,456 / 77,016 rows  1098s  accuracy so far 0.844
+19,712 / 77,016 rows  1117s  accuracy so far 0.844
+19,968 / 77,016 rows  1140s  accuracy so far 0.844
+20,224 / 77,016 rows  1162s  accuracy so far 0.843
+20,480 / 77,016 rows  1191s  accuracy so far 0.843
+20,736 / 77,016 rows  1218s  accuracy so far 0.842
+20,992 / 77,016 rows  1243s  accuracy so far 0.842
+21,248 / 77,016 rows  1268s  accuracy so far 0.841
+21,504 / 77,016 rows  1295s  accuracy so far 0.841
+21,760 / 77,016 rows  1318s  accuracy so far 0.841
+22,016 / 77,016 rows  1343s  accuracy so far 0.840
+22,272 / 77,016 rows  1368s  accuracy so far 0.840
+22,528 / 77,016 rows  1394s  accuracy so far 0.840
+22,784 / 77,016 rows  1419s  accuracy so far 0.839
+23,040 / 77,016 rows  1444s  accuracy so far 0.839
+23,296 / 77,016 rows  1466s  accuracy so far 0.839
+23,552 / 77,016 rows  1491s  accuracy so far 0.839
+23,808 / 77,016 rows  1516s  accuracy so far 0.839
+24,064 / 77,016 rows  1542s  accuracy so far 0.838
+24,320 / 77,016 rows  1568s  accuracy so far 0.838
+24,576 / 77,016 rows  1582s  accuracy so far 0.837
+24,832 / 77,016 rows  1602s  accuracy so far 0.837
+25,088 / 77,016 rows  1614s  accuracy so far 0.836
+25,344 / 77,016 rows  1625s  accuracy so far 0.835
+25,600 / 77,016 rows  1643s  accuracy so far 0.834
+25,856 / 77,016 rows  1659s  accuracy so far 0.833
+26,112 / 77,016 rows  1675s  accuracy so far 0.833
+26,368 / 77,016 rows  1689s  accuracy so far 0.832
+26,624 / 77,016 rows  1701s  accuracy so far 0.831
+26,880 / 77,016 rows  1713s  accuracy so far 0.830
+27,136 / 77,016 rows  1726s  accuracy so far 0.830
+27,392 / 77,016 rows  1741s  accuracy so far 0.829
+27,648 / 77,016 rows  1743s  accuracy so far 0.829
+27,904 / 77,016 rows  1745s  accuracy so far 0.829
+28,160 / 77,016 rows  1748s  accuracy so far 0.829
+28,416 / 77,016 rows  1752s  accuracy so far 0.829
+28,672 / 77,016 rows  1768s  accuracy so far 0.829
+28,928 / 77,016 rows  1784s  accuracy so far 0.829
+29,184 / 77,016 rows  1801s  accuracy so far 0.829
+29,440 / 77,016 rows  1815s  accuracy so far 0.829
+29,696 / 77,016 rows  1819s  accuracy so far 0.829
+29,952 / 77,016 rows  1823s  accuracy so far 0.829
+30,208 / 77,016 rows  1828s  accuracy so far 0.829
+30,464 / 77,016 rows  1832s  accuracy so far 0.829
+30,720 / 77,016 rows  1834s  accuracy so far 0.829
+30,976 / 77,016 rows  1837s  accuracy so far 0.827
+31,232 / 77,016 rows  1840s  accuracy so far 0.827
+31,488 / 77,016 rows  1843s  accuracy so far 0.826
+31,744 / 77,016 rows  1845s  accuracy so far 0.825
+32,000 / 77,016 rows  1848s  accuracy so far 0.824
+32,256 / 77,016 rows  1851s  accuracy so far 0.824
+32,512 / 77,016 rows  1853s  accuracy so far 0.824
+32,768 / 77,016 rows  1856s  accuracy so far 0.825
+33,024 / 77,016 rows  1859s  accuracy so far 0.825
+33,280 / 77,016 rows  1861s  accuracy so far 0.825
+33,536 / 77,016 rows  1864s  accuracy so far 0.825
+33,792 / 77,016 rows  1867s  accuracy so far 0.825
+34,048 / 77,016 rows  1875s  accuracy so far 0.825
+34,304 / 77,016 rows  1900s  accuracy so far 0.826
+34,560 / 77,016 rows  1926s  accuracy so far 0.826
+34,816 / 77,016 rows  1951s  accuracy so far 0.826
+35,072 / 77,016 rows  1968s  accuracy so far 0.826
+35,328 / 77,016 rows  1971s  accuracy so far 0.827
+35,584 / 77,016 rows  1973s  accuracy so far 0.827
+35,840 / 77,016 rows  1976s  accuracy so far 0.828
+36,096 / 77,016 rows  1979s  accuracy so far 0.827
+36,352 / 77,016 rows  1982s  accuracy so far 0.826
+36,608 / 77,016 rows  1985s  accuracy so far 0.824
+36,864 / 77,016 rows  1988s  accuracy so far 0.823
+37,120 / 77,016 rows  2016s  accuracy so far 0.821
+37,376 / 77,016 rows  2074s  accuracy so far 0.820
+37,632 / 77,016 rows  2127s  accuracy so far 0.818
+37,888 / 77,016 rows  2177s  accuracy so far 0.816
+38,144 / 77,016 rows  2229s  accuracy so far 0.815
+38,400 / 77,016 rows  2286s  accuracy so far 0.815
+38,656 / 77,016 rows  2340s  accuracy so far 0.814
+38,912 / 77,016 rows  2391s  accuracy so far 0.814
+39,168 / 77,016 rows  2414s  accuracy so far 0.813
+39,424 / 77,016 rows  2426s  accuracy so far 0.812
+39,680 / 77,016 rows  2438s  accuracy so far 0.812
+39,936 / 77,016 rows  2449s  accuracy so far 0.811
+40,192 / 77,016 rows  2461s  accuracy so far 0.812
+40,448 / 77,016 rows  2472s  accuracy so far 0.812
+40,704 / 77,016 rows  2483s  accuracy so far 0.812
+40,960 / 77,016 rows  2494s  accuracy so far 0.812
+41,216 / 77,016 rows  2500s  accuracy so far 0.811
+41,472 / 77,016 rows  2506s  accuracy so far 0.810
+41,728 / 77,016 rows  2512s  accuracy so far 0.809
+41,984 / 77,016 rows  2518s  accuracy so far 0.808
+42,240 / 77,016 rows  2522s  accuracy so far 0.807
+42,496 / 77,016 rows  2525s  accuracy so far 0.807
+42,752 / 77,016 rows  2529s  accuracy so far 0.806
+43,008 / 77,016 rows  2532s  accuracy so far 0.805
+43,264 / 77,016 rows  2535s  accuracy so far 0.805
+43,520 / 77,016 rows  2538s  accuracy so far 0.805
+43,776 / 77,016 rows  2541s  accuracy so far 0.805
+44,032 / 77,016 rows  2543s  accuracy so far 0.805
+44,288 / 77,016 rows  2545s  accuracy so far 0.805
+44,544 / 77,016 rows  2547s  accuracy so far 0.804
+44,800 / 77,016 rows  2550s  accuracy so far 0.804
+45,056 / 77,016 rows  2552s  accuracy so far 0.804
+45,312 / 77,016 rows  2553s  accuracy so far 0.805
+45,568 / 77,016 rows  2555s  accuracy so far 0.805
+45,824 / 77,016 rows  2557s  accuracy so far 0.805
+46,080 / 77,016 rows  2566s  accuracy so far 0.804
+46,336 / 77,016 rows  2574s  accuracy so far 0.803
+46,592 / 77,016 rows  2583s  accuracy so far 0.801
+46,848 / 77,016 rows  2590s  accuracy so far 0.800
+47,104 / 77,016 rows  2591s  accuracy so far 0.799
+47,360 / 77,016 rows  2592s  accuracy so far 0.798
+47,616 / 77,016 rows  2594s  accuracy so far 0.798
+47,872 / 77,016 rows  2596s  accuracy so far 0.798
+48,128 / 77,016 rows  2598s  accuracy so far 0.798
+48,384 / 77,016 rows  2609s  accuracy so far 0.797
+48,640 / 77,016 rows  2620s  accuracy so far 0.796
+48,896 / 77,016 rows  2631s  accuracy so far 0.795
+49,152 / 77,016 rows  2642s  accuracy so far 0.794
+49,408 / 77,016 rows  2663s  accuracy so far 0.793
+49,664 / 77,016 rows  2685s  accuracy so far 0.793
+49,920 / 77,016 rows  2707s  accuracy so far 0.793
+50,176 / 77,016 rows  2728s  accuracy so far 0.792
+50,432 / 77,016 rows  2737s  accuracy so far 0.792
+50,688 / 77,016 rows  2747s  accuracy so far 0.792
+50,944 / 77,016 rows  2757s  accuracy so far 0.792
+51,200 / 77,016 rows  2765s  accuracy so far 0.792
+51,456 / 77,016 rows  2775s  accuracy so far 0.792
+51,712 / 77,016 rows  2785s  accuracy so far 0.792
+51,968 / 77,016 rows  2795s  accuracy so far 0.792
+52,224 / 77,016 rows  2804s  accuracy so far 0.791
+52,480 / 77,016 rows  2815s  accuracy so far 0.792
+52,736 / 77,016 rows  2826s  accuracy so far 0.792
+52,992 / 77,016 rows  2837s  accuracy so far 0.792
+53,248 / 77,016 rows  2848s  accuracy so far 0.793
+53,504 / 77,016 rows  2859s  accuracy so far 0.793
+53,760 / 77,016 rows  2870s  accuracy so far 0.793
+54,016 / 77,016 rows  2881s  accuracy so far 0.793
+54,272 / 77,016 rows  2892s  accuracy so far 0.793
+54,528 / 77,016 rows  2903s  accuracy so far 0.794
+54,784 / 77,016 rows  2914s  accuracy so far 0.794
+55,040 / 77,016 rows  2925s  accuracy so far 0.794
+55,296 / 77,016 rows  2931s  accuracy so far 0.795
+55,552 / 77,016 rows  2934s  accuracy so far 0.795
+55,808 / 77,016 rows  2936s  accuracy so far 0.795
+56,064 / 77,016 rows  2939s  accuracy so far 0.795
+56,320 / 77,016 rows  2941s  accuracy so far 0.795
+56,576 / 77,016 rows  2944s  accuracy so far 0.795
+56,832 / 77,016 rows  2946s  accuracy so far 0.795
+57,088 / 77,016 rows  2949s  accuracy so far 0.795
+57,344 / 77,016 rows  2951s  accuracy so far 0.796
+57,600 / 77,016 rows  2954s  accuracy so far 0.796
+57,856 / 77,016 rows  2957s  accuracy so far 0.796
+58,112 / 77,016 rows  2959s  accuracy so far 0.796
+58,368 / 77,016 rows  2962s  accuracy so far 0.796
+58,624 / 77,016 rows  2965s  accuracy so far 0.796
+58,880 / 77,016 rows  2967s  accuracy so far 0.797
+59,136 / 77,016 rows  2970s  accuracy so far 0.797
+59,392 / 77,016 rows  2973s  accuracy so far 0.797
+59,648 / 77,016 rows  2975s  accuracy so far 0.797
+59,904 / 77,016 rows  2978s  accuracy so far 0.797
+60,160 / 77,016 rows  2981s  accuracy so far 0.797
+60,416 / 77,016 rows  2996s  accuracy so far 0.796
+60,672 / 77,016 rows  3013s  accuracy so far 0.796
+60,928 / 77,016 rows  3028s  accuracy so far 0.795
+61,184 / 77,016 rows  3042s  accuracy so far 0.795
+61,440 / 77,016 rows  3055s  accuracy so far 0.794
+61,696 / 77,016 rows  3071s  accuracy so far 0.794
+61,952 / 77,016 rows  3086s  accuracy so far 0.793
+62,208 / 77,016 rows  3100s  accuracy so far 0.792
+62,464 / 77,016 rows  3114s  accuracy so far 0.792
+62,720 / 77,016 rows  3130s  accuracy so far 0.791
+62,976 / 77,016 rows  3144s  accuracy so far 0.791
+63,232 / 77,016 rows  3159s  accuracy so far 0.790
+63,488 / 77,016 rows  3173s  accuracy so far 0.790
+63,744 / 77,016 rows  3188s  accuracy so far 0.790
+64,000 / 77,016 rows  3202s  accuracy so far 0.789
+64,256 / 77,016 rows  3218s  accuracy so far 0.789
+64,512 / 77,016 rows  3235s  accuracy so far 0.789
+64,768 / 77,016 rows  3250s  accuracy so far 0.788
+65,024 / 77,016 rows  3265s  accuracy so far 0.788
+65,280 / 77,016 rows  3275s  accuracy so far 0.787
+65,536 / 77,016 rows  3281s  accuracy so far 0.788
+65,792 / 77,016 rows  3287s  accuracy so far 0.788
+66,048 / 77,016 rows  3293s  accuracy so far 0.788
+66,304 / 77,016 rows  3299s  accuracy so far 0.788
+66,560 / 77,016 rows  3306s  accuracy so far 0.788
+66,816 / 77,016 rows  3312s  accuracy so far 0.789
+67,072 / 77,016 rows  3319s  accuracy so far 0.789
+67,328 / 77,016 rows  3326s  accuracy so far 0.789
+67,584 / 77,016 rows  3333s  accuracy so far 0.789
+67,840 / 77,016 rows  3339s  accuracy so far 0.789
+68,096 / 77,016 rows  3346s  accuracy so far 0.789
+68,352 / 77,016 rows  3355s  accuracy so far 0.789
+68,608 / 77,016 rows  3365s  accuracy so far 0.789
+68,864 / 77,016 rows  3373s  accuracy so far 0.790
+69,120 / 77,016 rows  3380s  accuracy so far 0.790
+69,376 / 77,016 rows  3387s  accuracy so far 0.790
+69,632 / 77,016 rows  3394s  accuracy so far 0.790
+69,888 / 77,016 rows  3401s  accuracy so far 0.790
+70,144 / 77,016 rows  3408s  accuracy so far 0.790
+70,400 / 77,016 rows  3415s  accuracy so far 0.791
+70,656 / 77,016 rows  3422s  accuracy so far 0.791
+70,912 / 77,016 rows  3429s  accuracy so far 0.791
+71,168 / 77,016 rows  3436s  accuracy so far 0.791
+71,424 / 77,016 rows  3443s  accuracy so far 0.792
+71,680 / 77,016 rows  3450s  accuracy so far 0.792
+71,936 / 77,016 rows  3457s  accuracy so far 0.792
+72,192 / 77,016 rows  3464s  accuracy so far 0.792
+72,448 / 77,016 rows  3471s  accuracy so far 0.793
+72,704 / 77,016 rows  3478s  accuracy so far 0.793
+72,960 / 77,016 rows  3486s  accuracy so far 0.793
+73,216 / 77,016 rows  3493s  accuracy so far 0.793
+73,472 / 77,016 rows  3500s  accuracy so far 0.794
+73,728 / 77,016 rows  3507s  accuracy so far 0.794
+73,984 / 77,016 rows  3514s  accuracy so far 0.794
+74,240 / 77,016 rows  3520s  accuracy so far 0.794
+74,496 / 77,016 rows  3527s  accuracy so far 0.795
+74,752 / 77,016 rows  3534s  accuracy so far 0.795
+75,008 / 77,016 rows  3540s  accuracy so far 0.795
+75,264 / 77,016 rows  3547s  accuracy so far 0.795
+75,520 / 77,016 rows  3554s  accuracy so far 0.795
+75,776 / 77,016 rows  3561s  accuracy so far 0.796
+76,032 / 77,016 rows  3569s  accuracy so far 0.796
+76,288 / 77,016 rows  3575s  accuracy so far 0.796
+76,544 / 77,016 rows  3582s  accuracy so far 0.796
+76,800 / 77,016 rows  3589s  accuracy so far 0.796
+77,016 / 77,016 rows  3595s  accuracy so far 0.796
+all {'n': 101716, 'accuracy': 0.7962, 'brier': 0.2653, 'nll': 0.53, 'ece': 0.0203}
+source/bev_tr {'n': 46320, 'accuracy': 0.8382, 'brier': 0.2215, 'nll': 0.3964, 'ece': 0.0092}
+source/bev_tr_en {'n': 5766, 'accuracy': 0.7542, 'brier': 0.3138, 'nll': 0.5674, 'ece': 0.0126}
+source/beyazperde_tr {'n': 5000, 'accuracy': 0.6336, 'brier': 0.4769, 'nll': 0.8667, 'ece': 0.0402}
+source/jev_bench_tr {'n': 22773, 'accuracy': 0.7078, 'brier': 0.3407, 'nll': 0.821, 'ece': 0.051}
+source/jev_bench_tr_en {'n': 2000, 'accuracy': 0.7705, 'brier': 0.2689, 'nll': 0.6936, 'ece': 0.0351}
+source/massive_tr {'n': 2974, 'accuracy': 0.8705, 'brier': 0.1941, 'nll': 0.5568, 'ece': 0.0362}
+source/squad_tr {'n': 11873, 'accuracy': 0.8619, 'brier': 0.2028, 'nll': 0.3322, 'ece': 0.0168}
+source/xnli_tr {'n': 5010, 'accuracy': 0.8315, 'brier': 0.2488, 'nll': 0.4509, 'ece': 0.0151}
+type/choice {'n': 39616, 'accuracy': 0.8135, 'brier': 0.2401, 'nll': 0.5701, 'ece': 0.0214}
+type/noul {'n': 39371, 'accuracy': 0.8746, 'brier': 0.1753, 'nll': 0.2943, 'ece': 0.0177}
+type/score {'n': 22729, 'accuracy': 0.6304, 'brier': 0.465, 'nll': 0.8684, 'ece': 0.0235}
+squad_tr/answerability {'auroc': 0.9397, 'acc_answerable': 0.9379, 'acc_unanswerable': 0.783}
+skipped decisions: {}
+00:56:41 eval: accuracy 0.7962, ECE 0.0203 (decider_run/eval_report.json)
+00:58:36 pushed: https://huggingface.co/hayriyigit/decider-4b-tr (241be87048b2)
+00:58:36 done
+EXIT 0

training/decider_lora.yaml

@@ -0,0 +1,45 @@
+# LoRA fine-tune of Mapika/decider-4b on the Turkish decision data that laya_pipeline built (configs/laya.yaml):
+#   uv run --script scripts/decider_lora.py --config configs/decider_lora.yaml
+# Everything goes to `workdir`; after a time limit, a crash or a STOP file run the same command again to resume.
+workdir: ./decider_run
+
+model:
+  repo: Mapika/decider-4b                       # v2.1: Qwen3.5-4B-Base, one forward pass reads every option's probability
+  revision: eb5fbdfc9448473ec25e399882912863afbdb70e
+
+data:
+  train: laya_run/data/all/train-00000-of-00001.parquet   # laya_pipeline prepare output (state / questions / gold)
+  test: laya_run/data/all/test-00000-of-00001.parquet     # the same test split as Laya: the reports compare row for row
+  rows:                                         # training rows sampled per source (laya_pipeline's source names)
+    bev_tr: 27000                               # bev-decision-150K-tr (domain-weighted, decontaminated); ~1,290 tokens/row
+    squad_tr: 20000                             # SQuAD 2.0-tr answerability; ~460 tokens/row
+    massive_tr: 5000                            # ~800 tokens/row: all 60 intents are options
+    xnli_tr: 2490                               # ~150 tokens/row
+    beyazperde_tr: 5000                         # ~915 tokens/row: one yes/no row per Score level
+    bev_tr_en: 6000                             # English rows: trained toward the base model's own answers (replay)
+                                                # ~62M tokens: ~4.6 h on one RTX 5090 (with causal-conv1d)
+  holdout_rows: 1500                            # Turkish rows, whole states: validation + temperature fit
+  max_state_tokens: 4096
+  seed: 0
+
+lora:                                           # decider-4b v2.1's own stage 2: rank 64, alpha 128, attention + MLP
+  r: 64
+  alpha: 128
+
+train:
+  epochs: 1                                     # ~65K different rows once rather than half of them twice
+  lr: 1.0e-4
+  warmup: 0.05
+  tokens_per_step: 65536                        # padded tokens per optimizer step (v2.1: 65,536)
+  micro_tokens: 16384                           # per forward pass: lower it on an out-of-memory error
+  label_smoothing: 0.0
+
+checkpoint:
+  every_minutes: 30
+
+limits:
+  max_hours: 6                                  # wall-clock budget of one invocation; 0 = none
+  reserve_minutes: 20
+
+export:
+  push_to_hub: {repo: hayriyigit/decider-4b-tr, private: true}

training/eval_10k_bf16.json

@@ -0,0 +1,800 @@
+{
+ "model": "decider-4b-v2.1-tr-lora",
+ "path": "/workspace/laya/decider_run/model",
+ "revision": "eb5fbdfc9448473ec25e399882912863afbdb70e",
+ "fp8_layers": 0,
+ "limit": 10000,
+ "skipped": {},
+ "seconds": 462,
+ "report": {
+  "all": {
+   "n": 13371,
+   "accuracy": 0.7968,
+   "brier": 0.2639,
+   "nll": 0.5223,
+   "ece": 0.0213
+  },
+  "source/bev_tr": {
+   "n": 6122,
+   "accuracy": 0.8383,
+   "brier": 0.2217,
+   "nll": 0.3967,
+   "ece": 0.0104
+  },
+  "source/bev_tr/choice": {
+   "n": 2464,
+   "accuracy": 0.8381,
+   "brier": 0.2187,
+   "nll": 0.4023,
+   "ece": 0.0167
+  },
+  "source/bev_tr/noul": {
+   "n": 2357,
+   "accuracy": 0.905,
+   "brier": 0.1368,
+   "nll": 0.2225,
+   "ece": 0.0084
+  },
+  "source/bev_tr/score": {
+   "n": 1301,
+   "accuracy": 0.7179,
+   "brier": 0.3814,
+   "nll": 0.7019,
+   "ece": 0.0301
+  },
+  "source/bev_tr_en": {
+   "n": 852,
+   "accuracy": 0.777,
+   "brier": 0.2821,
+   "nll": 0.5,
+   "ece": 0.0313
+  },
+  "source/bev_tr_en/choice": {
+   "n": 314,
+   "accuracy": 0.7548,
+   "brier": 0.3185,
+   "nll": 0.5896,
+   "ece": 0.038
+  },
+  "source/bev_tr_en/noul": {
+   "n": 364,
+   "accuracy": 0.8956,
+   "brier": 0.1319,
+   "nll": 0.2167,
+   "ece": 0.0393
+  },
+  "source/bev_tr_en/score": {
+   "n": 174,
+   "accuracy": 0.569,
+   "brier": 0.5307,
+   "nll": 0.9313,
+   "ece": 0.0733
+  },
+  "source/beyazperde_tr": {
+   "n": 631,
+   "accuracy": 0.6466,
+   "brier": 0.4678,
+   "nll": 0.8605,
+   "ece": 0.0735
+  },
+  "source/beyazperde_tr/score": {
+   "n": 631,
+   "accuracy": 0.6466,
+   "brier": 0.4678,
+   "nll": 0.8605,
+   "ece": 0.0735
+  },
+  "source/jev_bench_tr": {
+   "n": 2986,
+   "accuracy": 0.7029,
+   "brier": 0.3414,
+   "nll": 0.8115,
+   "ece": 0.0586
+  },
+  "source/jev_bench_tr/choice": {
+   "n": 1256,
+   "accuracy": 0.7301,
+   "brier": 0.2809,
+   "nll": 0.912,
+   "ece": 0.0693
+  },
+  "source/jev_bench_tr/noul": {
+   "n": 959,
+   "accuracy": 0.8092,
+   "brier": 0.2436,
+   "nll": 0.4284,
+   "ece": 0.0637
+  },
+  "source/jev_bench_tr/score": {
+   "n": 771,
+   "accuracy": 0.5266,
+   "brier": 0.5617,
+   "nll": 1.124,
+   "ece": 0.0547
+  },
+  "source/jev_bench_tr_en": {
+   "n": 278,
+   "accuracy": 0.7662,
+   "brier": 0.2858,
+   "nll": 0.7261,
+   "ece": 0.0676
+  },
+  "source/jev_bench_tr_en/choice": {
+   "n": 115,
+   "accuracy": 0.7565,
+   "brier": 0.2891,
+   "nll": 0.9776,
+   "ece": 0.093
+  },
+  "source/jev_bench_tr_en/noul": {
+   "n": 82,
+   "accuracy": 0.8902,
+   "brier": 0.1192,
+   "nll": 0.2343,
+   "ece": 0.1167
+  },
+  "source/jev_bench_tr_en/score": {
+   "n": 81,
+   "accuracy": 0.6543,
+   "brier": 0.45,
+   "nll": 0.8671,
+   "ece": 0.1334
+  },
+  "source/massive_tr": {
+   "n": 416,
+   "accuracy": 0.8894,
+   "brier": 0.158,
+   "nll": 0.3759,
+   "ece": 0.0261
+  },
+  "source/massive_tr/choice": {
+   "n": 416,
+   "accuracy": 0.8894,
+   "brier": 0.158,
+   "nll": 0.3759,
+   "ece": 0.0261
+  },
+  "source/squad_tr": {
+   "n": 1454,
+   "accuracy": 0.8583,
+   "brier": 0.2133,
+   "nll": 0.3509,
+   "ece": 0.0283
+  },
+  "source/squad_tr/noul": {
+   "n": 1454,
+   "accuracy": 0.8583,
+   "brier": 0.2133,
+   "nll": 0.3509,
+   "ece": 0.0283
+  },
+  "source/xnli_tr": {
+   "n": 632,
+   "accuracy": 0.8259,
+   "brier": 0.2537,
+   "nll": 0.4652,
+   "ece": 0.0423
+  },
+  "source/xnli_tr/choice": {
+   "n": 632,
+   "accuracy": 0.8259,
+   "brier": 0.2537,
+   "nll": 0.4652,
+   "ece": 0.0423
+  },
+  "type/choice": {
+   "n": 5197,
+   "accuracy": 0.8078,
+   "brier": 0.2407,
+   "nll": 0.5551,
+   "ece": 0.0268
+  },
+  "type/noul": {
+   "n": 5216,
+   "accuracy": 0.8735,
+   "brier": 0.1772,
+   "nll": 0.2959,
+   "ece": 0.0192
+  },
+  "type/score": {
+   "n": 2958,
+   "accuracy": 0.6423,
+   "brier": 0.4575,
+   "nll": 0.8638,
+   "ece": 0.0284
+  },
+  "workflow/bev_tr/Browser interaction": {
+   "n": 34,
+   "accuracy": 0.9412,
+   "brier": 0.0907,
+   "nll": 0.1661,
+   "ece": 0.1111
+  },
+  "workflow/bev_tr/Civic and safety operations": {
+   "n": 27,
+   "accuracy": 0.7037,
+   "brier": 0.3862,
+   "nll": 0.7932,
+   "ece": 0.1457
+  },
+  "workflow/bev_tr/Contract evidence": {
+   "n": 29,
+   "accuracy": 0.8621,
+   "brier": 0.222,
+   "nll": 0.4294,
+   "ece": 0.107
+  },
+  "workflow/bev_tr/Engagement and ranking": {
+   "n": 12,
+   "accuracy": 0.5833,
+   "brier": 0.5043,
+   "nll": 0.7431,
+   "ece": 0.2083
+  },
+  "workflow/bev_tr/Financial reporting and banking": {
+   "n": 210,
+   "accuracy": 0.8381,
+   "brier": 0.2199,
+   "nll": 0.3958,
+   "ece": 0.0434
+  },
+  "workflow/bev_tr/Game-state decisions": {
+   "n": 282,
+   "accuracy": 0.8723,
+   "brier": 0.146,
+   "nll": 0.2215,
+   "ece": 0.0367
+  },
+  "workflow/bev_tr/Reading comprehension": {
+   "n": 200,
+   "accuracy": 0.92,
+   "brier": 0.1237,
+   "nll": 0.2081,
+   "ece": 0.0396
+  },
+  "workflow/bev_tr/Response preference and quality": {
+   "n": 387,
+   "accuracy": 0.6822,
+   "brier": 0.4485,
+   "nll": 0.8363,
+   "ece": 0.0701
+  },
+  "workflow/bev_tr/Retail, product, and shopping": {
+   "n": 855,
+   "accuracy": 0.9275,
+   "brier": 0.1039,
+   "nll": 0.1844,
+   "ece": 0.018
+  },
+  "workflow/bev_tr/Scientific and paper understanding": {
+   "n": 71,
+   "accuracy": 0.7887,
+   "brier": 0.2864,
+   "nll": 0.4506,
+   "ece": 0.1094
+  },
+  "workflow/bev_tr/Sensory quality rating": {
+   "n": 41,
+   "accuracy": 0.3415,
+   "brier": 0.7407,
+   "nll": 1.4077,
+   "ece": 0.0664
+  },
+  "workflow/bev_tr/Sentiment, emotion, and moderation": {
+   "n": 1534,
+   "accuracy": 0.8696,
+   "brier": 0.1881,
+   "nll": 0.3356,
+   "ece": 0.0234
+  },
+  "workflow/bev_tr/Software engineering and code": {
+   "n": 252,
+   "accuracy": 0.7222,
+   "brier": 0.3646,
+   "nll": 0.6731,
+   "ece": 0.0413
+  },
+  "workflow/bev_tr/Software security": {
+   "n": 330,
+   "accuracy": 0.6848,
+   "brier": 0.4157,
+   "nll": 0.7087,
+   "ece": 0.0569
+  },
+  "workflow/bev_tr/Spam detection": {
+   "n": 37,
+   "accuracy": 1.0,
+   "brier": 0.0091,
+   "nll": 0.023,
+   "ece": 0.02
+  },
+  "workflow/bev_tr/Spatial and logical reasoning": {
+   "n": 1122,
+   "accuracy": 0.7576,
+   "brier": 0.326,
+   "nll": 0.5801,
+   "ece": 0.0301
+  },
+  "workflow/bev_tr/Support and intent routing": {
+   "n": 422,
+   "accuracy": 0.9787,
+   "brier": 0.0412,
+   "nll": 0.0858,
+   "ece": 0.0176
+  },
+  "workflow/bev_tr/Tool and workflow decisions": {
+   "n": 277,
+   "accuracy": 0.9892,
+   "brier": 0.0186,
+   "nll": 0.0572,
+   "ece": 0.0127
+  },
+  "workflow/bev_tr_en/Browser interaction": {
+   "n": 7,
+   "accuracy": 0.7143,
+   "brier": 0.3125,
+   "nll": 0.4928,
+   "ece": 0.3385
+  },
+  "workflow/bev_tr_en/Civic and safety operations": {
+   "n": 3,
+   "accuracy": 0.3333,
+   "brier": 0.829,
+   "nll": 1.1192,
+   "ece": 0.5844
+  },
+  "workflow/bev_tr_en/Contract evidence": {
+   "n": 5,
+   "accuracy": 1.0,
+   "brier": 0.1274,
+   "nll": 0.3017,
+   "ece": 0.2545
+  },
+  "workflow/bev_tr_en/Engagement and ranking": {
+   "n": 1,
+   "accuracy": 0.0,
+   "brier": 0.6791,
+   "nll": 0.8739,
+   "ece": 0.5827
+  },
+  "workflow/bev_tr_en/Financial reporting and banking": {
+   "n": 24,
+   "accuracy": 0.75,
+   "brier": 0.3628,
+   "nll": 0.6762,
+   "ece": 0.1383
+  },
+  "workflow/bev_tr_en/Game-state decisions": {
+   "n": 45,
+   "accuracy": 0.9556,
+   "brier": 0.1414,
+   "nll": 0.2709,
+   "ece": 0.17
+  },
+  "workflow/bev_tr_en/Reading comprehension": {
+   "n": 27,
+   "accuracy": 0.8889,
+   "brier": 0.2271,
+   "nll": 0.3849,
+   "ece": 0.1321
+  },
+  "workflow/bev_tr_en/Response preference and quality": {
+   "n": 34,
+   "accuracy": 0.5882,
+   "brier": 0.5002,
+   "nll": 0.8492,
+   "ece": 0.1697
+  },
+  "workflow/bev_tr_en/Retail, product, and shopping": {
+   "n": 129,
+   "accuracy": 0.6977,
+   "brier": 0.3353,
+   "nll": 0.561,
+   "ece": 0.1092
+  },
+  "workflow/bev_tr_en/Scientific and paper understanding": {
+   "n": 7,
+   "accuracy": 0.4286,
+   "brier": 0.5093,
+   "nll": 0.9023,
+   "ece": 0.3847
+  },
+  "workflow/bev_tr_en/Sensory quality rating": {
+   "n": 7,
+   "accuracy": 0.0,
+   "brier": 0.8219,
+   "nll": 1.6507,
+   "ece": 0.3153
+  },
+  "workflow/bev_tr_en/Sentiment, emotion, and moderation": {
+   "n": 262,
+   "accuracy": 0.8855,
+   "brier": 0.1727,
+   "nll": 0.3088,
+   "ece": 0.0688
+  },
+  "workflow/bev_tr_en/Software engineering and code": {
+   "n": 34,
+   "accuracy": 0.6471,
+   "brier": 0.4953,
+   "nll": 0.9538,
+   "ece": 0.1696
+  },
+  "workflow/bev_tr_en/Software security": {
+   "n": 34,
+   "accuracy": 0.5294,
+   "brier": 0.5426,
+   "nll": 0.8608,
+   "ece": 0.1167
+  },
+  "workflow/bev_tr_en/Spam detection": {
+   "n": 4,
+   "accuracy": 1.0,
+   "brier": 0.0002,
+   "nll": 0.0082,
+   "ece": 0.0081
+  },
+  "workflow/bev_tr_en/Spatial and logical reasoning": {
+   "n": 148,
+   "accuracy": 0.6486,
+   "brier": 0.4133,
+   "nll": 0.7589,
+   "ece": 0.062
+  },
+  "workflow/bev_tr_en/Support and intent routing": {
+   "n": 34,
+   "accuracy": 1.0,
+   "brier": 0.0532,
+   "nll": 0.1073,
+   "ece": 0.0803
+  },
+  "workflow/bev_tr_en/Tool and workflow decisions": {
+   "n": 47,
+   "accuracy": 1.0,
+   "brier": 0.0014,
+   "nll": 0.0067,
+   "ece": 0.0062
+  },
+  "workflow/beyazperde_tr": {
+   "n": 631,
+   "accuracy": 0.6466,
+   "brier": 0.4678,
+   "nll": 0.8605,
+   "ece": 0.0735
+  },
+  "workflow/jev_bench_tr/arc_challenge": {
+   "n": 120,
+   "accuracy": 0.8333,
+   "brier": 0.23,
+   "nll": 0.4115,
+   "ece": 0.056
+  },
+  "workflow/jev_bench_tr/banking77": {
+   "n": 141,
+   "accuracy": 0.8014,
+   "brier": 0.2653,
+   "nll": 0.5743,
+   "ece": 0.0687
+  },
+  "workflow/jev_bench_tr/boolq": {
+   "n": 133,
+   "accuracy": 0.8647,
+   "brier": 0.2,
+   "nll": 0.3307,
+   "ece": 0.0374
+  },
+  "workflow/jev_bench_tr/chaosnli": {
+   "n": 204,
+   "accuracy": 0.5882,
+   "brier": 0.2156,
+   "nll": 1.1708,
+   "ece": 0.1832
+  },
+  "workflow/jev_bench_tr/civil_comments": {
+   "n": 267,
+   "accuracy": 0.8577,
+   "brier": 0.1454,
+   "nll": 0.4001,
+   "ece": 0.0423
+  },
+  "workflow/jev_bench_tr/clinc150": {
+   "n": 129,
+   "accuracy": 0.845,
+   "brier": 0.2216,
+   "nll": 0.8302,
+   "ece": 0.0558
+  },
+  "workflow/jev_bench_tr/fever_evidence": {
+   "n": 139,
+   "accuracy": 0.8489,
+   "brier": 0.2078,
+   "nll": 0.3258,
+   "ece": 0.0788
+  },
+  "workflow/jev_bench_tr/go_emotions": {
+   "n": 119,
+   "accuracy": 0.4958,
+   "brier": 0.3178,
+   "nll": 2.3626,
+   "ece": 0.1529
+  },
+  "workflow/jev_bench_tr/helpsteer2_helpfulness": {
+   "n": 120,
+   "accuracy": 0.3417,
+   "brier": 0.7534,
+   "nll": 1.4327,
+   "ece": 0.1728
+  },
+  "workflow/jev_bench_tr/helpsteer2_verbosity": {
+   "n": 136,
+   "accuracy": 0.6324,
+   "brier": 0.541,
+   "nll": 0.9622,
+   "ece": 0.1695
+  },
+  "workflow/jev_bench_tr/ledgar": {
+   "n": 130,
+   "accuracy": 0.7,
+   "brier": 0.4419,
+   "nll": 1.1838,
+   "ece": 0.1384
+  },
+  "workflow/jev_bench_tr/massive": {
+   "n": 144,
+   "accuracy": 0.8958,
+   "brier": 0.1432,
+   "nll": 0.3663,
+   "ece": 0.043
+  },
+  "workflow/jev_bench_tr/measuring_hate_speech": {
+   "n": 142,
+   "accuracy": 0.5563,
+   "brier": 0.3266,
+   "nll": 1.062,
+   "ece": 0.1568
+  },
+  "workflow/jev_bench_tr/mmlu": {
+   "n": 124,
+   "accuracy": 0.6048,
+   "brier": 0.5092,
+   "nll": 0.9451,
+   "ece": 0.1036
+  },
+  "workflow/jev_bench_tr/mnli": {
+   "n": 145,
+   "accuracy": 0.8345,
+   "brier": 0.25,
+   "nll": 0.443,
+   "ece": 0.0808
+  },
+  "workflow/jev_bench_tr/paws": {
+   "n": 138,
+   "accuracy": 0.6957,
+   "brier": 0.4653,
+   "nll": 0.7604,
+   "ece": 0.1842
+  },
+  "workflow/jev_bench_tr/sms_spam": {
+   "n": 109,
+   "accuracy": 0.9908,
+   "brier": 0.0118,
+   "nll": 0.0272,
+   "ece": 0.0228
+  },
+  "workflow/jev_bench_tr/sst5": {
+   "n": 129,
+   "accuracy": 0.4264,
+   "brier": 0.6658,
+   "nll": 1.2069,
+   "ece": 0.1382
+  },
+  "workflow/jev_bench_tr/strategyqa_closed": {
+   "n": 96,
+   "accuracy": 0.5625,
+   "brier": 0.5143,
+   "nll": 0.7265,
+   "ece": 0.1302
+  },
+  "workflow/jev_bench_tr/strategyqa_grounded": {
+   "n": 77,
+   "accuracy": 0.7273,
+   "brier": 0.3174,
+   "nll": 0.482,
+   "ece": 0.1241
+  },
+  "workflow/jev_bench_tr/stsb": {
+   "n": 120,
+   "accuracy": 0.5,
+   "brier": 0.6592,
+   "nll": 1.298,
+   "ece": 0.1734
+  },
+  "workflow/jev_bench_tr/yelp5": {
+   "n": 124,
+   "accuracy": 0.6855,
+   "brier": 0.4654,
+   "nll": 0.8193,
+   "ece": 0.1218
+  },
+  "workflow/jev_bench_tr_en/arc_challenge": {
+   "n": 7,
+   "accuracy": 0.8571,
+   "brier": 0.1881,
+   "nll": 0.2846,
+   "ece": 0.115
+  },
+  "workflow/jev_bench_tr_en/banking77": {
+   "n": 16,
+   "accuracy": 0.75,
+   "brier": 0.384,
+   "nll": 1.0887,
+   "ece": 0.2165
+  },
+  "workflow/jev_bench_tr_en/boolq": {
+   "n": 9,
+   "accuracy": 1.0,
+   "brier": 0.0051,
+   "nll": 0.0379,
+   "ece": 0.0365
+  },
+  "workflow/jev_bench_tr_en/chaosnli": {
+   "n": 19,
+   "accuracy": 0.7895,
+   "brier": 0.1991,
+   "nll": 1.0873,
+   "ece": 0.1646
+  },
+  "workflow/jev_bench_tr_en/civil_comments": {
+   "n": 21,
+   "accuracy": 0.8095,
+   "brier": 0.1381,
+   "nll": 0.4076,
+   "ece": 0.1633
+  },
+  "workflow/jev_bench_tr_en/clinc150": {
+   "n": 8,
+   "accuracy": 1.0,
+   "brier": 0.0013,
+   "nll": 0.0225,
+   "ece": 0.0219
+  },
+  "workflow/jev_bench_tr_en/fever_evidence": {
+   "n": 13,
+   "accuracy": 1.0,
+   "brier": 0.0135,
+   "nll": 0.047,
+   "ece": 0.043
+  },
+  "workflow/jev_bench_tr_en/go_emotions": {
+   "n": 16,
+   "accuracy": 0.5,
+   "brier": 0.3503,
+   "nll": 2.451,
+   "ece": 0.2825
+  },
+  "workflow/jev_bench_tr_en/helpsteer2_helpfulness": {
+   "n": 12,
+   "accuracy": 0.5,
+   "brier": 0.5452,
+   "nll": 1.0448,
+   "ece": 0.217
+  },
+  "workflow/jev_bench_tr_en/helpsteer2_verbosity": {
+   "n": 12,
+   "accuracy": 0.9167,
+   "brier": 0.2796,
+   "nll": 0.5754,
+   "ece": 0.3113
+  },
+  "workflow/jev_bench_tr_en/ledgar": {
+   "n": 12,
+   "accuracy": 0.75,
+   "brier": 0.5091,
+   "nll": 1.0791,
+   "ece": 0.3892
+  },
+  "workflow/jev_bench_tr_en/massive": {
+   "n": 12,
+   "accuracy": 0.8333,
+   "brier": 0.2544,
+   "nll": 0.6022,
+   "ece": 0.1641
+  },
+  "workflow/jev_bench_tr_en/measuring_hate_speech": {
+   "n": 12,
+   "accuracy": 0.6667,
+   "brier": 0.3492,
+   "nll": 0.774,
+   "ece": 0.2877
+  },
+  "workflow/jev_bench_tr_en/mmlu": {
+   "n": 13,
+   "accuracy": 0.6923,
+   "brier": 0.3751,
+   "nll": 0.6949,
+   "ece": 0.2002
+  },
+  "workflow/jev_bench_tr_en/mnli": {
+   "n": 12,
+   "accuracy": 0.8333,
+   "brier": 0.1955,
+   "nll": 0.3119,
+   "ece": 0.2192
+  },
+  "workflow/jev_bench_tr_en/paws": {
+   "n": 5,
+   "accuracy": 1.0,
+   "brier": 0.0029,
+   "nll": 0.0349,
+   "ece": 0.0342
+  },
+  "workflow/jev_bench_tr_en/sms_spam": {
+   "n": 16,
+   "accuracy": 1.0,
+   "brier": 0.0062,
+   "nll": 0.0192,
+   "ece": 0.0173
+  },
+  "workflow/jev_bench_tr_en/sst5": {
+   "n": 10,
+   "accuracy": 0.7,
+   "brier": 0.5048,
+   "nll": 0.8854,
+   "ece": 0.1895
+  },
+  "workflow/jev_bench_tr_en/strategyqa_closed": {
+   "n": 11,
+   "accuracy": 0.7273,
+   "brier": 0.3491,
+   "nll": 0.503,
+   "ece": 0.291
+  },
+  "workflow/jev_bench_tr_en/strategyqa_grounded": {
+   "n": 7,
+   "accuracy": 0.7143,
+   "brier": 0.3853,
+   "nll": 0.5264,
+   "ece": 0.2675
+  },
+  "workflow/jev_bench_tr_en/stsb": {
+   "n": 15,
+   "accuracy": 0.4667,
+   "brier": 0.5584,
+   "nll": 1.0288,
+   "ece": 0.2153
+  },
+  "workflow/jev_bench_tr_en/yelp5": {
+   "n": 20,
+   "accuracy": 0.7,
+   "brier": 0.4468,
+   "nll": 0.8608,
+   "ece": 0.2496
+  },
+  "workflow/massive_tr": {
+   "n": 416,
+   "accuracy": 0.8894,
+   "brier": 0.158,
+   "nll": 0.3759,
+   "ece": 0.0261
+  },
+  "workflow/squad_tr": {
+   "n": 1454,
+   "accuracy": 0.8583,
+   "brier": 0.2133,
+   "nll": 0.3509,
+   "ece": 0.0283
+  },
+  "workflow/xnli_tr": {
+   "n": 632,
+   "accuracy": 0.8259,
+   "brier": 0.2537,
+   "nll": 0.4652,
+   "ece": 0.0423
+  },
+  "squad_tr/answerability": {
+   "auroc": 0.9316,
+   "acc_answerable": 0.9371,
+   "acc_unanswerable": 0.7759
+  }
+ }
+}

training/eval_10k_fp8.json

@@ -0,0 +1,800 @@
+{
+ "model": "decider-4b-v2.1-tr-lora",
+ "path": "/workspace/laya/decider_run/model",
+ "revision": "eb5fbdfc9448473ec25e399882912863afbdb70e",
+ "fp8_layers": 200,
+ "limit": 10000,
+ "skipped": {},
+ "seconds": 447,
+ "report": {
+  "all": {
+   "n": 13371,
+   "accuracy": 0.7965,
+   "brier": 0.2645,
+   "nll": 0.5237,
+   "ece": 0.0209
+  },
+  "source/bev_tr": {
+   "n": 6122,
+   "accuracy": 0.8381,
+   "brier": 0.2217,
+   "nll": 0.3972,
+   "ece": 0.0112
+  },
+  "source/bev_tr/choice": {
+   "n": 2464,
+   "accuracy": 0.8377,
+   "brier": 0.2183,
+   "nll": 0.4011,
+   "ece": 0.0154
+  },
+  "source/bev_tr/noul": {
+   "n": 2357,
+   "accuracy": 0.9079,
+   "brier": 0.137,
+   "nll": 0.224,
+   "ece": 0.0138
+  },
+  "source/bev_tr/score": {
+   "n": 1301,
+   "accuracy": 0.7125,
+   "brier": 0.3817,
+   "nll": 0.7037,
+   "ece": 0.0274
+  },
+  "source/bev_tr_en": {
+   "n": 852,
+   "accuracy": 0.7782,
+   "brier": 0.2807,
+   "nll": 0.497,
+   "ece": 0.0288
+  },
+  "source/bev_tr_en/choice": {
+   "n": 314,
+   "accuracy": 0.7516,
+   "brier": 0.3163,
+   "nll": 0.5854,
+   "ece": 0.0477
+  },
+  "source/bev_tr_en/noul": {
+   "n": 364,
+   "accuracy": 0.9011,
+   "brier": 0.1307,
+   "nll": 0.2156,
+   "ece": 0.0277
+  },
+  "source/bev_tr_en/score": {
+   "n": 174,
+   "accuracy": 0.569,
+   "brier": 0.5301,
+   "nll": 0.926,
+   "ece": 0.0572
+  },
+  "source/beyazperde_tr": {
+   "n": 631,
+   "accuracy": 0.6403,
+   "brier": 0.4695,
+   "nll": 0.8628,
+   "ece": 0.0631
+  },
+  "source/beyazperde_tr/score": {
+   "n": 631,
+   "accuracy": 0.6403,
+   "brier": 0.4695,
+   "nll": 0.8628,
+   "ece": 0.0631
+  },
+  "source/jev_bench_tr": {
+   "n": 2986,
+   "accuracy": 0.7036,
+   "brier": 0.3436,
+   "nll": 0.8157,
+   "ece": 0.0604
+  },
+  "source/jev_bench_tr/choice": {
+   "n": 1256,
+   "accuracy": 0.7253,
+   "brier": 0.2852,
+   "nll": 0.92,
+   "ece": 0.0718
+  },
+  "source/jev_bench_tr/noul": {
+   "n": 959,
+   "accuracy": 0.8123,
+   "brier": 0.2444,
+   "nll": 0.4304,
+   "ece": 0.0594
+  },
+  "source/jev_bench_tr/score": {
+   "n": 771,
+   "accuracy": 0.5331,
+   "brier": 0.5622,
+   "nll": 1.1249,
+   "ece": 0.07
+  },
+  "source/jev_bench_tr_en": {
+   "n": 278,
+   "accuracy": 0.7662,
+   "brier": 0.2867,
+   "nll": 0.7303,
+   "ece": 0.0868
+  },
+  "source/jev_bench_tr_en/choice": {
+   "n": 115,
+   "accuracy": 0.7652,
+   "brier": 0.2899,
+   "nll": 0.9846,
+   "ece": 0.0964
+  },
+  "source/jev_bench_tr_en/noul": {
+   "n": 82,
+   "accuracy": 0.878,
+   "brier": 0.1229,
+   "nll": 0.2394,
+   "ece": 0.1037
+  },
+  "source/jev_bench_tr_en/score": {
+   "n": 81,
+   "accuracy": 0.6543,
+   "brier": 0.4479,
+   "nll": 0.8662,
+   "ece": 0.2199
+  },
+  "source/massive_tr": {
+   "n": 416,
+   "accuracy": 0.8918,
+   "brier": 0.1612,
+   "nll": 0.3838,
+   "ece": 0.0236
+  },
+  "source/massive_tr/choice": {
+   "n": 416,
+   "accuracy": 0.8918,
+   "brier": 0.1612,
+   "nll": 0.3838,
+   "ece": 0.0236
+  },
+  "source/squad_tr": {
+   "n": 1454,
+   "accuracy": 0.8569,
+   "brier": 0.2134,
+   "nll": 0.351,
+   "ece": 0.0252
+  },
+  "source/squad_tr/noul": {
+   "n": 1454,
+   "accuracy": 0.8569,
+   "brier": 0.2134,
+   "nll": 0.351,
+   "ece": 0.0252
+  },
+  "source/xnli_tr": {
+   "n": 632,
+   "accuracy": 0.8244,
+   "brier": 0.2548,
+   "nll": 0.4658,
+   "ece": 0.0368
+  },
+  "source/xnli_tr/choice": {
+   "n": 632,
+   "accuracy": 0.8244,
+   "brier": 0.2548,
+   "nll": 0.4658,
+   "ece": 0.0368
+  },
+  "type/choice": {
+   "n": 5197,
+   "accuracy": 0.8064,
+   "brier": 0.2418,
+   "nll": 0.5571,
+   "ece": 0.0287
+  },
+  "type/noul": {
+   "n": 5216,
+   "accuracy": 0.8752,
+   "brier": 0.1774,
+   "nll": 0.297,
+   "ece": 0.0203
+  },
+  "type/score": {
+   "n": 2958,
+   "accuracy": 0.6403,
+   "brier": 0.458,
+   "nll": 0.865,
+   "ece": 0.0274
+  },
+  "workflow/bev_tr/Browser interaction": {
+   "n": 34,
+   "accuracy": 0.9412,
+   "brier": 0.1026,
+   "nll": 0.1858,
+   "ece": 0.0924
+  },
+  "workflow/bev_tr/Civic and safety operations": {
+   "n": 27,
+   "accuracy": 0.7037,
+   "brier": 0.3878,
+   "nll": 0.8065,
+   "ece": 0.172
+  },
+  "workflow/bev_tr/Contract evidence": {
+   "n": 29,
+   "accuracy": 0.8621,
+   "brier": 0.2234,
+   "nll": 0.4335,
+   "ece": 0.124
+  },
+  "workflow/bev_tr/Engagement and ranking": {
+   "n": 12,
+   "accuracy": 0.6667,
+   "brier": 0.518,
+   "nll": 0.7589,
+   "ece": 0.2632
+  },
+  "workflow/bev_tr/Financial reporting and banking": {
+   "n": 210,
+   "accuracy": 0.8333,
+   "brier": 0.2259,
+   "nll": 0.4079,
+   "ece": 0.0469
+  },
+  "workflow/bev_tr/Game-state decisions": {
+   "n": 282,
+   "accuracy": 0.8723,
+   "brier": 0.1467,
+   "nll": 0.223,
+   "ece": 0.0224
+  },
+  "workflow/bev_tr/Reading comprehension": {
+   "n": 200,
+   "accuracy": 0.925,
+   "brier": 0.1247,
+   "nll": 0.2108,
+   "ece": 0.0447
+  },
+  "workflow/bev_tr/Response preference and quality": {
+   "n": 387,
+   "accuracy": 0.6796,
+   "brier": 0.4472,
+   "nll": 0.8357,
+   "ece": 0.0669
+  },
+  "workflow/bev_tr/Retail, product, and shopping": {
+   "n": 855,
+   "accuracy": 0.931,
+   "brier": 0.1026,
+   "nll": 0.1808,
+   "ece": 0.0226
+  },
+  "workflow/bev_tr/Scientific and paper understanding": {
+   "n": 71,
+   "accuracy": 0.7887,
+   "brier": 0.2876,
+   "nll": 0.455,
+   "ece": 0.0831
+  },
+  "workflow/bev_tr/Sensory quality rating": {
+   "n": 41,
+   "accuracy": 0.2683,
+   "brier": 0.7343,
+   "nll": 1.3888,
+   "ece": 0.1425
+  },
+  "workflow/bev_tr/Sentiment, emotion, and moderation": {
+   "n": 1534,
+   "accuracy": 0.8703,
+   "brier": 0.1896,
+   "nll": 0.3393,
+   "ece": 0.0255
+  },
+  "workflow/bev_tr/Software engineering and code": {
+   "n": 252,
+   "accuracy": 0.7183,
+   "brier": 0.3628,
+   "nll": 0.67,
+   "ece": 0.0516
+  },
+  "workflow/bev_tr/Software security": {
+   "n": 330,
+   "accuracy": 0.6788,
+   "brier": 0.4166,
+   "nll": 0.7101,
+   "ece": 0.07
+  },
+  "workflow/bev_tr/Spam detection": {
+   "n": 37,
+   "accuracy": 1.0,
+   "brier": 0.0072,
+   "nll": 0.0209,
+   "ece": 0.0186
+  },
+  "workflow/bev_tr/Spatial and logical reasoning": {
+   "n": 1122,
+   "accuracy": 0.7585,
+   "brier": 0.324,
+   "nll": 0.5778,
+   "ece": 0.0316
+  },
+  "workflow/bev_tr/Support and intent routing": {
+   "n": 422,
+   "accuracy": 0.9787,
+   "brier": 0.0402,
+   "nll": 0.0863,
+   "ece": 0.0143
+  },
+  "workflow/bev_tr/Tool and workflow decisions": {
+   "n": 277,
+   "accuracy": 0.9892,
+   "brier": 0.0185,
+   "nll": 0.0543,
+   "ece": 0.0093
+  },
+  "workflow/bev_tr_en/Browser interaction": {
+   "n": 7,
+   "accuracy": 0.5714,
+   "brier": 0.3066,
+   "nll": 0.488,
+   "ece": 0.3347
+  },
+  "workflow/bev_tr_en/Civic and safety operations": {
+   "n": 3,
+   "accuracy": 0.3333,
+   "brier": 0.8266,
+   "nll": 1.109,
+   "ece": 0.4267
+  },
+  "workflow/bev_tr_en/Contract evidence": {
+   "n": 5,
+   "accuracy": 1.0,
+   "brier": 0.1152,
+   "nll": 0.2835,
+   "ece": 0.2417
+  },
+  "workflow/bev_tr_en/Engagement and ranking": {
+   "n": 1,
+   "accuracy": 0.0,
+   "brier": 0.6791,
+   "nll": 0.8739,
+   "ece": 0.5827
+  },
+  "workflow/bev_tr_en/Financial reporting and banking": {
+   "n": 24,
+   "accuracy": 0.75,
+   "brier": 0.369,
+   "nll": 0.683,
+   "ece": 0.1622
+  },
+  "workflow/bev_tr_en/Game-state decisions": {
+   "n": 45,
+   "accuracy": 0.9778,
+   "brier": 0.1366,
+   "nll": 0.2622,
+   "ece": 0.1834
+  },
+  "workflow/bev_tr_en/Reading comprehension": {
+   "n": 27,
+   "accuracy": 0.8889,
+   "brier": 0.2251,
+   "nll": 0.3871,
+   "ece": 0.1329
+  },
+  "workflow/bev_tr_en/Response preference and quality": {
+   "n": 34,
+   "accuracy": 0.5882,
+   "brier": 0.5046,
+   "nll": 0.8538,
+   "ece": 0.157
+  },
+  "workflow/bev_tr_en/Retail, product, and shopping": {
+   "n": 129,
+   "accuracy": 0.6977,
+   "brier": 0.3318,
+   "nll": 0.5527,
+   "ece": 0.102
+  },
+  "workflow/bev_tr_en/Scientific and paper understanding": {
+   "n": 7,
+   "accuracy": 0.4286,
+   "brier": 0.5201,
+   "nll": 0.9032,
+   "ece": 0.397
+  },
+  "workflow/bev_tr_en/Sensory quality rating": {
+   "n": 7,
+   "accuracy": 0.0,
+   "brier": 0.8088,
+   "nll": 1.6132,
+   "ece": 0.315
+  },
+  "workflow/bev_tr_en/Sentiment, emotion, and moderation": {
+   "n": 262,
+   "accuracy": 0.8893,
+   "brier": 0.1699,
+   "nll": 0.3039,
+   "ece": 0.0548
+  },
+  "workflow/bev_tr_en/Software engineering and code": {
+   "n": 34,
+   "accuracy": 0.6471,
+   "brier": 0.4888,
+   "nll": 0.947,
+   "ece": 0.1651
+  },
+  "workflow/bev_tr_en/Software security": {
+   "n": 34,
+   "accuracy": 0.5294,
+   "brier": 0.5471,
+   "nll": 0.8623,
+   "ece": 0.145
+  },
+  "workflow/bev_tr_en/Spam detection": {
+   "n": 4,
+   "accuracy": 1.0,
+   "brier": 0.0003,
+   "nll": 0.0097,
+   "ece": 0.0096
+  },
+  "workflow/bev_tr_en/Spatial and logical reasoning": {
+   "n": 148,
+   "accuracy": 0.6486,
+   "brier": 0.4134,
+   "nll": 0.7604,
+   "ece": 0.0696
+  },
+  "workflow/bev_tr_en/Support and intent routing": {
+   "n": 34,
+   "accuracy": 1.0,
+   "brier": 0.0574,
+   "nll": 0.1111,
+   "ece": 0.0812
+  },
+  "workflow/bev_tr_en/Tool and workflow decisions": {
+   "n": 47,
+   "accuracy": 1.0,
+   "brier": 0.0013,
+   "nll": 0.0064,
+   "ece": 0.006
+  },
+  "workflow/beyazperde_tr": {
+   "n": 631,
+   "accuracy": 0.6403,
+   "brier": 0.4695,
+   "nll": 0.8628,
+   "ece": 0.0631
+  },
+  "workflow/jev_bench_tr/arc_challenge": {
+   "n": 120,
+   "accuracy": 0.8167,
+   "brier": 0.2349,
+   "nll": 0.4166,
+   "ece": 0.0771
+  },
+  "workflow/jev_bench_tr/banking77": {
+   "n": 141,
+   "accuracy": 0.7801,
+   "brier": 0.2819,
+   "nll": 0.6119,
+   "ece": 0.107
+  },
+  "workflow/jev_bench_tr/boolq": {
+   "n": 133,
+   "accuracy": 0.8421,
+   "brier": 0.2069,
+   "nll": 0.3392,
+   "ece": 0.0428
+  },
+  "workflow/jev_bench_tr/chaosnli": {
+   "n": 204,
+   "accuracy": 0.5784,
+   "brier": 0.211,
+   "nll": 1.1579,
+   "ece": 0.1836
+  },
+  "workflow/jev_bench_tr/civil_comments": {
+   "n": 267,
+   "accuracy": 0.8539,
+   "brier": 0.1416,
+   "nll": 0.3936,
+   "ece": 0.0561
+  },
+  "workflow/jev_bench_tr/clinc150": {
+   "n": 129,
+   "accuracy": 0.8527,
+   "brier": 0.2251,
+   "nll": 0.8504,
+   "ece": 0.0754
+  },
+  "workflow/jev_bench_tr/fever_evidence": {
+   "n": 139,
+   "accuracy": 0.8633,
+   "brier": 0.2089,
+   "nll": 0.327,
+   "ece": 0.0611
+  },
+  "workflow/jev_bench_tr/go_emotions": {
+   "n": 119,
+   "accuracy": 0.5042,
+   "brier": 0.3195,
+   "nll": 2.3638,
+   "ece": 0.1197
+  },
+  "workflow/jev_bench_tr/helpsteer2_helpfulness": {
+   "n": 120,
+   "accuracy": 0.35,
+   "brier": 0.7576,
+   "nll": 1.4431,
+   "ece": 0.1645
+  },
+  "workflow/jev_bench_tr/helpsteer2_verbosity": {
+   "n": 136,
+   "accuracy": 0.6471,
+   "brier": 0.5413,
+   "nll": 0.9607,
+   "ece": 0.1735
+  },
+  "workflow/jev_bench_tr/ledgar": {
+   "n": 130,
+   "accuracy": 0.7,
+   "brier": 0.4511,
+   "nll": 1.2061,
+   "ece": 0.1448
+  },
+  "workflow/jev_bench_tr/massive": {
+   "n": 144,
+   "accuracy": 0.8889,
+   "brier": 0.1447,
+   "nll": 0.3666,
+   "ece": 0.0648
+  },
+  "workflow/jev_bench_tr/measuring_hate_speech": {
+   "n": 142,
+   "accuracy": 0.5634,
+   "brier": 0.3286,
+   "nll": 1.0651,
+   "ece": 0.1415
+  },
+  "workflow/jev_bench_tr/mmlu": {
+   "n": 124,
+   "accuracy": 0.6129,
+   "brier": 0.5136,
+   "nll": 0.9445,
+   "ece": 0.0948
+  },
+  "workflow/jev_bench_tr/mnli": {
+   "n": 145,
+   "accuracy": 0.8276,
+   "brier": 0.2553,
+   "nll": 0.4509,
+   "ece": 0.072
+  },
+  "workflow/jev_bench_tr/paws": {
+   "n": 138,
+   "accuracy": 0.6957,
+   "brier": 0.4684,
+   "nll": 0.7684,
+   "ece": 0.1833
+  },
+  "workflow/jev_bench_tr/sms_spam": {
+   "n": 109,
+   "accuracy": 0.9908,
+   "brier": 0.0135,
+   "nll": 0.0296,
+   "ece": 0.0243
+  },
+  "workflow/jev_bench_tr/sst5": {
+   "n": 129,
+   "accuracy": 0.4341,
+   "brier": 0.663,
+   "nll": 1.2004,
+   "ece": 0.1378
+  },
+  "workflow/jev_bench_tr/strategyqa_closed": {
+   "n": 96,
+   "accuracy": 0.5938,
+   "brier": 0.521,
+   "nll": 0.74,
+   "ece": 0.1296
+  },
+  "workflow/jev_bench_tr/strategyqa_grounded": {
+   "n": 77,
+   "accuracy": 0.7532,
+   "brier": 0.3097,
+   "nll": 0.4777,
+   "ece": 0.087
+  },
+  "workflow/jev_bench_tr/stsb": {
+   "n": 120,
+   "accuracy": 0.4917,
+   "brier": 0.6578,
+   "nll": 1.2984,
+   "ece": 0.1597
+  },
+  "workflow/jev_bench_tr/yelp5": {
+   "n": 124,
+   "accuracy": 0.6935,
+   "brier": 0.4662,
+   "nll": 0.8194,
+   "ece": 0.1309
+  },
+  "workflow/jev_bench_tr_en/arc_challenge": {
+   "n": 7,
+   "accuracy": 0.8571,
+   "brier": 0.1993,
+   "nll": 0.31,
+   "ece": 0.1203
+  },
+  "workflow/jev_bench_tr_en/banking77": {
+   "n": 16,
+   "accuracy": 0.75,
+   "brier": 0.3822,
+   "nll": 1.1184,
+   "ece": 0.2174
+  },
+  "workflow/jev_bench_tr_en/boolq": {
+   "n": 9,
+   "accuracy": 1.0,
+   "brier": 0.0049,
+   "nll": 0.0392,
+   "ece": 0.0379
+  },
+  "workflow/jev_bench_tr_en/chaosnli": {
+   "n": 19,
+   "accuracy": 0.7895,
+   "brier": 0.2056,
+   "nll": 1.0926,
+   "ece": 0.2334
+  },
+  "workflow/jev_bench_tr_en/civil_comments": {
+   "n": 21,
+   "accuracy": 0.8095,
+   "brier": 0.1395,
+   "nll": 0.412,
+   "ece": 0.1507
+  },
+  "workflow/jev_bench_tr_en/clinc150": {
+   "n": 8,
+   "accuracy": 1.0,
+   "brier": 0.0016,
+   "nll": 0.0232,
+   "ece": 0.0226
+  },
+  "workflow/jev_bench_tr_en/fever_evidence": {
+   "n": 13,
+   "accuracy": 1.0,
+   "brier": 0.0117,
+   "nll": 0.0447,
+   "ece": 0.0413
+  },
+  "workflow/jev_bench_tr_en/go_emotions": {
+   "n": 16,
+   "accuracy": 0.5,
+   "brier": 0.3472,
+   "nll": 2.4423,
+   "ece": 0.2566
+  },
+  "workflow/jev_bench_tr_en/helpsteer2_helpfulness": {
+   "n": 12,
+   "accuracy": 0.5833,
+   "brier": 0.5415,
+   "nll": 1.0402,
+   "ece": 0.3284
+  },
+  "workflow/jev_bench_tr_en/helpsteer2_verbosity": {
+   "n": 12,
+   "accuracy": 0.9167,
+   "brier": 0.2716,
+   "nll": 0.561,
+   "ece": 0.367
+  },
+  "workflow/jev_bench_tr_en/ledgar": {
+   "n": 12,
+   "accuracy": 0.75,
+   "brier": 0.5033,
+   "nll": 1.0739,
+   "ece": 0.3251
+  },
+  "workflow/jev_bench_tr_en/massive": {
+   "n": 12,
+   "accuracy": 0.9167,
+   "brier": 0.2471,
+   "nll": 0.5981,
+   "ece": 0.1823
+  },
+  "workflow/jev_bench_tr_en/measuring_hate_speech": {
+   "n": 12,
+   "accuracy": 0.6667,
+   "brier": 0.3689,
+   "nll": 0.8192,
+   "ece": 0.2886
+  },
+  "workflow/jev_bench_tr_en/mmlu": {
+   "n": 13,
+   "accuracy": 0.6923,
+   "brier": 0.39,
+   "nll": 0.7223,
+   "ece": 0.236
+  },
+  "workflow/jev_bench_tr_en/mnli": {
+   "n": 12,
+   "accuracy": 0.8333,
+   "brier": 0.1896,
+   "nll": 0.3074,
+   "ece": 0.1535
+  },
+  "workflow/jev_bench_tr_en/paws": {
+   "n": 5,
+   "accuracy": 1.0,
+   "brier": 0.0029,
+   "nll": 0.0341,
+   "ece": 0.0334
+  },
+  "workflow/jev_bench_tr_en/sms_spam": {
+   "n": 16,
+   "accuracy": 1.0,
+   "brier": 0.0109,
+   "nll": 0.0256,
+   "ece": 0.0222
+  },
+  "workflow/jev_bench_tr_en/sst5": {
+   "n": 10,
+   "accuracy": 0.7,
+   "brier": 0.5016,
+   "nll": 0.8804,
+   "ece": 0.2318
+  },
+  "workflow/jev_bench_tr_en/strategyqa_closed": {
+   "n": 11,
+   "accuracy": 0.6364,
+   "brier": 0.3726,
+   "nll": 0.5286,
+   "ece": 0.3522
+  },
+  "workflow/jev_bench_tr_en/strategyqa_grounded": {
+   "n": 7,
+   "accuracy": 0.7143,
+   "brier": 0.3812,
+   "nll": 0.522,
+   "ece": 0.2653
+  },
+  "workflow/jev_bench_tr_en/stsb": {
+   "n": 15,
+   "accuracy": 0.4,
+   "brier": 0.5529,
+   "nll": 1.0189,
+   "ece": 0.2745
+  },
+  "workflow/jev_bench_tr_en/yelp5": {
+   "n": 20,
+   "accuracy": 0.7,
+   "brier": 0.4393,
+   "nll": 0.8514,
+   "ece": 0.2851
+  },
+  "workflow/massive_tr": {
+   "n": 416,
+   "accuracy": 0.8918,
+   "brier": 0.1612,
+   "nll": 0.3838,
+   "ece": 0.0236
+  },
+  "workflow/squad_tr": {
+   "n": 1454,
+   "accuracy": 0.8569,
+   "brier": 0.2134,
+   "nll": 0.351,
+   "ece": 0.0252
+  },
+  "workflow/xnli_tr": {
+   "n": 632,
+   "accuracy": 0.8244,
+   "brier": 0.2548,
+   "nll": 0.4658,
+   "ece": 0.0368
+  },
+  "squad_tr/answerability": {
+   "auroc": 0.9318,
+   "acc_answerable": 0.9357,
+   "acc_unanswerable": 0.7732
+  }
+ }
+}

training/eval_report.json

@@ -0,0 +1,801 @@
+{
+ "model": "decider-4b-v2.1-tr-lora",
+ "path": "/workspace/laya/decider_run/model",
+ "revision": "eb5fbdfc9448473ec25e399882912863afbdb70e",
+ "fp8_layers": 0,
+ "limit": 0,
+ "skipped": {},
+ "seconds": 3595,
+ "report": {
+  "all": {
+   "n": 101716,
+   "accuracy": 0.7962,
+   "brier": 0.2653,
+   "nll": 0.53,
+   "ece": 0.0203
+  },
+  "source/bev_tr": {
+   "n": 46320,
+   "accuracy": 0.8382,
+   "brier": 0.2215,
+   "nll": 0.3964,
+   "ece": 0.0092
+  },
+  "source/bev_tr/choice": {
+   "n": 18808,
+   "accuracy": 0.8465,
+   "brier": 0.214,
+   "nll": 0.3982,
+   "ece": 0.009
+  },
+  "source/bev_tr/noul": {
+   "n": 17572,
+   "accuracy": 0.9029,
+   "brier": 0.1396,
+   "nll": 0.2281,
+   "ece": 0.0099
+  },
+  "source/bev_tr/score": {
+   "n": 9940,
+   "accuracy": 0.708,
+   "brier": 0.3805,
+   "nll": 0.6904,
+   "ece": 0.018
+  },
+  "source/bev_tr_en": {
+   "n": 5766,
+   "accuracy": 0.7542,
+   "brier": 0.3138,
+   "nll": 0.5674,
+   "ece": 0.0126
+  },
+  "source/bev_tr_en/choice": {
+   "n": 2386,
+   "accuracy": 0.7812,
+   "brier": 0.3036,
+   "nll": 0.5762,
+   "ece": 0.0289
+  },
+  "source/bev_tr_en/noul": {
+   "n": 2117,
+   "accuracy": 0.8734,
+   "brier": 0.1665,
+   "nll": 0.2696,
+   "ece": 0.0262
+  },
+  "source/bev_tr_en/score": {
+   "n": 1263,
+   "accuracy": 0.5036,
+   "brier": 0.5799,
+   "nll": 1.0501,
+   "ece": 0.0373
+  },
+  "source/beyazperde_tr": {
+   "n": 5000,
+   "accuracy": 0.6336,
+   "brier": 0.4769,
+   "nll": 0.8667,
+   "ece": 0.0402
+  },
+  "source/beyazperde_tr/score": {
+   "n": 5000,
+   "accuracy": 0.6336,
+   "brier": 0.4769,
+   "nll": 0.8667,
+   "ece": 0.0402
+  },
+  "source/jev_bench_tr": {
+   "n": 22773,
+   "accuracy": 0.7078,
+   "brier": 0.3407,
+   "nll": 0.821,
+   "ece": 0.051
+  },
+  "source/jev_bench_tr/choice": {
+   "n": 9599,
+   "accuracy": 0.7316,
+   "brier": 0.2873,
+   "nll": 0.9497,
+   "ece": 0.0593
+  },
+  "source/jev_bench_tr/noul": {
+   "n": 7174,
+   "accuracy": 0.8246,
+   "brier": 0.2226,
+   "nll": 0.4021,
+   "ece": 0.0458
+  },
+  "source/jev_bench_tr/score": {
+   "n": 6000,
+   "accuracy": 0.5298,
+   "brier": 0.5675,
+   "nll": 1.1159,
+   "ece": 0.0453
+  },
+  "source/jev_bench_tr_en": {
+   "n": 2000,
+   "accuracy": 0.7705,
+   "brier": 0.2689,
+   "nll": 0.6936,
+   "ece": 0.0351
+  },
+  "source/jev_bench_tr_en/choice": {
+   "n": 839,
+   "accuracy": 0.7926,
+   "brier": 0.2156,
+   "nll": 0.8206,
+   "ece": 0.0399
+  },
+  "source/jev_bench_tr_en/noul": {
+   "n": 635,
+   "accuracy": 0.8945,
+   "brier": 0.1434,
+   "nll": 0.2821,
+   "ece": 0.0208
+  },
+  "source/jev_bench_tr_en/score": {
+   "n": 526,
+   "accuracy": 0.5856,
+   "brier": 0.5053,
+   "nll": 0.988,
+   "ece": 0.0867
+  },
+  "source/massive_tr": {
+   "n": 2974,
+   "accuracy": 0.8705,
+   "brier": 0.1941,
+   "nll": 0.5568,
+   "ece": 0.0362
+  },
+  "source/massive_tr/choice": {
+   "n": 2974,
+   "accuracy": 0.8705,
+   "brier": 0.1941,
+   "nll": 0.5568,
+   "ece": 0.0362
+  },
+  "source/squad_tr": {
+   "n": 11873,
+   "accuracy": 0.8619,
+   "brier": 0.2028,
+   "nll": 0.3322,
+   "ece": 0.0168
+  },
+  "source/squad_tr/noul": {
+   "n": 11873,
+   "accuracy": 0.8619,
+   "brier": 0.2028,
+   "nll": 0.3322,
+   "ece": 0.0168
+  },
+  "source/xnli_tr": {
+   "n": 5010,
+   "accuracy": 0.8315,
+   "brier": 0.2488,
+   "nll": 0.4509,
+   "ece": 0.0151
+  },
+  "source/xnli_tr/choice": {
+   "n": 5010,
+   "accuracy": 0.8315,
+   "brier": 0.2488,
+   "nll": 0.4509,
+   "ece": 0.0151
+  },
+  "type/choice": {
+   "n": 39616,
+   "accuracy": 0.8135,
+   "brier": 0.2401,
+   "nll": 0.5701,
+   "ece": 0.0214
+  },
+  "type/noul": {
+   "n": 39371,
+   "accuracy": 0.8746,
+   "brier": 0.1753,
+   "nll": 0.2943,
+   "ece": 0.0177
+  },
+  "type/score": {
+   "n": 22729,
+   "accuracy": 0.6304,
+   "brier": 0.465,
+   "nll": 0.8684,
+   "ece": 0.0235
+  },
+  "workflow/bev_tr/Browser interaction": {
+   "n": 286,
+   "accuracy": 0.8497,
+   "brier": 0.1897,
+   "nll": 0.3352,
+   "ece": 0.0394
+  },
+  "workflow/bev_tr/Civic and safety operations": {
+   "n": 261,
+   "accuracy": 0.7701,
+   "brier": 0.3154,
+   "nll": 0.6527,
+   "ece": 0.0616
+  },
+  "workflow/bev_tr/Contract evidence": {
+   "n": 320,
+   "accuracy": 0.8375,
+   "brier": 0.2302,
+   "nll": 0.4253,
+   "ece": 0.0491
+  },
+  "workflow/bev_tr/Engagement and ranking": {
+   "n": 82,
+   "accuracy": 0.6463,
+   "brier": 0.4851,
+   "nll": 0.6918,
+   "ece": 0.1104
+  },
+  "workflow/bev_tr/Financial reporting and banking": {
+   "n": 1740,
+   "accuracy": 0.8264,
+   "brier": 0.2371,
+   "nll": 0.4219,
+   "ece": 0.0215
+  },
+  "workflow/bev_tr/Game-state decisions": {
+   "n": 2039,
+   "accuracy": 0.8794,
+   "brier": 0.1475,
+   "nll": 0.2267,
+   "ece": 0.0145
+  },
+  "workflow/bev_tr/Reading comprehension": {
+   "n": 1422,
+   "accuracy": 0.8643,
+   "brier": 0.1945,
+   "nll": 0.3188,
+   "ece": 0.0163
+  },
+  "workflow/bev_tr/Response preference and quality": {
+   "n": 3176,
+   "accuracy": 0.6382,
+   "brier": 0.4731,
+   "nll": 0.8563,
+   "ece": 0.0269
+  },
+  "workflow/bev_tr/Retail, product, and shopping": {
+   "n": 6601,
+   "accuracy": 0.9365,
+   "brier": 0.0922,
+   "nll": 0.1653,
+   "ece": 0.0122
+  },
+  "workflow/bev_tr/Scientific and paper understanding": {
+   "n": 508,
+   "accuracy": 0.8346,
+   "brier": 0.248,
+   "nll": 0.4421,
+   "ece": 0.0278
+  },
+  "workflow/bev_tr/Sensory quality rating": {
+   "n": 260,
+   "accuracy": 0.4,
+   "brier": 0.6968,
+   "nll": 1.3241,
+   "ece": 0.0533
+  },
+  "workflow/bev_tr/Sentiment, emotion, and moderation": {
+   "n": 11526,
+   "accuracy": 0.8647,
+   "brier": 0.1892,
+   "nll": 0.3338,
+   "ece": 0.0152
+  },
+  "workflow/bev_tr/Software engineering and code": {
+   "n": 2039,
+   "accuracy": 0.7229,
+   "brier": 0.3732,
+   "nll": 0.7028,
+   "ece": 0.0368
+  },
+  "workflow/bev_tr/Software security": {
+   "n": 2240,
+   "accuracy": 0.6879,
+   "brier": 0.4086,
+   "nll": 0.7059,
+   "ece": 0.0458
+  },
+  "workflow/bev_tr/Spam detection": {
+   "n": 271,
+   "accuracy": 0.9779,
+   "brier": 0.0284,
+   "nll": 0.0569,
+   "ece": 0.0159
+  },
+  "workflow/bev_tr/Spatial and logical reasoning": {
+   "n": 8384,
+   "accuracy": 0.7832,
+   "brier": 0.3054,
+   "nll": 0.5545,
+   "ece": 0.0198
+  },
+  "workflow/bev_tr/Support and intent routing": {
+   "n": 2858,
+   "accuracy": 0.971,
+   "brier": 0.0462,
+   "nll": 0.087,
+   "ece": 0.011
+  },
+  "workflow/bev_tr/Tool and workflow decisions": {
+   "n": 2307,
+   "accuracy": 0.9866,
+   "brier": 0.0197,
+   "nll": 0.0423,
+   "ece": 0.0068
+  },
+  "workflow/bev_tr_en/Browser interaction": {
+   "n": 39,
+   "accuracy": 0.7692,
+   "brier": 0.2566,
+   "nll": 0.4157,
+   "ece": 0.1178
+  },
+  "workflow/bev_tr_en/Civic and safety operations": {
+   "n": 31,
+   "accuracy": 0.5484,
+   "brier": 0.5519,
+   "nll": 0.9665,
+   "ece": 0.2049
+  },
+  "workflow/bev_tr_en/Contract evidence": {
+   "n": 49,
+   "accuracy": 0.7959,
+   "brier": 0.3148,
+   "nll": 0.5499,
+   "ece": 0.1107
+  },
+  "workflow/bev_tr_en/Engagement and ranking": {
+   "n": 6,
+   "accuracy": 0.6667,
+   "brier": 0.4327,
+   "nll": 0.6113,
+   "ece": 0.278
+  },
+  "workflow/bev_tr_en/Financial reporting and banking": {
+   "n": 196,
+   "accuracy": 0.6531,
+   "brier": 0.4856,
+   "nll": 0.975,
+   "ece": 0.1033
+  },
+  "workflow/bev_tr_en/Game-state decisions": {
+   "n": 303,
+   "accuracy": 0.835,
+   "brier": 0.2434,
+   "nll": 0.4021,
+   "ece": 0.0929
+  },
+  "workflow/bev_tr_en/Reading comprehension": {
+   "n": 168,
+   "accuracy": 0.9048,
+   "brier": 0.1584,
+   "nll": 0.2737,
+   "ece": 0.0486
+  },
+  "workflow/bev_tr_en/Response preference and quality": {
+   "n": 389,
+   "accuracy": 0.5604,
+   "brier": 0.5324,
+   "nll": 0.9589,
+   "ece": 0.0574
+  },
+  "workflow/bev_tr_en/Retail, product, and shopping": {
+   "n": 902,
+   "accuracy": 0.6896,
+   "brier": 0.3378,
+   "nll": 0.5532,
+   "ece": 0.1003
+  },
+  "workflow/bev_tr_en/Scientific and paper understanding": {
+   "n": 66,
+   "accuracy": 0.7424,
+   "brier": 0.3035,
+   "nll": 0.5148,
+   "ece": 0.1262
+  },
+  "workflow/bev_tr_en/Sensory quality rating": {
+   "n": 37,
+   "accuracy": 0.2432,
+   "brier": 0.7863,
+   "nll": 1.5804,
+   "ece": 0.1299
+  },
+  "workflow/bev_tr_en/Sentiment, emotion, and moderation": {
+   "n": 1439,
+   "accuracy": 0.8457,
+   "brier": 0.202,
+   "nll": 0.3659,
+   "ece": 0.0443
+  },
+  "workflow/bev_tr_en/Software engineering and code": {
+   "n": 260,
+   "accuracy": 0.7077,
+   "brier": 0.3893,
+   "nll": 0.7283,
+   "ece": 0.076
+  },
+  "workflow/bev_tr_en/Software security": {
+   "n": 234,
+   "accuracy": 0.5556,
+   "brier": 0.5142,
+   "nll": 0.8403,
+   "ece": 0.0889
+  },
+  "workflow/bev_tr_en/Spam detection": {
+   "n": 23,
+   "accuracy": 1.0,
+   "brier": 0.0001,
+   "nll": 0.006,
+   "ece": 0.006
+  },
+  "workflow/bev_tr_en/Spatial and logical reasoning": {
+   "n": 1042,
+   "accuracy": 0.6823,
+   "brier": 0.4407,
+   "nll": 0.8513,
+   "ece": 0.0635
+  },
+  "workflow/bev_tr_en/Support and intent routing": {
+   "n": 310,
+   "accuracy": 0.9581,
+   "brier": 0.0834,
+   "nll": 0.1668,
+   "ece": 0.0519
+  },
+  "workflow/bev_tr_en/Tool and workflow decisions": {
+   "n": 272,
+   "accuracy": 0.9779,
+   "brier": 0.0379,
+   "nll": 0.074,
+   "ece": 0.0203
+  },
+  "workflow/beyazperde_tr": {
+   "n": 5000,
+   "accuracy": 0.6336,
+   "brier": 0.4769,
+   "nll": 0.8667,
+   "ece": 0.0402
+  },
+  "workflow/jev_bench_tr/arc_challenge": {
+   "n": 1000,
+   "accuracy": 0.833,
+   "brier": 0.2373,
+   "nll": 0.452,
+   "ece": 0.0305
+  },
+  "workflow/jev_bench_tr/banking77": {
+   "n": 1000,
+   "accuracy": 0.81,
+   "brier": 0.2667,
+   "nll": 0.6704,
+   "ece": 0.0496
+  },
+  "workflow/jev_bench_tr/boolq": {
+   "n": 1000,
+   "accuracy": 0.877,
+   "brier": 0.1889,
+   "nll": 0.3099,
+   "ece": 0.0233
+  },
+  "workflow/jev_bench_tr/chaosnli": {
+   "n": 1599,
+   "accuracy": 0.6373,
+   "brier": 0.2039,
+   "nll": 1.1669,
+   "ece": 0.1133
+  },
+  "workflow/jev_bench_tr/civil_comments": {
+   "n": 2000,
+   "accuracy": 0.863,
+   "brier": 0.1263,
+   "nll": 0.3793,
+   "ece": 0.0208
+  },
+  "workflow/jev_bench_tr/clinc150": {
+   "n": 1000,
+   "accuracy": 0.873,
+   "brier": 0.1986,
+   "nll": 0.6114,
+   "ece": 0.0333
+  },
+  "workflow/jev_bench_tr/fever_evidence": {
+   "n": 1000,
+   "accuracy": 0.926,
+   "brier": 0.1063,
+   "nll": 0.1935,
+   "ece": 0.0218
+  },
+  "workflow/jev_bench_tr/go_emotions": {
+   "n": 1000,
+   "accuracy": 0.496,
+   "brier": 0.3399,
+   "nll": 2.3234,
+   "ece": 0.1412
+  },
+  "workflow/jev_bench_tr/helpsteer2_helpfulness": {
+   "n": 1000,
+   "accuracy": 0.403,
+   "brier": 0.7142,
+   "nll": 1.3625,
+   "ece": 0.1558
+  },
+  "workflow/jev_bench_tr/helpsteer2_verbosity": {
+   "n": 1000,
+   "accuracy": 0.661,
+   "brier": 0.5088,
+   "nll": 0.9137,
+   "ece": 0.1524
+  },
+  "workflow/jev_bench_tr/ledgar": {
+   "n": 1000,
+   "accuracy": 0.678,
+   "brier": 0.4474,
+   "nll": 1.2384,
+   "ece": 0.0563
+  },
+  "workflow/jev_bench_tr/massive": {
+   "n": 1000,
+   "accuracy": 0.867,
+   "brier": 0.1986,
+   "nll": 0.5556,
+   "ece": 0.0434
+  },
+  "workflow/jev_bench_tr/measuring_hate_speech": {
+   "n": 1000,
+   "accuracy": 0.534,
+   "brier": 0.3917,
+   "nll": 1.1279,
+   "ece": 0.1391
+  },
+  "workflow/jev_bench_tr/mmlu": {
+   "n": 1000,
+   "accuracy": 0.615,
+   "brier": 0.5026,
+   "nll": 0.9645,
+   "ece": 0.0798
+  },
+  "workflow/jev_bench_tr/mnli": {
+   "n": 1000,
+   "accuracy": 0.832,
+   "brier": 0.2406,
+   "nll": 0.4346,
+   "ece": 0.0289
+  },
+  "workflow/jev_bench_tr/paws": {
+   "n": 1000,
+   "accuracy": 0.694,
+   "brier": 0.4599,
+   "nll": 0.7411,
+   "ece": 0.1698
+  },
+  "workflow/jev_bench_tr/sms_spam": {
+   "n": 800,
+   "accuracy": 0.9587,
+   "brier": 0.0616,
+   "nll": 0.1232,
+   "ece": 0.0165
+  },
+  "workflow/jev_bench_tr/sst5": {
+   "n": 1000,
+   "accuracy": 0.428,
+   "brier": 0.6809,
+   "nll": 1.2517,
+   "ece": 0.0952
+  },
+  "workflow/jev_bench_tr/strategyqa_closed": {
+   "n": 687,
+   "accuracy": 0.5429,
+   "brier": 0.5218,
+   "nll": 0.7334,
+   "ece": 0.1441
+  },
+  "workflow/jev_bench_tr/strategyqa_grounded": {
+   "n": 687,
+   "accuracy": 0.8049,
+   "brier": 0.2641,
+   "nll": 0.4067,
+   "ece": 0.0331
+  },
+  "workflow/jev_bench_tr/stsb": {
+   "n": 1000,
+   "accuracy": 0.51,
+   "brier": 0.634,
+   "nll": 1.2248,
+   "ece": 0.0975
+  },
+  "workflow/jev_bench_tr/yelp5": {
+   "n": 1000,
+   "accuracy": 0.643,
+   "brier": 0.4752,
+   "nll": 0.8146,
+   "ece": 0.0633
+  },
+  "workflow/jev_bench_tr_en/arc_challenge": {
+   "n": 91,
+   "accuracy": 0.9121,
+   "brier": 0.1091,
+   "nll": 0.1997,
+   "ece": 0.0625
+  },
+  "workflow/jev_bench_tr_en/banking77": {
+   "n": 71,
+   "accuracy": 0.8592,
+   "brier": 0.2199,
+   "nll": 0.5531,
+   "ece": 0.0805
+  },
+  "workflow/jev_bench_tr_en/boolq": {
+   "n": 80,
+   "accuracy": 0.9,
+   "brier": 0.1214,
+   "nll": 0.2017,
+   "ece": 0.0737
+  },
+  "workflow/jev_bench_tr_en/chaosnli": {
+   "n": 151,
+   "accuracy": 0.7086,
+   "brier": 0.1814,
+   "nll": 1.1407,
+   "ece": 0.1201
+  },
+  "workflow/jev_bench_tr_en/civil_comments": {
+   "n": 171,
+   "accuracy": 0.8655,
+   "brier": 0.14,
+   "nll": 0.3978,
+   "ece": 0.0587
+  },
+  "workflow/jev_bench_tr_en/clinc150": {
+   "n": 96,
+   "accuracy": 0.8854,
+   "brier": 0.1594,
+   "nll": 0.3886,
+   "ece": 0.0521
+  },
+  "workflow/jev_bench_tr_en/fever_evidence": {
+   "n": 91,
+   "accuracy": 0.967,
+   "brier": 0.0561,
+   "nll": 0.1315,
+   "ece": 0.0348
+  },
+  "workflow/jev_bench_tr_en/go_emotions": {
+   "n": 93,
+   "accuracy": 0.5269,
+   "brier": 0.323,
+   "nll": 2.3535,
+   "ece": 0.1604
+  },
+  "workflow/jev_bench_tr_en/helpsteer2_helpfulness": {
+   "n": 89,
+   "accuracy": 0.4831,
+   "brier": 0.6573,
+   "nll": 1.1603,
+   "ece": 0.1857
+  },
+  "workflow/jev_bench_tr_en/helpsteer2_verbosity": {
+   "n": 79,
+   "accuracy": 0.7089,
+   "brier": 0.4661,
+   "nll": 0.8944,
+   "ece": 0.1493
+  },
+  "workflow/jev_bench_tr_en/ledgar": {
+   "n": 70,
+   "accuracy": 0.7857,
+   "brier": 0.3543,
+   "nll": 0.8834,
+   "ece": 0.1722
+  },
+  "workflow/jev_bench_tr_en/massive": {
+   "n": 95,
+   "accuracy": 0.8526,
+   "brier": 0.207,
+   "nll": 0.73,
+   "ece": 0.0683
+  },
+  "workflow/jev_bench_tr_en/measuring_hate_speech": {
+   "n": 96,
+   "accuracy": 0.5729,
+   "brier": 0.3906,
+   "nll": 1.1441,
+   "ece": 0.187
+  },
+  "workflow/jev_bench_tr_en/mmlu": {
+   "n": 80,
+   "accuracy": 0.725,
+   "brier": 0.3559,
+   "nll": 0.6658,
+   "ece": 0.071
+  },
+  "workflow/jev_bench_tr_en/mnli": {
+   "n": 92,
+   "accuracy": 0.9348,
+   "brier": 0.1053,
+   "nll": 0.1974,
+   "ece": 0.0927
+  },
+  "workflow/jev_bench_tr_en/paws": {
+   "n": 80,
+   "accuracy": 0.8375,
+   "brier": 0.2238,
+   "nll": 0.3598,
+   "ece": 0.0984
+  },
+  "workflow/jev_bench_tr_en/sms_spam": {
+   "n": 82,
+   "accuracy": 1.0,
+   "brier": 0.0055,
+   "nll": 0.0164,
+   "ece": 0.0147
+  },
+  "workflow/jev_bench_tr_en/sst5": {
+   "n": 73,
+   "accuracy": 0.5205,
+   "brier": 0.5669,
+   "nll": 0.9347,
+   "ece": 0.0965
+  },
+  "workflow/jev_bench_tr_en/strategyqa_closed": {
+   "n": 60,
+   "accuracy": 0.8333,
+   "brier": 0.3338,
+   "nll": 0.5123,
+   "ece": 0.1264
+  },
+  "workflow/jev_bench_tr_en/strategyqa_grounded": {
+   "n": 71,
+   "accuracy": 0.8592,
+   "brier": 0.1958,
+   "nll": 0.3115,
+   "ece": 0.0442
+  },
+  "workflow/jev_bench_tr_en/stsb": {
+   "n": 90,
+   "accuracy": 0.5111,
+   "brier": 0.577,
+   "nll": 1.098,
+   "ece": 0.1376
+  },
+  "workflow/jev_bench_tr_en/yelp5": {
+   "n": 99,
+   "accuracy": 0.7071,
+   "brier": 0.4005,
+   "nll": 0.6957,
+   "ece": 0.1338
+  },
+  "workflow/massive_tr": {
+   "n": 2974,
+   "accuracy": 0.8705,
+   "brier": 0.1941,
+   "nll": 0.5568,
+   "ece": 0.0362
+  },
+  "workflow/squad_tr": {
+   "n": 11873,
+   "accuracy": 0.8619,
+   "brier": 0.2028,
+   "nll": 0.3322,
+   "ece": 0.0168
+  },
+  "workflow/xnli_tr": {
+   "n": 5010,
+   "accuracy": 0.8315,
+   "brier": 0.2488,
+   "nll": 0.4509,
+   "ece": 0.0151
+  },
+  "squad_tr/answerability": {
+   "auroc": 0.9397,
+   "acc_answerable": 0.9379,
+   "acc_unanswerable": 0.783
+  }
+ },
+ "train_key": "411bcda4c97e6b5b"
+}

training/eval_report_base_decider-4b.json

@@ -0,0 +1,798 @@
+{
+ "model": "decider-4b-v2.1",
+ "path": "Mapika/decider-4b",
+ "revision": "eb5fbdfc9448473ec25e399882912863afbdb70e",
+ "skipped": {},
+ "seconds": 4083,
+ "report": {
+  "all": {
+   "n": 101716,
+   "accuracy": 0.689,
+   "brier": 0.3876,
+   "nll": 0.7511,
+   "ece": 0.0282
+  },
+  "source/bev_tr": {
+   "n": 46320,
+   "accuracy": 0.6687,
+   "brier": 0.416,
+   "nll": 0.7659,
+   "ece": 0.0272
+  },
+  "source/bev_tr/choice": {
+   "n": 18808,
+   "accuracy": 0.6687,
+   "brier": 0.4305,
+   "nll": 0.853,
+   "ece": 0.069
+  },
+  "source/bev_tr/noul": {
+   "n": 17572,
+   "accuracy": 0.8119,
+   "brier": 0.2643,
+   "nll": 0.4119,
+   "ece": 0.0192
+  },
+  "source/bev_tr/score": {
+   "n": 9940,
+   "accuracy": 0.4157,
+   "brier": 0.6566,
+   "nll": 1.227,
+   "ece": 0.0427
+  },
+  "source/bev_tr_en": {
+   "n": 5766,
+   "accuracy": 0.7197,
+   "brier": 0.352,
+   "nll": 0.653,
+   "ece": 0.0206
+  },
+  "source/bev_tr_en/choice": {
+   "n": 2386,
+   "accuracy": 0.715,
+   "brier": 0.3711,
+   "nll": 0.7288,
+   "ece": 0.0509
+  },
+  "source/bev_tr_en/noul": {
+   "n": 2117,
+   "accuracy": 0.8635,
+   "brier": 0.1836,
+   "nll": 0.3012,
+   "ece": 0.0466
+  },
+  "source/bev_tr_en/score": {
+   "n": 1263,
+   "accuracy": 0.4877,
+   "brier": 0.5983,
+   "nll": 1.0994,
+   "ece": 0.036
+  },
+  "source/beyazperde_tr": {
+   "n": 5000,
+   "accuracy": 0.5204,
+   "brier": 0.5888,
+   "nll": 1.0806,
+   "ece": 0.0503
+  },
+  "source/beyazperde_tr/score": {
+   "n": 5000,
+   "accuracy": 0.5204,
+   "brier": 0.5888,
+   "nll": 1.0806,
+   "ece": 0.0503
+  },
+  "source/jev_bench_tr": {
+   "n": 22773,
+   "accuracy": 0.71,
+   "brier": 0.3406,
+   "nll": 0.8272,
+   "ece": 0.0308
+  },
+  "source/jev_bench_tr/choice": {
+   "n": 9599,
+   "accuracy": 0.7209,
+   "brier": 0.3008,
+   "nll": 1.002,
+   "ece": 0.059
+  },
+  "source/jev_bench_tr/noul": {
+   "n": 7174,
+   "accuracy": 0.8433,
+   "brier": 0.2132,
+   "nll": 0.386,
+   "ece": 0.0263
+  },
+  "source/jev_bench_tr/score": {
+   "n": 6000,
+   "accuracy": 0.533,
+   "brier": 0.5565,
+   "nll": 1.0753,
+   "ece": 0.0604
+  },
+  "source/jev_bench_tr_en": {
+   "n": 2000,
+   "accuracy": 0.768,
+   "brier": 0.2677,
+   "nll": 0.6802,
+   "ece": 0.0338
+  },
+  "source/jev_bench_tr_en/choice": {
+   "n": 839,
+   "accuracy": 0.7926,
+   "brier": 0.2158,
+   "nll": 0.8053,
+   "ece": 0.0562
+  },
+  "source/jev_bench_tr_en/noul": {
+   "n": 635,
+   "accuracy": 0.8866,
+   "brier": 0.147,
+   "nll": 0.2887,
+   "ece": 0.0317
+  },
+  "source/jev_bench_tr_en/score": {
+   "n": 526,
+   "accuracy": 0.5856,
+   "brier": 0.4964,
+   "nll": 0.9534,
+   "ece": 0.0678
+  },
+  "source/massive_tr": {
+   "n": 2974,
+   "accuracy": 0.8285,
+   "brier": 0.2533,
+   "nll": 0.7069,
+   "ece": 0.0362
+  },
+  "source/massive_tr/choice": {
+   "n": 2974,
+   "accuracy": 0.8285,
+   "brier": 0.2533,
+   "nll": 0.7069,
+   "ece": 0.0362
+  },
+  "source/squad_tr": {
+   "n": 11873,
+   "accuracy": 0.6832,
+   "brier": 0.4016,
+   "nll": 0.585,
+   "ece": 0.0488
+  },
+  "source/squad_tr/noul": {
+   "n": 11873,
+   "accuracy": 0.6832,
+   "brier": 0.4016,
+   "nll": 0.585,
+   "ece": 0.0488
+  },
+  "source/xnli_tr": {
+   "n": 5010,
+   "accuracy": 0.8128,
+   "brier": 0.2741,
+   "nll": 0.4991,
+   "ece": 0.0356
+  },
+  "source/xnli_tr/choice": {
+   "n": 5010,
+   "accuracy": 0.8128,
+   "brier": 0.2741,
+   "nll": 0.4991,
+   "ece": 0.0356
+  },
+  "type/choice": {
+   "n": 39616,
+   "accuracy": 0.717,
+   "brier": 0.3579,
+   "nll": 0.8249,
+   "ece": 0.0563
+  },
+  "type/noul": {
+   "n": 39371,
+   "accuracy": 0.7828,
+   "brier": 0.2902,
+   "nll": 0.4514,
+   "ece": 0.0073
+  },
+  "type/score": {
+   "n": 22729,
+   "accuracy": 0.4776,
+   "brier": 0.6083,
+   "nll": 1.1413,
+   "ece": 0.0321
+  },
+  "workflow/bev_tr/Browser interaction": {
+   "n": 286,
+   "accuracy": 0.7622,
+   "brier": 0.3497,
+   "nll": 0.6694,
+   "ece": 0.0703
+  },
+  "workflow/bev_tr/Civic and safety operations": {
+   "n": 261,
+   "accuracy": 0.4598,
+   "brier": 0.7236,
+   "nll": 1.6077,
+   "ece": 0.0844
+  },
+  "workflow/bev_tr/Contract evidence": {
+   "n": 320,
+   "accuracy": 0.7,
+   "brier": 0.4117,
+   "nll": 0.7334,
+   "ece": 0.0947
+  },
+  "workflow/bev_tr/Engagement and ranking": {
+   "n": 82,
+   "accuracy": 0.5732,
+   "brier": 0.4994,
+   "nll": 0.6958,
+   "ece": 0.1553
+  },
+  "workflow/bev_tr/Financial reporting and banking": {
+   "n": 1740,
+   "accuracy": 0.573,
+   "brier": 0.5624,
+   "nll": 1.0865,
+   "ece": 0.0942
+  },
+  "workflow/bev_tr/Game-state decisions": {
+   "n": 2039,
+   "accuracy": 0.7013,
+   "brier": 0.4137,
+   "nll": 0.6803,
+   "ece": 0.0833
+  },
+  "workflow/bev_tr/Reading comprehension": {
+   "n": 1422,
+   "accuracy": 0.8601,
+   "brier": 0.2171,
+   "nll": 0.3537,
+   "ece": 0.0343
+  },
+  "workflow/bev_tr/Response preference and quality": {
+   "n": 3176,
+   "accuracy": 0.5107,
+   "brier": 0.5648,
+   "nll": 1.0151,
+   "ece": 0.0319
+  },
+  "workflow/bev_tr/Retail, product, and shopping": {
+   "n": 6601,
+   "accuracy": 0.6946,
+   "brier": 0.3684,
+   "nll": 0.6556,
+   "ece": 0.0808
+  },
+  "workflow/bev_tr/Scientific and paper understanding": {
+   "n": 508,
+   "accuracy": 0.6713,
+   "brier": 0.4589,
+   "nll": 0.7734,
+   "ece": 0.0884
+  },
+  "workflow/bev_tr/Sensory quality rating": {
+   "n": 260,
+   "accuracy": 0.2692,
+   "brier": 0.8003,
+   "nll": 1.6129,
+   "ece": 0.1629
+  },
+  "workflow/bev_tr/Sentiment, emotion, and moderation": {
+   "n": 11526,
+   "accuracy": 0.727,
+   "brier": 0.3386,
+   "nll": 0.594,
+   "ece": 0.0549
+  },
+  "workflow/bev_tr/Software engineering and code": {
+   "n": 2039,
+   "accuracy": 0.6709,
+   "brier": 0.428,
+   "nll": 0.8088,
+   "ece": 0.0432
+  },
+  "workflow/bev_tr/Software security": {
+   "n": 2240,
+   "accuracy": 0.5161,
+   "brier": 0.573,
+   "nll": 0.9591,
+   "ece": 0.034
+  },
+  "workflow/bev_tr/Spam detection": {
+   "n": 271,
+   "accuracy": 0.9852,
+   "brier": 0.0443,
+   "nll": 0.1076,
+   "ece": 0.0481
+  },
+  "workflow/bev_tr/Spatial and logical reasoning": {
+   "n": 8384,
+   "accuracy": 0.5081,
+   "brier": 0.621,
+   "nll": 1.2295,
+   "ece": 0.1028
+  },
+  "workflow/bev_tr/Support and intent routing": {
+   "n": 2858,
+   "accuracy": 0.8656,
+   "brier": 0.1927,
+   "nll": 0.3617,
+   "ece": 0.023
+  },
+  "workflow/bev_tr/Tool and workflow decisions": {
+   "n": 2307,
+   "accuracy": 0.9515,
+   "brier": 0.0784,
+   "nll": 0.1804,
+   "ece": 0.0202
+  },
+  "workflow/bev_tr_en/Browser interaction": {
+   "n": 39,
+   "accuracy": 0.7436,
+   "brier": 0.332,
+   "nll": 0.5797,
+   "ece": 0.2032
+  },
+  "workflow/bev_tr_en/Civic and safety operations": {
+   "n": 31,
+   "accuracy": 0.2903,
+   "brier": 0.7323,
+   "nll": 1.3491,
+   "ece": 0.2817
+  },
+  "workflow/bev_tr_en/Contract evidence": {
+   "n": 49,
+   "accuracy": 0.7959,
+   "brier": 0.3294,
+   "nll": 0.592,
+   "ece": 0.1754
+  },
+  "workflow/bev_tr_en/Engagement and ranking": {
+   "n": 6,
+   "accuracy": 0.5,
+   "brier": 0.5541,
+   "nll": 0.7408,
+   "ece": 0.2987
+  },
+  "workflow/bev_tr_en/Financial reporting and banking": {
+   "n": 196,
+   "accuracy": 0.6173,
+   "brier": 0.5212,
+   "nll": 1.0502,
+   "ece": 0.1278
+  },
+  "workflow/bev_tr_en/Game-state decisions": {
+   "n": 303,
+   "accuracy": 0.7987,
+   "brier": 0.2917,
+   "nll": 0.477,
+   "ece": 0.1155
+  },
+  "workflow/bev_tr_en/Reading comprehension": {
+   "n": 168,
+   "accuracy": 0.8988,
+   "brier": 0.1561,
+   "nll": 0.2729,
+   "ece": 0.0442
+  },
+  "workflow/bev_tr_en/Response preference and quality": {
+   "n": 389,
+   "accuracy": 0.5733,
+   "brier": 0.5404,
+   "nll": 0.9819,
+   "ece": 0.0858
+  },
+  "workflow/bev_tr_en/Retail, product, and shopping": {
+   "n": 902,
+   "accuracy": 0.6641,
+   "brier": 0.3671,
+   "nll": 0.6347,
+   "ece": 0.0989
+  },
+  "workflow/bev_tr_en/Scientific and paper understanding": {
+   "n": 66,
+   "accuracy": 0.6818,
+   "brier": 0.4373,
+   "nll": 0.6896,
+   "ece": 0.1184
+  },
+  "workflow/bev_tr_en/Sensory quality rating": {
+   "n": 37,
+   "accuracy": 0.2973,
+   "brier": 0.8005,
+   "nll": 1.645,
+   "ece": 0.026
+  },
+  "workflow/bev_tr_en/Sentiment, emotion, and moderation": {
+   "n": 1439,
+   "accuracy": 0.8367,
+   "brier": 0.2179,
+   "nll": 0.4039,
+   "ece": 0.0738
+  },
+  "workflow/bev_tr_en/Software engineering and code": {
+   "n": 260,
+   "accuracy": 0.6615,
+   "brier": 0.4328,
+   "nll": 0.7957,
+   "ece": 0.0837
+  },
+  "workflow/bev_tr_en/Software security": {
+   "n": 234,
+   "accuracy": 0.4829,
+   "brier": 0.5552,
+   "nll": 0.9091,
+   "ece": 0.0714
+  },
+  "workflow/bev_tr_en/Spam detection": {
+   "n": 23,
+   "accuracy": 1.0,
+   "brier": 0.014,
+   "nll": 0.0504,
+   "ece": 0.0463
+  },
+  "workflow/bev_tr_en/Spatial and logical reasoning": {
+   "n": 1042,
+   "accuracy": 0.5979,
+   "brier": 0.5265,
+   "nll": 1.0567,
+   "ece": 0.0797
+  },
+  "workflow/bev_tr_en/Support and intent routing": {
+   "n": 310,
+   "accuracy": 0.9032,
+   "brier": 0.1216,
+   "nll": 0.2252,
+   "ece": 0.0401
+  },
+  "workflow/bev_tr_en/Tool and workflow decisions": {
+   "n": 272,
+   "accuracy": 0.9669,
+   "brier": 0.0568,
+   "nll": 0.1371,
+   "ece": 0.0194
+  },
+  "workflow/beyazperde_tr": {
+   "n": 5000,
+   "accuracy": 0.5204,
+   "brier": 0.5888,
+   "nll": 1.0806,
+   "ece": 0.0503
+  },
+  "workflow/jev_bench_tr/arc_challenge": {
+   "n": 1000,
+   "accuracy": 0.844,
+   "brier": 0.2238,
+   "nll": 0.4245,
+   "ece": 0.0291
+  },
+  "workflow/jev_bench_tr/banking77": {
+   "n": 1000,
+   "accuracy": 0.786,
+   "brier": 0.3131,
+   "nll": 0.8613,
+   "ece": 0.0587
+  },
+  "workflow/jev_bench_tr/boolq": {
+   "n": 1000,
+   "accuracy": 0.862,
+   "brier": 0.2002,
+   "nll": 0.3298,
+   "ece": 0.0267
+  },
+  "workflow/jev_bench_tr/chaosnli": {
+   "n": 1599,
+   "accuracy": 0.5935,
+   "brier": 0.2505,
+   "nll": 1.2798,
+   "ece": 0.1838
+  },
+  "workflow/jev_bench_tr/civil_comments": {
+   "n": 2000,
+   "accuracy": 0.9375,
+   "brier": 0.0852,
+   "nll": 0.326,
+   "ece": 0.1395
+  },
+  "workflow/jev_bench_tr/clinc150": {
+   "n": 1000,
+   "accuracy": 0.877,
+   "brier": 0.2026,
+   "nll": 0.6939,
+   "ece": 0.0488
+  },
+  "workflow/jev_bench_tr/fever_evidence": {
+   "n": 1000,
+   "accuracy": 0.909,
+   "brier": 0.1436,
+   "nll": 0.2527,
+   "ece": 0.0384
+  },
+  "workflow/jev_bench_tr/go_emotions": {
+   "n": 1000,
+   "accuracy": 0.486,
+   "brier": 0.3056,
+   "nll": 2.2315,
+   "ece": 0.077
+  },
+  "workflow/jev_bench_tr/helpsteer2_helpfulness": {
+   "n": 1000,
+   "accuracy": 0.432,
+   "brier": 0.7229,
+   "nll": 1.3629,
+   "ece": 0.191
+  },
+  "workflow/jev_bench_tr/helpsteer2_verbosity": {
+   "n": 1000,
+   "accuracy": 0.668,
+   "brier": 0.4887,
+   "nll": 0.8986,
+   "ece": 0.1154
+  },
+  "workflow/jev_bench_tr/ledgar": {
+   "n": 1000,
+   "accuracy": 0.694,
+   "brier": 0.4402,
+   "nll": 1.2495,
+   "ece": 0.0355
+  },
+  "workflow/jev_bench_tr/massive": {
+   "n": 1000,
+   "accuracy": 0.832,
+   "brier": 0.2553,
+   "nll": 0.7084,
+   "ece": 0.0477
+  },
+  "workflow/jev_bench_tr/measuring_hate_speech": {
+   "n": 1000,
+   "accuracy": 0.46,
+   "brier": 0.3972,
+   "nll": 1.0576,
+   "ece": 0.1811
+  },
+  "workflow/jev_bench_tr/mmlu": {
+   "n": 1000,
+   "accuracy": 0.631,
+   "brier": 0.4863,
+   "nll": 0.9305,
+   "ece": 0.0703
+  },
+  "workflow/jev_bench_tr/mnli": {
+   "n": 1000,
+   "accuracy": 0.821,
+   "brier": 0.2604,
+   "nll": 0.472,
+   "ece": 0.0469
+  },
+  "workflow/jev_bench_tr/paws": {
+   "n": 1000,
+   "accuracy": 0.665,
+   "brier": 0.4888,
+   "nll": 0.7479,
+   "ece": 0.1876
+  },
+  "workflow/jev_bench_tr/sms_spam": {
+   "n": 800,
+   "accuracy": 0.9762,
+   "brier": 0.0415,
+   "nll": 0.088,
+   "ece": 0.0262
+  },
+  "workflow/jev_bench_tr/sst5": {
+   "n": 1000,
+   "accuracy": 0.477,
+   "brier": 0.624,
+   "nll": 1.1233,
+   "ece": 0.034
+  },
+  "workflow/jev_bench_tr/strategyqa_closed": {
+   "n": 687,
+   "accuracy": 0.5531,
+   "brier": 0.4841,
+   "nll": 0.6782,
+   "ece": 0.0609
+  },
+  "workflow/jev_bench_tr/strategyqa_grounded": {
+   "n": 687,
+   "accuracy": 0.8413,
+   "brier": 0.2337,
+   "nll": 0.3645,
+   "ece": 0.0358
+  },
+  "workflow/jev_bench_tr/stsb": {
+   "n": 1000,
+   "accuracy": 0.509,
+   "brier": 0.6394,
+   "nll": 1.2233,
+   "ece": 0.0989
+  },
+  "workflow/jev_bench_tr/yelp5": {
+   "n": 1000,
+   "accuracy": 0.652,
+   "brier": 0.467,
+   "nll": 0.7862,
+   "ece": 0.0543
+  },
+  "workflow/jev_bench_tr_en/arc_challenge": {
+   "n": 91,
+   "accuracy": 0.9451,
+   "brier": 0.0923,
+   "nll": 0.1834,
+   "ece": 0.0425
+  },
+  "workflow/jev_bench_tr_en/banking77": {
+   "n": 71,
+   "accuracy": 0.8592,
+   "brier": 0.2192,
+   "nll": 0.5844,
+   "ece": 0.104
+  },
+  "workflow/jev_bench_tr_en/boolq": {
+   "n": 80,
+   "accuracy": 0.9125,
+   "brier": 0.1191,
+   "nll": 0.202,
+   "ece": 0.0567
+  },
+  "workflow/jev_bench_tr_en/chaosnli": {
+   "n": 151,
+   "accuracy": 0.6755,
+   "brier": 0.2455,
+   "nll": 1.3287,
+   "ece": 0.1913
+  },
+  "workflow/jev_bench_tr_en/civil_comments": {
+   "n": 171,
+   "accuracy": 0.8538,
+   "brier": 0.1343,
+   "nll": 0.3876,
+   "ece": 0.0761
+  },
+  "workflow/jev_bench_tr_en/clinc150": {
+   "n": 96,
+   "accuracy": 0.9062,
+   "brier": 0.1405,
+   "nll": 0.3832,
+   "ece": 0.0887
+  },
+  "workflow/jev_bench_tr_en/fever_evidence": {
+   "n": 91,
+   "accuracy": 0.956,
+   "brier": 0.0721,
+   "nll": 0.1511,
+   "ece": 0.0373
+  },
+  "workflow/jev_bench_tr_en/go_emotions": {
+   "n": 93,
+   "accuracy": 0.5161,
+   "brier": 0.283,
+   "nll": 2.1674,
+   "ece": 0.1568
+  },
+  "workflow/jev_bench_tr_en/helpsteer2_helpfulness": {
+   "n": 89,
+   "accuracy": 0.427,
+   "brier": 0.6556,
+   "nll": 1.1478,
+   "ece": 0.1033
+  },
+  "workflow/jev_bench_tr_en/helpsteer2_verbosity": {
+   "n": 79,
+   "accuracy": 0.7089,
+   "brier": 0.4528,
+   "nll": 0.8678,
+   "ece": 0.1142
+  },
+  "workflow/jev_bench_tr_en/ledgar": {
+   "n": 70,
+   "accuracy": 0.8,
+   "brier": 0.3384,
+   "nll": 0.8433,
+   "ece": 0.1181
+  },
+  "workflow/jev_bench_tr_en/massive": {
+   "n": 95,
+   "accuracy": 0.8105,
+   "brier": 0.2327,
+   "nll": 0.5822,
+   "ece": 0.0781
+  },
+  "workflow/jev_bench_tr_en/measuring_hate_speech": {
+   "n": 96,
+   "accuracy": 0.5312,
+   "brier": 0.3995,
+   "nll": 1.0836,
+   "ece": 0.1889
+  },
+  "workflow/jev_bench_tr_en/mmlu": {
+   "n": 80,
+   "accuracy": 0.7625,
+   "brier": 0.3231,
+   "nll": 0.603,
+   "ece": 0.1016
+  },
+  "workflow/jev_bench_tr_en/mnli": {
+   "n": 92,
+   "accuracy": 0.9457,
+   "brier": 0.0929,
+   "nll": 0.1728,
+   "ece": 0.0731
+  },
+  "workflow/jev_bench_tr_en/paws": {
+   "n": 80,
+   "accuracy": 0.825,
+   "brier": 0.2561,
+   "nll": 0.413,
+   "ece": 0.0911
+  },
+  "workflow/jev_bench_tr_en/sms_spam": {
+   "n": 82,
+   "accuracy": 0.9634,
+   "brier": 0.0418,
+   "nll": 0.0741,
+   "ece": 0.0379
+  },
+  "workflow/jev_bench_tr_en/sst5": {
+   "n": 73,
+   "accuracy": 0.589,
+   "brier": 0.5294,
+   "nll": 0.8771,
+   "ece": 0.1161
+  },
+  "workflow/jev_bench_tr_en/strategyqa_closed": {
+   "n": 60,
+   "accuracy": 0.8,
+   "brier": 0.3278,
+   "nll": 0.5048,
+   "ece": 0.0974
+  },
+  "workflow/jev_bench_tr_en/strategyqa_grounded": {
+   "n": 71,
+   "accuracy": 0.9014,
+   "brier": 0.1506,
+   "nll": 0.2501,
+   "ece": 0.0844
+  },
+  "workflow/jev_bench_tr_en/stsb": {
+   "n": 90,
+   "accuracy": 0.5556,
+   "brier": 0.5762,
+   "nll": 1.0837,
+   "ece": 0.167
+  },
+  "workflow/jev_bench_tr_en/yelp5": {
+   "n": 99,
+   "accuracy": 0.7071,
+   "brier": 0.3851,
+   "nll": 0.6587,
+   "ece": 0.1091
+  },
+  "workflow/massive_tr": {
+   "n": 2974,
+   "accuracy": 0.8285,
+   "brier": 0.2533,
+   "nll": 0.7069,
+   "ece": 0.0362
+  },
+  "workflow/squad_tr": {
+   "n": 11873,
+   "accuracy": 0.6832,
+   "brier": 0.4016,
+   "nll": 0.585,
+   "ece": 0.0488
+  },
+  "workflow/xnli_tr": {
+   "n": 5010,
+   "accuracy": 0.8128,
+   "brier": 0.2741,
+   "nll": 0.4991,
+   "ece": 0.0356
+  },
+  "squad_tr/answerability": {
+   "auroc": 0.8093,
+   "acc_answerable": 0.9178,
+   "acc_unanswerable": 0.439
+  }
+ }
+}

training/export.json

@@ -0,0 +1,30 @@
+{
+ "train_key": "411bcda4c97e6b5b",
+ "calibration": {
+  "temperature_by_type": {
+   "choice": 0.936,
+   "noul": 1.146,
+   "score": 1.093
+  },
+  "temperature": 1.051,
+  "rows": {
+   "choice": 638,
+   "noul": 1024,
+   "score": 456
+  },
+  "nll": {
+   "choice": {
+    "T=1": 0.3116,
+    "fitted": 0.3107
+   },
+   "noul": {
+    "T=1": 0.2288,
+    "fitted": 0.2265
+   },
+   "score": {
+    "T=1": 0.6997,
+    "fitted": 0.6972
+   }
+  }
+ }
+}

training/fp8.log

@@ -0,0 +1,115 @@
+decider-4b-v2.1-tr-lora on 10,000 rows (13,371 decisions)
+256 / 10,000 rows  14s  accuracy so far 0.778
+512 / 10,000 rows  28s  accuracy so far 0.774
+768 / 10,000 rows  38s  accuracy so far 0.788
+1,024 / 10,000 rows  52s  accuracy so far 0.787
+1,280 / 10,000 rows  62s  accuracy so far 0.787
+1,536 / 10,000 rows  75s  accuracy so far 0.787
+1,792 / 10,000 rows  89s  accuracy so far 0.792
+2,048 / 10,000 rows  101s  accuracy so far 0.793
+2,304 / 10,000 rows  111s  accuracy so far 0.790
+2,560 / 10,000 rows  122s  accuracy so far 0.789
+2,816 / 10,000 rows  133s  accuracy so far 0.790
+3,072 / 10,000 rows  143s  accuracy so far 0.790
+3,328 / 10,000 rows  155s  accuracy so far 0.790
+3,584 / 10,000 rows  167s  accuracy so far 0.794
+3,840 / 10,000 rows  178s  accuracy so far 0.798
+4,096 / 10,000 rows  189s  accuracy so far 0.798
+4,352 / 10,000 rows  202s  accuracy so far 0.797
+4,608 / 10,000 rows  216s  accuracy so far 0.795
+4,864 / 10,000 rows  226s  accuracy so far 0.795
+5,120 / 10,000 rows  237s  accuracy so far 0.796
+5,376 / 10,000 rows  252s  accuracy so far 0.797
+5,632 / 10,000 rows  265s  accuracy so far 0.797
+5,888 / 10,000 rows  277s  accuracy so far 0.797
+6,144 / 10,000 rows  290s  accuracy so far 0.797
+6,400 / 10,000 rows  300s  accuracy so far 0.797
+6,656 / 10,000 rows  313s  accuracy so far 0.797
+6,912 / 10,000 rows  326s  accuracy so far 0.799
+7,168 / 10,000 rows  337s  accuracy so far 0.798
+7,424 / 10,000 rows  348s  accuracy so far 0.797
+7,680 / 10,000 rows  361s  accuracy so far 0.795
+7,936 / 10,000 rows  373s  accuracy so far 0.795
+8,192 / 10,000 rows  382s  accuracy so far 0.796
+8,448 / 10,000 rows  393s  accuracy so far 0.796
+8,704 / 10,000 rows  404s  accuracy so far 0.796
+8,960 / 10,000 rows  414s  accuracy so far 0.796
+9,216 / 10,000 rows  427s  accuracy so far 0.796
+9,472 / 10,000 rows  437s  accuracy so far 0.797
+9,728 / 10,000 rows  449s  accuracy so far 0.797
+9,984 / 10,000 rows  461s  accuracy so far 0.797
+10,000 / 10,000 rows  462s  accuracy so far 0.797
+all {'n': 13371, 'accuracy': 0.7968, 'brier': 0.2639, 'nll': 0.5223, 'ece': 0.0213}
+source/bev_tr {'n': 6122, 'accuracy': 0.8383, 'brier': 0.2217, 'nll': 0.3967, 'ece': 0.0104}
+source/bev_tr_en {'n': 852, 'accuracy': 0.777, 'brier': 0.2821, 'nll': 0.5, 'ece': 0.0313}
+source/beyazperde_tr {'n': 631, 'accuracy': 0.6466, 'brier': 0.4678, 'nll': 0.8605, 'ece': 0.0735}
+source/jev_bench_tr {'n': 2986, 'accuracy': 0.7029, 'brier': 0.3414, 'nll': 0.8115, 'ece': 0.0586}
+source/jev_bench_tr_en {'n': 278, 'accuracy': 0.7662, 'brier': 0.2858, 'nll': 0.7261, 'ece': 0.0676}
+source/massive_tr {'n': 416, 'accuracy': 0.8894, 'brier': 0.158, 'nll': 0.3759, 'ece': 0.0261}
+source/squad_tr {'n': 1454, 'accuracy': 0.8583, 'brier': 0.2133, 'nll': 0.3509, 'ece': 0.0283}
+source/xnli_tr {'n': 632, 'accuracy': 0.8259, 'brier': 0.2537, 'nll': 0.4652, 'ece': 0.0423}
+type/choice {'n': 5197, 'accuracy': 0.8078, 'brier': 0.2407, 'nll': 0.5551, 'ece': 0.0268}
+type/noul {'n': 5216, 'accuracy': 0.8735, 'brier': 0.1772, 'nll': 0.2959, 'ece': 0.0192}
+type/score {'n': 2958, 'accuracy': 0.6423, 'brier': 0.4575, 'nll': 0.8638, 'ece': 0.0284}
+squad_tr/answerability {'auroc': 0.9316, 'acc_answerable': 0.9371, 'acc_unanswerable': 0.7759}
+skipped decisions: {}
+FP8: 200 linear layers
+decider-4b-v2.1-tr-lora on 10,000 rows (13,371 decisions)
+256 / 10,000 rows  14s  accuracy so far 0.783
+512 / 10,000 rows  27s  accuracy so far 0.779
+768 / 10,000 rows  37s  accuracy so far 0.790
+1,024 / 10,000 rows  51s  accuracy so far 0.789
+1,280 / 10,000 rows  61s  accuracy so far 0.786
+1,536 / 10,000 rows  74s  accuracy so far 0.786
+1,792 / 10,000 rows  87s  accuracy so far 0.794
+2,048 / 10,000 rows  99s  accuracy so far 0.792
+2,304 / 10,000 rows  109s  accuracy so far 0.789
+2,560 / 10,000 rows  119s  accuracy so far 0.787
+2,816 / 10,000 rows  130s  accuracy so far 0.789
+3,072 / 10,000 rows  140s  accuracy so far 0.788
+3,328 / 10,000 rows  151s  accuracy so far 0.788
+3,584 / 10,000 rows  162s  accuracy so far 0.793
+3,840 / 10,000 rows  173s  accuracy so far 0.796
+4,096 / 10,000 rows  184s  accuracy so far 0.798
+4,352 / 10,000 rows  196s  accuracy so far 0.796
+4,608 / 10,000 rows  209s  accuracy so far 0.796
+4,864 / 10,000 rows  219s  accuracy so far 0.796
+5,120 / 10,000 rows  230s  accuracy so far 0.796
+5,376 / 10,000 rows  244s  accuracy so far 0.797
+5,632 / 10,000 rows  257s  accuracy so far 0.798
+5,888 / 10,000 rows  268s  accuracy so far 0.797
+6,144 / 10,000 rows  281s  accuracy so far 0.797
+6,400 / 10,000 rows  291s  accuracy so far 0.797
+6,656 / 10,000 rows  303s  accuracy so far 0.797
+6,912 / 10,000 rows  316s  accuracy so far 0.798
+7,168 / 10,000 rows  326s  accuracy so far 0.797
+7,424 / 10,000 rows  337s  accuracy so far 0.796
+7,680 / 10,000 rows  349s  accuracy so far 0.795
+7,936 / 10,000 rows  360s  accuracy so far 0.794
+8,192 / 10,000 rows  368s  accuracy so far 0.796
+8,448 / 10,000 rows  379s  accuracy so far 0.795
+8,704 / 10,000 rows  390s  accuracy so far 0.796
+8,960 / 10,000 rows  400s  accuracy so far 0.796
+9,216 / 10,000 rows  412s  accuracy so far 0.795
+9,472 / 10,000 rows  422s  accuracy so far 0.796
+9,728 / 10,000 rows  434s  accuracy so far 0.797
+9,984 / 10,000 rows  446s  accuracy so far 0.796
+10,000 / 10,000 rows  447s  accuracy so far 0.796
+all {'n': 13371, 'accuracy': 0.7965, 'brier': 0.2645, 'nll': 0.5237, 'ece': 0.0209}
+source/bev_tr {'n': 6122, 'accuracy': 0.8381, 'brier': 0.2217, 'nll': 0.3972, 'ece': 0.0112}
+source/bev_tr_en {'n': 852, 'accuracy': 0.7782, 'brier': 0.2807, 'nll': 0.497, 'ece': 0.0288}
+source/beyazperde_tr {'n': 631, 'accuracy': 0.6403, 'brier': 0.4695, 'nll': 0.8628, 'ece': 0.0631}
+source/jev_bench_tr {'n': 2986, 'accuracy': 0.7036, 'brier': 0.3436, 'nll': 0.8157, 'ece': 0.0604}
+source/jev_bench_tr_en {'n': 278, 'accuracy': 0.7662, 'brier': 0.2867, 'nll': 0.7303, 'ece': 0.0868}
+source/massive_tr {'n': 416, 'accuracy': 0.8918, 'brier': 0.1612, 'nll': 0.3838, 'ece': 0.0236}
+source/squad_tr {'n': 1454, 'accuracy': 0.8569, 'brier': 0.2134, 'nll': 0.351, 'ece': 0.0252}
+source/xnli_tr {'n': 632, 'accuracy': 0.8244, 'brier': 0.2548, 'nll': 0.4658, 'ece': 0.0368}
+type/choice {'n': 5197, 'accuracy': 0.8064, 'brier': 0.2418, 'nll': 0.5571, 'ece': 0.0287}
+type/noul {'n': 5216, 'accuracy': 0.8752, 'brier': 0.1774, 'nll': 0.297, 'ece': 0.0203}
+type/score {'n': 2958, 'accuracy': 0.6403, 'brier': 0.458, 'nll': 0.865, 'ece': 0.0274}
+squad_tr/answerability {'auroc': 0.9318, 'acc_answerable': 0.9357, 'acc_unanswerable': 0.7732}
+skipped decisions: {}
+Installed 57 packages in 342ms
+{"fp8": false, "fp8_layers": 0, "requests": 300, "decisions": 432, "latency_ms_p50": 23.5, "latency_ms_p90": 83.2, "decisions_per_s": 4.0, "gpu": "NVIDIA GeForce RTX 5090"}
+{"fp8": true, "fp8_layers": 200, "requests": 300, "decisions": 432, "latency_ms_p50": 14.7, "latency_ms_p90": 90.2, "decisions_per_s": 5.3, "gpu": "NVIDIA GeForce RTX 5090"}
+FP8 DONE

training/meta.json

@@ -0,0 +1,21 @@
+{
+ "key": "435f5337f79d0547",
+ "train": {
+  "rows": 63990,
+  "items": 169657,
+  "replay_items": 22217,
+  "tokens": 56858516,
+  "tokens_p50": 218,
+  "tokens_max": 4163,
+  "skipped": {}
+ },
+ "holdout": {
+  "rows": 1500,
+  "items": 3826,
+  "replay_items": 0,
+  "tokens": 1277818,
+  "tokens_p50": 226,
+  "tokens_max": 4163,
+  "skipped": {}
+ }
+}

training/train_log.jsonl

@@ -0,0 +1 @@
+{"epoch_end": 1, "val_rows": 3826, "val_slot_accuracy": 0.8907, "val_ce": 0.2603}