05:11:09 hotpot_tr: 18,202 of 18,202 rows 05:11:09 musique_tr: 6,068 of 6,068 rows 05:11:09 replay_bev_tr: 4,000 of 4,000 rows 05:11:09 replay_bev_tr_en: 3,000 of 3,000 rows 05:11:09 squad_tr: 12,000 of 12,000 rows 05:11:09 synth_tr: 21,548 of 21,548 rows 05:11:09 wiki2_tr: 24,000 of 24,000 rows 05:11:54 items train: 87,318 rows -> 106,021 prompt rows, 49,019,781 tokens 05:11:55 items holdout: 1,500 rows -> 1,500 prompt rows, 772,790 tokens 05:12:03 LoRA r=64: 121.9M trainable parameters [transformers] `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`. 05:19:07 train epoch=1 step=20 of=660 ce=0.1464 kl=0.0000 acc=0.956 lr=6.06e-05 tok_per_s=3774 eta_h=3.77 05:25:25 train epoch=1 step=40 of=660 ce=0.2318 kl=0.0000 acc=0.906 lr=1.00e-04 tok_per_s=4000 eta_h=3.45 05:31:38 train epoch=1 step=60 of=660 ce=0.0799 kl=0.0000 acc=0.984 lr=9.95e-05 tok_per_s=4065 eta_h=3.26 05:37:58 train epoch=1 step=80 of=660 ce=0.1024 kl=0.2941 acc=0.960 lr=9.86e-05 tok_per_s=4094 eta_h=3.13 05:42:16 checkpoint: step 94 / 660 05:44:06 train epoch=1 step=100 of=660 ce=0.1029 kl=0.0000 acc=0.960 lr=9.72e-05 tok_per_s=4135 eta_h=2.99 05:50:20 train epoch=1 step=120 of=660 ce=0.1264 kl=0.3063 acc=0.973 lr=9.53e-05 tok_per_s=4145 eta_h=2.87 05:56:29 train epoch=1 step=140 of=660 ce=0.0482 kl=0.3015 acc=0.975 lr=9.30e-05 tok_per_s=4173 eta_h=2.75 06:02:38 train epoch=1 step=160 of=660 ce=0.0612 kl=0.3076 acc=0.962 lr=9.02e-05 tok_per_s=4192 eta_h=2.63 06:08:49 train epoch=1 step=180 of=660 ce=0.0780 kl=0.3077 acc=0.969 lr=8.70e-05 tok_per_s=4206 eta_h=2.52 06:12:28 checkpoint: step 192 / 660 06:14:55 train epoch=1 step=200 of=660 ce=0.0092 kl=0.2882 acc=1.000 lr=8.35e-05 tok_per_s=4219 eta_h=2.41 06:21:05 train epoch=1 step=220 of=660 ce=0.0616 kl=0.0000 acc=0.971 lr=7.96e-05 tok_per_s=4229 eta_h=2.30 06:27:14 train epoch=1 step=240 of=660 ce=0.0167 kl=0.2768 acc=1.000 lr=7.54e-05 tok_per_s=4235 eta_h=2.19 06:33:19 train epoch=1 step=260 of=660 ce=0.0510 kl=0.2752 acc=0.987 lr=7.10e-05 tok_per_s=4241 eta_h=2.08 06:39:26 train epoch=1 step=280 of=660 ce=0.0538 kl=0.0000 acc=0.976 lr=6.64e-05 tok_per_s=4250 eta_h=1.98 06:42:30 checkpoint: step 290 / 660 06:45:29 train epoch=1 step=300 of=660 ce=0.0165 kl=0.0000 acc=0.994 lr=6.15e-05 tok_per_s=4257 eta_h=1.87 06:51:38 train epoch=1 step=320 of=660 ce=0.0986 kl=0.0000 acc=0.951 lr=5.66e-05 tok_per_s=4261 eta_h=1.76 06:57:33 train epoch=1 step=340 of=660 ce=0.0847 kl=0.0000 acc=0.975 lr=5.16e-05 tok_per_s=4263 eta_h=1.65 07:03:42 train epoch=1 step=360 of=660 ce=0.0057 kl=0.2554 acc=0.981 lr=4.66e-05 tok_per_s=4264 eta_h=1.55 07:09:45 train epoch=1 step=380 of=660 ce=0.0736 kl=1.0762 acc=0.957 lr=4.16e-05 tok_per_s=4269 eta_h=1.45 07:12:47 checkpoint: step 390 / 660 07:15:44 train epoch=1 step=400 of=660 ce=0.0026 kl=0.2418 acc=1.000 lr=3.68e-05 tok_per_s=4272 eta_h=1.34 07:21:55 train epoch=1 step=420 of=660 ce=0.0876 kl=0.2455 acc=0.985 lr=3.20e-05 tok_per_s=4273 eta_h=1.24 07:28:02 train epoch=1 step=440 of=660 ce=0.0553 kl=0.4678 acc=0.990 lr=2.74e-05 tok_per_s=4274 eta_h=1.13 07:34:11 train epoch=1 step=460 of=660 ce=0.0090 kl=0.3076 acc=0.963 lr=2.31e-05 tok_per_s=4276 eta_h=1.03 07:40:18 train epoch=1 step=480 of=660 ce=0.0804 kl=0.0000 acc=0.981 lr=1.90e-05 tok_per_s=4278 eta_h=0.93 07:42:49 checkpoint: step 488 / 660 07:46:26 train epoch=1 step=500 of=660 ce=0.0376 kl=0.0000 acc=0.987 lr=1.52e-05 tok_per_s=4280 eta_h=0.82 07:52:30 train epoch=1 step=520 of=660 ce=0.0204 kl=0.0000 acc=1.000 lr=1.18e-05 tok_per_s=4284 eta_h=0.72 07:58:29 train epoch=1 step=540 of=660 ce=0.0495 kl=0.0000 acc=0.990 lr=8.77e-06 tok_per_s=4287 eta_h=0.62 08:04:35 train epoch=1 step=560 of=660 ce=0.0728 kl=0.2521 acc=0.989 lr=6.15e-06 tok_per_s=4291 eta_h=0.51 08:10:43 train epoch=1 step=580 of=660 ce=0.0332 kl=0.0000 acc=0.988 lr=3.96e-06 tok_per_s=4293 eta_h=0.41 08:12:57 checkpoint: step 587 / 660 08:16:58 train epoch=1 step=600 of=660 ce=0.0545 kl=0.2792 acc=0.978 lr=2.24e-06 tok_per_s=4293 eta_h=0.31 08:23:00 train epoch=1 step=620 of=660 ce=0.0631 kl=0.6144 acc=0.965 lr=1.00e-06 tok_per_s=4292 eta_h=0.21 08:29:10 train epoch=1 step=640 of=660 ce=0.0139 kl=0.4028 acc=0.994 lr=2.51e-07 tok_per_s=4292 eta_h=0.10 08:35:16 train epoch=1 step=660 of=660 ce=0.0369 kl=0.0000 acc=0.987 lr=0.00e+00 tok_per_s=4294 eta_h=0.00 08:36:11 validation epoch_end=1 val_rows=1500 val_slot_accuracy=0.9813 val_ce=0.0555 08:36:11 checkpoint: step 660 / 660, training complete 08:37:11 temperatures on 1,500 held-out answers: {'noul': 1.097} (pooled 1.097) 08:37:11 exported: answerability_run/model (load with Decider('answerability_run/model')) decider-4b-v2.1-tr-lora-tr-lora on 16,243 rows (16,243 decisions) 256 / 16,243 rows 8s accuracy so far 0.871 512 / 16,243 rows 16s accuracy so far 0.879 768 / 16,243 rows 24s accuracy so far 0.872 1,024 / 16,243 rows 32s accuracy so far 0.883 1,280 / 16,243 rows 40s accuracy so far 0.885 1,536 / 16,243 rows 48s accuracy so far 0.889 1,792 / 16,243 rows 56s accuracy so far 0.884 2,048 / 16,243 rows 64s accuracy so far 0.882 2,304 / 16,243 rows 72s accuracy so far 0.884 2,560 / 16,243 rows 80s accuracy so far 0.881 2,816 / 16,243 rows 88s accuracy so far 0.881 3,072 / 16,243 rows 95s accuracy so far 0.882 3,328 / 16,243 rows 103s accuracy so far 0.884 3,584 / 16,243 rows 111s accuracy so far 0.884 3,840 / 16,243 rows 119s accuracy so far 0.885 4,096 / 16,243 rows 127s accuracy so far 0.885 4,352 / 16,243 rows 135s accuracy so far 0.887 4,608 / 16,243 rows 143s accuracy so far 0.888 4,864 / 16,243 rows 151s accuracy so far 0.889 5,120 / 16,243 rows 159s accuracy so far 0.890 5,376 / 16,243 rows 167s accuracy so far 0.889 5,632 / 16,243 rows 174s accuracy so far 0.889 5,888 / 16,243 rows 182s accuracy so far 0.891 6,144 / 16,243 rows 190s accuracy so far 0.891 6,400 / 16,243 rows 197s accuracy so far 0.891 6,656 / 16,243 rows 205s accuracy so far 0.890 6,912 / 16,243 rows 213s accuracy so far 0.891 7,168 / 16,243 rows 221s accuracy so far 0.892 7,424 / 16,243 rows 229s accuracy so far 0.891 7,680 / 16,243 rows 237s accuracy so far 0.891 7,936 / 16,243 rows 244s accuracy so far 0.890 8,192 / 16,243 rows 253s accuracy so far 0.890 8,448 / 16,243 rows 260s accuracy so far 0.889 8,704 / 16,243 rows 268s accuracy so far 0.889 8,960 / 16,243 rows 276s accuracy so far 0.890 9,216 / 16,243 rows 284s accuracy so far 0.891 9,472 / 16,243 rows 292s accuracy so far 0.891 9,728 / 16,243 rows 300s accuracy so far 0.892 9,984 / 16,243 rows 307s accuracy so far 0.892 10,240 / 16,243 rows 316s accuracy so far 0.892 10,496 / 16,243 rows 324s accuracy so far 0.892 10,752 / 16,243 rows 332s accuracy so far 0.892 11,008 / 16,243 rows 339s accuracy so far 0.891 11,264 / 16,243 rows 347s accuracy so far 0.892 11,520 / 16,243 rows 355s accuracy so far 0.892 11,776 / 16,243 rows 363s accuracy so far 0.892 12,032 / 16,243 rows 371s accuracy so far 0.892 12,288 / 16,243 rows 379s accuracy so far 0.892 12,544 / 16,243 rows 386s accuracy so far 0.892 12,800 / 16,243 rows 394s accuracy so far 0.892 13,056 / 16,243 rows 403s accuracy so far 0.892 13,312 / 16,243 rows 411s accuracy so far 0.893 13,568 / 16,243 rows 418s accuracy so far 0.893 13,824 / 16,243 rows 426s accuracy so far 0.893 14,080 / 16,243 rows 434s accuracy so far 0.893 14,336 / 16,243 rows 441s accuracy so far 0.893 14,592 / 16,243 rows 449s accuracy so far 0.894 14,848 / 16,243 rows 458s accuracy so far 0.894 15,104 / 16,243 rows 466s accuracy so far 0.893 15,360 / 16,243 rows 474s accuracy so far 0.893 15,616 / 16,243 rows 482s accuracy so far 0.893 15,872 / 16,243 rows 490s accuracy so far 0.893 16,128 / 16,243 rows 498s accuracy so far 0.893 16,243 / 16,243 rows 502s accuracy so far 0.893 all {'n': 16243, 'accuracy': 0.8931, 'brier': 0.1653, 'nll': 0.2898, 'ece': 0.0426} source/hotpot_tr {'n': 1500, 'accuracy': 0.98, 'brier': 0.032, 'nll': 0.0605, 'ece': 0.0063} source/musique_tr {'n': 500, 'accuracy': 0.898, 'brier': 0.1518, 'nll': 0.2529, 'ece': 0.0296} source/squad_tr {'n': 11873, 'accuracy': 0.8614, 'brier': 0.2146, 'nll': 0.3759, 'ece': 0.0572} source/synth_tr {'n': 1170, 'accuracy': 0.9923, 'brier': 0.0113, 'nll': 0.023, 'ece': 0.0038} source/wiki2_tr {'n': 1200, 'accuracy': 1.0, 'brier': 0.0001, 'nll': 0.0006, 'ece': 0.0006} type/noul {'n': 16243, 'accuracy': 0.8931, 'brier': 0.1653, 'nll': 0.2898, 'ece': 0.0426} hotpot_tr/answerability {'n': 1500, 'acc_answerable': 0.97, 'acc_unanswerable': 0.984, 'auroc': 0.997} musique_tr/answerability {'n': 500, 'acc_answerable': 0.832, 'acc_unanswerable': 0.972, 'auroc': 0.9783} squad_tr/answerability {'n': 11873, 'acc_answerable': 0.9239, 'acc_unanswerable': 0.7978, 'auroc': 0.9346} synth_tr/answerability {'n': 1170, 'acc_answerable': 0.983, 'acc_unanswerable': 0.9963, 'auroc': 0.9996} wiki2_tr/answerability {'n': 1200, 'acc_answerable': 1.0, 'acc_unanswerable': 1.0, 'auroc': 1.0} skipped decisions: {} 08:45:39 eval: accuracy 0.8931, ECE 0.0426 (answerability_run/eval_report.json) 08:46:52 pushed: https://huggingface.co/hayriyigit/decider-4b-tr-rag (8fe5d78d3bb3) 08:46:52 done EXIT 0