| 05:11:09 hotpot_tr: 18,202 of 18,202 rows |
| 05:11:09 musique_tr: 6,068 of 6,068 rows |
| 05:11:09 replay_bev_tr: 4,000 of 4,000 rows |
| 05:11:09 replay_bev_tr_en: 3,000 of 3,000 rows |
| 05:11:09 squad_tr: 12,000 of 12,000 rows |
| 05:11:09 synth_tr: 21,548 of 21,548 rows |
| 05:11:09 wiki2_tr: 24,000 of 24,000 rows |
| 05:11:54 items train: 87,318 rows -> 106,021 prompt rows, 49,019,781 tokens |
| 05:11:55 items holdout: 1,500 rows -> 1,500 prompt rows, 772,790 tokens |
| 05:12:03 LoRA r=64: 121.9M trainable parameters |
| [transformers] `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`. |
| 05:19:07 train epoch=1 step=20 of=660 ce=0.1464 kl=0.0000 acc=0.956 lr=6.06e-05 tok_per_s=3774 eta_h=3.77 |
| 05:25:25 train epoch=1 step=40 of=660 ce=0.2318 kl=0.0000 acc=0.906 lr=1.00e-04 tok_per_s=4000 eta_h=3.45 |
| 05:31:38 train epoch=1 step=60 of=660 ce=0.0799 kl=0.0000 acc=0.984 lr=9.95e-05 tok_per_s=4065 eta_h=3.26 |
| 05:37:58 train epoch=1 step=80 of=660 ce=0.1024 kl=0.2941 acc=0.960 lr=9.86e-05 tok_per_s=4094 eta_h=3.13 |
| 05:42:16 checkpoint: step 94 / 660 |
| 05:44:06 train epoch=1 step=100 of=660 ce=0.1029 kl=0.0000 acc=0.960 lr=9.72e-05 tok_per_s=4135 eta_h=2.99 |
| 05:50:20 train epoch=1 step=120 of=660 ce=0.1264 kl=0.3063 acc=0.973 lr=9.53e-05 tok_per_s=4145 eta_h=2.87 |
| 05:56:29 train epoch=1 step=140 of=660 ce=0.0482 kl=0.3015 acc=0.975 lr=9.30e-05 tok_per_s=4173 eta_h=2.75 |
| 06:02:38 train epoch=1 step=160 of=660 ce=0.0612 kl=0.3076 acc=0.962 lr=9.02e-05 tok_per_s=4192 eta_h=2.63 |
| 06:08:49 train epoch=1 step=180 of=660 ce=0.0780 kl=0.3077 acc=0.969 lr=8.70e-05 tok_per_s=4206 eta_h=2.52 |
| 06:12:28 checkpoint: step 192 / 660 |
| 06:14:55 train epoch=1 step=200 of=660 ce=0.0092 kl=0.2882 acc=1.000 lr=8.35e-05 tok_per_s=4219 eta_h=2.41 |
| 06:21:05 train epoch=1 step=220 of=660 ce=0.0616 kl=0.0000 acc=0.971 lr=7.96e-05 tok_per_s=4229 eta_h=2.30 |
| 06:27:14 train epoch=1 step=240 of=660 ce=0.0167 kl=0.2768 acc=1.000 lr=7.54e-05 tok_per_s=4235 eta_h=2.19 |
| 06:33:19 train epoch=1 step=260 of=660 ce=0.0510 kl=0.2752 acc=0.987 lr=7.10e-05 tok_per_s=4241 eta_h=2.08 |
| 06:39:26 train epoch=1 step=280 of=660 ce=0.0538 kl=0.0000 acc=0.976 lr=6.64e-05 tok_per_s=4250 eta_h=1.98 |
| 06:42:30 checkpoint: step 290 / 660 |
| 06:45:29 train epoch=1 step=300 of=660 ce=0.0165 kl=0.0000 acc=0.994 lr=6.15e-05 tok_per_s=4257 eta_h=1.87 |
| 06:51:38 train epoch=1 step=320 of=660 ce=0.0986 kl=0.0000 acc=0.951 lr=5.66e-05 tok_per_s=4261 eta_h=1.76 |
| 06:57:33 train epoch=1 step=340 of=660 ce=0.0847 kl=0.0000 acc=0.975 lr=5.16e-05 tok_per_s=4263 eta_h=1.65 |
| 07:03:42 train epoch=1 step=360 of=660 ce=0.0057 kl=0.2554 acc=0.981 lr=4.66e-05 tok_per_s=4264 eta_h=1.55 |
| 07:09:45 train epoch=1 step=380 of=660 ce=0.0736 kl=1.0762 acc=0.957 lr=4.16e-05 tok_per_s=4269 eta_h=1.45 |
| 07:12:47 checkpoint: step 390 / 660 |
| 07:15:44 train epoch=1 step=400 of=660 ce=0.0026 kl=0.2418 acc=1.000 lr=3.68e-05 tok_per_s=4272 eta_h=1.34 |
| 07:21:55 train epoch=1 step=420 of=660 ce=0.0876 kl=0.2455 acc=0.985 lr=3.20e-05 tok_per_s=4273 eta_h=1.24 |
| 07:28:02 train epoch=1 step=440 of=660 ce=0.0553 kl=0.4678 acc=0.990 lr=2.74e-05 tok_per_s=4274 eta_h=1.13 |
| 07:34:11 train epoch=1 step=460 of=660 ce=0.0090 kl=0.3076 acc=0.963 lr=2.31e-05 tok_per_s=4276 eta_h=1.03 |
| 07:40:18 train epoch=1 step=480 of=660 ce=0.0804 kl=0.0000 acc=0.981 lr=1.90e-05 tok_per_s=4278 eta_h=0.93 |
| 07:42:49 checkpoint: step 488 / 660 |
| 07:46:26 train epoch=1 step=500 of=660 ce=0.0376 kl=0.0000 acc=0.987 lr=1.52e-05 tok_per_s=4280 eta_h=0.82 |
| 07:52:30 train epoch=1 step=520 of=660 ce=0.0204 kl=0.0000 acc=1.000 lr=1.18e-05 tok_per_s=4284 eta_h=0.72 |
| 07:58:29 train epoch=1 step=540 of=660 ce=0.0495 kl=0.0000 acc=0.990 lr=8.77e-06 tok_per_s=4287 eta_h=0.62 |
| 08:04:35 train epoch=1 step=560 of=660 ce=0.0728 kl=0.2521 acc=0.989 lr=6.15e-06 tok_per_s=4291 eta_h=0.51 |
| 08:10:43 train epoch=1 step=580 of=660 ce=0.0332 kl=0.0000 acc=0.988 lr=3.96e-06 tok_per_s=4293 eta_h=0.41 |
| 08:12:57 checkpoint: step 587 / 660 |
| 08:16:58 train epoch=1 step=600 of=660 ce=0.0545 kl=0.2792 acc=0.978 lr=2.24e-06 tok_per_s=4293 eta_h=0.31 |
| 08:23:00 train epoch=1 step=620 of=660 ce=0.0631 kl=0.6144 acc=0.965 lr=1.00e-06 tok_per_s=4292 eta_h=0.21 |
| 08:29:10 train epoch=1 step=640 of=660 ce=0.0139 kl=0.4028 acc=0.994 lr=2.51e-07 tok_per_s=4292 eta_h=0.10 |
| 08:35:16 train epoch=1 step=660 of=660 ce=0.0369 kl=0.0000 acc=0.987 lr=0.00e+00 tok_per_s=4294 eta_h=0.00 |
| 08:36:11 validation epoch_end=1 val_rows=1500 val_slot_accuracy=0.9813 val_ce=0.0555 |
| 08:36:11 checkpoint: step 660 / 660, training complete |
| 08:37:11 temperatures on 1,500 held-out answers: {'noul': 1.097} (pooled 1.097) |
| 08:37:11 exported: answerability_run/model (load with Decider('answerability_run/model')) |
| decider-4b-v2.1-tr-lora-tr-lora on 16,243 rows (16,243 decisions) |
| 256 / 16,243 rows 8s accuracy so far 0.871 |
| 512 / 16,243 rows 16s accuracy so far 0.879 |
| 768 / 16,243 rows 24s accuracy so far 0.872 |
| 1,024 / 16,243 rows 32s accuracy so far 0.883 |
| 1,280 / 16,243 rows 40s accuracy so far 0.885 |
| 1,536 / 16,243 rows 48s accuracy so far 0.889 |
| 1,792 / 16,243 rows 56s accuracy so far 0.884 |
| 2,048 / 16,243 rows 64s accuracy so far 0.882 |
| 2,304 / 16,243 rows 72s accuracy so far 0.884 |
| 2,560 / 16,243 rows 80s accuracy so far 0.881 |
| 2,816 / 16,243 rows 88s accuracy so far 0.881 |
| 3,072 / 16,243 rows 95s accuracy so far 0.882 |
| 3,328 / 16,243 rows 103s accuracy so far 0.884 |
| 3,584 / 16,243 rows 111s accuracy so far 0.884 |
| 3,840 / 16,243 rows 119s accuracy so far 0.885 |
| 4,096 / 16,243 rows 127s accuracy so far 0.885 |
| 4,352 / 16,243 rows 135s accuracy so far 0.887 |
| 4,608 / 16,243 rows 143s accuracy so far 0.888 |
| 4,864 / 16,243 rows 151s accuracy so far 0.889 |
| 5,120 / 16,243 rows 159s accuracy so far 0.890 |
| 5,376 / 16,243 rows 167s accuracy so far 0.889 |
| 5,632 / 16,243 rows 174s accuracy so far 0.889 |
| 5,888 / 16,243 rows 182s accuracy so far 0.891 |
| 6,144 / 16,243 rows 190s accuracy so far 0.891 |
| 6,400 / 16,243 rows 197s accuracy so far 0.891 |
| 6,656 / 16,243 rows 205s accuracy so far 0.890 |
| 6,912 / 16,243 rows 213s accuracy so far 0.891 |
| 7,168 / 16,243 rows 221s accuracy so far 0.892 |
| 7,424 / 16,243 rows 229s accuracy so far 0.891 |
| 7,680 / 16,243 rows 237s accuracy so far 0.891 |
| 7,936 / 16,243 rows 244s accuracy so far 0.890 |
| 8,192 / 16,243 rows 253s accuracy so far 0.890 |
| 8,448 / 16,243 rows 260s accuracy so far 0.889 |
| 8,704 / 16,243 rows 268s accuracy so far 0.889 |
| 8,960 / 16,243 rows 276s accuracy so far 0.890 |
| 9,216 / 16,243 rows 284s accuracy so far 0.891 |
| 9,472 / 16,243 rows 292s accuracy so far 0.891 |
| 9,728 / 16,243 rows 300s accuracy so far 0.892 |
| 9,984 / 16,243 rows 307s accuracy so far 0.892 |
| 10,240 / 16,243 rows 316s accuracy so far 0.892 |
| 10,496 / 16,243 rows 324s accuracy so far 0.892 |
| 10,752 / 16,243 rows 332s accuracy so far 0.892 |
| 11,008 / 16,243 rows 339s accuracy so far 0.891 |
| 11,264 / 16,243 rows 347s accuracy so far 0.892 |
| 11,520 / 16,243 rows 355s accuracy so far 0.892 |
| 11,776 / 16,243 rows 363s accuracy so far 0.892 |
| 12,032 / 16,243 rows 371s accuracy so far 0.892 |
| 12,288 / 16,243 rows 379s accuracy so far 0.892 |
| 12,544 / 16,243 rows 386s accuracy so far 0.892 |
| 12,800 / 16,243 rows 394s accuracy so far 0.892 |
| 13,056 / 16,243 rows 403s accuracy so far 0.892 |
| 13,312 / 16,243 rows 411s accuracy so far 0.893 |
| 13,568 / 16,243 rows 418s accuracy so far 0.893 |
| 13,824 / 16,243 rows 426s accuracy so far 0.893 |
| 14,080 / 16,243 rows 434s accuracy so far 0.893 |
| 14,336 / 16,243 rows 441s accuracy so far 0.893 |
| 14,592 / 16,243 rows 449s accuracy so far 0.894 |
| 14,848 / 16,243 rows 458s accuracy so far 0.894 |
| 15,104 / 16,243 rows 466s accuracy so far 0.893 |
| 15,360 / 16,243 rows 474s accuracy so far 0.893 |
| 15,616 / 16,243 rows 482s accuracy so far 0.893 |
| 15,872 / 16,243 rows 490s accuracy so far 0.893 |
| 16,128 / 16,243 rows 498s accuracy so far 0.893 |
| 16,243 / 16,243 rows 502s accuracy so far 0.893 |
| all {'n': 16243, 'accuracy': 0.8931, 'brier': 0.1653, 'nll': 0.2898, 'ece': 0.0426} |
| source/hotpot_tr {'n': 1500, 'accuracy': 0.98, 'brier': 0.032, 'nll': 0.0605, 'ece': 0.0063} |
| source/musique_tr {'n': 500, 'accuracy': 0.898, 'brier': 0.1518, 'nll': 0.2529, 'ece': 0.0296} |
| source/squad_tr {'n': 11873, 'accuracy': 0.8614, 'brier': 0.2146, 'nll': 0.3759, 'ece': 0.0572} |
| source/synth_tr {'n': 1170, 'accuracy': 0.9923, 'brier': 0.0113, 'nll': 0.023, 'ece': 0.0038} |
| source/wiki2_tr {'n': 1200, 'accuracy': 1.0, 'brier': 0.0001, 'nll': 0.0006, 'ece': 0.0006} |
| type/noul {'n': 16243, 'accuracy': 0.8931, 'brier': 0.1653, 'nll': 0.2898, 'ece': 0.0426} |
| hotpot_tr/answerability {'n': 1500, 'acc_answerable': 0.97, 'acc_unanswerable': 0.984, 'auroc': 0.997} |
| musique_tr/answerability {'n': 500, 'acc_answerable': 0.832, 'acc_unanswerable': 0.972, 'auroc': 0.9783} |
| squad_tr/answerability {'n': 11873, 'acc_answerable': 0.9239, 'acc_unanswerable': 0.7978, 'auroc': 0.9346} |
| synth_tr/answerability {'n': 1170, 'acc_answerable': 0.983, 'acc_unanswerable': 0.9963, 'auroc': 0.9996} |
| wiki2_tr/answerability {'n': 1200, 'acc_answerable': 1.0, 'acc_unanswerable': 1.0, 'auroc': 1.0} |
| skipped decisions: {} |
| 08:45:39 eval: accuracy 0.8931, ECE 0.0426 (answerability_run/eval_report.json) |
| 08:46:52 pushed: https://huggingface.co/hayriyigit/decider-4b-tr-rag (8fe5d78d3bb3) |
| 08:46:52 done |
| EXIT 0 |
|
|
