2025-11-22 18:00:28,752 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-22 18:00:28,762 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json 2025-11-22 18:00:28,763 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 18:00:28,763 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 2025-11-22 18:00:28,766 - __main__ - INFO - 🎲 Random seed: 42 2025-11-22 18:00:28,766 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-22 18:00:28,767 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 18:00:28,767 - __main__ - INFO - Push strategy: checkpoint 2025-11-22 18:00:28,767 - __main__ - INFO - → Will push after each checkpoint save 2025-11-22 18:00:28,768 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-22 18:00:28,768 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-22 18:00:28,769 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-22 18:00:28,783 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-22 18:00:28,785 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 18:00:29,963 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-22 18:00:30,946 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_180030 2025-11-22 18:00:30,947 - __main__ - INFO - Run ID: 5daca621f4734a9d83a4026d35c5be11 2025-11-22 18:00:30,948 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 18:00:31,382 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow 2025-11-22 18:00:31,384 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-22 18:00:31,394 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-22 18:00:31,395 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-22 18:00:31,396 - __main__ - INFO - 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 18:00:39,649 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-22 18:00:39,649 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-22 18:00:39,688 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file 2025-11-22 18:00:40,202 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file 2025-11-22 18:00:40,215 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl 2025-11-22 18:00:40,813 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) 2025-11-22 18:00:44,710 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 2025-11-22 18:00:47,919 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 2025-11-22 18:00:48,496 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) 2025-11-22 18:00:48,503 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl 2025-11-22 18:00:53,190 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) 2025-11-22 18:00:57,043 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 2025-11-22 18:01:00,317 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 2025-11-22 18:01:03,440 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 2025-11-22 18:01:06,523 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 2025-11-22 18:01:09,601 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 2025-11-22 18:01:12,744 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 2025-11-22 18:01:15,855 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 2025-11-22 18:01:18,947 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 2025-11-22 18:01:21,980 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 2025-11-22 18:01:25,052 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 2025-11-22 18:01:28,125 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 2025-11-22 18:01:31,191 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 2025-11-22 18:01:34,271 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 2025-11-22 18:01:37,341 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 2025-11-22 18:01:40,328 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 2025-11-22 18:01:43,407 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 2025-11-22 18:01:46,462 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 2025-11-22 18:01:49,521 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 2025-11-22 18:01:52,563 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 2025-11-22 18:01:55,616 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 2025-11-22 18:01:58,692 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 2025-11-22 18:02:01,773 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 2025-11-22 18:02:04,835 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 2025-11-22 18:02:07,850 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 2025-11-22 18:02:10,864 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 2025-11-22 18:02:13,929 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 2025-11-22 18:02:16,981 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 2025-11-22 18:02:20,014 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 2025-11-22 18:02:23,050 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 2025-11-22 18:02:25,312 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) 2025-11-22 18:02:25,370 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 2025-11-22 18:02:25,371 - __main__ - INFO - 📝 Sample data check: 2025-11-22 18:02:25,372 - __main__ - INFO - Sample 1: 2025-11-22 18:02:25,372 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... 2025-11-22 18:02:25,373 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... 2025-11-22 18:02:25,373 - __main__ - INFO - Sample 2: 2025-11-22 18:02:25,374 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... 2025-11-22 18:02:25,375 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... 2025-11-22 18:02:25,376 - __main__ - INFO - Sample 3: 2025-11-22 18:02:25,378 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... 2025-11-22 18:02:25,378 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience; they are indeed the ones that ... 2025-11-22 18:02:29,507 - __main__ - INFO - Filtering statistics: {'kept': 3190457} 2025-11-22 18:02:29,519 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 2025-11-22 18:02:32,819 - __main__ - INFO - Removed 116 duplicates 2025-11-22 18:02:32,990 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 2025-11-22 18:02:38,144 - __main__ - INFO - ====================================================================== 2025-11-22 18:02:38,144 - __main__ - INFO - ✅ Dataset splits: 2025-11-22 18:02:38,145 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 18:02:38,146 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 18:02:38,146 - __main__ - INFO - Test: 159,518 samples 2025-11-22 18:02:38,147 - __main__ - INFO - ====================================================================== 2025-11-22 18:02:38,518 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-22 18:05:00,120 - __main__ - INFO - ✅ Dataset validation passed 2025-11-22 18:05:00,123 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 18:05:00,123 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 18:05:00,124 - __main__ - INFO - Test: 159,518 samples 2025-11-22 18:05:02,062 - __main__ - INFO - 🔧 Initializing COMET-QE model... 2025-11-22 18:05:02,275 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. 2025-11-22 18:05:03,804 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` 2025-11-22 18:05:10,989 - comet.models.base - INFO - Encoder model frozen. 2025-11-22 18:05:12,932 - __main__ - INFO - ✅ COMET-QE model loaded successfully 2025-11-22 18:05:12,936 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) 2025-11-22 18:05:12,937 - __main__ - INFO - Next run will skip tokenization and load from cache 2025-11-22 18:05:12,938 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... 2025-11-22 18:06:05,795 - __main__ - ERROR - ❌ Pipeline failed: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. Traceback (most recent call last): File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1987, in main trainer_manager = TranslationTrainer(config, model, tokenizer, dataset_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1330, in __init__ self.tokenized_datasets = self.dataset_dict.map( ^^^^^^^^^^^^^^^^^^^^^^ File "/root/venv/lib/python3.12/site-packages/datasets/dataset_dict.py", line 953, in map dataset_dict[split] = dataset.map( ^^^^^^^^^^^^ File "/root/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 562, in wrapper out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3332, in map for rank, done, content in iflatmap_unordered( File "/root/venv/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 619, in iflatmap_unordered raise RuntimeError( RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. 2025-11-22 18:06:07,362 - __main__ - INFO - ✅ MLflow run ended: 5daca621f4734a9d83a4026d35c5be11 2025-11-22 18:09:39,437 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-22 18:09:39,447 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json 2025-11-22 18:09:39,447 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 18:09:39,448 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 2025-11-22 18:09:39,450 - __main__ - INFO - 🎲 Random seed: 42 2025-11-22 18:09:39,451 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-22 18:09:39,451 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 18:09:39,452 - __main__ - INFO - Push strategy: checkpoint 2025-11-22 18:09:39,452 - __main__ - INFO - → Will push after each checkpoint save 2025-11-22 18:09:39,453 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-22 18:09:39,453 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-22 18:09:39,453 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-22 18:09:39,471 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-22 18:09:39,472 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 18:09:40,614 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-22 18:09:41,671 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_180941 2025-11-22 18:09:41,673 - __main__ - INFO - Run ID: e2133803031f4db38199a0e4895e1a60 2025-11-22 18:09:41,674 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 18:09:42,139 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow 2025-11-22 18:09:42,145 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-22 18:09:42,155 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-22 18:09:42,156 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-22 18:09:42,156 - __main__ - INFO - 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 18:09:45,323 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-22 18:09:45,324 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-22 18:09:45,411 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file 2025-11-22 18:09:45,926 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file 2025-11-22 18:09:45,927 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl 2025-11-22 18:09:46,151 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) 2025-11-22 18:09:49,794 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 2025-11-22 18:09:52,995 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 2025-11-22 18:09:53,438 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) 2025-11-22 18:09:53,442 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl 2025-11-22 18:09:54,507 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) 2025-11-22 18:09:57,526 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 2025-11-22 18:10:00,517 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 2025-11-22 18:10:03,537 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 2025-11-22 18:10:06,547 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 2025-11-22 18:10:09,537 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 2025-11-22 18:10:12,525 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 2025-11-22 18:10:15,566 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 2025-11-22 18:10:18,576 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 2025-11-22 18:10:21,624 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 2025-11-22 18:10:24,650 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 2025-11-22 18:10:27,674 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 2025-11-22 18:10:30,691 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 2025-11-22 18:10:33,741 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 2025-11-22 18:10:36,783 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 2025-11-22 18:10:39,789 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 2025-11-22 18:10:42,854 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 2025-11-22 18:10:45,927 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 2025-11-22 18:11:30,196 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-22 18:11:30,219 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json 2025-11-22 18:11:30,220 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 18:11:30,221 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 2025-11-22 18:11:30,226 - __main__ - INFO - 🎲 Random seed: 42 2025-11-22 18:11:30,226 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-22 18:11:30,227 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 18:11:30,227 - __main__ - INFO - Push strategy: checkpoint 2025-11-22 18:11:30,228 - __main__ - INFO - → Will push after each checkpoint save 2025-11-22 18:11:30,228 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-22 18:11:30,228 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-22 18:11:30,229 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-22 18:11:30,248 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-22 18:11:30,248 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 18:11:31,402 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-22 18:11:32,515 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_181131 2025-11-22 18:11:32,516 - __main__ - INFO - Run ID: 1ea1fc3d7e6841139ec6d2af5ac6f07b 2025-11-22 18:11:32,517 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 18:11:32,984 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow 2025-11-22 18:11:32,986 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-22 18:11:32,992 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-22 18:11:32,992 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-22 18:11:32,993 - __main__ - INFO - 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 18:11:35,172 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-22 18:11:35,173 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-22 18:11:35,211 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file 2025-11-22 18:11:35,717 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file 2025-11-22 18:11:35,718 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl 2025-11-22 18:11:35,859 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) 2025-11-22 18:11:39,616 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 2025-11-22 18:11:42,804 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 2025-11-22 18:11:43,278 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) 2025-11-22 18:11:43,285 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl 2025-11-22 18:11:43,482 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) 2025-11-22 18:11:46,608 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 2025-11-22 18:11:49,715 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 2025-11-22 18:11:52,824 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 2025-11-22 18:11:55,973 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 2025-11-22 18:11:59,154 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 2025-11-22 18:12:02,267 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 2025-11-22 18:12:05,413 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 2025-11-22 18:12:08,545 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 2025-11-22 18:12:11,668 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 2025-11-22 18:12:14,810 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 2025-11-22 18:12:17,960 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 2025-11-22 18:12:21,093 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 2025-11-22 18:12:24,516 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 2025-11-22 18:12:27,835 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 2025-11-22 18:12:30,928 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 2025-11-22 18:12:34,075 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 2025-11-22 18:12:37,224 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 2025-11-22 18:12:40,407 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 2025-11-22 18:12:43,527 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 2025-11-22 18:12:46,651 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 2025-11-22 18:12:49,754 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 2025-11-22 18:12:52,862 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 2025-11-22 18:12:55,955 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 2025-11-22 18:12:59,034 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 2025-11-22 18:13:02,074 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 2025-11-22 18:13:05,178 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 2025-11-22 18:13:08,270 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 2025-11-22 18:13:11,352 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 2025-11-22 18:13:14,433 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 2025-11-22 18:13:16,768 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) 2025-11-22 18:13:16,819 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 2025-11-22 18:13:16,819 - __main__ - INFO - 📝 Sample data check: 2025-11-22 18:13:16,820 - __main__ - INFO - Sample 1: 2025-11-22 18:13:16,821 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... 2025-11-22 18:13:16,821 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... 2025-11-22 18:13:16,822 - __main__ - INFO - Sample 2: 2025-11-22 18:13:16,822 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... 2025-11-22 18:13:16,822 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... 2025-11-22 18:13:16,823 - __main__ - INFO - Sample 3: 2025-11-22 18:13:16,823 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... 2025-11-22 18:13:16,824 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience; they are indeed the ones that ... 2025-11-22 18:13:20,795 - __main__ - INFO - Filtering statistics: {'kept': 3190457} 2025-11-22 18:13:20,796 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 2025-11-22 18:13:24,142 - __main__ - INFO - Removed 116 duplicates 2025-11-22 18:13:24,311 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 2025-11-22 18:13:29,401 - __main__ - INFO - ====================================================================== 2025-11-22 18:13:29,418 - __main__ - INFO - ✅ Dataset splits: 2025-11-22 18:13:29,419 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 18:13:29,419 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 18:13:29,420 - __main__ - INFO - Test: 159,518 samples 2025-11-22 18:13:29,421 - __main__ - INFO - ====================================================================== 2025-11-22 18:13:29,781 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-22 18:15:50,493 - __main__ - INFO - ✅ Dataset validation passed 2025-11-22 18:15:50,495 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 18:15:50,496 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 18:15:50,496 - __main__ - INFO - Test: 159,518 samples 2025-11-22 18:15:52,526 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) 2025-11-22 18:15:52,527 - __main__ - INFO - Next run will skip tokenization and load from cache 2025-11-22 18:15:52,528 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... 2025-11-22 18:16:28,894 - __main__ - ERROR - ❌ Pipeline failed: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. Traceback (most recent call last): File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1990, in main trainer_manager = TranslationTrainer(config, model, tokenizer, dataset_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1329, in __init__ self.tokenized_datasets = self.dataset_dict.map( ^^^^^^^^^^^^^^^^^^^^^^ File "/root/venv/lib/python3.12/site-packages/datasets/dataset_dict.py", line 953, in map dataset_dict[split] = dataset.map( ^^^^^^^^^^^^ File "/root/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 562, in wrapper out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3332, in map for rank, done, content in iflatmap_unordered( File "/root/venv/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 619, in iflatmap_unordered raise RuntimeError( RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. 2025-11-22 18:16:30,480 - __main__ - INFO - ✅ MLflow run ended: 1ea1fc3d7e6841139ec6d2af5ac6f07b 2025-11-22 18:32:47,906 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-22 18:32:47,938 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json 2025-11-22 18:32:47,939 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 18:32:47,939 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 2025-11-22 18:32:47,942 - __main__ - INFO - 🎲 Random seed: 42 2025-11-22 18:32:47,943 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-22 18:32:47,943 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 18:32:47,944 - __main__ - INFO - Push strategy: checkpoint 2025-11-22 18:32:47,944 - __main__ - INFO - → Will push after each checkpoint save 2025-11-22 18:32:47,946 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-22 18:32:47,946 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-22 18:32:47,947 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-22 18:32:47,963 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-22 18:32:47,964 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 18:32:48,787 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-22 18:32:50,249 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_183249 2025-11-22 18:32:50,251 - __main__ - INFO - Run ID: 4bd900a24a6e48ac94ba57dbfabfade2 2025-11-22 18:32:50,251 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 18:32:50,617 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow 2025-11-22 18:32:50,621 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-22 18:32:50,637 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-22 18:32:50,638 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-22 18:32:50,639 - __main__ - INFO - 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 18:32:52,949 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-22 18:32:52,950 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-22 18:32:52,990 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file 2025-11-22 18:32:53,498 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file 2025-11-22 18:32:53,499 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl 2025-11-22 18:32:53,780 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) 2025-11-22 18:32:57,511 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 2025-11-22 18:33:00,602 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 2025-11-22 18:33:01,086 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) 2025-11-22 18:33:01,090 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl 2025-11-22 18:33:02,496 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) 2025-11-22 18:33:05,650 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 2025-11-22 18:33:08,753 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 2025-11-22 18:33:11,895 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 2025-11-22 18:33:15,188 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 2025-11-22 18:33:18,337 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 2025-11-22 18:33:21,451 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 2025-11-22 18:33:24,913 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 2025-11-22 18:33:28,019 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 2025-11-22 18:33:31,323 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 2025-11-22 18:33:34,635 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 2025-11-22 18:33:37,935 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 2025-11-22 18:33:41,203 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 2025-11-22 18:33:44,494 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 2025-11-22 18:33:47,922 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 2025-11-22 18:33:51,815 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 2025-11-22 18:33:55,183 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 2025-11-22 18:33:58,359 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 2025-11-22 18:34:01,528 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 2025-11-22 18:34:04,728 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 2025-11-22 18:34:07,921 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 2025-11-22 18:34:11,102 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 2025-11-22 18:34:14,245 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 2025-11-22 18:34:17,409 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 2025-11-22 18:34:20,593 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 2025-11-22 18:34:23,682 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 2025-11-22 18:34:26,823 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 2025-11-22 18:34:29,975 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 2025-11-22 18:34:33,136 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 2025-11-22 18:34:36,287 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 2025-11-22 18:34:38,668 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) 2025-11-22 18:34:38,717 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 2025-11-22 18:34:38,718 - __main__ - INFO - 📝 Sample data check: 2025-11-22 18:34:38,719 - __main__ - INFO - Sample 1: 2025-11-22 18:34:38,720 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... 2025-11-22 18:34:38,721 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... 2025-11-22 18:34:38,721 - __main__ - INFO - Sample 2: 2025-11-22 18:34:38,722 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... 2025-11-22 18:34:38,722 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... 2025-11-22 18:34:38,723 - __main__ - INFO - Sample 3: 2025-11-22 18:34:38,723 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... 2025-11-22 18:34:38,724 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience; they are indeed the ones that ... 2025-11-22 18:34:42,893 - __main__ - INFO - Filtering statistics: {'kept': 3190457} 2025-11-22 18:34:42,894 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 2025-11-22 18:34:46,294 - __main__ - INFO - Removed 116 duplicates 2025-11-22 18:34:46,468 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 2025-11-22 18:34:51,669 - __main__ - INFO - ====================================================================== 2025-11-22 18:34:51,670 - __main__ - INFO - ✅ Dataset splits: 2025-11-22 18:34:51,671 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 18:34:51,671 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 18:34:51,672 - __main__ - INFO - Test: 159,518 samples 2025-11-22 18:34:51,672 - __main__ - INFO - ====================================================================== 2025-11-22 18:34:52,022 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-22 18:37:19,923 - __main__ - INFO - ✅ Dataset validation passed 2025-11-22 18:37:19,926 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 18:37:19,927 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 18:37:19,927 - __main__ - INFO - Test: 159,518 samples 2025-11-22 18:37:21,281 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) 2025-11-22 18:37:21,281 - __main__ - INFO - Next run will skip tokenization and load from cache 2025-11-22 18:37:21,282 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... 2025-11-22 18:37:53,621 - __main__ - INFO - ✅ Tokenization complete! 2025-11-22 18:37:53,623 - __main__ - INFO - 🔧 Initializing COMET-QE model... 2025-11-22 18:37:53,732 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. 2025-11-22 18:37:54,829 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` 2025-11-22 18:38:01,229 - comet.models.base - INFO - Encoder model frozen. 2025-11-22 18:38:01,704 - __main__ - INFO - ✅ COMET-QE model loaded successfully 2025-11-22 18:38:01,723 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ 2025-11-22 18:38:01,863 - __main__ - INFO - ✅ Trainer initialized 2025-11-22 18:38:01,864 - __main__ - INFO - 🔬 STEP 1: Baseline Evaluation 2025-11-22 18:38:02,507 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 2025-11-22 18:38:02,515 - __main__ - INFO - ====================================================================== 2025-11-22 18:38:02,517 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... 2025-11-22 18:38:02,517 - __main__ - INFO - ====================================================================== 2025-11-22 18:38:23,653 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.0198 2025-11-22 18:38:23,662 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-22 18:38:23,715 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... 2025-11-22 18:38:23,717 - __main__ - INFO - Batch 1/16... 2025-11-22 18:38:25,543 - __main__ - INFO - Batch 5/16... 2025-11-22 18:38:27,277 - __main__ - INFO - Batch 10/16... 2025-11-22 18:38:28,933 - __main__ - INFO - Batch 15/16... 2025-11-22 18:38:29,534 - __main__ - INFO - Running COMET-QE model... 2025-11-22 18:38:29,545 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-22 18:38:29,554 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-22 18:38:29,555 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-22 18:38:29,557 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision 2025-11-22 18:38:29,558 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-22 18:38:32,978 - __main__ - INFO - ✅ COMET-QE: 0.0383 ± 0.1441 2025-11-22 18:38:32,979 - __main__ - INFO - ✅ Baseline Results (Test Set): 2025-11-22 18:38:32,980 - __main__ - INFO - BLEU: 0.0198 2025-11-22 18:38:32,981 - __main__ - INFO - chrF: 17.59 2025-11-22 18:38:32,981 - __main__ - INFO - COMET: 0.0383 2025-11-22 18:38:32,982 - __main__ - INFO - ====================================================================== 2025-11-22 18:38:32,982 - __main__ - INFO - ====================================================================== 2025-11-22 18:38:32,983 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-22 18:38:32,983 - __main__ - INFO - ====================================================================== 2025-11-22 18:38:32,983 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-22 18:38:32,987 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-22 18:38:34,322 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-22 18:38:34,394 - __main__ - INFO - ✅ Domain chrF: 0.93 2025-11-22 18:38:34,395 - __main__ - INFO - Computing COMET-QE... 2025-11-22 18:38:34,420 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-22 18:38:34,429 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-22 18:38:34,435 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-22 18:38:34,438 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-22 18:38:36,561 - __main__ - INFO - ✅ Domain COMET-QE: 0.0000 2025-11-22 18:38:36,562 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-22 18:38:36,562 - __main__ - INFO - BLEU: 0.0000 2025-11-22 18:38:36,563 - __main__ - INFO - chrF: 0.93 2025-11-22 18:38:36,563 - __main__ - INFO - COMET-QE: 0.0000 2025-11-22 18:38:36,564 - __main__ - INFO - ====================================================================== 2025-11-22 18:38:36,953 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 2025-11-22 18:38:36,969 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase1/baseline_results.json 2025-11-22 18:38:36,976 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics 2025-11-22 18:38:36,976 - __main__ - INFO - 🚀 STEP 2: Fine-tuning 2025-11-22 18:38:36,977 - __main__ - INFO - Metrics are being logged to MLflow in real-time 2025-11-22 18:38:36,978 - __main__ - INFO - MLflow URL: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 18:38:36,979 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1/logs 2025-11-22 18:38:36,979 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') 2025-11-22 18:38:36,981 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 18:38:36,981 - __main__ - INFO - ====================================================================== 2025-11-22 18:38:36,982 - __main__ - INFO - 🚀 Starting training... 2025-11-22 18:38:36,983 - __main__ - INFO - ====================================================================== 2025-11-22 18:38:38,036 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) 2025-11-22 18:39:02,025 - __main__ - INFO - 📈 Epoch progress: 0.01/8 (0.1%) 2025-11-22 18:39:25,763 - __main__ - INFO - 📈 Epoch progress: 0.02/8 (0.2%) 2025-11-22 18:39:50,193 - __main__ - INFO - 📈 Epoch progress: 0.03/8 (0.4%) 2025-11-22 18:40:14,121 - __main__ - INFO - 📈 Epoch progress: 0.04/8 (0.5%) 2025-11-22 18:40:38,244 - __main__ - INFO - 📈 Epoch progress: 0.05/8 (0.6%) 2025-11-22 18:41:10,169 - __main__ - INFO - 📈 Epoch progress: 0.06/8 (0.7%) 2025-11-22 18:41:40,768 - __main__ - INFO - 📈 Epoch progress: 0.07/8 (0.8%) 2025-11-22 18:42:11,121 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (0.9%) 2025-11-22 18:42:41,739 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (1.1%) 2025-11-22 18:43:12,016 - __main__ - INFO - 📈 Epoch progress: 0.09/8 (1.2%) 2025-11-22 19:10:25,471 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-22 19:10:25,481 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json 2025-11-22 19:10:25,481 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 19:10:25,482 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 2025-11-22 19:10:25,484 - __main__ - INFO - 🎲 Random seed: 42 2025-11-22 19:10:25,484 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-22 19:10:25,485 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 19:10:25,485 - __main__ - INFO - Push strategy: checkpoint 2025-11-22 19:10:25,486 - __main__ - INFO - → Will push after each checkpoint save 2025-11-22 19:10:25,486 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-22 19:10:25,487 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-22 19:10:25,487 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-22 19:10:25,503 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-22 19:10:25,517 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 19:10:26,702 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-22 19:10:27,695 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_191027 2025-11-22 19:10:27,696 - __main__ - INFO - Run ID: bdc7acccde764a5f94d2bfab879011bc 2025-11-22 19:10:27,697 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 19:10:28,059 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow 2025-11-22 19:10:28,062 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-22 19:10:28,072 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-22 19:10:28,073 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-22 19:10:28,073 - __main__ - INFO - 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 19:10:30,734 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-22 19:10:30,735 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-22 19:10:30,774 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file 2025-11-22 19:10:31,285 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file 2025-11-22 19:10:31,289 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl 2025-11-22 19:10:31,550 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) 2025-11-22 19:10:35,181 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 2025-11-22 19:11:10,118 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-22 19:11:10,126 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json 2025-11-22 19:11:10,126 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 19:11:10,127 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 2025-11-22 19:11:10,129 - __main__ - INFO - 🎲 Random seed: 42 2025-11-22 19:11:10,129 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-22 19:11:10,130 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 19:11:10,130 - __main__ - INFO - Push strategy: checkpoint 2025-11-22 19:11:10,131 - __main__ - INFO - → Will push after each checkpoint save 2025-11-22 19:11:10,131 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-22 19:11:10,132 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-22 19:11:10,132 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-22 19:11:10,152 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-22 19:11:10,153 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 19:11:10,578 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-22 19:11:11,452 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_191110 2025-11-22 19:11:11,454 - __main__ - INFO - Run ID: b71909a6bb4441a98279b91c6d4fd466 2025-11-22 19:11:11,454 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 19:11:11,821 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow 2025-11-22 19:11:11,823 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-22 19:11:11,830 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-22 19:11:11,831 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-22 19:11:11,831 - __main__ - INFO - 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 19:11:13,547 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-22 19:11:13,548 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-22 19:11:13,588 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file 2025-11-22 19:11:14,104 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file 2025-11-22 19:11:14,114 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl 2025-11-22 19:11:14,258 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) 2025-11-22 19:11:18,111 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 2025-11-22 19:11:21,459 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 2025-11-22 19:11:21,919 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) 2025-11-22 19:11:21,924 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl 2025-11-22 19:11:23,966 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) 2025-11-22 19:11:27,726 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 2025-11-22 19:11:30,808 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 2025-11-22 19:11:33,880 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 2025-11-22 19:11:36,898 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 2025-11-22 19:11:39,913 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 2025-11-22 19:11:42,928 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 2025-11-22 19:11:45,982 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 2025-11-22 19:11:48,978 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 2025-11-22 19:11:52,010 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 2025-11-22 19:11:55,087 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 2025-11-22 19:11:58,183 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 2025-11-22 19:12:01,268 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 2025-11-22 19:12:04,380 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 2025-11-22 19:12:07,505 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 2025-11-22 19:12:10,549 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 2025-11-22 19:12:13,640 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 2025-11-22 19:12:16,770 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 2025-11-22 19:12:19,953 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 2025-11-22 19:12:23,052 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 2025-11-22 19:12:26,150 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 2025-11-22 19:12:29,224 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 2025-11-22 19:12:32,306 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 2025-11-22 19:12:35,411 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 2025-11-22 19:12:38,474 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 2025-11-22 19:12:41,509 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 2025-11-22 19:12:44,604 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 2025-11-22 19:12:47,696 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 2025-11-22 19:12:50,792 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 2025-11-22 19:12:53,890 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 2025-11-22 19:12:56,211 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) 2025-11-22 19:12:56,262 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 2025-11-22 19:12:56,263 - __main__ - INFO - 📝 Sample data check: 2025-11-22 19:12:56,264 - __main__ - INFO - Sample 1: 2025-11-22 19:12:56,264 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... 2025-11-22 19:12:56,265 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... 2025-11-22 19:12:56,265 - __main__ - INFO - Sample 2: 2025-11-22 19:12:56,266 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... 2025-11-22 19:12:56,266 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... 2025-11-22 19:12:56,267 - __main__ - INFO - Sample 3: 2025-11-22 19:12:56,267 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... 2025-11-22 19:12:56,267 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience; they are indeed the ones that ... 2025-11-22 19:13:00,365 - __main__ - INFO - Filtering statistics: {'kept': 3190457} 2025-11-22 19:13:00,366 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 2025-11-22 19:13:03,687 - __main__ - INFO - Removed 116 duplicates 2025-11-22 19:13:03,864 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 2025-11-22 19:13:09,098 - __main__ - INFO - ====================================================================== 2025-11-22 19:13:09,115 - __main__ - INFO - ✅ Dataset splits: 2025-11-22 19:13:09,117 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 19:13:09,118 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 19:13:09,118 - __main__ - INFO - Test: 159,518 samples 2025-11-22 19:13:09,119 - __main__ - INFO - ====================================================================== 2025-11-22 19:13:09,479 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-22 19:15:30,511 - __main__ - INFO - ✅ Dataset validation passed 2025-11-22 19:15:30,520 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 19:15:30,521 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 19:15:30,522 - __main__ - INFO - Test: 159,518 samples 2025-11-22 19:15:31,951 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) 2025-11-22 19:15:31,952 - __main__ - INFO - Next run will skip tokenization and load from cache 2025-11-22 19:15:31,953 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... 2025-11-22 19:16:05,189 - __main__ - INFO - ✅ Tokenization complete! 2025-11-22 19:16:05,191 - __main__ - INFO - 🔧 Initializing COMET-QE model... 2025-11-22 19:16:05,318 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. 2025-11-22 19:16:06,472 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` 2025-11-22 19:16:13,255 - comet.models.base - INFO - Encoder model frozen. 2025-11-22 19:16:13,713 - __main__ - INFO - ✅ COMET-QE model loaded successfully 2025-11-22 19:16:13,714 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ 2025-11-22 19:16:13,716 - __main__ - ERROR - ❌ Pipeline failed: --load_best_model_at_end requires the saving steps to be a round multiple of the evaluation steps, but found 1000, which is not a round multiple of 2000. Traceback (most recent call last): File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1990, in main trainer_manager = TranslationTrainer(config, model, tokenizer, dataset_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1345, in __init__ self._setup_trainer() File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1371, in _setup_trainer training_args = Seq2SeqTrainingArguments( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 140, in __init__ File "/root/venv/lib/python3.12/site-packages/transformers/training_args.py", line 1709, in __post_init__ raise ValueError( ValueError: --load_best_model_at_end requires the saving steps to be a round multiple of the evaluation steps, but found 1000, which is not a round multiple of 2000. 2025-11-22 19:16:15,287 - __main__ - INFO - ✅ MLflow run ended: b71909a6bb4441a98279b91c6d4fd466 2025-11-22 19:21:07,378 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-22 19:21:07,387 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json 2025-11-22 19:21:07,388 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 19:21:07,388 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 2025-11-22 19:21:07,390 - __main__ - INFO - 🎲 Random seed: 42 2025-11-22 19:21:07,391 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-22 19:21:07,391 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 19:21:07,392 - __main__ - INFO - Push strategy: checkpoint 2025-11-22 19:21:07,392 - __main__ - INFO - → Will push after each checkpoint save 2025-11-22 19:21:07,393 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-22 19:21:07,393 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-22 19:21:07,394 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-22 19:21:07,411 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-22 19:21:07,414 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 19:21:08,151 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-22 19:21:09,160 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_192108 2025-11-22 19:21:09,162 - __main__ - INFO - Run ID: 1832aa44606e4efbaecf3870d9bab685 2025-11-22 19:21:09,162 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 19:21:09,658 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow 2025-11-22 19:21:09,779 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-22 19:21:09,798 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-22 19:21:09,798 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-22 19:21:09,814 - __main__ - INFO - 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 19:21:11,918 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-22 19:21:11,919 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-22 19:21:11,957 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file 2025-11-22 19:21:12,465 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file 2025-11-22 19:21:12,466 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl 2025-11-22 19:21:12,699 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) 2025-11-22 19:21:16,476 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 2025-11-22 19:21:20,071 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 2025-11-22 19:21:20,529 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) 2025-11-22 19:21:20,532 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl 2025-11-22 19:21:22,650 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) 2025-11-22 19:21:25,817 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 2025-11-22 19:21:28,834 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 2025-11-22 19:21:31,885 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 2025-11-22 19:21:34,948 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 2025-11-22 19:21:37,976 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 2025-11-22 19:21:40,997 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 2025-11-22 19:21:44,060 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 2025-11-22 19:21:47,079 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 2025-11-22 19:21:50,055 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 2025-11-22 19:21:53,049 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 2025-11-22 19:21:56,050 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 2025-11-22 19:21:59,055 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 2025-11-22 19:22:02,074 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 2025-11-22 19:22:05,077 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 2025-11-22 19:22:08,050 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 2025-11-22 19:22:11,082 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 2025-11-22 19:22:14,106 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 2025-11-22 19:22:17,112 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 2025-11-22 19:22:20,199 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 2025-11-22 19:22:23,273 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 2025-11-22 19:22:26,310 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 2025-11-22 19:22:29,364 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 2025-11-22 19:22:32,401 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 2025-11-22 19:22:35,454 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 2025-11-22 19:22:38,432 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 2025-11-22 19:22:41,458 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 2025-11-22 19:22:44,469 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 2025-11-22 19:22:47,487 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 2025-11-22 19:22:50,507 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 2025-11-22 19:22:52,804 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) 2025-11-22 19:22:52,874 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 2025-11-22 19:22:52,875 - __main__ - INFO - 📝 Sample data check: 2025-11-22 19:22:52,875 - __main__ - INFO - Sample 1: 2025-11-22 19:22:52,876 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... 2025-11-22 19:22:52,876 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... 2025-11-22 19:22:52,877 - __main__ - INFO - Sample 2: 2025-11-22 19:22:52,877 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... 2025-11-22 19:22:52,878 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... 2025-11-22 19:22:52,879 - __main__ - INFO - Sample 3: 2025-11-22 19:22:52,879 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... 2025-11-22 19:22:52,880 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience; they are indeed the ones that ... 2025-11-22 19:22:57,109 - __main__ - INFO - Filtering statistics: {'kept': 3190457} 2025-11-22 19:22:57,114 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 2025-11-22 19:23:00,518 - __main__ - INFO - Removed 116 duplicates 2025-11-22 19:23:00,691 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 2025-11-22 19:23:05,980 - __main__ - INFO - ====================================================================== 2025-11-22 19:23:05,981 - __main__ - INFO - ✅ Dataset splits: 2025-11-22 19:23:05,982 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 19:23:05,982 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 19:23:05,983 - __main__ - INFO - Test: 159,518 samples 2025-11-22 19:23:05,983 - __main__ - INFO - ====================================================================== 2025-11-22 19:23:06,360 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-22 19:25:35,062 - __main__ - INFO - ✅ Dataset validation passed 2025-11-22 19:25:35,065 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 19:25:35,066 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 19:25:35,067 - __main__ - INFO - Test: 159,518 samples 2025-11-22 19:25:36,537 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) 2025-11-22 19:25:36,538 - __main__ - INFO - Next run will skip tokenization and load from cache 2025-11-22 19:25:36,538 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... 2025-11-22 19:26:10,347 - __main__ - INFO - ✅ Tokenization complete! 2025-11-22 19:26:10,351 - __main__ - INFO - 🔧 Initializing COMET-QE model... 2025-11-22 19:26:10,467 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. 2025-11-22 19:26:11,722 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` 2025-11-22 19:26:19,194 - comet.models.base - INFO - Encoder model frozen. 2025-11-22 19:26:20,010 - __main__ - INFO - ✅ COMET-QE model loaded successfully 2025-11-22 19:26:20,021 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ 2025-11-22 19:26:20,149 - __main__ - INFO - ✅ Trainer initialized 2025-11-22 19:26:20,150 - __main__ - INFO - 🔬 STEP 1: Baseline Evaluation 2025-11-22 19:26:20,766 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 2025-11-22 19:26:20,767 - __main__ - INFO - ====================================================================== 2025-11-22 19:26:20,767 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... 2025-11-22 19:26:20,768 - __main__ - INFO - ====================================================================== 2025-11-22 19:26:42,556 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.0198 2025-11-22 19:26:42,557 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-22 19:26:42,595 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... 2025-11-22 19:26:42,596 - __main__ - INFO - Batch 1/16... 2025-11-22 19:26:44,238 - __main__ - INFO - Batch 5/16... 2025-11-22 19:26:45,863 - __main__ - INFO - Batch 10/16... 2025-11-22 19:26:47,539 - __main__ - INFO - Batch 15/16... 2025-11-22 19:26:48,147 - __main__ - INFO - Running COMET-QE model... 2025-11-22 19:26:48,160 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-22 19:26:48,168 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-22 19:26:48,169 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-22 19:26:48,171 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision 2025-11-22 19:26:48,172 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-22 19:26:51,610 - __main__ - INFO - ✅ COMET-QE: 0.0383 ± 0.1441 2025-11-22 19:26:51,614 - __main__ - INFO - ✅ Baseline Results (Test Set): 2025-11-22 19:26:51,615 - __main__ - INFO - BLEU: 0.0198 2025-11-22 19:26:51,616 - __main__ - INFO - chrF: 17.59 2025-11-22 19:26:51,616 - __main__ - INFO - COMET: 0.0383 2025-11-22 19:26:51,617 - __main__ - INFO - ====================================================================== 2025-11-22 19:26:51,617 - __main__ - INFO - ====================================================================== 2025-11-22 19:26:51,618 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-22 19:26:51,618 - __main__ - INFO - ====================================================================== 2025-11-22 19:26:51,619 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-22 19:26:51,620 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-22 19:26:52,723 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-22 19:26:52,790 - __main__ - INFO - ✅ Domain chrF: 0.93 2025-11-22 19:26:52,797 - __main__ - INFO - Computing COMET-QE... 2025-11-22 19:26:52,823 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-22 19:26:52,831 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-22 19:26:52,831 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-22 19:26:52,833 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-22 19:26:55,743 - __main__ - INFO - ✅ Domain COMET-QE: 0.0000 2025-11-22 19:26:55,744 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-22 19:26:55,744 - __main__ - INFO - BLEU: 0.0000 2025-11-22 19:26:55,745 - __main__ - INFO - chrF: 0.93 2025-11-22 19:26:55,745 - __main__ - INFO - COMET-QE: 0.0000 2025-11-22 19:26:55,746 - __main__ - INFO - ====================================================================== 2025-11-22 19:26:56,216 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 2025-11-22 19:26:56,226 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase1/baseline_results.json 2025-11-22 19:26:56,238 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics 2025-11-22 19:26:56,239 - __main__ - INFO - 🚀 STEP 2: Fine-tuning 2025-11-22 19:26:56,240 - __main__ - INFO - Metrics are being logged to MLflow in real-time 2025-11-22 19:26:56,240 - __main__ - INFO - MLflow URL: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 19:26:56,241 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1/logs 2025-11-22 19:26:56,242 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') 2025-11-22 19:26:56,242 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 19:26:56,243 - __main__ - INFO - ====================================================================== 2025-11-22 19:26:56,244 - __main__ - INFO - 🚀 Starting training... 2025-11-22 19:26:56,244 - __main__ - INFO - ====================================================================== 2025-11-22 19:26:57,393 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) 2025-11-22 19:27:21,616 - __main__ - INFO - 📈 Epoch progress: 0.01/8 (0.1%) 2025-11-22 19:27:45,820 - __main__ - INFO - 📈 Epoch progress: 0.02/8 (0.2%) 2025-11-22 19:28:10,085 - __main__ - INFO - 📈 Epoch progress: 0.03/8 (0.4%) 2025-11-22 19:28:34,780 - __main__ - INFO - 📈 Epoch progress: 0.04/8 (0.5%) 2025-11-22 19:28:59,138 - __main__ - INFO - 📈 Epoch progress: 0.05/8 (0.6%) 2025-11-22 19:29:23,444 - __main__ - INFO - 📈 Epoch progress: 0.06/8 (0.7%) 2025-11-22 19:29:47,282 - __main__ - INFO - 📈 Epoch progress: 0.07/8 (0.8%) 2025-11-22 19:30:10,983 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (0.9%) 2025-11-22 19:30:34,764 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (1.1%) 2025-11-22 19:30:58,732 - __main__ - INFO - 📈 Epoch progress: 0.09/8 (1.2%) 2025-11-22 19:31:22,641 - __main__ - INFO - 📈 Epoch progress: 0.10/8 (1.3%) 2025-11-22 19:31:47,140 - __main__ - INFO - 📈 Epoch progress: 0.11/8 (1.4%) 2025-11-22 19:32:10,785 - __main__ - INFO - 📈 Epoch progress: 0.12/8 (1.5%) 2025-11-22 19:32:34,598 - __main__ - INFO - 📈 Epoch progress: 0.13/8 (1.7%) 2025-11-22 19:32:58,470 - __main__ - INFO - 📈 Epoch progress: 0.14/8 (1.8%) 2025-11-22 19:33:22,516 - __main__ - INFO - 📈 Epoch progress: 0.15/8 (1.9%) 2025-11-22 19:33:46,506 - __main__ - INFO - 📈 Epoch progress: 0.16/8 (2.0%) 2025-11-22 19:34:10,370 - __main__ - INFO - 📈 Epoch progress: 0.17/8 (2.1%) 2025-11-22 19:34:34,008 - __main__ - INFO - 📈 Epoch progress: 0.18/8 (2.2%) 2025-11-22 19:34:57,884 - __main__ - INFO - 📈 Epoch progress: 0.19/8 (2.4%) 2025-11-22 19:54:08,127 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-22 19:54:08,138 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json 2025-11-22 19:54:08,138 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 19:54:08,139 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 2025-11-22 19:54:08,142 - __main__ - INFO - 🎲 Random seed: 42 2025-11-22 19:54:08,143 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-22 19:54:08,144 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 19:54:08,145 - __main__ - INFO - Push strategy: checkpoint 2025-11-22 19:54:08,146 - __main__ - INFO - → Will push after each checkpoint save 2025-11-22 19:54:08,146 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-22 19:54:08,147 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-22 19:54:08,148 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-22 19:54:08,163 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-22 19:54:08,169 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 19:54:08,905 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-22 19:54:09,890 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_195409 2025-11-22 19:54:09,891 - __main__ - INFO - Run ID: 8b5deaed89ec48e986246e5b801158e3 2025-11-22 19:54:09,892 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 19:54:10,331 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow 2025-11-22 19:54:10,334 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-22 19:54:10,345 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-22 19:54:10,346 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-22 19:54:10,347 - __main__ - INFO - 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en 2025-11-22 19:54:13,899 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-22 19:54:13,916 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-22 19:54:13,956 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file 2025-11-22 19:54:14,464 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file 2025-11-22 19:54:14,465 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl 2025-11-22 19:54:14,729 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) 2025-11-22 19:54:18,636 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 2025-11-22 19:54:22,160 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 2025-11-22 19:54:22,658 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) 2025-11-22 19:54:22,663 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl 2025-11-22 19:54:24,643 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) 2025-11-22 19:54:27,963 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 2025-11-22 19:54:31,277 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 2025-11-22 19:54:34,536 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 2025-11-22 19:54:37,749 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 2025-11-22 19:54:40,962 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 2025-11-22 19:54:44,154 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 2025-11-22 19:54:47,382 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 2025-11-22 19:54:50,550 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 2025-11-22 19:54:53,754 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 2025-11-22 19:54:56,995 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 2025-11-22 19:55:00,239 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 2025-11-22 19:55:03,493 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 2025-11-22 19:55:06,729 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 2025-11-22 19:55:09,937 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 2025-11-22 19:55:13,139 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 2025-11-22 19:55:16,411 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 2025-11-22 19:55:19,744 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 2025-11-22 19:55:23,053 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 2025-11-22 19:55:26,373 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 2025-11-22 19:55:29,693 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 2025-11-22 19:55:32,966 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 2025-11-22 19:55:36,255 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 2025-11-22 19:55:39,548 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 2025-11-22 19:55:42,824 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 2025-11-22 19:55:46,080 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 2025-11-22 19:55:49,373 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 2025-11-22 19:55:52,653 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 2025-11-22 19:55:55,930 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 2025-11-22 19:55:59,183 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 2025-11-22 19:56:01,628 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) 2025-11-22 19:56:01,684 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 2025-11-22 19:56:01,686 - __main__ - INFO - 📝 Sample data check: 2025-11-22 19:56:01,687 - __main__ - INFO - Sample 1: 2025-11-22 19:56:01,688 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... 2025-11-22 19:56:01,688 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... 2025-11-22 19:56:01,690 - __main__ - INFO - Sample 2: 2025-11-22 19:56:01,690 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... 2025-11-22 19:56:01,691 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... 2025-11-22 19:56:01,692 - __main__ - INFO - Sample 3: 2025-11-22 19:56:01,693 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... 2025-11-22 19:56:01,693 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience; they are indeed the ones that ... 2025-11-22 19:56:05,923 - __main__ - INFO - Filtering statistics: {'kept': 3190457} 2025-11-22 19:56:05,925 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 2025-11-22 19:56:09,451 - __main__ - INFO - Removed 116 duplicates 2025-11-22 19:56:09,641 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 2025-11-22 19:56:15,040 - __main__ - INFO - ====================================================================== 2025-11-22 19:56:15,041 - __main__ - INFO - ✅ Dataset splits: 2025-11-22 19:56:15,041 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 19:56:15,042 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 19:56:15,042 - __main__ - INFO - Test: 159,518 samples 2025-11-22 19:56:15,043 - __main__ - INFO - ====================================================================== 2025-11-22 19:56:15,413 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-22 19:58:46,518 - __main__ - INFO - ✅ Dataset validation passed 2025-11-22 19:58:46,523 - __main__ - INFO - Train: 2,711,789 samples 2025-11-22 19:58:46,523 - __main__ - INFO - Validation: 319,034 samples 2025-11-22 19:58:46,524 - __main__ - INFO - Test: 159,518 samples 2025-11-22 19:58:48,125 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) 2025-11-22 19:58:48,126 - __main__ - INFO - Next run will skip tokenization and load from cache 2025-11-22 19:58:48,126 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... 2025-11-22 19:59:26,483 - __main__ - INFO - ✅ Tokenization complete! 2025-11-22 19:59:26,485 - __main__ - INFO - 🔧 Initializing COMET-QE model... 2025-11-22 19:59:26,617 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. 2025-11-22 19:59:27,759 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` 2025-11-22 19:59:34,104 - comet.models.base - INFO - Encoder model frozen. 2025-11-22 19:59:34,612 - __main__ - INFO - ✅ COMET-QE model loaded successfully 2025-11-22 19:59:34,618 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ 2025-11-22 19:59:35,142 - __main__ - INFO - 📊 Limiting evaluation dataset from 319046 to 1000 samples 2025-11-22 19:59:35,223 - __main__ - INFO - ✅ Trainer initialized 2025-11-22 19:59:35,230 - __main__ - INFO - 🔬 STEP 1: Baseline Evaluation 2025-11-22 19:59:35,817 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 2025-11-22 19:59:35,819 - __main__ - INFO - ====================================================================== 2025-11-22 19:59:35,820 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... 2025-11-22 19:59:35,820 - __main__ - INFO - ====================================================================== 2025-11-22 19:59:57,876 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.0198 2025-11-22 19:59:57,877 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-22 19:59:57,910 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... 2025-11-22 19:59:57,916 - __main__ - INFO - Batch 1/16... 2025-11-22 19:59:59,612 - __main__ - INFO - Batch 5/16... 2025-11-22 20:00:01,281 - __main__ - INFO - Batch 10/16... 2025-11-22 20:00:02,999 - __main__ - INFO - Batch 15/16... 2025-11-22 20:00:03,636 - __main__ - INFO - Running COMET-QE model... 2025-11-22 20:00:03,653 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-22 20:00:03,662 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-22 20:00:03,663 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-22 20:00:03,664 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision 2025-11-22 20:00:03,666 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-22 20:00:07,052 - __main__ - INFO - ✅ COMET-QE: 0.0383 ± 0.1441 2025-11-22 20:00:07,052 - __main__ - INFO - ✅ Baseline Results (Test Set): 2025-11-22 20:00:07,053 - __main__ - INFO - BLEU: 0.0198 2025-11-22 20:00:07,054 - __main__ - INFO - chrF: 17.59 2025-11-22 20:00:07,055 - __main__ - INFO - COMET: 0.0383 2025-11-22 20:00:07,055 - __main__ - INFO - ====================================================================== 2025-11-22 20:00:07,056 - __main__ - INFO - ====================================================================== 2025-11-22 20:00:07,056 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-22 20:00:07,057 - __main__ - INFO - ====================================================================== 2025-11-22 20:00:07,057 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-22 20:00:07,059 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-22 20:00:08,185 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-22 20:00:08,262 - __main__ - INFO - ✅ Domain chrF: 0.93 2025-11-22 20:00:08,263 - __main__ - INFO - Computing COMET-QE... 2025-11-22 20:00:08,270 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-22 20:00:08,278 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-22 20:00:08,278 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-22 20:00:08,280 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-22 20:00:10,965 - __main__ - INFO - ✅ Domain COMET-QE: 0.0000 2025-11-22 20:00:10,966 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-22 20:00:10,967 - __main__ - INFO - BLEU: 0.0000 2025-11-22 20:00:10,967 - __main__ - INFO - chrF: 0.93 2025-11-22 20:00:10,967 - __main__ - INFO - COMET-QE: 0.0000 2025-11-22 20:00:10,968 - __main__ - INFO - ====================================================================== 2025-11-22 20:00:11,420 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 2025-11-22 20:00:11,430 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase1/baseline_results.json 2025-11-22 20:00:11,435 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics 2025-11-22 20:00:11,436 - __main__ - INFO - 🚀 STEP 2: Fine-tuning 2025-11-22 20:00:11,437 - __main__ - INFO - Metrics are being logged to MLflow in real-time 2025-11-22 20:00:11,437 - __main__ - INFO - MLflow URL: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 20:00:11,438 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1/logs 2025-11-22 20:00:11,439 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') 2025-11-22 20:00:11,440 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 20:00:11,440 - __main__ - INFO - ====================================================================== 2025-11-22 20:00:11,441 - __main__ - INFO - 🚀 Starting training... 2025-11-22 20:00:11,442 - __main__ - INFO - ====================================================================== 2025-11-22 20:00:13,055 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) 2025-11-22 20:00:38,350 - __main__ - INFO - 📈 Epoch progress: 0.01/8 (0.1%) 2025-11-22 20:01:08,931 - __main__ - INFO - 📈 Epoch progress: 0.02/8 (0.2%) 2025-11-22 20:01:40,702 - __main__ - INFO - 📈 Epoch progress: 0.03/8 (0.4%) 2025-11-22 20:02:07,267 - __main__ - INFO - 📈 Epoch progress: 0.04/8 (0.5%) 2025-11-22 20:02:31,080 - __main__ - INFO - 📈 Epoch progress: 0.05/8 (0.6%) 2025-11-22 20:02:55,120 - __main__ - INFO - 📈 Epoch progress: 0.06/8 (0.7%) 2025-11-22 20:03:18,911 - __main__ - INFO - 📈 Epoch progress: 0.07/8 (0.8%) 2025-11-22 20:03:42,670 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (0.9%) 2025-11-22 20:04:06,358 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (1.1%) 2025-11-22 20:04:30,209 - __main__ - INFO - 📈 Epoch progress: 0.09/8 (1.2%) 2025-11-22 20:04:53,982 - __main__ - INFO - 📈 Epoch progress: 0.10/8 (1.3%) 2025-11-22 20:05:18,464 - __main__ - INFO - 📈 Epoch progress: 0.11/8 (1.4%) 2025-11-22 20:05:42,236 - __main__ - INFO - 📈 Epoch progress: 0.12/8 (1.5%) 2025-11-22 20:06:06,369 - __main__ - INFO - 📈 Epoch progress: 0.13/8 (1.7%) 2025-11-22 20:06:30,324 - __main__ - INFO - 📈 Epoch progress: 0.14/8 (1.8%) 2025-11-22 20:06:54,545 - __main__ - INFO - 📈 Epoch progress: 0.15/8 (1.9%) 2025-11-22 20:07:17,941 - __main__ - INFO - 📈 Epoch progress: 0.16/8 (2.0%) 2025-11-22 20:07:41,589 - __main__ - INFO - 📈 Epoch progress: 0.17/8 (2.1%) 2025-11-22 20:08:05,091 - __main__ - INFO - 📈 Epoch progress: 0.18/8 (2.2%) 2025-11-22 20:08:28,918 - __main__ - INFO - 📈 Epoch progress: 0.19/8 (2.4%) 2025-11-22 20:08:41,181 - __main__ - INFO - 📊 Evaluation 2: BLEU = 0.1085 2025-11-22 20:08:41,601 - __main__ - INFO - 📊 Evaluation at step 2000 2025-11-22 20:08:53,479 - __main__ - INFO - 📤 Pushing checkpoint 2000 to HuggingFace Hub... 2025-11-22 20:08:53,480 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 20:09:03,565 - __main__ - WARNING - ⚠️ Failed to push checkpoint to hub: 404 Client Error. (Request ID: Root=1-6922185f-2864f6c94f85721857318bbc;38a69bf4-c6b2-4079-9449-b6a171029895) Repository Not Found for url: https://huggingface.co/api/models/brendaogutu/mul-sw-en-translation-phase1/preupload/main. Please make sure you specified the correct `repo_id` and `repo_type`. If you are trying to access a private or gated repo, make sure you are authenticated. For more details, see https://huggingface.co/docs/huggingface_hub/authentication Note: Creating a commit assumes that the repo already exists on the Huggingface Hub. Please use `create_repo` if it's not the case. 2025-11-22 20:09:27,139 - __main__ - INFO - 📈 Epoch progress: 0.20/8 (2.5%) 2025-11-22 20:09:50,958 - __main__ - INFO - 📈 Epoch progress: 0.21/8 (2.6%) 2025-11-22 20:10:14,586 - __main__ - INFO - 📈 Epoch progress: 0.22/8 (2.7%) 2025-11-22 20:10:38,782 - __main__ - INFO - 📈 Epoch progress: 0.23/8 (2.8%) 2025-11-22 20:11:02,476 - __main__ - INFO - 📈 Epoch progress: 0.24/8 (2.9%) 2025-11-22 20:11:25,551 - __main__ - INFO - 📈 Epoch progress: 0.25/8 (3.1%) 2025-11-22 20:11:49,041 - __main__ - INFO - 📈 Epoch progress: 0.25/8 (3.2%) 2025-11-22 20:12:12,789 - __main__ - INFO - 📈 Epoch progress: 0.26/8 (3.3%) 2025-11-22 20:12:37,301 - __main__ - INFO - 📈 Epoch progress: 0.27/8 (3.4%) 2025-11-22 20:13:01,463 - __main__ - INFO - 📈 Epoch progress: 0.28/8 (3.5%) 2025-11-22 20:13:24,947 - __main__ - INFO - 📈 Epoch progress: 0.29/8 (3.7%) 2025-11-22 20:13:48,819 - __main__ - INFO - 📈 Epoch progress: 0.30/8 (3.8%) 2025-11-22 20:14:12,698 - __main__ - INFO - 📈 Epoch progress: 0.31/8 (3.9%) 2025-11-22 20:14:36,547 - __main__ - INFO - 📈 Epoch progress: 0.32/8 (4.0%) 2025-11-22 20:15:09,789 - __main__ - INFO - 📈 Epoch progress: 0.33/8 (4.1%) 2025-11-22 20:15:33,688 - __main__ - INFO - 📈 Epoch progress: 0.34/8 (4.2%) 2025-11-22 20:15:57,310 - __main__ - INFO - 📈 Epoch progress: 0.35/8 (4.4%) 2025-11-22 20:16:21,091 - __main__ - INFO - 📈 Epoch progress: 0.36/8 (4.5%) 2025-11-22 20:16:44,861 - __main__ - INFO - 📈 Epoch progress: 0.37/8 (4.6%) 2025-11-22 20:17:08,559 - __main__ - INFO - 📈 Epoch progress: 0.38/8 (4.7%) 2025-11-22 20:17:24,279 - __main__ - INFO - 📊 Evaluation 3: BLEU = 0.1403 2025-11-22 20:17:24,620 - __main__ - INFO - 📊 Evaluation at step 4000 2025-11-22 20:17:30,144 - __main__ - INFO - 📤 Pushing checkpoint 4000 to HuggingFace Hub... 2025-11-22 20:17:30,145 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 20:17:40,192 - __main__ - INFO - ✅ Checkpoint 4000 pushed successfully! 2025-11-22 20:17:40,193 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-4000 2025-11-22 20:18:03,497 - __main__ - INFO - 📈 Epoch progress: 0.39/8 (4.8%) 2025-11-22 20:18:27,502 - __main__ - INFO - 📈 Epoch progress: 0.40/8 (5.0%) 2025-11-22 20:18:50,959 - __main__ - INFO - 📈 Epoch progress: 0.41/8 (5.1%) 2025-11-22 20:19:14,786 - __main__ - INFO - 📈 Epoch progress: 0.42/8 (5.2%) 2025-11-22 20:19:38,083 - __main__ - INFO - 📈 Epoch progress: 0.42/8 (5.3%) 2025-11-22 20:20:01,787 - __main__ - INFO - 📈 Epoch progress: 0.43/8 (5.4%) 2025-11-22 20:20:25,229 - __main__ - INFO - 📈 Epoch progress: 0.44/8 (5.5%) 2025-11-22 20:20:48,594 - __main__ - INFO - 📈 Epoch progress: 0.45/8 (5.7%) 2025-11-22 20:21:12,168 - __main__ - INFO - 📈 Epoch progress: 0.46/8 (5.8%) 2025-11-22 20:21:36,348 - __main__ - INFO - 📈 Epoch progress: 0.47/8 (5.9%) 2025-11-22 20:21:59,806 - __main__ - INFO - 📈 Epoch progress: 0.48/8 (6.0%) 2025-11-22 20:22:23,780 - __main__ - INFO - 📈 Epoch progress: 0.49/8 (6.1%) 2025-11-22 20:22:47,397 - __main__ - INFO - 📈 Epoch progress: 0.50/8 (6.3%) 2025-11-22 20:23:11,216 - __main__ - INFO - 📈 Epoch progress: 0.51/8 (6.4%) 2025-11-22 20:23:34,702 - __main__ - INFO - 📈 Epoch progress: 0.52/8 (6.5%) 2025-11-22 20:23:58,147 - __main__ - INFO - 📈 Epoch progress: 0.53/8 (6.6%) 2025-11-22 20:24:21,926 - __main__ - INFO - 📈 Epoch progress: 0.54/8 (6.7%) 2025-11-22 20:24:45,551 - __main__ - INFO - 📈 Epoch progress: 0.55/8 (6.8%) 2025-11-22 20:25:09,313 - __main__ - INFO - 📈 Epoch progress: 0.56/8 (7.0%) 2025-11-22 20:25:33,561 - __main__ - INFO - 📈 Epoch progress: 0.57/8 (7.1%) 2025-11-22 20:25:52,923 - __main__ - INFO - 📊 Evaluation 4: BLEU = 0.1460 2025-11-22 20:25:53,358 - __main__ - INFO - 📊 Evaluation at step 6000 2025-11-22 20:25:57,061 - __main__ - INFO - 📤 Pushing checkpoint 6000 to HuggingFace Hub... 2025-11-22 20:25:57,062 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 20:26:07,651 - __main__ - INFO - ✅ Checkpoint 6000 pushed successfully! 2025-11-22 20:26:07,652 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-6000 2025-11-22 20:26:31,166 - __main__ - INFO - 📈 Epoch progress: 0.58/8 (7.2%) 2025-11-22 20:26:54,743 - __main__ - INFO - 📈 Epoch progress: 0.59/8 (7.3%) 2025-11-22 20:27:18,278 - __main__ - INFO - 📈 Epoch progress: 0.59/8 (7.4%) 2025-11-22 20:27:42,270 - __main__ - INFO - 📈 Epoch progress: 0.60/8 (7.6%) 2025-11-22 20:28:06,898 - __main__ - INFO - 📈 Epoch progress: 0.61/8 (7.7%) 2025-11-22 20:28:30,498 - __main__ - INFO - 📈 Epoch progress: 0.62/8 (7.8%) 2025-11-22 20:28:54,751 - __main__ - INFO - 📈 Epoch progress: 0.63/8 (7.9%) 2025-11-22 20:29:18,490 - __main__ - INFO - 📈 Epoch progress: 0.64/8 (8.0%) 2025-11-22 20:29:41,950 - __main__ - INFO - 📈 Epoch progress: 0.65/8 (8.1%) 2025-11-22 20:30:05,838 - __main__ - INFO - 📈 Epoch progress: 0.66/8 (8.3%) 2025-11-22 20:30:29,703 - __main__ - INFO - 📈 Epoch progress: 0.67/8 (8.4%) 2025-11-22 20:30:53,615 - __main__ - INFO - 📈 Epoch progress: 0.68/8 (8.5%) 2025-11-22 20:31:17,704 - __main__ - INFO - 📈 Epoch progress: 0.69/8 (8.6%) 2025-11-22 20:31:41,537 - __main__ - INFO - 📈 Epoch progress: 0.70/8 (8.7%) 2025-11-22 20:32:05,353 - __main__ - INFO - 📈 Epoch progress: 0.71/8 (8.8%) 2025-11-22 20:32:37,028 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 20:32:37,029 - __main__ - INFO - 📈 Epoch progress: 0.72/8 (9.0%) 2025-11-22 20:32:37,375 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 20:32:49,028 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 20:33:00,709 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 20:33:00,714 - __main__ - INFO - 📈 Epoch progress: 0.73/8 (9.1%) 2025-11-22 20:33:01,151 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 20:33:13,027 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 20:33:24,387 - __main__ - INFO - 📈 Epoch progress: 0.74/8 (9.2%) 2025-11-22 20:33:47,835 - __main__ - INFO - 📈 Epoch progress: 0.75/8 (9.3%) 2025-11-22 20:34:11,110 - __main__ - INFO - 📈 Epoch progress: 0.76/8 (9.4%) 2025-11-22 20:34:26,961 - __main__ - INFO - 📊 Evaluation 5: BLEU = 0.1538 2025-11-22 20:34:27,322 - __main__ - INFO - 📊 Evaluation at step 8000 2025-11-22 20:34:33,717 - __main__ - INFO - 📤 Pushing checkpoint 8000 to HuggingFace Hub... 2025-11-22 20:34:33,718 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 20:34:44,612 - __main__ - INFO - ✅ Checkpoint 8000 pushed successfully! 2025-11-22 20:34:44,615 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-8000 2025-11-22 20:35:07,956 - __main__ - INFO - 📈 Epoch progress: 0.76/8 (9.6%) 2025-11-22 20:35:31,667 - __main__ - INFO - 📈 Epoch progress: 0.77/8 (9.7%) 2025-11-22 20:35:55,140 - __main__ - INFO - 📈 Epoch progress: 0.78/8 (9.8%) 2025-11-22 20:36:18,945 - __main__ - INFO - 📈 Epoch progress: 0.79/8 (9.9%) 2025-11-22 20:36:43,642 - __main__ - INFO - 📈 Epoch progress: 0.80/8 (10.0%) 2025-11-22 20:37:07,303 - __main__ - INFO - 📈 Epoch progress: 0.81/8 (10.1%) 2025-11-22 20:37:31,040 - __main__ - INFO - 📈 Epoch progress: 0.82/8 (10.3%) 2025-11-22 20:37:54,752 - __main__ - INFO - 📈 Epoch progress: 0.83/8 (10.4%) 2025-11-22 20:38:18,185 - __main__ - INFO - 📈 Epoch progress: 0.84/8 (10.5%) 2025-11-22 20:38:41,713 - __main__ - INFO - 📈 Epoch progress: 0.85/8 (10.6%) 2025-11-22 20:39:05,292 - __main__ - INFO - 📈 Epoch progress: 0.86/8 (10.7%) 2025-11-22 20:39:28,872 - __main__ - INFO - 📈 Epoch progress: 0.87/8 (10.9%) 2025-11-22 20:39:52,309 - __main__ - INFO - 📈 Epoch progress: 0.88/8 (11.0%) 2025-11-22 20:40:16,091 - __main__ - INFO - 📈 Epoch progress: 0.89/8 (11.1%) 2025-11-22 20:40:39,485 - __main__ - INFO - 📈 Epoch progress: 0.90/8 (11.2%) 2025-11-22 20:41:03,175 - __main__ - INFO - 📈 Epoch progress: 0.91/8 (11.3%) 2025-11-22 20:41:26,580 - __main__ - INFO - 📈 Epoch progress: 0.92/8 (11.4%) 2025-11-22 20:41:50,347 - __main__ - INFO - 📈 Epoch progress: 0.93/8 (11.6%) 2025-11-22 20:42:13,732 - __main__ - INFO - 📈 Epoch progress: 0.93/8 (11.7%) 2025-11-22 20:42:37,615 - __main__ - INFO - 📈 Epoch progress: 0.94/8 (11.8%) 2025-11-22 20:42:52,788 - __main__ - INFO - 📊 Evaluation 6: BLEU = 0.1541 2025-11-22 20:42:53,140 - __main__ - INFO - 📊 Evaluation at step 10000 2025-11-22 20:42:57,687 - __main__ - INFO - 📤 Pushing checkpoint 10000 to HuggingFace Hub... 2025-11-22 20:42:57,688 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 20:43:08,616 - __main__ - INFO - ✅ Checkpoint 10000 pushed successfully! 2025-11-22 20:43:08,619 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-10000 2025-11-22 20:43:31,749 - __main__ - INFO - 📈 Epoch progress: 0.95/8 (11.9%) 2025-11-22 20:43:55,608 - __main__ - INFO - 📈 Epoch progress: 0.96/8 (12.0%) 2025-11-22 20:44:19,143 - __main__ - INFO - 📈 Epoch progress: 0.97/8 (12.2%) 2025-11-22 20:44:42,575 - __main__ - INFO - 📈 Epoch progress: 0.98/8 (12.3%) 2025-11-22 20:45:06,633 - __main__ - INFO - 📈 Epoch progress: 0.99/8 (12.4%) 2025-11-22 20:45:30,901 - __main__ - INFO - 📈 Epoch progress: 1.00/8 (12.5%) 2025-11-22 20:45:54,569 - __main__ - INFO - 📈 Epoch progress: 1.01/8 (12.6%) 2025-11-22 20:46:18,121 - __main__ - INFO - 📈 Epoch progress: 1.02/8 (12.7%) 2025-11-22 20:46:41,888 - __main__ - INFO - 📈 Epoch progress: 1.03/8 (12.9%) 2025-11-22 20:47:06,029 - __main__ - INFO - 📈 Epoch progress: 1.04/8 (13.0%) 2025-11-22 20:47:29,508 - __main__ - INFO - 📈 Epoch progress: 1.05/8 (13.1%) 2025-11-22 20:47:52,465 - __main__ - INFO - 📈 Epoch progress: 1.06/8 (13.2%) 2025-11-22 20:48:15,982 - __main__ - INFO - 📈 Epoch progress: 1.07/8 (13.3%) 2025-11-22 20:48:39,691 - __main__ - INFO - 📈 Epoch progress: 1.08/8 (13.5%) 2025-11-22 20:49:03,378 - __main__ - INFO - 📈 Epoch progress: 1.09/8 (13.6%) 2025-11-22 20:49:26,543 - __main__ - INFO - 📈 Epoch progress: 1.09/8 (13.7%) 2025-11-22 20:49:49,871 - __main__ - INFO - 📈 Epoch progress: 1.10/8 (13.8%) 2025-11-22 20:50:13,500 - __main__ - INFO - 📈 Epoch progress: 1.11/8 (13.9%) 2025-11-22 20:50:37,515 - __main__ - INFO - 📈 Epoch progress: 1.12/8 (14.0%) 2025-11-22 20:51:01,136 - __main__ - INFO - 📈 Epoch progress: 1.13/8 (14.2%) 2025-11-22 20:51:25,040 - __main__ - INFO - 📊 Evaluation 7: BLEU = 0.1463 2025-11-22 20:51:25,397 - __main__ - INFO - 📊 Evaluation at step 12000 2025-11-22 20:51:30,017 - __main__ - INFO - 📤 Pushing checkpoint 12000 to HuggingFace Hub... 2025-11-22 20:51:30,018 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 20:51:39,904 - __main__ - INFO - ✅ Checkpoint 12000 pushed successfully! 2025-11-22 20:51:39,914 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-12000 2025-11-22 20:52:03,513 - __main__ - INFO - 📈 Epoch progress: 1.14/8 (14.3%) 2025-11-22 20:52:27,285 - __main__ - INFO - 📈 Epoch progress: 1.15/8 (14.4%) 2025-11-22 20:52:50,714 - __main__ - INFO - 📈 Epoch progress: 1.16/8 (14.5%) 2025-11-22 20:53:14,707 - __main__ - INFO - 📈 Epoch progress: 1.17/8 (14.6%) 2025-11-22 20:53:38,315 - __main__ - INFO - 📈 Epoch progress: 1.18/8 (14.7%) 2025-11-22 20:54:02,170 - __main__ - INFO - 📈 Epoch progress: 1.19/8 (14.9%) 2025-11-22 20:54:25,745 - __main__ - INFO - 📈 Epoch progress: 1.20/8 (15.0%) 2025-11-22 20:54:56,587 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 20:54:56,589 - __main__ - INFO - 📈 Epoch progress: 1.21/8 (15.1%) 2025-11-22 20:54:56,904 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 20:55:20,217 - __main__ - INFO - 📈 Epoch progress: 1.22/8 (15.2%) 2025-11-22 20:55:43,949 - __main__ - INFO - 📈 Epoch progress: 1.23/8 (15.3%) 2025-11-22 20:56:07,794 - __main__ - INFO - 📈 Epoch progress: 1.24/8 (15.5%) 2025-11-22 20:56:31,239 - __main__ - INFO - 📈 Epoch progress: 1.25/8 (15.6%) 2025-11-22 20:56:54,645 - __main__ - INFO - 📈 Epoch progress: 1.26/8 (15.7%) 2025-11-22 20:57:17,680 - __main__ - INFO - 📈 Epoch progress: 1.26/8 (15.8%) 2025-11-22 20:57:41,619 - __main__ - INFO - 📈 Epoch progress: 1.27/8 (15.9%) 2025-11-22 20:58:05,294 - __main__ - INFO - 📈 Epoch progress: 1.28/8 (16.0%) 2025-11-22 20:58:28,616 - __main__ - INFO - 📈 Epoch progress: 1.29/8 (16.2%) 2025-11-22 20:58:52,621 - __main__ - INFO - 📈 Epoch progress: 1.30/8 (16.3%) 2025-11-22 20:59:16,154 - __main__ - INFO - 📈 Epoch progress: 1.31/8 (16.4%) 2025-11-22 20:59:39,666 - __main__ - INFO - 📈 Epoch progress: 1.32/8 (16.5%) 2025-11-22 20:59:56,828 - __main__ - INFO - 📊 Evaluation 8: BLEU = 0.1585 2025-11-22 20:59:57,166 - __main__ - INFO - 📊 Evaluation at step 14000 2025-11-22 21:00:00,516 - __main__ - INFO - 📤 Pushing checkpoint 14000 to HuggingFace Hub... 2025-11-22 21:00:00,517 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 21:00:12,536 - __main__ - INFO - ✅ Checkpoint 14000 pushed successfully! 2025-11-22 21:00:12,538 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-14000 2025-11-22 21:00:36,120 - __main__ - INFO - 📈 Epoch progress: 1.33/8 (16.6%) 2025-11-22 21:00:59,239 - __main__ - INFO - 📈 Epoch progress: 1.34/8 (16.8%) 2025-11-22 21:01:23,166 - __main__ - INFO - 📈 Epoch progress: 1.35/8 (16.9%) 2025-11-22 21:01:46,902 - __main__ - INFO - 📈 Epoch progress: 1.36/8 (17.0%) 2025-11-22 21:02:10,577 - __main__ - INFO - 📈 Epoch progress: 1.37/8 (17.1%) 2025-11-22 21:02:36,636 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:02:36,638 - __main__ - INFO - 📈 Epoch progress: 1.38/8 (17.2%) 2025-11-22 21:02:36,940 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:03:00,616 - __main__ - INFO - 📈 Epoch progress: 1.39/8 (17.3%) 2025-11-22 21:03:24,289 - __main__ - INFO - 📈 Epoch progress: 1.40/8 (17.5%) 2025-11-22 21:03:47,887 - __main__ - INFO - 📈 Epoch progress: 1.41/8 (17.6%) 2025-11-22 21:04:11,815 - __main__ - INFO - 📈 Epoch progress: 1.42/8 (17.7%) 2025-11-22 21:04:35,610 - __main__ - INFO - 📈 Epoch progress: 1.43/8 (17.8%) 2025-11-22 21:04:59,042 - __main__ - INFO - 📈 Epoch progress: 1.43/8 (17.9%) 2025-11-22 21:05:22,497 - __main__ - INFO - 📈 Epoch progress: 1.44/8 (18.1%) 2025-11-22 21:05:45,903 - __main__ - INFO - 📈 Epoch progress: 1.45/8 (18.2%) 2025-11-22 21:06:09,033 - __main__ - INFO - 📈 Epoch progress: 1.46/8 (18.3%) 2025-11-22 21:06:32,773 - __main__ - INFO - 📈 Epoch progress: 1.47/8 (18.4%) 2025-11-22 21:06:56,585 - __main__ - INFO - 📈 Epoch progress: 1.48/8 (18.5%) 2025-11-22 21:07:20,075 - __main__ - INFO - 📈 Epoch progress: 1.49/8 (18.6%) 2025-11-22 21:07:43,689 - __main__ - INFO - 📈 Epoch progress: 1.50/8 (18.8%) 2025-11-22 21:08:07,130 - __main__ - INFO - 📈 Epoch progress: 1.51/8 (18.9%) 2025-11-22 21:08:22,384 - __main__ - INFO - 📊 Evaluation 9: BLEU = 0.1666 2025-11-22 21:08:22,727 - __main__ - INFO - 📊 Evaluation at step 16000 2025-11-22 21:08:27,275 - __main__ - INFO - 📤 Pushing checkpoint 16000 to HuggingFace Hub... 2025-11-22 21:08:27,276 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 21:08:43,289 - __main__ - INFO - ✅ Checkpoint 16000 pushed successfully! 2025-11-22 21:08:43,290 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-16000 2025-11-22 21:09:06,478 - __main__ - INFO - 📈 Epoch progress: 1.52/8 (19.0%) 2025-11-22 21:09:30,626 - __main__ - INFO - 📈 Epoch progress: 1.53/8 (19.1%) 2025-11-22 21:09:54,017 - __main__ - INFO - 📈 Epoch progress: 1.54/8 (19.2%) 2025-11-22 21:10:17,768 - __main__ - INFO - 📈 Epoch progress: 1.55/8 (19.4%) 2025-11-22 21:10:41,731 - __main__ - INFO - 📈 Epoch progress: 1.56/8 (19.5%) 2025-11-22 21:11:05,567 - __main__ - INFO - 📈 Epoch progress: 1.57/8 (19.6%) 2025-11-22 21:11:29,040 - __main__ - INFO - 📈 Epoch progress: 1.58/8 (19.7%) 2025-11-22 21:11:53,073 - __main__ - INFO - 📈 Epoch progress: 1.59/8 (19.8%) 2025-11-22 21:12:17,169 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:12:17,177 - __main__ - INFO - 📈 Epoch progress: 1.60/8 (19.9%) 2025-11-22 21:12:17,481 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:12:29,345 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:12:41,169 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:12:41,170 - __main__ - INFO - 📈 Epoch progress: 1.60/8 (20.1%) 2025-11-22 21:12:41,465 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:13:04,807 - __main__ - INFO - 📈 Epoch progress: 1.61/8 (20.2%) 2025-11-22 21:13:28,403 - __main__ - INFO - 📈 Epoch progress: 1.62/8 (20.3%) 2025-11-22 21:13:52,653 - __main__ - INFO - 📈 Epoch progress: 1.63/8 (20.4%) 2025-11-22 21:14:15,499 - __main__ - INFO - 📈 Epoch progress: 1.64/8 (20.5%) 2025-11-22 21:14:40,686 - __main__ - INFO - 📈 Epoch progress: 1.65/8 (20.6%) 2025-11-22 21:15:05,650 - __main__ - INFO - 📈 Epoch progress: 1.66/8 (20.8%) 2025-11-22 21:15:29,482 - __main__ - INFO - 📈 Epoch progress: 1.67/8 (20.9%) 2025-11-22 21:15:52,896 - __main__ - INFO - 📈 Epoch progress: 1.68/8 (21.0%) 2025-11-22 21:16:16,430 - __main__ - INFO - 📈 Epoch progress: 1.69/8 (21.1%) 2025-11-22 21:16:40,428 - __main__ - INFO - 📈 Epoch progress: 1.70/8 (21.2%) 2025-11-22 21:16:58,183 - __main__ - INFO - 📊 Evaluation 10: BLEU = 0.1674 2025-11-22 21:16:58,533 - __main__ - INFO - 📊 Evaluation at step 18000 2025-11-22 21:17:07,291 - __main__ - INFO - 📤 Pushing checkpoint 18000 to HuggingFace Hub... 2025-11-22 21:17:07,292 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 21:17:24,438 - __main__ - INFO - ✅ Checkpoint 18000 pushed successfully! 2025-11-22 21:17:24,439 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-18000 2025-11-22 21:17:47,446 - __main__ - INFO - 📈 Epoch progress: 1.71/8 (21.4%) 2025-11-22 21:18:11,052 - __main__ - INFO - 📈 Epoch progress: 1.72/8 (21.5%) 2025-11-22 21:18:35,198 - __main__ - INFO - 📈 Epoch progress: 1.73/8 (21.6%) 2025-11-22 21:18:59,250 - __main__ - INFO - 📈 Epoch progress: 1.74/8 (21.7%) 2025-11-22 21:19:22,877 - __main__ - INFO - 📈 Epoch progress: 1.75/8 (21.8%) 2025-11-22 21:19:46,356 - __main__ - INFO - 📈 Epoch progress: 1.76/8 (21.9%) 2025-11-22 21:20:09,961 - __main__ - INFO - 📈 Epoch progress: 1.77/8 (22.1%) 2025-11-22 21:20:34,266 - __main__ - INFO - 📈 Epoch progress: 1.77/8 (22.2%) 2025-11-22 21:20:58,401 - __main__ - INFO - 📈 Epoch progress: 1.78/8 (22.3%) 2025-11-22 21:21:23,028 - __main__ - INFO - 📈 Epoch progress: 1.79/8 (22.4%) 2025-11-22 21:21:46,582 - __main__ - INFO - 📈 Epoch progress: 1.80/8 (22.5%) 2025-11-22 21:22:10,298 - __main__ - INFO - 📈 Epoch progress: 1.81/8 (22.7%) 2025-11-22 21:22:38,997 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:22:38,998 - __main__ - INFO - 📈 Epoch progress: 1.82/8 (22.8%) 2025-11-22 21:22:39,347 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:22:51,039 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:23:02,730 - __main__ - INFO - 📈 Epoch progress: 1.83/8 (22.9%) 2025-11-22 21:23:25,793 - __main__ - INFO - 📈 Epoch progress: 1.84/8 (23.0%) 2025-11-22 21:23:49,391 - __main__ - INFO - 📈 Epoch progress: 1.85/8 (23.1%) 2025-11-22 21:24:13,588 - __main__ - INFO - 📈 Epoch progress: 1.86/8 (23.2%) 2025-11-22 21:24:37,046 - __main__ - INFO - 📈 Epoch progress: 1.87/8 (23.4%) 2025-11-22 21:25:00,995 - __main__ - INFO - 📈 Epoch progress: 1.88/8 (23.5%) 2025-11-22 21:25:24,480 - __main__ - INFO - 📈 Epoch progress: 1.89/8 (23.6%) 2025-11-22 21:25:46,690 - __main__ - INFO - 📊 Evaluation 11: BLEU = 0.1619 2025-11-22 21:25:47,056 - __main__ - INFO - 📊 Evaluation at step 20000 2025-11-22 21:25:53,323 - __main__ - INFO - 📤 Pushing checkpoint 20000 to HuggingFace Hub... 2025-11-22 21:25:53,324 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 21:26:05,726 - __main__ - INFO - ✅ Checkpoint 20000 pushed successfully! 2025-11-22 21:26:05,728 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-20000 2025-11-22 21:26:29,812 - __main__ - INFO - 📈 Epoch progress: 1.90/8 (23.7%) 2025-11-22 21:26:53,847 - __main__ - INFO - 📈 Epoch progress: 1.91/8 (23.8%) 2025-11-22 21:27:17,422 - __main__ - INFO - 📈 Epoch progress: 1.92/8 (24.0%) 2025-11-22 21:27:42,006 - __main__ - INFO - 📈 Epoch progress: 1.93/8 (24.1%) 2025-11-22 21:28:05,446 - __main__ - INFO - 📈 Epoch progress: 1.94/8 (24.2%) 2025-11-22 21:28:29,168 - __main__ - INFO - 📈 Epoch progress: 1.94/8 (24.3%) 2025-11-22 21:28:52,701 - __main__ - INFO - 📈 Epoch progress: 1.95/8 (24.4%) 2025-11-22 21:29:16,950 - __main__ - INFO - 📈 Epoch progress: 1.96/8 (24.5%) 2025-11-22 21:29:40,772 - __main__ - INFO - 📈 Epoch progress: 1.97/8 (24.7%) 2025-11-22 21:30:04,317 - __main__ - INFO - 📈 Epoch progress: 1.98/8 (24.8%) 2025-11-22 21:30:27,826 - __main__ - INFO - 📈 Epoch progress: 1.99/8 (24.9%) 2025-11-22 21:30:52,170 - __main__ - INFO - 📈 Epoch progress: 2.00/8 (25.0%) 2025-11-22 21:31:15,774 - __main__ - INFO - 📈 Epoch progress: 2.01/8 (25.1%) 2025-11-22 21:31:39,640 - __main__ - INFO - 📈 Epoch progress: 2.02/8 (25.3%) 2025-11-22 21:32:02,847 - __main__ - INFO - 📈 Epoch progress: 2.03/8 (25.4%) 2025-11-22 21:32:27,451 - __main__ - INFO - 📈 Epoch progress: 2.04/8 (25.5%) 2025-11-22 21:32:51,684 - __main__ - INFO - 📈 Epoch progress: 2.05/8 (25.6%) 2025-11-22 21:33:15,308 - __main__ - INFO - 📈 Epoch progress: 2.06/8 (25.7%) 2025-11-22 21:33:38,738 - __main__ - INFO - 📈 Epoch progress: 2.07/8 (25.8%) 2025-11-22 21:34:02,226 - __main__ - INFO - 📈 Epoch progress: 2.08/8 (26.0%) 2025-11-22 21:34:20,000 - __main__ - INFO - 📊 Evaluation 12: BLEU = 0.1677 2025-11-22 21:34:20,369 - __main__ - INFO - 📊 Evaluation at step 22000 2025-11-22 21:34:24,871 - __main__ - INFO - 📤 Pushing checkpoint 22000 to HuggingFace Hub... 2025-11-22 21:34:24,872 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 21:34:34,963 - __main__ - INFO - ✅ Checkpoint 22000 pushed successfully! 2025-11-22 21:34:34,965 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-22000 2025-11-22 21:34:58,374 - __main__ - INFO - 📈 Epoch progress: 2.09/8 (26.1%) 2025-11-22 21:35:22,174 - __main__ - INFO - 📈 Epoch progress: 2.10/8 (26.2%) 2025-11-22 21:35:45,215 - __main__ - INFO - 📈 Epoch progress: 2.10/8 (26.3%) 2025-11-22 21:36:08,584 - __main__ - INFO - 📈 Epoch progress: 2.11/8 (26.4%) 2025-11-22 21:36:32,144 - __main__ - INFO - 📈 Epoch progress: 2.12/8 (26.5%) 2025-11-22 21:36:56,099 - __main__ - INFO - 📈 Epoch progress: 2.13/8 (26.7%) 2025-11-22 21:37:19,843 - __main__ - INFO - 📈 Epoch progress: 2.14/8 (26.8%) 2025-11-22 21:37:43,099 - __main__ - INFO - 📈 Epoch progress: 2.15/8 (26.9%) 2025-11-22 21:38:07,047 - __main__ - INFO - 📈 Epoch progress: 2.16/8 (27.0%) 2025-11-22 21:38:30,112 - __main__ - INFO - 📈 Epoch progress: 2.17/8 (27.1%) 2025-11-22 21:38:53,928 - __main__ - INFO - 📈 Epoch progress: 2.18/8 (27.3%) 2025-11-22 21:39:18,208 - __main__ - INFO - 📈 Epoch progress: 2.19/8 (27.4%) 2025-11-22 21:39:42,762 - __main__ - INFO - 📈 Epoch progress: 2.20/8 (27.5%) 2025-11-22 21:40:06,488 - __main__ - INFO - 📈 Epoch progress: 2.21/8 (27.6%) 2025-11-22 21:40:30,471 - __main__ - INFO - 📈 Epoch progress: 2.22/8 (27.7%) 2025-11-22 21:40:53,743 - __main__ - INFO - 📈 Epoch progress: 2.23/8 (27.8%) 2025-11-22 21:41:17,384 - __main__ - INFO - 📈 Epoch progress: 2.24/8 (28.0%) 2025-11-22 21:41:40,963 - __main__ - INFO - 📈 Epoch progress: 2.25/8 (28.1%) 2025-11-22 21:42:05,101 - __main__ - INFO - 📈 Epoch progress: 2.26/8 (28.2%) 2025-11-22 21:42:28,584 - __main__ - INFO - 📈 Epoch progress: 2.27/8 (28.3%) 2025-11-22 21:42:49,035 - __main__ - INFO - 📊 Evaluation 13: BLEU = 0.1630 2025-11-22 21:42:49,388 - __main__ - INFO - 📊 Evaluation at step 24000 2025-11-22 21:42:54,416 - __main__ - INFO - 📤 Pushing checkpoint 24000 to HuggingFace Hub... 2025-11-22 21:42:54,417 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 21:43:11,101 - __main__ - INFO - ✅ Checkpoint 24000 pushed successfully! 2025-11-22 21:43:11,114 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-24000 2025-11-22 21:43:34,133 - __main__ - INFO - 📈 Epoch progress: 2.27/8 (28.4%) 2025-11-22 21:43:58,647 - __main__ - INFO - 📈 Epoch progress: 2.28/8 (28.6%) 2025-11-22 21:44:22,526 - __main__ - INFO - 📈 Epoch progress: 2.29/8 (28.7%) 2025-11-22 21:44:56,326 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:44:56,328 - __main__ - INFO - 📈 Epoch progress: 2.30/8 (28.8%) 2025-11-22 21:44:56,642 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 21:45:20,598 - __main__ - INFO - 📈 Epoch progress: 2.31/8 (28.9%) 2025-11-22 21:45:44,794 - __main__ - INFO - 📈 Epoch progress: 2.32/8 (29.0%) 2025-11-22 21:46:08,169 - __main__ - INFO - 📈 Epoch progress: 2.33/8 (29.1%) 2025-11-22 21:46:31,960 - __main__ - INFO - 📈 Epoch progress: 2.34/8 (29.3%) 2025-11-22 21:46:55,735 - __main__ - INFO - 📈 Epoch progress: 2.35/8 (29.4%) 2025-11-22 21:47:19,450 - __main__ - INFO - 📈 Epoch progress: 2.36/8 (29.5%) 2025-11-22 21:47:42,862 - __main__ - INFO - 📈 Epoch progress: 2.37/8 (29.6%) 2025-11-22 21:48:07,254 - __main__ - INFO - 📈 Epoch progress: 2.38/8 (29.7%) 2025-11-22 21:48:31,170 - __main__ - INFO - 📈 Epoch progress: 2.39/8 (29.9%) 2025-11-22 21:48:55,953 - __main__ - INFO - 📈 Epoch progress: 2.40/8 (30.0%) 2025-11-22 21:49:19,508 - __main__ - INFO - 📈 Epoch progress: 2.41/8 (30.1%) 2025-11-22 21:49:43,336 - __main__ - INFO - 📈 Epoch progress: 2.42/8 (30.2%) 2025-11-22 21:50:07,127 - __main__ - INFO - 📈 Epoch progress: 2.43/8 (30.3%) 2025-11-22 21:50:31,261 - __main__ - INFO - 📈 Epoch progress: 2.44/8 (30.4%) 2025-11-22 21:50:55,006 - __main__ - INFO - 📈 Epoch progress: 2.44/8 (30.6%) 2025-11-22 21:51:18,988 - __main__ - INFO - 📈 Epoch progress: 2.45/8 (30.7%) 2025-11-22 21:51:41,415 - __main__ - INFO - 📊 Evaluation 14: BLEU = 0.1721 2025-11-22 21:51:41,771 - __main__ - INFO - 📊 Evaluation at step 26000 2025-11-22 21:51:47,075 - __main__ - INFO - 📤 Pushing checkpoint 26000 to HuggingFace Hub... 2025-11-22 21:51:47,076 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 21:51:58,317 - __main__ - INFO - ✅ Checkpoint 26000 pushed successfully! 2025-11-22 21:51:58,323 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-26000 2025-11-22 21:52:21,800 - __main__ - INFO - 📈 Epoch progress: 2.46/8 (30.8%) 2025-11-22 21:52:45,848 - __main__ - INFO - 📈 Epoch progress: 2.47/8 (30.9%) 2025-11-22 21:53:09,666 - __main__ - INFO - 📈 Epoch progress: 2.48/8 (31.0%) 2025-11-22 21:53:33,443 - __main__ - INFO - 📈 Epoch progress: 2.49/8 (31.1%) 2025-11-22 21:53:56,636 - __main__ - INFO - 📈 Epoch progress: 2.50/8 (31.3%) 2025-11-22 21:54:20,060 - __main__ - INFO - 📈 Epoch progress: 2.51/8 (31.4%) 2025-11-22 21:54:44,973 - __main__ - INFO - 📈 Epoch progress: 2.52/8 (31.5%) 2025-11-22 21:55:08,222 - __main__ - INFO - 📈 Epoch progress: 2.53/8 (31.6%) 2025-11-22 21:55:31,852 - __main__ - INFO - 📈 Epoch progress: 2.54/8 (31.7%) 2025-11-22 21:55:55,256 - __main__ - INFO - 📈 Epoch progress: 2.55/8 (31.9%) 2025-11-22 21:56:18,899 - __main__ - INFO - 📈 Epoch progress: 2.56/8 (32.0%) 2025-11-22 21:56:42,547 - __main__ - INFO - 📈 Epoch progress: 2.57/8 (32.1%) 2025-11-22 21:57:06,146 - __main__ - INFO - 📈 Epoch progress: 2.58/8 (32.2%) 2025-11-22 21:57:30,165 - __main__ - INFO - 📈 Epoch progress: 2.59/8 (32.3%) 2025-11-22 21:57:53,581 - __main__ - INFO - 📈 Epoch progress: 2.60/8 (32.4%) 2025-11-22 21:58:17,968 - __main__ - INFO - 📈 Epoch progress: 2.61/8 (32.6%) 2025-11-22 21:58:41,705 - __main__ - INFO - 📈 Epoch progress: 2.61/8 (32.7%) 2025-11-22 21:59:05,247 - __main__ - INFO - 📈 Epoch progress: 2.62/8 (32.8%) 2025-11-22 21:59:28,518 - __main__ - INFO - 📈 Epoch progress: 2.63/8 (32.9%) 2025-11-22 21:59:52,385 - __main__ - INFO - 📈 Epoch progress: 2.64/8 (33.0%) 2025-11-22 22:00:16,924 - __main__ - INFO - 📊 Evaluation 15: BLEU = 0.1706 2025-11-22 22:00:17,285 - __main__ - INFO - 📊 Evaluation at step 28000 2025-11-22 22:00:22,699 - __main__ - INFO - 📤 Pushing checkpoint 28000 to HuggingFace Hub... 2025-11-22 22:00:22,718 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 22:00:34,097 - __main__ - INFO - ✅ Checkpoint 28000 pushed successfully! 2025-11-22 22:00:34,099 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-28000 2025-11-22 22:00:57,572 - __main__ - INFO - 📈 Epoch progress: 2.65/8 (33.2%) 2025-11-22 22:01:21,910 - __main__ - INFO - 📈 Epoch progress: 2.66/8 (33.3%) 2025-11-22 22:01:45,653 - __main__ - INFO - 📈 Epoch progress: 2.67/8 (33.4%) 2025-11-22 22:02:09,324 - __main__ - INFO - 📈 Epoch progress: 2.68/8 (33.5%) 2025-11-22 22:02:32,388 - __main__ - INFO - 📈 Epoch progress: 2.69/8 (33.6%) 2025-11-22 22:02:55,915 - __main__ - INFO - 📈 Epoch progress: 2.70/8 (33.7%) 2025-11-22 22:03:19,709 - __main__ - INFO - 📈 Epoch progress: 2.71/8 (33.9%) 2025-11-22 22:03:42,372 - __main__ - INFO - 📈 Epoch progress: 2.72/8 (34.0%) 2025-11-22 22:04:14,921 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 22:04:14,922 - __main__ - INFO - 📈 Epoch progress: 2.73/8 (34.1%) 2025-11-22 22:04:15,222 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 22:04:39,431 - __main__ - INFO - 📈 Epoch progress: 2.74/8 (34.2%) 2025-11-22 22:05:03,047 - __main__ - INFO - 📈 Epoch progress: 2.75/8 (34.3%) 2025-11-22 22:05:26,149 - __main__ - INFO - 📈 Epoch progress: 2.76/8 (34.5%) 2025-11-22 22:05:49,543 - __main__ - INFO - 📈 Epoch progress: 2.77/8 (34.6%) 2025-11-22 22:06:13,007 - __main__ - INFO - 📈 Epoch progress: 2.78/8 (34.7%) 2025-11-22 22:06:36,910 - __main__ - INFO - 📈 Epoch progress: 2.78/8 (34.8%) 2025-11-22 22:07:00,093 - __main__ - INFO - 📈 Epoch progress: 2.79/8 (34.9%) 2025-11-22 22:07:23,534 - __main__ - INFO - 📈 Epoch progress: 2.80/8 (35.0%) 2025-11-22 22:07:47,807 - __main__ - INFO - 📈 Epoch progress: 2.81/8 (35.2%) 2025-11-22 22:08:11,112 - __main__ - INFO - 📈 Epoch progress: 2.82/8 (35.3%) 2025-11-22 22:08:34,882 - __main__ - INFO - 📈 Epoch progress: 2.83/8 (35.4%) 2025-11-22 22:08:54,341 - __main__ - INFO - 📊 Evaluation 16: BLEU = 0.1744 2025-11-22 22:08:54,690 - __main__ - INFO - 📊 Evaluation at step 30000 2025-11-22 22:08:58,267 - __main__ - INFO - 📤 Pushing checkpoint 30000 to HuggingFace Hub... 2025-11-22 22:08:58,267 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 22:09:10,043 - __main__ - INFO - ✅ Checkpoint 30000 pushed successfully! 2025-11-22 22:09:10,045 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-30000 2025-11-22 22:09:33,448 - __main__ - INFO - 📈 Epoch progress: 2.84/8 (35.5%) 2025-11-22 22:09:57,022 - __main__ - INFO - 📈 Epoch progress: 2.85/8 (35.6%) 2025-11-22 22:10:20,696 - __main__ - INFO - 📈 Epoch progress: 2.86/8 (35.8%) 2025-11-22 22:10:44,433 - __main__ - INFO - 📈 Epoch progress: 2.87/8 (35.9%) 2025-11-22 22:11:08,015 - __main__ - INFO - 📈 Epoch progress: 2.88/8 (36.0%) 2025-11-22 22:11:31,148 - __main__ - INFO - 📈 Epoch progress: 2.89/8 (36.1%) 2025-11-22 22:11:55,004 - __main__ - INFO - 📈 Epoch progress: 2.90/8 (36.2%) 2025-11-22 22:12:18,689 - __main__ - INFO - 📈 Epoch progress: 2.91/8 (36.3%) 2025-11-22 22:12:42,132 - __main__ - INFO - 📈 Epoch progress: 2.92/8 (36.5%) 2025-11-22 22:13:05,631 - __main__ - INFO - 📈 Epoch progress: 2.93/8 (36.6%) 2025-11-22 22:13:28,949 - __main__ - INFO - 📈 Epoch progress: 2.94/8 (36.7%) 2025-11-22 22:13:52,766 - __main__ - INFO - 📈 Epoch progress: 2.95/8 (36.8%) 2025-11-22 22:14:16,537 - __main__ - INFO - 📈 Epoch progress: 2.95/8 (36.9%) 2025-11-22 22:14:40,633 - __main__ - INFO - 📈 Epoch progress: 2.96/8 (37.0%) 2025-11-22 22:15:04,766 - __main__ - INFO - 📈 Epoch progress: 2.97/8 (37.2%) 2025-11-22 22:15:28,401 - __main__ - INFO - 📈 Epoch progress: 2.98/8 (37.3%) 2025-11-22 22:15:51,913 - __main__ - INFO - 📈 Epoch progress: 2.99/8 (37.4%) 2025-11-22 22:16:16,816 - __main__ - INFO - 📈 Epoch progress: 3.00/8 (37.5%) 2025-11-22 22:16:41,053 - __main__ - INFO - 📈 Epoch progress: 3.01/8 (37.6%) 2025-11-22 22:17:05,221 - __main__ - INFO - 📈 Epoch progress: 3.02/8 (37.8%) 2025-11-22 22:17:20,019 - __main__ - INFO - 📊 Evaluation 17: BLEU = 0.1803 2025-11-22 22:17:20,363 - __main__ - INFO - 📊 Evaluation at step 32000 2025-11-22 22:17:24,622 - __main__ - INFO - 📤 Pushing checkpoint 32000 to HuggingFace Hub... 2025-11-22 22:17:24,623 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 22:17:36,404 - __main__ - INFO - ✅ Checkpoint 32000 pushed successfully! 2025-11-22 22:17:36,419 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-32000 2025-11-22 22:18:00,147 - __main__ - INFO - 📈 Epoch progress: 3.03/8 (37.9%) 2025-11-22 22:18:24,208 - __main__ - INFO - 📈 Epoch progress: 3.04/8 (38.0%) 2025-11-22 22:18:47,490 - __main__ - INFO - 📈 Epoch progress: 3.05/8 (38.1%) 2025-11-22 22:19:11,236 - __main__ - INFO - 📈 Epoch progress: 3.06/8 (38.2%) 2025-11-22 22:19:34,490 - __main__ - INFO - 📈 Epoch progress: 3.07/8 (38.3%) 2025-11-22 22:19:58,382 - __main__ - INFO - 📈 Epoch progress: 3.08/8 (38.5%) 2025-11-22 22:20:22,181 - __main__ - INFO - 📈 Epoch progress: 3.09/8 (38.6%) 2025-11-22 22:20:46,069 - __main__ - INFO - 📈 Epoch progress: 3.10/8 (38.7%) 2025-11-22 22:21:10,006 - __main__ - INFO - 📈 Epoch progress: 3.11/8 (38.8%) 2025-11-22 22:21:33,569 - __main__ - INFO - 📈 Epoch progress: 3.11/8 (38.9%) 2025-11-22 22:21:56,826 - __main__ - INFO - 📈 Epoch progress: 3.12/8 (39.1%) 2025-11-22 22:22:20,689 - __main__ - INFO - 📈 Epoch progress: 3.13/8 (39.2%) 2025-11-22 22:22:44,409 - __main__ - INFO - 📈 Epoch progress: 3.14/8 (39.3%) 2025-11-22 22:23:08,312 - __main__ - INFO - 📈 Epoch progress: 3.15/8 (39.4%) 2025-11-22 22:23:32,702 - __main__ - INFO - 📈 Epoch progress: 3.16/8 (39.5%) 2025-11-22 22:23:55,904 - __main__ - INFO - 📈 Epoch progress: 3.17/8 (39.6%) 2025-11-22 22:24:20,172 - __main__ - INFO - 📈 Epoch progress: 3.18/8 (39.8%) 2025-11-22 22:24:44,449 - __main__ - INFO - 📈 Epoch progress: 3.19/8 (39.9%) 2025-11-22 22:25:08,639 - __main__ - INFO - 📈 Epoch progress: 3.20/8 (40.0%) 2025-11-22 22:25:32,297 - __main__ - INFO - 📈 Epoch progress: 3.21/8 (40.1%) 2025-11-22 22:25:48,079 - __main__ - INFO - 📊 Evaluation 18: BLEU = 0.1742 2025-11-22 22:25:48,433 - __main__ - INFO - 📊 Evaluation at step 34000 2025-11-22 22:25:52,773 - __main__ - INFO - 📤 Pushing checkpoint 34000 to HuggingFace Hub... 2025-11-22 22:25:52,774 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 22:26:02,999 - __main__ - INFO - ✅ Checkpoint 34000 pushed successfully! 2025-11-22 22:26:03,015 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-34000 2025-11-22 22:26:32,695 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 22:26:32,696 - __main__ - INFO - 📈 Epoch progress: 3.22/8 (40.2%) 2025-11-22 22:26:33,015 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 22:26:57,007 - __main__ - INFO - 📈 Epoch progress: 3.23/8 (40.4%) 2025-11-22 22:27:21,160 - __main__ - INFO - 📈 Epoch progress: 3.24/8 (40.5%) 2025-11-22 22:27:45,784 - __main__ - INFO - 📈 Epoch progress: 3.25/8 (40.6%) 2025-11-22 22:28:09,851 - __main__ - INFO - 📈 Epoch progress: 3.26/8 (40.7%) 2025-11-22 22:28:33,187 - __main__ - INFO - 📈 Epoch progress: 3.27/8 (40.8%) 2025-11-22 22:28:56,968 - __main__ - INFO - 📈 Epoch progress: 3.28/8 (40.9%) 2025-11-22 22:29:20,427 - __main__ - INFO - 📈 Epoch progress: 3.28/8 (41.1%) 2025-11-22 22:29:44,210 - __main__ - INFO - 📈 Epoch progress: 3.29/8 (41.2%) 2025-11-22 22:30:08,158 - __main__ - INFO - 📈 Epoch progress: 3.30/8 (41.3%) 2025-11-22 22:30:31,651 - __main__ - INFO - 📈 Epoch progress: 3.31/8 (41.4%) 2025-11-22 22:30:55,813 - __main__ - INFO - 📈 Epoch progress: 3.32/8 (41.5%) 2025-11-22 22:31:19,823 - __main__ - INFO - 📈 Epoch progress: 3.33/8 (41.7%) 2025-11-22 22:31:43,557 - __main__ - INFO - 📈 Epoch progress: 3.34/8 (41.8%) 2025-11-22 22:32:07,188 - __main__ - INFO - 📈 Epoch progress: 3.35/8 (41.9%) 2025-11-22 22:32:37,715 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 22:32:37,718 - __main__ - INFO - 📈 Epoch progress: 3.36/8 (42.0%) 2025-11-22 22:32:38,127 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 22:33:01,680 - __main__ - INFO - 📈 Epoch progress: 3.37/8 (42.1%) 2025-11-22 22:33:25,806 - __main__ - INFO - 📈 Epoch progress: 3.38/8 (42.2%) 2025-11-22 22:33:49,888 - __main__ - INFO - 📈 Epoch progress: 3.39/8 (42.4%) 2025-11-22 22:34:13,802 - __main__ - INFO - 📈 Epoch progress: 3.40/8 (42.5%) 2025-11-22 22:34:29,581 - __main__ - INFO - 📊 Evaluation 19: BLEU = 0.1791 2025-11-22 22:34:29,919 - __main__ - INFO - 📊 Evaluation at step 36000 2025-11-22 22:34:35,623 - __main__ - INFO - 📤 Pushing checkpoint 36000 to HuggingFace Hub... 2025-11-22 22:34:35,624 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 22:34:46,345 - __main__ - INFO - ✅ Checkpoint 36000 pushed successfully! 2025-11-22 22:34:46,347 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-36000 2025-11-22 22:35:09,516 - __main__ - INFO - 📈 Epoch progress: 3.41/8 (42.6%) 2025-11-22 22:35:33,257 - __main__ - INFO - 📈 Epoch progress: 3.42/8 (42.7%) 2025-11-22 22:35:57,242 - __main__ - INFO - 📈 Epoch progress: 3.43/8 (42.8%) 2025-11-22 22:36:21,693 - __main__ - INFO - 📈 Epoch progress: 3.44/8 (42.9%) 2025-11-22 22:36:45,861 - __main__ - INFO - 📈 Epoch progress: 3.45/8 (43.1%) 2025-11-22 22:37:10,943 - __main__ - INFO - 📈 Epoch progress: 3.45/8 (43.2%) 2025-11-22 22:37:34,716 - __main__ - INFO - 📈 Epoch progress: 3.46/8 (43.3%) 2025-11-22 22:37:59,177 - __main__ - INFO - 📈 Epoch progress: 3.47/8 (43.4%) 2025-11-22 22:38:23,278 - __main__ - INFO - 📈 Epoch progress: 3.48/8 (43.5%) 2025-11-22 22:38:47,089 - __main__ - INFO - 📈 Epoch progress: 3.49/8 (43.7%) 2025-11-22 22:39:11,123 - __main__ - INFO - 📈 Epoch progress: 3.50/8 (43.8%) 2025-11-22 22:39:35,715 - __main__ - INFO - 📈 Epoch progress: 3.51/8 (43.9%) 2025-11-22 22:39:59,298 - __main__ - INFO - 📈 Epoch progress: 3.52/8 (44.0%) 2025-11-22 22:40:22,780 - __main__ - INFO - 📈 Epoch progress: 3.53/8 (44.1%) 2025-11-22 22:40:46,271 - __main__ - INFO - 📈 Epoch progress: 3.54/8 (44.2%) 2025-11-22 22:41:10,602 - __main__ - INFO - 📈 Epoch progress: 3.55/8 (44.4%) 2025-11-22 22:41:35,020 - __main__ - INFO - 📈 Epoch progress: 3.56/8 (44.5%) 2025-11-22 22:41:58,896 - __main__ - INFO - 📈 Epoch progress: 3.57/8 (44.6%) 2025-11-22 22:42:22,712 - __main__ - INFO - 📈 Epoch progress: 3.58/8 (44.7%) 2025-11-22 22:42:46,271 - __main__ - INFO - 📈 Epoch progress: 3.59/8 (44.8%) 2025-11-22 22:43:08,440 - __main__ - INFO - 📊 Evaluation 20: BLEU = 0.1705 2025-11-22 22:43:08,780 - __main__ - INFO - 📊 Evaluation at step 38000 2025-11-22 22:43:13,372 - __main__ - INFO - 📤 Pushing checkpoint 38000 to HuggingFace Hub... 2025-11-22 22:43:13,373 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 22:43:24,549 - __main__ - INFO - ✅ Checkpoint 38000 pushed successfully! 2025-11-22 22:43:24,551 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-38000 2025-11-22 22:43:47,855 - __main__ - INFO - 📈 Epoch progress: 3.60/8 (45.0%) 2025-11-22 22:44:12,036 - __main__ - INFO - 📈 Epoch progress: 3.61/8 (45.1%) 2025-11-22 22:44:35,675 - __main__ - INFO - 📈 Epoch progress: 3.62/8 (45.2%) 2025-11-22 22:44:54,430 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 22:45:06,039 - __main__ - INFO - 📈 Epoch progress: 3.62/8 (45.3%) 2025-11-22 22:45:30,388 - __main__ - INFO - 📈 Epoch progress: 3.63/8 (45.4%) 2025-11-22 22:45:54,473 - __main__ - INFO - 📈 Epoch progress: 3.64/8 (45.5%) 2025-11-22 22:46:18,311 - __main__ - INFO - 📈 Epoch progress: 3.65/8 (45.7%) 2025-11-22 22:46:42,472 - __main__ - INFO - 📈 Epoch progress: 3.66/8 (45.8%) 2025-11-22 22:47:07,409 - __main__ - INFO - 📈 Epoch progress: 3.67/8 (45.9%) 2025-11-22 22:47:30,886 - __main__ - INFO - 📈 Epoch progress: 3.68/8 (46.0%) 2025-11-22 22:47:54,827 - __main__ - INFO - 📈 Epoch progress: 3.69/8 (46.1%) 2025-11-22 22:48:18,009 - __main__ - INFO - 📈 Epoch progress: 3.70/8 (46.3%) 2025-11-22 22:48:41,563 - __main__ - INFO - 📈 Epoch progress: 3.71/8 (46.4%) 2025-11-22 22:49:06,015 - __main__ - INFO - 📈 Epoch progress: 3.72/8 (46.5%) 2025-11-22 22:49:30,312 - __main__ - INFO - 📈 Epoch progress: 3.73/8 (46.6%) 2025-11-22 22:49:54,384 - __main__ - INFO - 📈 Epoch progress: 3.74/8 (46.7%) 2025-11-22 22:50:18,637 - __main__ - INFO - 📈 Epoch progress: 3.75/8 (46.8%) 2025-11-22 22:50:42,692 - __main__ - INFO - 📈 Epoch progress: 3.76/8 (47.0%) 2025-11-22 22:51:06,477 - __main__ - INFO - 📈 Epoch progress: 3.77/8 (47.1%) 2025-11-22 22:51:30,876 - __main__ - INFO - 📈 Epoch progress: 3.78/8 (47.2%) 2025-11-22 22:51:49,849 - __main__ - INFO - 📊 Evaluation 21: BLEU = 0.1801 2025-11-22 22:51:50,268 - __main__ - INFO - 📊 Evaluation at step 40000 2025-11-22 22:51:55,274 - __main__ - INFO - 📤 Pushing checkpoint 40000 to HuggingFace Hub... 2025-11-22 22:51:55,275 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 22:52:05,418 - __main__ - INFO - ✅ Checkpoint 40000 pushed successfully! 2025-11-22 22:52:05,419 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-40000 2025-11-22 22:52:28,376 - __main__ - INFO - 📈 Epoch progress: 3.79/8 (47.3%) 2025-11-22 22:52:52,172 - __main__ - INFO - 📈 Epoch progress: 3.79/8 (47.4%) 2025-11-22 22:53:16,023 - __main__ - INFO - 📈 Epoch progress: 3.80/8 (47.6%) 2025-11-22 22:53:39,673 - __main__ - INFO - 📈 Epoch progress: 3.81/8 (47.7%) 2025-11-22 22:54:03,075 - __main__ - INFO - 📈 Epoch progress: 3.82/8 (47.8%) 2025-11-22 22:54:27,337 - __main__ - INFO - 📈 Epoch progress: 3.83/8 (47.9%) 2025-11-22 22:54:57,783 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 22:54:57,790 - __main__ - INFO - 📈 Epoch progress: 3.84/8 (48.0%) 2025-11-22 22:54:58,152 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-22 22:55:21,366 - __main__ - INFO - 📈 Epoch progress: 3.85/8 (48.1%) 2025-11-22 22:55:45,172 - __main__ - INFO - 📈 Epoch progress: 3.86/8 (48.3%) 2025-11-22 22:56:08,569 - __main__ - INFO - 📈 Epoch progress: 3.87/8 (48.4%) 2025-11-22 22:56:32,826 - __main__ - INFO - 📈 Epoch progress: 3.88/8 (48.5%) 2025-11-22 22:56:56,673 - __main__ - INFO - 📈 Epoch progress: 3.89/8 (48.6%) 2025-11-22 22:57:20,679 - __main__ - INFO - 📈 Epoch progress: 3.90/8 (48.7%) 2025-11-22 22:57:44,496 - __main__ - INFO - 📈 Epoch progress: 3.91/8 (48.8%) 2025-11-22 22:58:08,328 - __main__ - INFO - 📈 Epoch progress: 3.92/8 (49.0%) 2025-11-22 22:58:31,781 - __main__ - INFO - 📈 Epoch progress: 3.93/8 (49.1%) 2025-11-22 22:58:55,466 - __main__ - INFO - 📈 Epoch progress: 3.94/8 (49.2%) 2025-11-22 22:59:19,046 - __main__ - INFO - 📈 Epoch progress: 3.95/8 (49.3%) 2025-11-22 22:59:42,251 - __main__ - INFO - 📈 Epoch progress: 3.96/8 (49.4%) 2025-11-22 23:00:06,289 - __main__ - INFO - 📈 Epoch progress: 3.96/8 (49.6%) 2025-11-22 23:00:33,348 - __main__ - INFO - 📊 Evaluation 22: BLEU = 0.1746 2025-11-22 23:00:34,193 - __main__ - INFO - 📊 Evaluation at step 42000 2025-11-22 23:00:38,079 - __main__ - INFO - 📤 Pushing checkpoint 42000 to HuggingFace Hub... 2025-11-22 23:00:38,080 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 23:00:48,483 - __main__ - INFO - ✅ Checkpoint 42000 pushed successfully! 2025-11-22 23:00:48,485 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-42000 2025-11-22 23:00:51,556 - __main__ - INFO - ✅ MLflow: Training finished - status_code=2.0 (FINISHED) 2025-11-22 23:00:51,853 - __main__ - INFO - 💾 Saving model to models/finetuned-sw-en-phase1 2025-11-22 23:00:53,397 - __main__ - INFO - ✅ Training complete! 2025-11-22 23:00:53,398 - __main__ - INFO - 📊 STEP 3: Final Evaluation 2025-11-22 23:00:53,761 - __main__ - INFO - ✅ MLflow: Final evaluation starting - status_code=3.0 2025-11-22 23:00:53,762 - __main__ - INFO - ====================================================================== 2025-11-22 23:00:53,762 - __main__ - INFO - 📊 Running FINAL evaluation on test set... 2025-11-22 23:00:53,763 - __main__ - INFO - ====================================================================== 2025-11-22 23:00:53,764 - __main__ - INFO - Tokenizing test dataset... 2025-11-22 23:00:55,719 - datasets.fingerprint - WARNING - Parameter 'function'=> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only shown once. Subsequent hashing failures won't be shown. 2025-11-22 23:49:59,784 - __main__ - INFO - 📊 Evaluation 23: BLEU = 0.1867 2025-11-22 23:50:00,135 - __main__ - INFO - 📊 Evaluation at step 42000 2025-11-22 23:50:00,257 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-22 23:50:00,313 - __main__ - INFO - Computing predictions for COMET (1000 samples, batch_size=32)... 2025-11-22 23:50:00,319 - __main__ - INFO - Batch 1/32... 2025-11-22 23:50:04,256 - __main__ - INFO - Batch 5/32... 2025-11-22 23:50:05,629 - __main__ - INFO - Batch 10/32... 2025-11-22 23:50:06,935 - __main__ - INFO - Batch 15/32... 2025-11-22 23:50:08,526 - __main__ - INFO - Batch 20/32... 2025-11-22 23:50:12,080 - __main__ - INFO - Batch 25/32... 2025-11-22 23:50:13,411 - __main__ - INFO - Batch 30/32... 2025-11-22 23:50:13,939 - __main__ - INFO - Running COMET-QE model... 2025-11-22 23:50:13,949 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-22 23:50:13,961 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-22 23:50:13,962 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-22 23:50:13,964 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-22 23:50:19,326 - __main__ - INFO - ✅ COMET-QE: 0.2124 ± 0.2721 2025-11-22 23:50:19,327 - __main__ - INFO - ✅ FINAL Test Results: 2025-11-22 23:50:19,328 - __main__ - INFO - BLEU: 0.1867 2025-11-22 23:50:19,328 - __main__ - INFO - chrF: 38.35 2025-11-22 23:50:19,329 - __main__ - INFO - COMET-QE: 0.2124 2025-11-22 23:50:19,329 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:19,330 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:19,331 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-22 23:50:19,331 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:19,331 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-22 23:50:19,332 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-22 23:50:20,433 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-22 23:50:20,518 - __main__ - INFO - ✅ Domain chrF: 1.22 2025-11-22 23:50:20,518 - __main__ - INFO - Computing COMET-QE... 2025-11-22 23:50:20,526 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-22 23:50:20,535 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-22 23:50:20,536 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-22 23:50:20,537 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-22 23:50:23,127 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 2025-11-22 23:50:23,127 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-22 23:50:23,128 - __main__ - INFO - BLEU: 0.0000 2025-11-22 23:50:23,129 - __main__ - INFO - chrF: 1.22 2025-11-22 23:50:23,129 - __main__ - INFO - COMET-QE: 0.0001 2025-11-22 23:50:23,129 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:23,137 - __main__ - INFO - 💾 Final results saved to models/finetuned-sw-en-phase1/test_results.json 2025-11-22 23:50:23,141 - __main__ - INFO - ✅ Verified test_results.json with 14 metrics 2025-11-22 23:50:23,141 - __main__ - INFO - 📊 Logging all metrics to TensorBoard... 2025-11-22 23:50:23,278 - __main__ - INFO - ✅ All metrics logged to TensorBoard 2025-11-22 23:50:23,278 - __main__ - INFO - Baseline metrics: 13 2025-11-22 23:50:23,279 - __main__ - INFO - Final test metrics: 10 2025-11-22 23:50:23,280 - __main__ - INFO - Final domain metrics: 4 2025-11-22 23:50:23,281 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:23,281 - __main__ - INFO - 📊 BASELINE vs FINE-TUNED Comparison 2025-11-22 23:50:23,282 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:23,282 - __main__ - INFO - 📈 Test Set Metrics: 2025-11-22 23:50:23,283 - __main__ - INFO - Metric Baseline Fine-tuned Improvement 2025-11-22 23:50:23,283 - __main__ - INFO - ------------------------------------------------------------ 2025-11-22 23:50:23,284 - __main__ - INFO - eval_bleu 0.0000 0.1867 N/A 2025-11-22 23:50:23,290 - __main__ - INFO - eval_chrf 0.0000 38.3529 N/A 2025-11-22 23:50:23,291 - __main__ - INFO - eval_comet 0.0000 0.2124 N/A 2025-11-22 23:50:23,291 - __main__ - INFO - 📈 Domain Metrics (Real Call Transcriptions): 2025-11-22 23:50:23,292 - __main__ - INFO - Metric Baseline Fine-tuned Improvement 2025-11-22 23:50:23,293 - __main__ - INFO - ------------------------------------------------------------ 2025-11-22 23:50:23,293 - __main__ - INFO - domain_bleu 0.0000 0.0000 N/A 2025-11-22 23:50:23,294 - __main__ - INFO - domain_chrf 0.9342 1.2250 +31.1% 2025-11-22 23:50:23,294 - __main__ - INFO - domain_comet 0.0000 0.0001 +33.3% 2025-11-22 23:50:23,295 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:23,664 - __main__ - INFO - ✅ Logged 14 final metrics to MLflow 2025-11-22 23:50:23,665 - __main__ - INFO - Visit MLflow: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 23:50:23,666 - __main__ - INFO - 📊 Captured 863 training log entries 2025-11-22 23:50:23,718 - __main__ - INFO - 💾 Saved comprehensive metrics to models/finetuned-sw-en-phase1/training_metrics.json 2025-11-22 23:50:23,734 - __main__ - INFO - 💾 Saved training config to models/finetuned-sw-en-phase1/training_config_used.json 2025-11-22 23:50:23,758 - __main__ - INFO - 💾 Copied training.log to output directory 2025-11-22 23:50:23,759 - __main__ - INFO - 🤖 HUGGINGFACE HUB 2025-11-22 23:50:23,760 - __main__ - INFO - ℹ️ Checkpoints were pushed during training (push_strategy='checkpoint') 2025-11-22 23:50:23,761 - __main__ - INFO - 📤 Pushing final model to hub... 2025-11-22 23:50:23,761 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:23,762 - __main__ - INFO - 📤 Uploading model to HuggingFace Hub... 2025-11-22 23:50:23,763 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:23,763 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 23:50:23,774 - __main__ - INFO - ✅ Model card generated 2025-11-22 23:50:24,095 - huggingface_hub._login - WARNING - Note: Environment variable`HF_TOKEN` is set and is the current active token independently from the token you've just configured. 2025-11-22 23:50:24,096 - __main__ - INFO - Uploading files... 2025-11-22 23:50:24,170 - __main__ - INFO - ✅ Repository created/verified: brendaogutu/mul-sw-en-translation-phase1 2025-11-22 23:50:24,174 - __main__ - INFO - 📊 Including TensorBoard logs (5 files) 2025-11-22 23:50:24,176 - __main__ - INFO - 📁 Uploading 21 items from models/finetuned-sw-en-phase1 2025-11-22 23:50:24,177 - __main__ - INFO - 📄 README.md 2025-11-22 23:50:24,178 - __main__ - INFO - 📄 training.log 2025-11-22 23:50:24,179 - __main__ - INFO - 📄 training_config_used.json 2025-11-22 23:50:24,181 - __main__ - INFO - 📄 training_metrics.json 2025-11-22 23:50:24,182 - __main__ - INFO - 📄 test_results.json 2025-11-22 23:50:24,183 - __main__ - INFO - 📄 train_results.json 2025-11-22 23:50:24,184 - __main__ - INFO - 📄 translation_config.json 2025-11-22 23:50:24,185 - __main__ - INFO - 📄 training_args.bin 2025-11-22 23:50:24,186 - __main__ - INFO - 📄 target.spm 2025-11-22 23:50:24,187 - __main__ - INFO - 📄 source.spm 2025-11-22 23:50:24,188 - __main__ - INFO - 📄 vocab.json 2025-11-22 23:50:24,189 - __main__ - INFO - 📄 special_tokens_map.json 2025-11-22 23:50:24,191 - __main__ - INFO - 📄 tokenizer_config.json 2025-11-22 23:50:24,192 - __main__ - INFO - 📄 model.safetensors 2025-11-22 23:50:24,193 - __main__ - INFO - 📄 generation_config.json 2025-11-22 23:50:24,194 - __main__ - INFO - 📄 config.json 2025-11-22 23:50:24,195 - __main__ - INFO - 📂 checkpoint-42000 2025-11-22 23:50:24,196 - __main__ - INFO - 📂 checkpoint-40000 2025-11-22 23:50:24,197 - __main__ - INFO - 📂 checkpoint-32000 2025-11-22 23:50:24,198 - __main__ - INFO - 📄 baseline_results.json 2025-11-22 23:50:24,198 - __main__ - INFO - 📂 logs 2025-11-22 23:50:43,662 - __main__ - INFO - ✅ Model uploaded successfully! 2025-11-22 23:50:43,664 - __main__ - INFO - 🔗 View model at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1 2025-11-22 23:50:43,667 - __main__ - INFO - 📊 TensorBoard logs at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/logs 2025-11-22 23:50:43,667 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:43,668 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:43,669 - __main__ - INFO - ✅ Pipeline completed successfully! 2025-11-22 23:50:43,669 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:43,670 - __main__ - INFO - 📋 PIPELINE SUMMARY 2025-11-22 23:50:43,671 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:43,671 - __main__ - INFO - ✓ Baseline Evaluation 2025-11-22 23:50:43,672 - __main__ - INFO - - Test set: 159518 samples 2025-11-22 23:50:43,672 - __main__ - INFO - - Domain data: 5 samples 2025-11-22 23:50:43,673 - __main__ - INFO - ✓ Training 2025-11-22 23:50:43,674 - __main__ - INFO - - Epochs: 8 2025-11-22 23:50:43,674 - __main__ - INFO - - Train samples: 2,711,789 2025-11-22 23:50:43,675 - __main__ - INFO - - Validation samples: 319,034 2025-11-22 23:50:43,676 - __main__ - INFO - ✓ Final Evaluation 2025-11-22 23:50:43,676 - __main__ - INFO - - Test set BLEU: 0.1867 2025-11-22 23:50:43,677 - __main__ - INFO - - Domain BLEU: 0.0000 2025-11-22 23:50:43,677 - __main__ - INFO - 📁 Output Files 2025-11-22 23:50:43,679 - __main__ - INFO - - Models: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 2025-11-22 23:50:43,679 - __main__ - INFO - - Baseline results: models/finetuned-sw-en-phase1/baseline_results.json 2025-11-22 23:50:43,680 - __main__ - INFO - - Test results: models/finetuned-sw-en-phase1/test_results.json 2025-11-22 23:50:43,681 - __main__ - INFO - - Training logs: /workspace/franklin/openchs_rnd/tasks/translation/swahili/training.log 2025-11-22 23:50:43,681 - __main__ - INFO - 📈 MLflow Tracking 2025-11-22 23:50:43,682 - __main__ - INFO - - URL: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 23:50:43,683 - __main__ - INFO - - Run ID: 8b5deaed89ec48e986246e5b801158e3 2025-11-22 23:50:43,683 - __main__ - INFO - 📊 TensorBoard Logs 2025-11-22 23:50:43,684 - __main__ - INFO - - View with: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1/logs 2025-11-22 23:50:43,684 - __main__ - INFO - - All metrics logged: baseline, test, domain, and COMET scores 2025-11-22 23:50:43,686 - __main__ - INFO - ====================================================================== 2025-11-22 23:50:43,686 - __main__ - INFO - 📈 MLflow Tracking: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-22 23:50:43,687 - __main__ - INFO - Status codes: 2025-11-22 23:50:43,687 - __main__ - INFO - - 0.5 = Baseline evaluation 2025-11-22 23:50:43,688 - __main__ - INFO - - 1.0 = Training started 2025-11-22 23:50:43,689 - __main__ - INFO - - 2.0 = Training finished 2025-11-22 23:50:43,689 - __main__ - INFO - - 3.0 = Final evaluation 2025-11-22 23:50:44,048 - __main__ - INFO - ✅ MLflow: Pipeline completed - status_code=4.0 2025-11-22 23:50:44,889 - __main__ - INFO - ✅ MLflow run ended: 8b5deaed89ec48e986246e5b801158e3 2025-11-22 23:50:44,891 - __main__ - INFO - ====================================================================== 2025-11-23 04:05:43,732 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-23 04:05:43,743 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json 2025-11-23 04:05:43,744 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-23 04:05:43,745 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-23 04:05:43,747 - __main__ - INFO - 🎲 Random seed: 42 2025-11-23 04:05:43,748 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-23 04:05:43,748 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 04:05:43,749 - __main__ - INFO - Push strategy: checkpoint 2025-11-23 04:05:43,749 - __main__ - INFO - → Will push after each checkpoint save 2025-11-23 04:05:43,749 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-23 04:05:43,750 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-23 04:05:43,751 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-23 04:05:43,760 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-23 04:05:43,761 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 04:05:44,443 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-23 04:05:45,579 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251123_040544 2025-11-23 04:05:45,580 - __main__ - INFO - Run ID: e70d46acae184a60a902a20b85021668 2025-11-23 04:05:45,581 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 04:05:45,962 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow 2025-11-23 04:05:45,965 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-23 04:05:45,976 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-23 04:05:45,977 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-23 04:05:45,978 - __main__ - INFO - 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-23 04:05:54,359 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-23 04:05:54,361 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-23 04:05:54,696 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file 2025-11-23 04:05:54,923 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file 2025-11-23 04:05:54,924 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl 2025-11-23 04:05:57,916 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) 2025-11-23 04:06:00,884 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 2025-11-23 04:06:03,675 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 2025-11-23 04:06:06,082 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 2025-11-23 04:06:08,442 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 2025-11-23 04:06:10,807 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 2025-11-23 04:06:13,193 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 2025-11-23 04:06:15,538 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 2025-11-23 04:06:17,888 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 2025-11-23 04:06:20,255 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 2025-11-23 04:06:22,631 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 2025-11-23 04:06:25,011 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 2025-11-23 04:06:27,409 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 2025-11-23 04:06:29,801 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 2025-11-23 04:06:32,196 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 2025-11-23 04:06:34,590 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 2025-11-23 04:06:36,955 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 2025-11-23 04:06:39,341 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 2025-11-23 04:06:41,730 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 2025-11-23 04:06:44,125 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 2025-11-23 04:06:46,526 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 2025-11-23 04:06:48,904 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 2025-11-23 04:06:51,289 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 2025-11-23 04:06:53,684 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 2025-11-23 04:06:56,086 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 2025-11-23 04:06:58,486 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 2025-11-23 04:06:58,719 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) 2025-11-23 04:06:58,754 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl 2025-11-23 04:07:00,804 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) 2025-11-23 04:07:03,765 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 2025-11-23 04:07:06,444 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 2025-11-23 04:07:08,822 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 2025-11-23 04:07:11,186 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 2025-11-23 04:07:13,575 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 2025-11-23 04:07:15,965 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 2025-11-23 04:07:18,459 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 2025-11-23 04:07:20,871 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 2025-11-23 04:07:23,259 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 2025-11-23 04:07:25,654 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 2025-11-23 04:07:28,055 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 2025-11-23 04:07:30,458 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 2025-11-23 04:07:32,855 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 2025-11-23 04:07:35,252 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 2025-11-23 04:07:37,634 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 2025-11-23 04:07:40,067 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 2025-11-23 04:07:42,278 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) 2025-11-23 04:07:42,325 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 2025-11-23 04:07:42,326 - __main__ - INFO - 📝 Sample data check: 2025-11-23 04:07:42,327 - __main__ - INFO - Sample 1: 2025-11-23 04:07:42,327 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... 2025-11-23 04:07:42,328 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... 2025-11-23 04:07:42,328 - __main__ - INFO - Sample 2: 2025-11-23 04:07:42,329 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... 2025-11-23 04:07:42,329 - __main__ - INFO - Target (6 words): Job had said no such thing.... 2025-11-23 04:07:42,330 - __main__ - INFO - Sample 3: 2025-11-23 04:07:42,331 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... 2025-11-23 04:07:42,331 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... 2025-11-23 04:07:47,761 - __main__ - INFO - Filtering statistics: {'kept': 4202243} 2025-11-23 04:07:47,762 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 2025-11-23 04:07:52,179 - __main__ - INFO - Removed 518 duplicates 2025-11-23 04:07:52,414 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 2025-11-23 04:07:59,017 - __main__ - INFO - ====================================================================== 2025-11-23 04:07:59,023 - __main__ - INFO - ✅ Dataset splits: 2025-11-23 04:07:59,025 - __main__ - INFO - Train: 3,571,465 samples 2025-11-23 04:07:59,027 - __main__ - INFO - Validation: 420,173 samples 2025-11-23 04:07:59,027 - __main__ - INFO - Test: 210,087 samples 2025-11-23 04:07:59,028 - __main__ - INFO - ====================================================================== 2025-11-23 04:07:59,463 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-23 04:11:21,132 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-23 04:11:21,139 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json 2025-11-23 04:11:21,140 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-23 04:11:21,141 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-23 04:11:21,143 - __main__ - INFO - 🎲 Random seed: 42 2025-11-23 04:11:21,144 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-23 04:11:21,144 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 04:11:21,145 - __main__ - INFO - Push strategy: checkpoint 2025-11-23 04:11:21,145 - __main__ - INFO - → Will push after each checkpoint save 2025-11-23 04:11:21,146 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-23 04:11:21,146 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-23 04:11:21,147 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-23 04:11:21,158 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-23 04:11:21,159 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 04:11:21,913 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-23 04:11:22,843 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251123_041122 2025-11-23 04:11:22,845 - __main__ - INFO - Run ID: e2e357df5cbe463abba0139445600679 2025-11-23 04:11:22,845 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 04:11:23,226 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow 2025-11-23 04:11:23,231 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-23 04:11:23,239 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-23 04:11:23,241 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-23 04:11:23,241 - __main__ - INFO - 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-23 04:11:26,716 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-23 04:11:26,717 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-23 04:11:27,122 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file 2025-11-23 04:11:27,390 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file 2025-11-23 04:11:27,393 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl 2025-11-23 04:11:28,190 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) 2025-11-23 04:11:31,094 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 2025-11-23 04:11:33,656 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 2025-11-23 04:11:36,508 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 2025-11-23 04:11:38,901 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 2025-11-23 04:11:41,318 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 2025-11-23 04:11:43,721 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 2025-11-23 04:11:46,121 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 2025-11-23 04:11:48,539 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 2025-11-23 04:11:50,925 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 2025-11-23 04:11:53,348 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 2025-11-23 04:11:55,767 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 2025-11-23 04:11:58,204 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 2025-11-23 04:12:00,636 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 2025-11-23 04:12:03,051 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 2025-11-23 04:12:05,459 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 2025-11-23 04:12:07,868 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 2025-11-23 04:12:10,285 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 2025-11-23 04:12:12,718 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 2025-11-23 04:12:15,153 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 2025-11-23 04:12:17,596 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 2025-11-23 04:12:20,000 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 2025-11-23 04:12:22,421 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 2025-11-23 04:12:24,843 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 2025-11-23 04:12:27,269 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 2025-11-23 04:12:29,700 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 2025-11-23 04:12:29,952 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) 2025-11-23 04:12:29,998 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl 2025-11-23 04:12:30,726 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) 2025-11-23 04:12:33,185 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 2025-11-23 04:12:35,566 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 2025-11-23 04:12:37,932 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 2025-11-23 04:12:40,292 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 2025-11-23 04:12:42,663 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 2025-11-23 04:12:45,025 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 2025-11-23 04:12:47,392 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 2025-11-23 04:12:49,787 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 2025-11-23 04:12:52,193 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 2025-11-23 04:12:54,609 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 2025-11-23 04:12:57,036 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 2025-11-23 04:12:59,440 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 2025-11-23 04:13:01,849 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 2025-11-23 04:13:04,266 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 2025-11-23 04:13:06,685 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 2025-11-23 04:13:09,101 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 2025-11-23 04:13:11,345 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) 2025-11-23 04:13:11,386 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 2025-11-23 04:13:11,387 - __main__ - INFO - 📝 Sample data check: 2025-11-23 04:13:11,387 - __main__ - INFO - Sample 1: 2025-11-23 04:13:11,388 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... 2025-11-23 04:13:11,388 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... 2025-11-23 04:13:11,389 - __main__ - INFO - Sample 2: 2025-11-23 04:13:11,389 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... 2025-11-23 04:13:11,389 - __main__ - INFO - Target (6 words): Job had said no such thing.... 2025-11-23 04:13:11,390 - __main__ - INFO - Sample 3: 2025-11-23 04:13:11,391 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... 2025-11-23 04:13:11,391 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... 2025-11-23 04:13:16,729 - __main__ - INFO - Filtering statistics: {'kept': 4202243} 2025-11-23 04:13:16,730 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 2025-11-23 04:13:21,152 - __main__ - INFO - Removed 518 duplicates 2025-11-23 04:13:21,390 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 2025-11-23 04:13:27,939 - __main__ - INFO - ====================================================================== 2025-11-23 04:13:27,941 - __main__ - INFO - ✅ Dataset splits: 2025-11-23 04:13:27,942 - __main__ - INFO - Train: 3,571,465 samples 2025-11-23 04:13:27,942 - __main__ - INFO - Validation: 420,173 samples 2025-11-23 04:13:27,943 - __main__ - INFO - Test: 210,087 samples 2025-11-23 04:13:27,943 - __main__ - INFO - ====================================================================== 2025-11-23 04:13:28,390 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-23 04:16:33,982 - __main__ - INFO - ✅ Dataset validation passed 2025-11-23 04:16:33,984 - __main__ - INFO - Train: 3,571,465 samples 2025-11-23 04:16:33,985 - __main__ - INFO - Validation: 420,173 samples 2025-11-23 04:16:33,986 - __main__ - INFO - Test: 210,087 samples 2025-11-23 04:16:35,491 - __main__ - INFO - ✅ Found cached tokenized datasets (hash: 59e2a5c4) 2025-11-23 04:16:35,492 - __main__ - INFO - Skipping tokenization - this saves significant time! 2025-11-23 04:16:35,493 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... 2025-11-23 04:17:11,756 - __main__ - INFO - ✅ Tokenization complete! 2025-11-23 04:17:11,757 - __main__ - INFO - 🔧 Initializing COMET-QE model... 2025-11-23 04:17:11,859 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. 2025-11-23 04:17:12,851 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` 2025-11-23 04:17:18,905 - comet.models.base - INFO - Encoder model frozen. 2025-11-23 04:17:19,462 - __main__ - INFO - ✅ COMET-QE model loaded successfully 2025-11-23 04:17:19,463 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ 2025-11-23 04:17:19,494 - __main__ - INFO - 📊 Limiting evaluation dataset from 420225 to 2000 samples 2025-11-23 04:17:19,516 - __main__ - INFO - ✅ Trainer initialized 2025-11-23 04:17:19,517 - __main__ - INFO - 🔬 STEP 1: Baseline Evaluation 2025-11-23 04:17:20,164 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 2025-11-23 04:17:20,165 - __main__ - INFO - ====================================================================== 2025-11-23 04:17:20,165 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... 2025-11-23 04:17:20,166 - __main__ - INFO - ====================================================================== 2025-11-23 04:17:52,543 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.1215 2025-11-23 04:17:52,544 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-23 04:17:52,576 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... 2025-11-23 04:17:52,578 - __main__ - INFO - Batch 1/16... 2025-11-23 04:17:53,920 - __main__ - INFO - Batch 5/16... 2025-11-23 04:17:54,977 - __main__ - INFO - Batch 10/16... 2025-11-23 04:17:56,189 - __main__ - INFO - Batch 15/16... 2025-11-23 04:17:56,576 - __main__ - INFO - Running COMET-QE model... 2025-11-23 04:17:56,588 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-23 04:17:56,596 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-23 04:17:56,597 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-23 04:17:56,598 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision 2025-11-23 04:17:56,599 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-23 04:17:59,616 - __main__ - INFO - ✅ COMET-QE: 0.1977 ± 0.2784 2025-11-23 04:17:59,619 - __main__ - INFO - ✅ Baseline Results (Test Set): 2025-11-23 04:17:59,620 - __main__ - INFO - BLEU: 0.1215 2025-11-23 04:17:59,622 - __main__ - INFO - chrF: 32.36 2025-11-23 04:17:59,622 - __main__ - INFO - COMET: 0.1977 2025-11-23 04:17:59,622 - __main__ - INFO - ====================================================================== 2025-11-23 04:17:59,623 - __main__ - INFO - ====================================================================== 2025-11-23 04:17:59,624 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-23 04:17:59,624 - __main__ - INFO - ====================================================================== 2025-11-23 04:17:59,625 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-23 04:17:59,626 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-23 04:18:00,687 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-23 04:18:00,771 - __main__ - INFO - ✅ Domain chrF: 1.22 2025-11-23 04:18:00,771 - __main__ - INFO - Computing COMET-QE... 2025-11-23 04:18:00,781 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-23 04:18:00,788 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-23 04:18:00,789 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-23 04:18:00,791 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-23 04:18:02,815 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 2025-11-23 04:18:02,818 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-23 04:18:02,820 - __main__ - INFO - BLEU: 0.0000 2025-11-23 04:18:02,821 - __main__ - INFO - chrF: 1.22 2025-11-23 04:18:02,821 - __main__ - INFO - COMET-QE: 0.0001 2025-11-23 04:18:02,822 - __main__ - INFO - ====================================================================== 2025-11-23 04:18:03,214 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 2025-11-23 04:18:03,234 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase2/baseline_results.json 2025-11-23 04:18:03,239 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics 2025-11-23 04:18:03,240 - __main__ - INFO - 🚀 STEP 2: Fine-tuning 2025-11-23 04:18:03,240 - __main__ - INFO - Metrics are being logged to MLflow in real-time 2025-11-23 04:18:03,241 - __main__ - INFO - MLflow URL: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 04:18:03,241 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs 2025-11-23 04:18:03,242 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') 2025-11-23 04:18:03,243 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 04:18:03,243 - __main__ - INFO - ====================================================================== 2025-11-23 04:18:03,244 - __main__ - INFO - 🚀 Starting training... 2025-11-23 04:18:03,244 - __main__ - INFO - ====================================================================== 2025-11-23 04:18:04,245 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) 2025-11-23 04:18:25,258 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.1%) 2025-11-23 04:18:45,682 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.2%) 2025-11-23 04:19:06,543 - __main__ - INFO - 📈 Epoch progress: 0.02/6 (0.4%) 2025-11-23 04:19:26,980 - __main__ - INFO - 📈 Epoch progress: 0.03/6 (0.5%) 2025-11-23 04:19:47,541 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.6%) 2025-11-23 04:20:07,833 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.7%) 2025-11-23 04:20:29,177 - __main__ - INFO - 📈 Epoch progress: 0.05/6 (0.8%) 2025-11-23 04:20:49,579 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.0%) 2025-11-23 04:21:10,348 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.1%) 2025-11-23 04:21:30,953 - __main__ - INFO - 📈 Epoch progress: 0.07/6 (1.2%) 2025-11-23 04:21:51,628 - __main__ - INFO - 📈 Epoch progress: 0.08/6 (1.3%) 2025-11-23 04:22:12,093 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.4%) 2025-11-23 04:22:35,662 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 04:22:35,664 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.6%) 2025-11-23 04:22:35,984 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 04:22:56,478 - __main__ - INFO - 📈 Epoch progress: 0.10/6 (1.7%) 2025-11-23 04:23:17,467 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.8%) 2025-11-23 04:23:38,091 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.9%) 2025-11-23 04:23:58,922 - __main__ - INFO - 📈 Epoch progress: 0.12/6 (2.0%) 2025-11-23 04:24:19,633 - __main__ - INFO - 📈 Epoch progress: 0.13/6 (2.2%) 2025-11-23 04:24:40,816 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.3%) 2025-11-23 04:25:01,487 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.4%) 2025-11-23 04:25:50,552 - __main__ - INFO - 📊 Evaluation 2: BLEU = 0.1169 2025-11-23 04:25:50,914 - __main__ - INFO - 📊 Evaluation at step 2000 2025-11-23 04:25:55,509 - __main__ - INFO - ✅ Repository brendaogutu/mul-sw-en-translation-phase2 ready 2025-11-23 04:25:55,558 - __main__ - INFO - 📤 Pushing checkpoint 2000 to HuggingFace Hub... 2025-11-23 04:25:55,564 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 04:26:08,358 - __main__ - INFO - ✅ Checkpoint 2000 pushed successfully! 2025-11-23 04:26:08,360 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-2000 2025-11-23 04:26:29,312 - __main__ - INFO - 📈 Epoch progress: 0.15/6 (2.5%) 2025-11-23 04:26:49,842 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.6%) 2025-11-23 04:27:10,452 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.7%) 2025-11-23 04:27:30,891 - __main__ - INFO - 📈 Epoch progress: 0.17/6 (2.9%) 2025-11-23 04:27:51,404 - __main__ - INFO - 📈 Epoch progress: 0.18/6 (3.0%) 2025-11-23 04:28:11,815 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.1%) 2025-11-23 04:28:31,947 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.2%) 2025-11-23 04:28:52,529 - __main__ - INFO - 📈 Epoch progress: 0.20/6 (3.3%) 2025-11-23 04:29:12,902 - __main__ - INFO - 📈 Epoch progress: 0.21/6 (3.5%) 2025-11-23 04:29:34,164 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.6%) 2025-11-23 04:29:54,671 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.7%) 2025-11-23 04:30:15,398 - __main__ - INFO - 📈 Epoch progress: 0.23/6 (3.8%) 2025-11-23 04:30:36,458 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (3.9%) 2025-11-23 04:30:57,150 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (4.1%) 2025-11-23 04:31:17,954 - __main__ - INFO - 📈 Epoch progress: 0.25/6 (4.2%) 2025-11-23 04:31:38,767 - __main__ - INFO - 📈 Epoch progress: 0.26/6 (4.3%) 2025-11-23 04:31:59,507 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.4%) 2025-11-23 04:32:20,095 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.5%) 2025-11-23 04:32:35,746 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 04:32:45,647 - __main__ - INFO - 📈 Epoch progress: 0.28/6 (4.7%) 2025-11-23 04:33:06,504 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.8%) 2025-11-23 04:33:52,383 - __main__ - INFO - 📊 Evaluation 3: BLEU = 0.1186 2025-11-23 04:33:52,744 - __main__ - INFO - 📊 Evaluation at step 4000 2025-11-23 04:33:56,093 - __main__ - INFO - 📤 Pushing checkpoint 4000 to HuggingFace Hub... 2025-11-23 04:33:56,095 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 04:34:06,396 - __main__ - INFO - ✅ Checkpoint 4000 pushed successfully! 2025-11-23 04:34:06,398 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-4000 2025-11-23 04:34:26,610 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.9%) 2025-11-23 04:34:47,206 - __main__ - INFO - 📈 Epoch progress: 0.30/6 (5.0%) 2025-11-23 04:35:07,510 - __main__ - INFO - 📈 Epoch progress: 0.31/6 (5.1%) 2025-11-23 04:35:27,928 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.3%) 2025-11-23 04:35:48,227 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.4%) 2025-11-23 04:36:08,893 - __main__ - INFO - 📈 Epoch progress: 0.33/6 (5.5%) 2025-11-23 04:36:29,539 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.6%) 2025-11-23 04:36:50,343 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.7%) 2025-11-23 04:37:10,968 - __main__ - INFO - 📈 Epoch progress: 0.35/6 (5.9%) 2025-11-23 04:37:31,505 - __main__ - INFO - 📈 Epoch progress: 0.36/6 (6.0%) 2025-11-23 04:37:52,113 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.1%) 2025-11-23 04:38:12,698 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.2%) 2025-11-23 04:38:33,129 - __main__ - INFO - 📈 Epoch progress: 0.38/6 (6.3%) 2025-11-23 04:38:54,508 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.5%) 2025-11-23 04:39:15,066 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.6%) 2025-11-23 04:39:35,569 - __main__ - INFO - 📈 Epoch progress: 0.40/6 (6.7%) 2025-11-23 04:39:56,468 - __main__ - INFO - 📈 Epoch progress: 0.41/6 (6.8%) 2025-11-23 04:40:16,968 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (6.9%) 2025-11-23 04:40:29,982 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 04:40:40,394 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (7.0%) 2025-11-23 04:41:00,393 - __main__ - INFO - 📈 Epoch progress: 0.43/6 (7.2%) 2025-11-23 04:41:48,848 - __main__ - INFO - 📊 Evaluation 4: BLEU = 0.1098 2025-11-23 04:41:49,164 - __main__ - INFO - 📊 Evaluation at step 6000 2025-11-23 04:41:52,562 - __main__ - INFO - 📤 Pushing checkpoint 6000 to HuggingFace Hub... 2025-11-23 04:41:52,563 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 04:42:01,604 - __main__ - INFO - ✅ Checkpoint 6000 pushed successfully! 2025-11-23 04:42:01,616 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-6000 2025-11-23 04:42:21,350 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.3%) 2025-11-23 04:42:41,788 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.4%) 2025-11-23 04:43:02,549 - __main__ - INFO - 📈 Epoch progress: 0.45/6 (7.5%) 2025-11-23 04:43:23,339 - __main__ - INFO - 📈 Epoch progress: 0.46/6 (7.6%) 2025-11-23 04:43:43,873 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.8%) 2025-11-23 04:44:04,296 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.9%) 2025-11-23 04:44:24,756 - __main__ - INFO - 📈 Epoch progress: 0.48/6 (8.0%) 2025-11-23 04:44:44,869 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.1%) 2025-11-23 04:45:05,284 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.2%) 2025-11-23 04:45:25,479 - __main__ - INFO - 📈 Epoch progress: 0.50/6 (8.4%) 2025-11-23 04:45:46,062 - __main__ - INFO - 📈 Epoch progress: 0.51/6 (8.5%) 2025-11-23 04:46:06,580 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.6%) 2025-11-23 04:46:27,033 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.7%) 2025-11-23 04:46:47,558 - __main__ - INFO - 📈 Epoch progress: 0.53/6 (8.8%) 2025-11-23 04:47:08,065 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.0%) 2025-11-23 04:47:28,658 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.1%) 2025-11-23 04:47:49,153 - __main__ - INFO - 📈 Epoch progress: 0.55/6 (9.2%) 2025-11-23 04:48:10,185 - __main__ - INFO - 📈 Epoch progress: 0.56/6 (9.3%) 2025-11-23 04:48:30,182 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.4%) 2025-11-23 04:48:50,420 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.6%) 2025-11-23 04:49:36,565 - __main__ - INFO - 📊 Evaluation 5: BLEU = 0.1175 2025-11-23 04:49:36,883 - __main__ - INFO - 📊 Evaluation at step 8000 2025-11-23 04:49:40,471 - __main__ - INFO - 📤 Pushing checkpoint 8000 to HuggingFace Hub... 2025-11-23 04:49:40,472 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 04:49:52,284 - __main__ - INFO - ✅ Checkpoint 8000 pushed successfully! 2025-11-23 04:49:52,285 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-8000 2025-11-23 04:50:12,334 - __main__ - INFO - 📈 Epoch progress: 0.58/6 (9.7%) 2025-11-23 04:50:32,901 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.8%) 2025-11-23 04:50:53,451 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.9%) 2025-11-23 04:51:13,926 - __main__ - INFO - 📈 Epoch progress: 0.60/6 (10.0%) 2025-11-23 04:51:33,994 - __main__ - INFO - 📈 Epoch progress: 0.61/6 (10.2%) 2025-11-23 04:51:54,443 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.3%) 2025-11-23 04:52:06,094 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 04:52:16,233 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.4%) 2025-11-23 04:52:36,718 - __main__ - INFO - 📈 Epoch progress: 0.63/6 (10.5%) 2025-11-23 04:52:57,184 - __main__ - INFO - 📈 Epoch progress: 0.64/6 (10.6%) 2025-11-23 04:53:18,002 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.8%) 2025-11-23 04:53:38,479 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.9%) 2025-11-23 04:53:58,840 - __main__ - INFO - 📈 Epoch progress: 0.66/6 (11.0%) 2025-11-23 04:54:19,816 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.1%) 2025-11-23 04:54:40,670 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.2%) 2025-11-23 04:55:00,739 - __main__ - INFO - 📈 Epoch progress: 0.68/6 (11.3%) 2025-11-23 04:55:21,422 - __main__ - INFO - 📈 Epoch progress: 0.69/6 (11.5%) 2025-11-23 04:55:42,073 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.6%) 2025-11-23 04:56:02,920 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.7%) 2025-11-23 04:56:23,022 - __main__ - INFO - 📈 Epoch progress: 0.71/6 (11.8%) 2025-11-23 04:56:44,055 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (11.9%) 2025-11-23 04:57:32,881 - __main__ - INFO - 📊 Evaluation 6: BLEU = 0.1201 2025-11-23 04:57:33,225 - __main__ - INFO - 📊 Evaluation at step 10000 2025-11-23 04:57:36,863 - __main__ - INFO - 📤 Pushing checkpoint 10000 to HuggingFace Hub... 2025-11-23 04:57:36,864 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 04:57:46,391 - __main__ - INFO - ✅ Checkpoint 10000 pushed successfully! 2025-11-23 04:57:46,392 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-10000 2025-11-23 04:58:07,139 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (12.1%) 2025-11-23 04:58:27,867 - __main__ - INFO - 📈 Epoch progress: 0.73/6 (12.2%) 2025-11-23 04:58:48,310 - __main__ - INFO - 📈 Epoch progress: 0.74/6 (12.3%) 2025-11-23 04:59:08,967 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.4%) 2025-11-23 04:59:29,328 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.5%) 2025-11-23 04:59:50,033 - __main__ - INFO - 📈 Epoch progress: 0.76/6 (12.7%) 2025-11-23 05:00:10,362 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.8%) 2025-11-23 05:00:31,242 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.9%) 2025-11-23 05:00:52,420 - __main__ - INFO - 📈 Epoch progress: 0.78/6 (13.0%) 2025-11-23 05:01:13,259 - __main__ - INFO - 📈 Epoch progress: 0.79/6 (13.1%) 2025-11-23 05:01:33,648 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.3%) 2025-11-23 05:01:53,923 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.4%) 2025-11-23 05:02:14,830 - __main__ - INFO - 📈 Epoch progress: 0.81/6 (13.5%) 2025-11-23 05:02:35,660 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.6%) 2025-11-23 05:02:55,972 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.7%) 2025-11-23 05:03:16,924 - __main__ - INFO - 📈 Epoch progress: 0.83/6 (13.9%) 2025-11-23 05:03:37,496 - __main__ - INFO - 📈 Epoch progress: 0.84/6 (14.0%) 2025-11-23 05:03:57,884 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.1%) 2025-11-23 05:04:18,312 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.2%) 2025-11-23 05:04:38,492 - __main__ - INFO - 📈 Epoch progress: 0.86/6 (14.3%) 2025-11-23 05:05:32,672 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:05:32,673 - __main__ - INFO - 📊 Evaluation 7: BLEU = 0.1172 2025-11-23 05:05:33,486 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:05:33,487 - __main__ - INFO - 📊 Evaluation at step 12000 2025-11-23 05:05:38,273 - __main__ - INFO - 📤 Pushing checkpoint 12000 to HuggingFace Hub... 2025-11-23 05:05:38,274 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 05:05:50,516 - __main__ - INFO - ✅ Checkpoint 12000 pushed successfully! 2025-11-23 05:05:50,517 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-12000 2025-11-23 05:06:10,571 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.5%) 2025-11-23 05:06:30,760 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.6%) 2025-11-23 05:06:51,435 - __main__ - INFO - 📈 Epoch progress: 0.88/6 (14.7%) 2025-11-23 05:07:11,929 - __main__ - INFO - 📈 Epoch progress: 0.89/6 (14.8%) 2025-11-23 05:07:32,329 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (14.9%) 2025-11-23 05:07:52,931 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (15.1%) 2025-11-23 05:08:13,396 - __main__ - INFO - 📈 Epoch progress: 0.91/6 (15.2%) 2025-11-23 05:08:34,125 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.3%) 2025-11-23 05:08:54,743 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.4%) 2025-11-23 05:09:15,545 - __main__ - INFO - 📈 Epoch progress: 0.93/6 (15.5%) 2025-11-23 05:09:35,761 - __main__ - INFO - 📈 Epoch progress: 0.94/6 (15.6%) 2025-11-23 05:09:56,678 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.8%) 2025-11-23 05:10:17,145 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.9%) 2025-11-23 05:10:37,457 - __main__ - INFO - 📈 Epoch progress: 0.96/6 (16.0%) 2025-11-23 05:10:57,757 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.1%) 2025-11-23 05:11:18,234 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.2%) 2025-11-23 05:11:39,210 - __main__ - INFO - 📈 Epoch progress: 0.98/6 (16.4%) 2025-11-23 05:11:59,688 - __main__ - INFO - 📈 Epoch progress: 0.99/6 (16.5%) 2025-11-23 05:12:19,771 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.6%) 2025-11-23 05:12:41,806 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.7%) 2025-11-23 05:13:28,678 - __main__ - INFO - 📊 Evaluation 8: BLEU = 0.1178 2025-11-23 05:13:29,017 - __main__ - INFO - 📊 Evaluation at step 14000 2025-11-23 05:13:33,956 - __main__ - INFO - 📤 Pushing checkpoint 14000 to HuggingFace Hub... 2025-11-23 05:13:33,956 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 05:13:44,412 - __main__ - INFO - ✅ Checkpoint 14000 pushed successfully! 2025-11-23 05:13:44,414 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-14000 2025-11-23 05:14:04,241 - __main__ - INFO - 📈 Epoch progress: 1.01/6 (16.8%) 2025-11-23 05:14:24,517 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.0%) 2025-11-23 05:14:44,594 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.1%) 2025-11-23 05:15:05,254 - __main__ - INFO - 📈 Epoch progress: 1.03/6 (17.2%) 2025-11-23 05:15:25,675 - __main__ - INFO - 📈 Epoch progress: 1.04/6 (17.3%) 2025-11-23 05:15:46,194 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.4%) 2025-11-23 05:16:06,354 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.6%) 2025-11-23 05:16:26,813 - __main__ - INFO - 📈 Epoch progress: 1.06/6 (17.7%) 2025-11-23 05:16:47,591 - __main__ - INFO - 📈 Epoch progress: 1.07/6 (17.8%) 2025-11-23 05:17:08,129 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (17.9%) 2025-11-23 05:17:28,440 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (18.0%) 2025-11-23 05:17:48,715 - __main__ - INFO - 📈 Epoch progress: 1.09/6 (18.2%) 2025-11-23 05:18:09,243 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.3%) 2025-11-23 05:18:29,913 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.4%) 2025-11-23 05:18:51,203 - __main__ - INFO - 📈 Epoch progress: 1.11/6 (18.5%) 2025-11-23 05:19:11,745 - __main__ - INFO - 📈 Epoch progress: 1.12/6 (18.6%) 2025-11-23 05:19:32,497 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.8%) 2025-11-23 05:19:53,187 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.9%) 2025-11-23 05:20:13,965 - __main__ - INFO - 📈 Epoch progress: 1.14/6 (19.0%) 2025-11-23 05:20:34,523 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.1%) 2025-11-23 05:21:24,032 - __main__ - INFO - 📊 Evaluation 9: BLEU = 0.1183 2025-11-23 05:21:24,377 - __main__ - INFO - 📊 Evaluation at step 16000 2025-11-23 05:21:28,472 - __main__ - INFO - 📤 Pushing checkpoint 16000 to HuggingFace Hub... 2025-11-23 05:21:28,473 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 05:21:40,266 - __main__ - INFO - ✅ Checkpoint 16000 pushed successfully! 2025-11-23 05:21:40,267 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-16000 2025-11-23 05:22:00,311 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.2%) 2025-11-23 05:22:21,476 - __main__ - INFO - 📈 Epoch progress: 1.16/6 (19.4%) 2025-11-23 05:22:42,000 - __main__ - INFO - 📈 Epoch progress: 1.17/6 (19.5%) 2025-11-23 05:23:02,752 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.6%) 2025-11-23 05:23:23,438 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.7%) 2025-11-23 05:23:44,156 - __main__ - INFO - 📈 Epoch progress: 1.19/6 (19.8%) 2025-11-23 05:24:04,702 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (19.9%) 2025-11-23 05:24:25,167 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (20.1%) 2025-11-23 05:24:45,698 - __main__ - INFO - 📈 Epoch progress: 1.21/6 (20.2%) 2025-11-23 05:25:06,038 - __main__ - INFO - 📈 Epoch progress: 1.22/6 (20.3%) 2025-11-23 05:25:26,872 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.4%) 2025-11-23 05:25:47,382 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.5%) 2025-11-23 05:26:07,913 - __main__ - INFO - 📈 Epoch progress: 1.24/6 (20.7%) 2025-11-23 05:26:28,562 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.8%) 2025-11-23 05:26:49,511 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.9%) 2025-11-23 05:27:10,199 - __main__ - INFO - 📈 Epoch progress: 1.26/6 (21.0%) 2025-11-23 05:27:30,936 - __main__ - INFO - 📈 Epoch progress: 1.27/6 (21.1%) 2025-11-23 05:27:51,405 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.3%) 2025-11-23 05:28:11,636 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.4%) 2025-11-23 05:28:32,412 - __main__ - INFO - 📈 Epoch progress: 1.29/6 (21.5%) 2025-11-23 05:29:18,518 - __main__ - INFO - 📊 Evaluation 10: BLEU = 0.1219 2025-11-23 05:29:18,888 - __main__ - INFO - 📊 Evaluation at step 18000 2025-11-23 05:29:22,587 - __main__ - INFO - 📤 Pushing checkpoint 18000 to HuggingFace Hub... 2025-11-23 05:29:22,588 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 05:29:32,167 - __main__ - INFO - ✅ Checkpoint 18000 pushed successfully! 2025-11-23 05:29:32,168 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-18000 2025-11-23 05:29:52,949 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.6%) 2025-11-23 05:30:13,314 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.7%) 2025-11-23 05:30:33,392 - __main__ - INFO - 📈 Epoch progress: 1.31/6 (21.9%) 2025-11-23 05:30:53,705 - __main__ - INFO - 📈 Epoch progress: 1.32/6 (22.0%) 2025-11-23 05:31:22,316 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.1%) 2025-11-23 05:31:43,345 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.2%) 2025-11-23 05:32:03,977 - __main__ - INFO - 📈 Epoch progress: 1.34/6 (22.3%) 2025-11-23 05:32:24,473 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.5%) 2025-11-23 05:32:45,183 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.6%) 2025-11-23 05:33:05,536 - __main__ - INFO - 📈 Epoch progress: 1.36/6 (22.7%) 2025-11-23 05:33:26,144 - __main__ - INFO - 📈 Epoch progress: 1.37/6 (22.8%) 2025-11-23 05:33:46,767 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (22.9%) 2025-11-23 05:34:07,141 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (23.1%) 2025-11-23 05:34:27,441 - __main__ - INFO - 📈 Epoch progress: 1.39/6 (23.2%) 2025-11-23 05:34:48,175 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.3%) 2025-11-23 05:35:08,694 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.4%) 2025-11-23 05:35:29,100 - __main__ - INFO - 📈 Epoch progress: 1.41/6 (23.5%) 2025-11-23 05:35:49,987 - __main__ - INFO - 📈 Epoch progress: 1.42/6 (23.7%) 2025-11-23 05:36:10,311 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.8%) 2025-11-23 05:36:30,472 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.9%) 2025-11-23 05:37:17,929 - __main__ - INFO - 📊 Evaluation 11: BLEU = 0.1225 2025-11-23 05:37:18,295 - __main__ - INFO - 📊 Evaluation at step 20000 2025-11-23 05:37:22,372 - __main__ - INFO - 📤 Pushing checkpoint 20000 to HuggingFace Hub... 2025-11-23 05:37:22,373 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 05:37:33,166 - __main__ - INFO - ✅ Checkpoint 20000 pushed successfully! 2025-11-23 05:37:33,167 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-20000 2025-11-23 05:37:54,117 - __main__ - INFO - 📈 Epoch progress: 1.44/6 (24.0%) 2025-11-23 05:38:14,559 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.1%) 2025-11-23 05:38:35,292 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.2%) 2025-11-23 05:38:55,844 - __main__ - INFO - 📈 Epoch progress: 1.46/6 (24.4%) 2025-11-23 05:39:17,013 - __main__ - INFO - 📈 Epoch progress: 1.47/6 (24.5%) 2025-11-23 05:39:37,798 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.6%) 2025-11-23 05:39:58,948 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.7%) 2025-11-23 05:40:19,537 - __main__ - INFO - 📈 Epoch progress: 1.49/6 (24.8%) 2025-11-23 05:40:40,399 - __main__ - INFO - 📈 Epoch progress: 1.50/6 (25.0%) 2025-11-23 05:41:00,822 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.1%) 2025-11-23 05:41:21,593 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.2%) 2025-11-23 05:41:42,212 - __main__ - INFO - 📈 Epoch progress: 1.52/6 (25.3%) 2025-11-23 05:42:03,403 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.4%) 2025-11-23 05:42:23,842 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.6%) 2025-11-23 05:42:44,097 - __main__ - INFO - 📈 Epoch progress: 1.54/6 (25.7%) 2025-11-23 05:43:04,918 - __main__ - INFO - 📈 Epoch progress: 1.55/6 (25.8%) 2025-11-23 05:43:25,164 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (25.9%) 2025-11-23 05:43:54,430 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:43:54,432 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (26.0%) 2025-11-23 05:43:54,742 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:44:14,598 - __main__ - INFO - 📈 Epoch progress: 1.57/6 (26.2%) 2025-11-23 05:44:35,347 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.3%) 2025-11-23 05:45:31,338 - __main__ - INFO - 📊 Evaluation 12: BLEU = 0.1133 2025-11-23 05:45:31,674 - __main__ - INFO - 📊 Evaluation at step 22000 2025-11-23 05:45:35,719 - __main__ - INFO - 📤 Pushing checkpoint 22000 to HuggingFace Hub... 2025-11-23 05:45:35,721 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 05:45:44,938 - __main__ - INFO - ✅ Checkpoint 22000 pushed successfully! 2025-11-23 05:45:44,939 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-22000 2025-11-23 05:46:05,000 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.4%) 2025-11-23 05:46:25,834 - __main__ - INFO - 📈 Epoch progress: 1.59/6 (26.5%) 2025-11-23 05:46:46,517 - __main__ - INFO - 📈 Epoch progress: 1.60/6 (26.6%) 2025-11-23 05:47:06,755 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.8%) 2025-11-23 05:47:27,574 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.9%) 2025-11-23 05:47:47,894 - __main__ - INFO - 📈 Epoch progress: 1.62/6 (27.0%) 2025-11-23 05:48:08,503 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.1%) 2025-11-23 05:48:29,258 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.2%) 2025-11-23 05:48:50,206 - __main__ - INFO - 📈 Epoch progress: 1.64/6 (27.4%) 2025-11-23 05:49:11,012 - __main__ - INFO - 📈 Epoch progress: 1.65/6 (27.5%) 2025-11-23 05:49:22,154 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:49:32,090 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:49:32,091 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.6%) 2025-11-23 05:49:32,420 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:49:43,023 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:49:52,768 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.7%) 2025-11-23 05:50:13,180 - __main__ - INFO - 📈 Epoch progress: 1.67/6 (27.8%) 2025-11-23 05:50:33,663 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.0%) 2025-11-23 05:50:54,100 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.1%) 2025-11-23 05:51:14,972 - __main__ - INFO - 📈 Epoch progress: 1.69/6 (28.2%) 2025-11-23 05:51:35,547 - __main__ - INFO - 📈 Epoch progress: 1.70/6 (28.3%) 2025-11-23 05:51:55,721 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.4%) 2025-11-23 05:52:16,479 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.5%) 2025-11-23 05:52:36,928 - __main__ - INFO - 📈 Epoch progress: 1.72/6 (28.7%) 2025-11-23 05:53:20,026 - __main__ - INFO - 📊 Evaluation 13: BLEU = 0.1197 2025-11-23 05:53:20,363 - __main__ - INFO - 📊 Evaluation at step 24000 2025-11-23 05:53:24,050 - __main__ - INFO - 📤 Pushing checkpoint 24000 to HuggingFace Hub... 2025-11-23 05:53:24,050 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 05:53:35,770 - __main__ - INFO - ✅ Checkpoint 24000 pushed successfully! 2025-11-23 05:53:35,771 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-24000 2025-11-23 05:53:55,952 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.8%) 2025-11-23 05:54:16,149 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.9%) 2025-11-23 05:54:36,530 - __main__ - INFO - 📈 Epoch progress: 1.74/6 (29.0%) 2025-11-23 05:54:57,769 - __main__ - INFO - 📈 Epoch progress: 1.75/6 (29.1%) 2025-11-23 05:55:17,644 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.3%) 2025-11-23 05:55:38,341 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.4%) 2025-11-23 05:55:58,622 - __main__ - INFO - 📈 Epoch progress: 1.77/6 (29.5%) 2025-11-23 05:56:18,961 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.6%) 2025-11-23 05:56:39,998 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.7%) 2025-11-23 05:57:00,353 - __main__ - INFO - 📈 Epoch progress: 1.79/6 (29.9%) 2025-11-23 05:57:20,656 - __main__ - INFO - 📈 Epoch progress: 1.80/6 (30.0%) 2025-11-23 05:57:36,826 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:57:46,733 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:57:46,734 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.1%) 2025-11-23 05:57:47,030 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:57:56,836 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:58:06,747 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:58:06,747 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.2%) 2025-11-23 05:58:07,049 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:58:16,831 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 05:58:27,177 - __main__ - INFO - 📈 Epoch progress: 1.82/6 (30.3%) 2025-11-23 05:58:48,335 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.5%) 2025-11-23 05:59:09,475 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.6%) 2025-11-23 05:59:29,797 - __main__ - INFO - 📈 Epoch progress: 1.84/6 (30.7%) 2025-11-23 05:59:50,498 - __main__ - INFO - 📈 Epoch progress: 1.85/6 (30.8%) 2025-11-23 06:00:11,935 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (30.9%) 2025-11-23 06:00:32,557 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (31.1%) 2025-11-23 06:01:16,076 - __main__ - INFO - 📊 Evaluation 14: BLEU = 0.1220 2025-11-23 06:01:16,758 - __main__ - INFO - 📊 Evaluation at step 26000 2025-11-23 06:01:20,119 - __main__ - INFO - 📤 Pushing checkpoint 26000 to HuggingFace Hub... 2025-11-23 06:01:20,120 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 06:01:29,811 - __main__ - INFO - ✅ Checkpoint 26000 pushed successfully! 2025-11-23 06:01:29,814 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-26000 2025-11-23 06:01:49,787 - __main__ - INFO - 📈 Epoch progress: 1.87/6 (31.2%) 2025-11-23 06:02:10,995 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.3%) 2025-11-23 06:02:31,913 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.4%) 2025-11-23 06:02:53,351 - __main__ - INFO - 📈 Epoch progress: 1.89/6 (31.5%) 2025-11-23 06:03:13,904 - __main__ - INFO - 📈 Epoch progress: 1.90/6 (31.7%) 2025-11-23 06:03:35,786 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.8%) 2025-11-23 06:03:55,853 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.9%) 2025-11-23 06:04:16,517 - __main__ - INFO - 📈 Epoch progress: 1.92/6 (32.0%) 2025-11-23 06:04:37,380 - __main__ - INFO - 📈 Epoch progress: 1.93/6 (32.1%) 2025-11-23 06:05:08,205 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.3%) 2025-11-23 06:05:28,854 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.4%) 2025-11-23 06:05:49,622 - __main__ - INFO - 📈 Epoch progress: 1.95/6 (32.5%) 2025-11-23 06:06:10,932 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.6%) 2025-11-23 06:06:31,737 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.7%) 2025-11-23 06:06:51,966 - __main__ - INFO - 📈 Epoch progress: 1.97/6 (32.8%) 2025-11-23 06:07:12,494 - __main__ - INFO - 📈 Epoch progress: 1.98/6 (33.0%) 2025-11-23 06:07:33,444 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.1%) 2025-11-23 06:07:55,079 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.2%) 2025-11-23 06:08:15,763 - __main__ - INFO - 📈 Epoch progress: 2.00/6 (33.3%) 2025-11-23 06:08:37,725 - __main__ - INFO - 📈 Epoch progress: 2.01/6 (33.4%) 2025-11-23 06:09:16,114 - __main__ - INFO - 📊 Evaluation 15: BLEU = 0.1212 2025-11-23 06:09:16,813 - __main__ - INFO - 📊 Evaluation at step 28000 2025-11-23 06:09:20,719 - __main__ - INFO - 📤 Pushing checkpoint 28000 to HuggingFace Hub... 2025-11-23 06:09:20,719 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 06:09:30,705 - __main__ - INFO - ✅ Checkpoint 28000 pushed successfully! 2025-11-23 06:09:30,717 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-28000 2025-11-23 06:09:34,395 - __main__ - INFO - ✅ MLflow: Training finished - status_code=2.0 (FINISHED) 2025-11-23 06:09:34,733 - __main__ - INFO - 💾 Saving model to models/finetuned-sw-en-phase2 2025-11-23 06:09:36,180 - __main__ - INFO - ✅ Training complete! 2025-11-23 06:09:36,181 - __main__ - INFO - 📊 STEP 3: Final Evaluation 2025-11-23 06:09:36,530 - __main__ - INFO - ✅ MLflow: Final evaluation starting - status_code=3.0 2025-11-23 06:09:36,531 - __main__ - INFO - ====================================================================== 2025-11-23 06:09:36,532 - __main__ - INFO - 📊 Running FINAL evaluation on test set... 2025-11-23 06:09:36,533 - __main__ - INFO - ====================================================================== 2025-11-23 06:09:36,534 - __main__ - INFO - Tokenizing test dataset... 2025-11-23 06:09:38,520 - datasets.fingerprint - WARNING - Parameter 'function'=> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only shown once. Subsequent hashing failures won't be shown. 2025-11-23 06:41:24,155 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-23 06:41:24,162 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json 2025-11-23 06:41:24,163 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-23 06:41:24,163 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-23 06:41:24,166 - __main__ - INFO - 🎲 Random seed: 42 2025-11-23 06:41:24,166 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-23 06:41:24,167 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 06:41:24,167 - __main__ - INFO - Push strategy: checkpoint 2025-11-23 06:41:24,168 - __main__ - INFO - → Will push after each checkpoint save 2025-11-23 06:41:24,168 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-23 06:41:24,169 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-23 06:41:24,170 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-23 06:41:24,178 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-23 06:41:24,179 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 06:41:24,827 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-23 06:41:25,688 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251123_064125 2025-11-23 06:41:25,690 - __main__ - INFO - Run ID: 54a97666ebd34e46a8ce61941ee6341c 2025-11-23 06:41:25,691 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 06:41:26,048 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow 2025-11-23 06:41:26,051 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-23 06:41:26,061 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-23 06:41:26,061 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-23 06:41:26,062 - __main__ - INFO - 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-23 06:41:31,514 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-23 06:41:31,515 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-23 06:41:31,921 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file 2025-11-23 06:41:32,192 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file 2025-11-23 06:41:32,193 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl 2025-11-23 06:41:33,774 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) 2025-11-23 06:41:36,713 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 2025-11-23 06:41:39,276 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 2025-11-23 06:41:41,691 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 2025-11-23 06:41:44,079 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 2025-11-23 06:41:46,461 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 2025-11-23 06:41:48,840 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 2025-11-23 06:41:51,248 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 2025-11-23 06:41:53,590 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 2025-11-23 06:41:55,986 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 2025-11-23 06:41:58,421 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 2025-11-23 06:42:00,853 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 2025-11-23 06:42:03,291 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 2025-11-23 06:42:05,726 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 2025-11-23 06:42:08,157 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 2025-11-23 06:42:10,577 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 2025-11-23 06:42:12,974 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 2025-11-23 06:42:15,394 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 2025-11-23 06:42:17,840 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 2025-11-23 06:42:20,309 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 2025-11-23 06:42:22,760 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 2025-11-23 06:42:25,196 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 2025-11-23 06:42:27,644 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 2025-11-23 06:42:30,074 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 2025-11-23 06:42:32,478 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 2025-11-23 06:42:34,882 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 2025-11-23 06:42:35,119 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) 2025-11-23 06:42:35,156 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl 2025-11-23 06:42:36,942 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) 2025-11-23 06:42:39,956 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 2025-11-23 06:42:42,426 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 2025-11-23 06:42:44,876 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 2025-11-23 06:42:47,303 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 2025-11-23 06:42:49,779 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 2025-11-23 06:42:52,232 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 2025-11-23 06:42:54,688 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 2025-11-23 06:42:57,144 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 2025-11-23 06:42:59,590 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 2025-11-23 06:43:02,033 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 2025-11-23 06:43:04,493 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 2025-11-23 06:43:06,941 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 2025-11-23 06:43:09,378 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 2025-11-23 06:43:11,817 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 2025-11-23 06:43:14,265 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 2025-11-23 06:43:16,716 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 2025-11-23 06:43:18,994 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) 2025-11-23 06:43:19,040 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 2025-11-23 06:43:19,041 - __main__ - INFO - 📝 Sample data check: 2025-11-23 06:43:19,042 - __main__ - INFO - Sample 1: 2025-11-23 06:43:19,043 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... 2025-11-23 06:43:19,044 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... 2025-11-23 06:43:19,045 - __main__ - INFO - Sample 2: 2025-11-23 06:43:19,046 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... 2025-11-23 06:43:19,046 - __main__ - INFO - Target (6 words): Job had said no such thing.... 2025-11-23 06:43:19,047 - __main__ - INFO - Sample 3: 2025-11-23 06:43:19,048 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... 2025-11-23 06:43:19,048 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... 2025-11-23 06:43:24,401 - __main__ - INFO - Filtering statistics: {'kept': 4202243} 2025-11-23 06:43:24,417 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 2025-11-23 06:43:28,817 - __main__ - INFO - Removed 518 duplicates 2025-11-23 06:43:29,060 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 2025-11-23 06:43:35,738 - __main__ - INFO - ====================================================================== 2025-11-23 06:43:35,740 - __main__ - INFO - ✅ Dataset splits: 2025-11-23 06:43:35,740 - __main__ - INFO - Train: 3,571,465 samples 2025-11-23 06:43:35,741 - __main__ - INFO - Validation: 420,173 samples 2025-11-23 06:43:35,742 - __main__ - INFO - Test: 210,087 samples 2025-11-23 06:43:35,743 - __main__ - INFO - ====================================================================== 2025-11-23 06:43:36,203 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-23 06:46:40,471 - __main__ - INFO - ✅ Dataset validation passed 2025-11-23 06:46:40,473 - __main__ - INFO - Train: 3,571,465 samples 2025-11-23 06:46:40,474 - __main__ - INFO - Validation: 420,173 samples 2025-11-23 06:46:40,475 - __main__ - INFO - Test: 210,087 samples 2025-11-23 06:46:41,873 - __main__ - INFO - ✅ Found cached tokenized datasets (hash: 59e2a5c4) 2025-11-23 06:46:41,874 - __main__ - INFO - Skipping tokenization - this saves significant time! 2025-11-23 06:46:41,875 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... 2025-11-23 06:47:24,470 - __main__ - INFO - ✅ Tokenization complete! 2025-11-23 06:47:24,472 - __main__ - INFO - 🔧 Initializing COMET-QE model... 2025-11-23 06:47:24,839 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. 2025-11-23 06:47:25,934 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` 2025-11-23 06:47:32,709 - comet.models.base - INFO - Encoder model frozen. 2025-11-23 06:47:33,265 - __main__ - INFO - ✅ COMET-QE model loaded successfully 2025-11-23 06:47:33,266 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ 2025-11-23 06:47:33,298 - __main__ - INFO - 📊 Limiting evaluation dataset from 420225 to 2000 samples 2025-11-23 06:47:33,335 - __main__ - INFO - ✅ Trainer initialized 2025-11-23 06:47:33,336 - __main__ - INFO - 🔬 STEP 1: Baseline Evaluation 2025-11-23 06:47:33,836 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 2025-11-23 06:47:33,837 - __main__ - INFO - ====================================================================== 2025-11-23 06:47:33,838 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... 2025-11-23 06:47:33,839 - __main__ - INFO - ====================================================================== 2025-11-23 06:48:23,063 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.1215 2025-11-23 06:48:23,065 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-23 06:48:23,106 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... 2025-11-23 06:48:23,115 - __main__ - INFO - Batch 1/16... 2025-11-23 06:48:25,740 - __main__ - INFO - Batch 5/16... 2025-11-23 06:48:27,878 - __main__ - INFO - Batch 10/16... 2025-11-23 06:48:30,442 - __main__ - INFO - Batch 15/16... 2025-11-23 06:48:31,207 - __main__ - INFO - Running COMET-QE model... 2025-11-23 06:48:31,228 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-23 06:48:31,237 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-23 06:48:31,238 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-23 06:48:31,239 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision 2025-11-23 06:48:31,241 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-23 06:48:36,590 - __main__ - INFO - ✅ COMET-QE: 0.1977 ± 0.2784 2025-11-23 06:48:36,591 - __main__ - INFO - ✅ Baseline Results (Test Set): 2025-11-23 06:48:36,592 - __main__ - INFO - BLEU: 0.1215 2025-11-23 06:48:36,592 - __main__ - INFO - chrF: 32.36 2025-11-23 06:48:36,593 - __main__ - INFO - COMET: 0.1977 2025-11-23 06:48:36,594 - __main__ - INFO - ====================================================================== 2025-11-23 06:48:36,595 - __main__ - INFO - ====================================================================== 2025-11-23 06:48:36,596 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-23 06:48:36,596 - __main__ - INFO - ====================================================================== 2025-11-23 06:48:36,597 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-23 06:48:36,599 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-23 06:48:38,317 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-23 06:48:38,378 - __main__ - INFO - ✅ Domain chrF: 1.22 2025-11-23 06:48:38,380 - __main__ - INFO - Computing COMET-QE... 2025-11-23 06:48:38,386 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-23 06:48:38,394 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-23 06:48:38,395 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-23 06:48:38,397 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-23 06:48:41,106 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 2025-11-23 06:48:41,115 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-23 06:48:41,116 - __main__ - INFO - BLEU: 0.0000 2025-11-23 06:48:41,117 - __main__ - INFO - chrF: 1.22 2025-11-23 06:48:41,118 - __main__ - INFO - COMET-QE: 0.0001 2025-11-23 06:48:41,120 - __main__ - INFO - ====================================================================== 2025-11-23 06:48:41,726 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 2025-11-23 06:48:41,737 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase2/baseline_results.json 2025-11-23 06:48:41,742 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics 2025-11-23 06:48:41,744 - __main__ - INFO - 🚀 STEP 2: Fine-tuning 2025-11-23 06:48:41,745 - __main__ - INFO - Metrics are being logged to MLflow in real-time 2025-11-23 06:48:41,746 - __main__ - INFO - MLflow URL: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 06:48:41,747 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs 2025-11-23 06:48:41,749 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') 2025-11-23 06:48:41,750 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 06:48:41,751 - __main__ - INFO - ====================================================================== 2025-11-23 06:48:41,752 - __main__ - INFO - 🚀 Starting training... 2025-11-23 06:48:41,753 - __main__ - INFO - ====================================================================== 2025-11-23 06:48:42,779 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) 2025-11-23 06:49:20,694 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.1%) 2025-11-23 06:49:56,400 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.2%) 2025-11-23 06:50:32,830 - __main__ - INFO - 📈 Epoch progress: 0.02/6 (0.4%) 2025-11-23 06:51:08,804 - __main__ - INFO - 📈 Epoch progress: 0.03/6 (0.5%) 2025-11-23 06:51:44,666 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.6%) 2025-11-23 06:52:20,882 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.7%) 2025-11-23 06:52:59,996 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 06:52:59,998 - __main__ - INFO - 📈 Epoch progress: 0.05/6 (0.8%) 2025-11-23 06:53:00,318 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 06:53:17,708 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 06:53:35,669 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.0%) 2025-11-23 06:54:11,635 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.1%) 2025-11-23 06:54:47,036 - __main__ - INFO - 📈 Epoch progress: 0.07/6 (1.2%) 2025-11-23 06:55:22,872 - __main__ - INFO - 📈 Epoch progress: 0.08/6 (1.3%) 2025-11-23 06:55:57,782 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.4%) 2025-11-23 06:56:34,083 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.6%) 2025-11-23 06:57:10,153 - __main__ - INFO - 📈 Epoch progress: 0.10/6 (1.7%) 2025-11-23 06:57:45,970 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.8%) 2025-11-23 06:58:21,543 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 06:58:21,545 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.9%) 2025-11-23 06:58:21,879 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 06:58:57,226 - __main__ - INFO - 📈 Epoch progress: 0.12/6 (2.0%) 2025-11-23 06:59:32,715 - __main__ - INFO - 📈 Epoch progress: 0.13/6 (2.2%) 2025-11-23 07:00:08,615 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.3%) 2025-11-23 07:00:45,098 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.4%) 2025-11-23 07:02:10,816 - __main__ - INFO - 📊 Evaluation 2: BLEU = 0.1169 2025-11-23 07:02:11,162 - __main__ - INFO - 📊 Evaluation at step 2000 2025-11-23 07:02:15,403 - __main__ - INFO - ✅ Repository brendaogutu/mul-sw-en-translation-phase2 ready 2025-11-23 07:02:15,420 - __main__ - INFO - 📤 Pushing checkpoint 2000 to HuggingFace Hub... 2025-11-23 07:02:15,421 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 07:02:30,795 - __main__ - INFO - ✅ Checkpoint 2000 pushed successfully! 2025-11-23 07:02:30,800 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-2000 2025-11-23 07:03:07,071 - __main__ - INFO - 📈 Epoch progress: 0.15/6 (2.5%) 2025-11-23 07:03:43,908 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.6%) 2025-11-23 07:04:20,289 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.7%) 2025-11-23 07:04:56,501 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:04:56,516 - __main__ - INFO - 📈 Epoch progress: 0.17/6 (2.9%) 2025-11-23 07:04:56,852 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:05:32,222 - __main__ - INFO - 📈 Epoch progress: 0.18/6 (3.0%) 2025-11-23 07:06:08,173 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.1%) 2025-11-23 07:06:43,702 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.2%) 2025-11-23 07:07:20,094 - __main__ - INFO - 📈 Epoch progress: 0.20/6 (3.3%) 2025-11-23 07:07:55,646 - __main__ - INFO - 📈 Epoch progress: 0.21/6 (3.5%) 2025-11-23 07:08:31,361 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.6%) 2025-11-23 07:09:06,938 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.7%) 2025-11-23 07:09:42,535 - __main__ - INFO - 📈 Epoch progress: 0.23/6 (3.8%) 2025-11-23 07:10:19,180 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (3.9%) 2025-11-23 07:10:55,369 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (4.1%) 2025-11-23 07:11:31,824 - __main__ - INFO - 📈 Epoch progress: 0.25/6 (4.2%) 2025-11-23 07:12:08,087 - __main__ - INFO - 📈 Epoch progress: 0.26/6 (4.3%) 2025-11-23 07:12:44,805 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:12:44,816 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.4%) 2025-11-23 07:12:45,137 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:13:21,013 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.5%) 2025-11-23 07:13:55,915 - __main__ - INFO - 📈 Epoch progress: 0.28/6 (4.7%) 2025-11-23 07:14:32,742 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.8%) 2025-11-23 07:15:53,704 - __main__ - INFO - 📊 Evaluation 3: BLEU = 0.1186 2025-11-23 07:15:54,113 - __main__ - INFO - 📊 Evaluation at step 4000 2025-11-23 07:15:58,436 - __main__ - INFO - 📤 Pushing checkpoint 4000 to HuggingFace Hub... 2025-11-23 07:15:58,437 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 07:16:05,731 - __main__ - INFO - ✅ Checkpoint 4000 pushed successfully! 2025-11-23 07:16:05,732 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-4000 2025-11-23 07:16:42,057 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.9%) 2025-11-23 07:17:17,223 - __main__ - INFO - 📈 Epoch progress: 0.30/6 (5.0%) 2025-11-23 07:17:52,523 - __main__ - INFO - 📈 Epoch progress: 0.31/6 (5.1%) 2025-11-23 07:18:28,936 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.3%) 2025-11-23 07:19:05,115 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.4%) 2025-11-23 07:19:40,549 - __main__ - INFO - 📈 Epoch progress: 0.33/6 (5.5%) 2025-11-23 07:20:16,666 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.6%) 2025-11-23 07:20:52,558 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.7%) 2025-11-23 07:21:28,256 - __main__ - INFO - 📈 Epoch progress: 0.35/6 (5.9%) 2025-11-23 07:22:04,280 - __main__ - INFO - 📈 Epoch progress: 0.36/6 (6.0%) 2025-11-23 07:22:40,312 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.1%) 2025-11-23 07:23:15,817 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.2%) 2025-11-23 07:23:51,373 - __main__ - INFO - 📈 Epoch progress: 0.38/6 (6.3%) 2025-11-23 07:24:28,475 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.5%) 2025-11-23 07:25:05,401 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.6%) 2025-11-23 07:25:41,747 - __main__ - INFO - 📈 Epoch progress: 0.40/6 (6.7%) 2025-11-23 07:26:17,592 - __main__ - INFO - 📈 Epoch progress: 0.41/6 (6.8%) 2025-11-23 07:26:53,822 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (6.9%) 2025-11-23 07:27:29,503 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (7.0%) 2025-11-23 07:28:04,934 - __main__ - INFO - 📈 Epoch progress: 0.43/6 (7.2%) 2025-11-23 07:29:35,171 - __main__ - INFO - 📊 Evaluation 4: BLEU = 0.1098 2025-11-23 07:29:35,497 - __main__ - INFO - 📊 Evaluation at step 6000 2025-11-23 07:29:39,457 - __main__ - INFO - 📤 Pushing checkpoint 6000 to HuggingFace Hub... 2025-11-23 07:29:39,458 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 07:29:47,255 - __main__ - INFO - ✅ Checkpoint 6000 pushed successfully! 2025-11-23 07:29:47,257 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-6000 2025-11-23 07:30:22,257 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.3%) 2025-11-23 07:30:58,369 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.4%) 2025-11-23 07:31:35,126 - __main__ - INFO - 📈 Epoch progress: 0.45/6 (7.5%) 2025-11-23 07:32:12,216 - __main__ - INFO - 📈 Epoch progress: 0.46/6 (7.6%) 2025-11-23 07:32:37,401 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:32:55,033 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:32:55,036 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.8%) 2025-11-23 07:32:55,345 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:33:32,031 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.9%) 2025-11-23 07:34:08,553 - __main__ - INFO - 📈 Epoch progress: 0.48/6 (8.0%) 2025-11-23 07:34:45,333 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.1%) 2025-11-23 07:35:21,813 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.2%) 2025-11-23 07:35:57,659 - __main__ - INFO - 📈 Epoch progress: 0.50/6 (8.4%) 2025-11-23 07:36:34,656 - __main__ - INFO - 📈 Epoch progress: 0.51/6 (8.5%) 2025-11-23 07:37:11,328 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.6%) 2025-11-23 07:37:48,164 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.7%) 2025-11-23 07:38:25,336 - __main__ - INFO - 📈 Epoch progress: 0.53/6 (8.8%) 2025-11-23 07:39:03,247 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.0%) 2025-11-23 07:39:22,404 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:39:40,831 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:39:40,833 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.1%) 2025-11-23 07:39:41,190 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:39:59,010 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: ' Cloudflare Tunnel error | food-accepts-blocked-refinance.trycloudflare.com | Cloudflare

Error 1033

Ray ID: 9a2f040dc6afa2c4 • 2025-11-23 07:39:59 UTC

Cloudflare Tunnel error

What happened?

You've requested a page on a website (food-accepts-blocked-refinance.trycloudflare.com) that is on the Cloudflare network. The host (food-accepts-blocked-refinance.trycloudflare.com) is configured as a Cloudflare Tunnel, and Cloudflare is currently unable to resolve it.

What can I do?

If you are a visitor of this website:
Please try again in a few minutes.

If you are the owner of this website:
Ensure that cloudflared is running and can reach the network. You may wish to enable load balancing for your tunnel.

Cloudflare Ray ID: 9a2f040dc6afa2c4 • Performance & security by Cloudflare

' 2025-11-23 07:40:16,962 - __main__ - INFO - 📈 Epoch progress: 0.55/6 (9.2%) 2025-11-23 07:40:52,752 - __main__ - INFO - 📈 Epoch progress: 0.56/6 (9.3%) 2025-11-23 07:41:28,407 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.4%) 2025-11-23 07:42:04,220 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.6%) 2025-11-23 07:43:25,561 - __main__ - INFO - 📊 Evaluation 5: BLEU = 0.1175 2025-11-23 07:43:25,888 - __main__ - INFO - 📊 Evaluation at step 8000 2025-11-23 07:43:31,017 - __main__ - INFO - 📤 Pushing checkpoint 8000 to HuggingFace Hub... 2025-11-23 07:43:31,018 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 07:43:39,393 - __main__ - INFO - ✅ Checkpoint 8000 pushed successfully! 2025-11-23 07:43:39,395 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-8000 2025-11-23 07:43:59,385 - __main__ - INFO - 📈 Epoch progress: 0.58/6 (9.7%) 2025-11-23 07:44:19,912 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.8%) 2025-11-23 07:44:40,386 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.9%) 2025-11-23 07:45:00,848 - __main__ - INFO - 📈 Epoch progress: 0.60/6 (10.0%) 2025-11-23 07:45:20,858 - __main__ - INFO - 📈 Epoch progress: 0.61/6 (10.2%) 2025-11-23 07:45:41,250 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.3%) 2025-11-23 07:46:01,676 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.4%) 2025-11-23 07:46:21,983 - __main__ - INFO - 📈 Epoch progress: 0.63/6 (10.5%) 2025-11-23 07:46:42,221 - __main__ - INFO - 📈 Epoch progress: 0.64/6 (10.6%) 2025-11-23 07:47:02,695 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.8%) 2025-11-23 07:47:22,928 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.9%) 2025-11-23 07:47:37,482 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:47:47,339 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:47:47,340 - __main__ - INFO - 📈 Epoch progress: 0.66/6 (11.0%) 2025-11-23 07:47:47,677 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:47:57,719 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 07:48:07,924 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.1%) 2025-11-23 07:48:29,384 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.2%) 2025-11-23 07:48:49,792 - __main__ - INFO - 📈 Epoch progress: 0.68/6 (11.3%) 2025-11-23 07:49:10,658 - __main__ - INFO - 📈 Epoch progress: 0.69/6 (11.5%) 2025-11-23 07:49:31,428 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.6%) 2025-11-23 07:49:52,589 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.7%) 2025-11-23 07:50:13,540 - __main__ - INFO - 📈 Epoch progress: 0.71/6 (11.8%) 2025-11-23 07:50:33,579 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (11.9%) 2025-11-23 07:51:21,978 - __main__ - INFO - 📊 Evaluation 6: BLEU = 0.1201 2025-11-23 07:51:22,310 - __main__ - INFO - 📊 Evaluation at step 10000 2025-11-23 07:51:26,467 - __main__ - INFO - 📤 Pushing checkpoint 10000 to HuggingFace Hub... 2025-11-23 07:51:26,468 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 07:51:36,822 - __main__ - INFO - ✅ Checkpoint 10000 pushed successfully! 2025-11-23 07:51:36,824 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-10000 2025-11-23 07:51:57,838 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (12.1%) 2025-11-23 07:52:19,175 - __main__ - INFO - 📈 Epoch progress: 0.73/6 (12.2%) 2025-11-23 07:52:39,834 - __main__ - INFO - 📈 Epoch progress: 0.74/6 (12.3%) 2025-11-23 07:53:01,284 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.4%) 2025-11-23 07:53:21,589 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.5%) 2025-11-23 07:53:42,477 - __main__ - INFO - 📈 Epoch progress: 0.76/6 (12.7%) 2025-11-23 07:54:04,122 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.8%) 2025-11-23 07:54:25,232 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.9%) 2025-11-23 07:54:46,194 - __main__ - INFO - 📈 Epoch progress: 0.78/6 (13.0%) 2025-11-23 07:55:07,609 - __main__ - INFO - 📈 Epoch progress: 0.79/6 (13.1%) 2025-11-23 07:55:28,579 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.3%) 2025-11-23 07:55:49,099 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.4%) 2025-11-23 07:56:10,055 - __main__ - INFO - 📈 Epoch progress: 0.81/6 (13.5%) 2025-11-23 07:56:31,036 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.6%) 2025-11-23 07:56:51,675 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.7%) 2025-11-23 07:57:12,699 - __main__ - INFO - 📈 Epoch progress: 0.83/6 (13.9%) 2025-11-23 07:57:33,405 - __main__ - INFO - 📈 Epoch progress: 0.84/6 (14.0%) 2025-11-23 07:57:54,084 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.1%) 2025-11-23 07:58:15,527 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.2%) 2025-11-23 07:58:36,159 - __main__ - INFO - 📈 Epoch progress: 0.86/6 (14.3%) 2025-11-23 07:59:31,105 - __main__ - INFO - 📊 Evaluation 7: BLEU = 0.1172 2025-11-23 07:59:31,811 - __main__ - INFO - 📊 Evaluation at step 12000 2025-11-23 07:59:36,349 - __main__ - INFO - 📤 Pushing checkpoint 12000 to HuggingFace Hub... 2025-11-23 07:59:36,350 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 07:59:43,697 - __main__ - INFO - ✅ Checkpoint 12000 pushed successfully! 2025-11-23 07:59:43,698 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-12000 2025-11-23 07:59:55,944 - __main__ - INFO - 📊 Evaluation 16: BLEU = 0.1289 2025-11-23 07:59:56,301 - __main__ - INFO - 📊 Evaluation at step 28000 2025-11-23 07:59:56,472 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-23 07:59:56,528 - __main__ - INFO - Computing predictions for COMET (1000 samples, batch_size=32)... 2025-11-23 07:59:56,529 - __main__ - INFO - Batch 1/32... 2025-11-23 08:00:04,186 - __main__ - INFO - Batch 5/32... 2025-11-23 08:00:06,674 - __main__ - INFO - Batch 10/32... 2025-11-23 08:00:08,376 - __main__ - INFO - Batch 15/32... 2025-11-23 08:00:08,387 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.5%) 2025-11-23 08:00:10,797 - __main__ - INFO - Batch 20/32... 2025-11-23 08:00:13,377 - __main__ - INFO - Batch 25/32... 2025-11-23 08:00:15,932 - __main__ - INFO - Batch 30/32... 2025-11-23 08:00:17,281 - __main__ - INFO - Running COMET-QE model... 2025-11-23 08:00:17,287 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-23 08:00:17,296 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-23 08:00:17,297 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-23 08:00:17,299 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-23 08:00:25,701 - __main__ - INFO - ✅ COMET-QE: 0.1763 ± 0.2628 2025-11-23 08:00:25,716 - __main__ - INFO - ✅ FINAL Test Results: 2025-11-23 08:00:25,718 - __main__ - INFO - BLEU: 0.1289 2025-11-23 08:00:25,718 - __main__ - INFO - chrF: 32.25 2025-11-23 08:00:25,719 - __main__ - INFO - COMET-QE: 0.1763 2025-11-23 08:00:25,720 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:25,721 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:25,721 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-23 08:00:25,722 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:25,723 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-23 08:00:25,724 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-23 08:00:27,792 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-23 08:00:27,871 - __main__ - INFO - ✅ Domain chrF: 1.18 2025-11-23 08:00:27,872 - __main__ - INFO - Computing COMET-QE... 2025-11-23 08:00:27,876 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-23 08:00:27,885 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-23 08:00:27,886 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-23 08:00:27,888 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-23 08:00:30,990 - __main__ - INFO - ✅ Domain COMET-QE: 0.0000 2025-11-23 08:00:30,991 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-23 08:00:30,992 - __main__ - INFO - BLEU: 0.0000 2025-11-23 08:00:30,992 - __main__ - INFO - chrF: 1.18 2025-11-23 08:00:30,992 - __main__ - INFO - COMET-QE: 0.0000 2025-11-23 08:00:30,993 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:31,015 - __main__ - INFO - 💾 Final results saved to models/finetuned-sw-en-phase2/test_results.json 2025-11-23 08:00:31,022 - __main__ - INFO - ✅ Verified test_results.json with 14 metrics 2025-11-23 08:00:31,023 - __main__ - INFO - 📊 Logging all metrics to TensorBoard... 2025-11-23 08:00:31,177 - __main__ - INFO - ✅ All metrics logged to TensorBoard 2025-11-23 08:00:31,179 - __main__ - INFO - Baseline metrics: 13 2025-11-23 08:00:31,179 - __main__ - INFO - Final test metrics: 10 2025-11-23 08:00:31,180 - __main__ - INFO - Final domain metrics: 4 2025-11-23 08:00:31,181 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:31,181 - __main__ - INFO - 📊 BASELINE vs FINE-TUNED Comparison 2025-11-23 08:00:31,182 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:31,183 - __main__ - INFO - 📈 Test Set Metrics: 2025-11-23 08:00:31,183 - __main__ - INFO - Metric Baseline Fine-tuned Improvement 2025-11-23 08:00:31,184 - __main__ - INFO - ------------------------------------------------------------ 2025-11-23 08:00:31,185 - __main__ - INFO - eval_bleu 0.0000 0.1289 N/A 2025-11-23 08:00:31,186 - __main__ - INFO - eval_chrf 0.0000 32.2491 N/A 2025-11-23 08:00:31,186 - __main__ - INFO - eval_comet 0.0000 0.1763 N/A 2025-11-23 08:00:31,187 - __main__ - INFO - 📈 Domain Metrics (Real Call Transcriptions): 2025-11-23 08:00:31,188 - __main__ - INFO - Metric Baseline Fine-tuned Improvement 2025-11-23 08:00:31,189 - __main__ - INFO - ------------------------------------------------------------ 2025-11-23 08:00:31,190 - __main__ - INFO - domain_bleu 0.0000 0.0000 +1417.7% 2025-11-23 08:00:31,191 - __main__ - INFO - domain_chrf 1.2250 1.1786 -3.8% 2025-11-23 08:00:31,192 - __main__ - INFO - domain_comet 0.0001 0.0000 -26.2% 2025-11-23 08:00:31,192 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:32,089 - __main__ - INFO - ✅ Logged 14 final metrics to MLflow 2025-11-23 08:00:32,090 - __main__ - INFO - Visit MLflow: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 08:00:32,091 - __main__ - INFO - 📊 Captured 576 training log entries 2025-11-23 08:00:32,126 - __main__ - INFO - 💾 Saved comprehensive metrics to models/finetuned-sw-en-phase2/training_metrics.json 2025-11-23 08:00:32,143 - __main__ - INFO - 💾 Saved training config to models/finetuned-sw-en-phase2/training_config_used.json 2025-11-23 08:00:32,178 - __main__ - INFO - 💾 Copied training.log to output directory 2025-11-23 08:00:32,179 - __main__ - INFO - 🤖 HUGGINGFACE HUB 2025-11-23 08:00:32,180 - __main__ - INFO - ℹ️ Checkpoints were pushed during training (push_strategy='checkpoint') 2025-11-23 08:00:32,181 - __main__ - INFO - 📤 Pushing final model to hub... 2025-11-23 08:00:32,181 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:32,182 - __main__ - INFO - 📤 Uploading model to HuggingFace Hub... 2025-11-23 08:00:32,182 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:32,183 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 08:00:32,190 - __main__ - INFO - ✅ Model card generated 2025-11-23 08:00:32,333 - huggingface_hub._login - WARNING - Note: Environment variable`HF_TOKEN` is set and is the current active token independently from the token you've just configured. 2025-11-23 08:00:32,334 - __main__ - INFO - Uploading files... 2025-11-23 08:00:32,369 - __main__ - INFO - ✅ Repository created/verified: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 08:00:32,374 - __main__ - INFO - 📊 Including TensorBoard logs (4 files) 2025-11-23 08:00:32,376 - __main__ - INFO - 📁 Uploading 21 items from models/finetuned-sw-en-phase2 2025-11-23 08:00:32,377 - __main__ - INFO - 📄 README.md 2025-11-23 08:00:32,379 - __main__ - INFO - 📄 training.log 2025-11-23 08:00:32,380 - __main__ - INFO - 📄 training_config_used.json 2025-11-23 08:00:32,381 - __main__ - INFO - 📄 training_metrics.json 2025-11-23 08:00:32,382 - __main__ - INFO - 📄 test_results.json 2025-11-23 08:00:32,383 - __main__ - INFO - 📂 checkpoint-12000 2025-11-23 08:00:32,385 - __main__ - INFO - 📂 checkpoint-10000 2025-11-23 08:00:32,386 - __main__ - INFO - 📂 checkpoint-8000 2025-11-23 08:00:32,387 - __main__ - INFO - 📄 train_results.json 2025-11-23 08:00:32,388 - __main__ - INFO - 📄 translation_config.json 2025-11-23 08:00:32,390 - __main__ - INFO - 📄 training_args.bin 2025-11-23 08:00:32,391 - __main__ - INFO - 📄 target.spm 2025-11-23 08:00:32,392 - __main__ - INFO - 📄 source.spm 2025-11-23 08:00:32,393 - __main__ - INFO - 📄 vocab.json 2025-11-23 08:00:32,394 - __main__ - INFO - 📄 special_tokens_map.json 2025-11-23 08:00:32,395 - __main__ - INFO - 📄 tokenizer_config.json 2025-11-23 08:00:32,397 - __main__ - INFO - 📄 model.safetensors 2025-11-23 08:00:32,398 - __main__ - INFO - 📄 generation_config.json 2025-11-23 08:00:32,398 - __main__ - INFO - 📄 config.json 2025-11-23 08:00:32,415 - __main__ - INFO - 📄 baseline_results.json 2025-11-23 08:00:32,416 - __main__ - INFO - 📂 logs 2025-11-23 08:00:34,114 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.6%) 2025-11-23 08:00:50,055 - __main__ - INFO - ✅ Model uploaded successfully! 2025-11-23 08:00:50,057 - __main__ - INFO - 🔗 View model at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2 2025-11-23 08:00:50,059 - __main__ - INFO - 📊 TensorBoard logs at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/logs 2025-11-23 08:00:50,060 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:50,061 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:50,062 - __main__ - INFO - ✅ Pipeline completed successfully! 2025-11-23 08:00:50,062 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:50,063 - __main__ - INFO - 📋 PIPELINE SUMMARY 2025-11-23 08:00:50,064 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:50,065 - __main__ - INFO - ✓ Baseline Evaluation 2025-11-23 08:00:50,066 - __main__ - INFO - - Test set: 210087 samples 2025-11-23 08:00:50,067 - __main__ - INFO - - Domain data: 5 samples 2025-11-23 08:00:50,068 - __main__ - INFO - ✓ Training 2025-11-23 08:00:50,069 - __main__ - INFO - - Epochs: 6 2025-11-23 08:00:50,069 - __main__ - INFO - - Train samples: 3,571,465 2025-11-23 08:00:50,070 - __main__ - INFO - - Validation samples: 420,173 2025-11-23 08:00:50,071 - __main__ - INFO - ✓ Final Evaluation 2025-11-23 08:00:50,072 - __main__ - INFO - - Test set BLEU: 0.1289 2025-11-23 08:00:50,073 - __main__ - INFO - - Domain BLEU: 0.0000 2025-11-23 08:00:50,073 - __main__ - INFO - 📁 Output Files 2025-11-23 08:00:50,081 - __main__ - INFO - - Models: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-23 08:00:50,083 - __main__ - INFO - - Baseline results: models/finetuned-sw-en-phase2/baseline_results.json 2025-11-23 08:00:50,083 - __main__ - INFO - - Test results: models/finetuned-sw-en-phase2/test_results.json 2025-11-23 08:00:50,084 - __main__ - INFO - - Training logs: /workspace/franklin/openchs_rnd/tasks/translation/swahili/training.log 2025-11-23 08:00:50,085 - __main__ - INFO - 📈 MLflow Tracking 2025-11-23 08:00:50,086 - __main__ - INFO - - URL: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 08:00:50,087 - __main__ - INFO - - Run ID: e2e357df5cbe463abba0139445600679 2025-11-23 08:00:50,088 - __main__ - INFO - 📊 TensorBoard Logs 2025-11-23 08:00:50,089 - __main__ - INFO - - View with: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs 2025-11-23 08:00:50,089 - __main__ - INFO - - All metrics logged: baseline, test, domain, and COMET scores 2025-11-23 08:00:50,090 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:50,091 - __main__ - INFO - 📈 MLflow Tracking: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 08:00:50,092 - __main__ - INFO - Status codes: 2025-11-23 08:00:50,093 - __main__ - INFO - - 0.5 = Baseline evaluation 2025-11-23 08:00:50,093 - __main__ - INFO - - 1.0 = Training started 2025-11-23 08:00:50,094 - __main__ - INFO - - 2.0 = Training finished 2025-11-23 08:00:50,096 - __main__ - INFO - - 3.0 = Final evaluation 2025-11-23 08:00:50,849 - __main__ - INFO - ✅ MLflow: Pipeline completed - status_code=4.0 2025-11-23 08:00:51,659 - __main__ - INFO - ✅ MLflow run ended: e2e357df5cbe463abba0139445600679 2025-11-23 08:00:51,660 - __main__ - INFO - ====================================================================== 2025-11-23 08:00:55,562 - __main__ - INFO - 📈 Epoch progress: 0.88/6 (14.7%) 2025-11-23 08:01:15,956 - __main__ - INFO - 📈 Epoch progress: 0.89/6 (14.8%) 2025-11-23 08:01:36,402 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (14.9%) 2025-11-23 08:01:58,072 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:02:07,990 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:02:07,991 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (15.1%) 2025-11-23 08:02:08,717 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:02:18,960 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:02:29,571 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:02:29,572 - __main__ - INFO - 📈 Epoch progress: 0.91/6 (15.2%) 2025-11-23 08:02:29,906 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:02:39,798 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: ' Cloudflare Tunnel error | food-accepts-blocked-refinance.trycloudflare.com | Cloudflare

Error 1033

Ray ID: 9a2f2546a63da2c4 • 2025-11-23 08:02:39 UTC

Cloudflare Tunnel error

What happened?

You've requested a page on a website (food-accepts-blocked-refinance.trycloudflare.com) that is on the Cloudflare network. The host (food-accepts-blocked-refinance.trycloudflare.com) is configured as a Cloudflare Tunnel, and Cloudflare is currently unable to resolve it.

What can I do?

If you are a visitor of this website:
Please try again in a few minutes.

If you are the owner of this website:
Ensure that cloudflared is running and can reach the network. You may wish to enable load balancing for your tunnel.

Cloudflare Ray ID: 9a2f2546a63da2c4 • Performance & security by Cloudflare

' 2025-11-23 08:02:49,395 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: ' Cloudflare Tunnel error | food-accepts-blocked-refinance.trycloudflare.com | Cloudflare

Error 1033

Ray ID: 9a2f2582a0f5a2c4 • 2025-11-23 08:02:49 UTC

Cloudflare Tunnel error

What happened?

You've requested a page on a website (food-accepts-blocked-refinance.trycloudflare.com) that is on the Cloudflare network. The host (food-accepts-blocked-refinance.trycloudflare.com) is configured as a Cloudflare Tunnel, and Cloudflare is currently unable to resolve it.

What can I do?

If you are a visitor of this website:
Please try again in a few minutes.

If you are the owner of this website:
Ensure that cloudflared is running and can reach the network. You may wish to enable load balancing for your tunnel.

Cloudflare Ray ID: 9a2f2582a0f5a2c4 • Performance & security by Cloudflare

' 2025-11-23 08:02:49,399 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.3%) 2025-11-23 08:02:49,477 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: ' Cloudflare Tunnel error | food-accepts-blocked-refinance.trycloudflare.com | Cloudflare

Error 1033

Ray ID: 9a2f25832124a2c4 • 2025-11-23 08:02:49 UTC

Cloudflare Tunnel error

What happened?

You've requested a page on a website (food-accepts-blocked-refinance.trycloudflare.com) that is on the Cloudflare network. The host (food-accepts-blocked-refinance.trycloudflare.com) is configured as a Cloudflare Tunnel, and Cloudflare is currently unable to resolve it.

What can I do?

If you are a visitor of this website:
Please try again in a few minutes.

If you are the owner of this website:
Ensure that cloudflared is running and can reach the network. You may wish to enable load balancing for your tunnel.

Cloudflare Ray ID: 9a2f25832124a2c4 • Performance & security by Cloudflare

' 2025-11-23 08:03:10,086 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.4%) 2025-11-23 08:03:30,868 - __main__ - INFO - 📈 Epoch progress: 0.93/6 (15.5%) 2025-11-23 08:03:51,127 - __main__ - INFO - 📈 Epoch progress: 0.94/6 (15.6%) 2025-11-23 08:04:11,310 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.8%) 2025-11-23 08:04:31,944 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.9%) 2025-11-23 08:04:52,229 - __main__ - INFO - 📈 Epoch progress: 0.96/6 (16.0%) 2025-11-23 08:05:12,433 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.1%) 2025-11-23 08:05:32,947 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.2%) 2025-11-23 08:05:53,251 - __main__ - INFO - 📈 Epoch progress: 0.98/6 (16.4%) 2025-11-23 08:06:13,503 - __main__ - INFO - 📈 Epoch progress: 0.99/6 (16.5%) 2025-11-23 08:06:33,647 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.6%) 2025-11-23 08:06:56,076 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.7%) 2025-11-23 08:07:42,358 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:07:42,361 - __main__ - INFO - 📊 Evaluation 8: BLEU = 0.1178 2025-11-23 08:07:42,660 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:07:42,661 - __main__ - INFO - 📊 Evaluation at step 14000 2025-11-23 08:07:50,686 - __main__ - INFO - 📤 Pushing checkpoint 14000 to HuggingFace Hub... 2025-11-23 08:07:50,687 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 08:08:00,297 - __main__ - INFO - ✅ Checkpoint 14000 pushed successfully! 2025-11-23 08:08:00,298 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-14000 2025-11-23 08:08:20,043 - __main__ - INFO - 📈 Epoch progress: 1.01/6 (16.8%) 2025-11-23 08:08:40,333 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.0%) 2025-11-23 08:09:00,458 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.1%) 2025-11-23 08:09:20,954 - __main__ - INFO - 📈 Epoch progress: 1.03/6 (17.2%) 2025-11-23 08:09:41,182 - __main__ - INFO - 📈 Epoch progress: 1.04/6 (17.3%) 2025-11-23 08:10:01,521 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.4%) 2025-11-23 08:10:22,100 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.6%) 2025-11-23 08:10:42,424 - __main__ - INFO - 📈 Epoch progress: 1.06/6 (17.7%) 2025-11-23 08:11:03,105 - __main__ - INFO - 📈 Epoch progress: 1.07/6 (17.8%) 2025-11-23 08:11:23,389 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (17.9%) 2025-11-23 08:11:43,462 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (18.0%) 2025-11-23 08:12:03,440 - __main__ - INFO - 📈 Epoch progress: 1.09/6 (18.2%) 2025-11-23 08:12:23,799 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.3%) 2025-11-23 08:12:35,540 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:12:45,484 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:12:45,484 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.4%) 2025-11-23 08:12:45,795 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:13:06,414 - __main__ - INFO - 📈 Epoch progress: 1.11/6 (18.5%) 2025-11-23 08:13:27,011 - __main__ - INFO - 📈 Epoch progress: 1.12/6 (18.6%) 2025-11-23 08:13:47,468 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.8%) 2025-11-23 08:14:07,883 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.9%) 2025-11-23 08:14:28,371 - __main__ - INFO - 📈 Epoch progress: 1.14/6 (19.0%) 2025-11-23 08:14:48,689 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.1%) 2025-11-23 08:15:36,568 - __main__ - INFO - 📊 Evaluation 9: BLEU = 0.1183 2025-11-23 08:15:36,909 - __main__ - INFO - 📊 Evaluation at step 16000 2025-11-23 08:15:40,691 - __main__ - INFO - 📤 Pushing checkpoint 16000 to HuggingFace Hub... 2025-11-23 08:15:40,692 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 08:15:48,673 - __main__ - INFO - ✅ Checkpoint 16000 pushed successfully! 2025-11-23 08:15:48,674 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-16000 2025-11-23 08:16:08,416 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.2%) 2025-11-23 08:16:29,051 - __main__ - INFO - 📈 Epoch progress: 1.16/6 (19.4%) 2025-11-23 08:16:49,428 - __main__ - INFO - 📈 Epoch progress: 1.17/6 (19.5%) 2025-11-23 08:17:10,033 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.6%) 2025-11-23 08:17:30,608 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.7%) 2025-11-23 08:17:51,070 - __main__ - INFO - 📈 Epoch progress: 1.19/6 (19.8%) 2025-11-23 08:18:12,240 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (19.9%) 2025-11-23 08:18:32,443 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (20.1%) 2025-11-23 08:18:52,757 - __main__ - INFO - 📈 Epoch progress: 1.21/6 (20.2%) 2025-11-23 08:19:12,512 - __main__ - INFO - 📈 Epoch progress: 1.22/6 (20.3%) 2025-11-23 08:19:33,339 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.4%) 2025-11-23 08:19:54,208 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.5%) 2025-11-23 08:20:14,632 - __main__ - INFO - 📈 Epoch progress: 1.24/6 (20.7%) 2025-11-23 08:20:35,260 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.8%) 2025-11-23 08:20:55,982 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.9%) 2025-11-23 08:21:16,552 - __main__ - INFO - 📈 Epoch progress: 1.26/6 (21.0%) 2025-11-23 08:21:37,005 - __main__ - INFO - 📈 Epoch progress: 1.27/6 (21.1%) 2025-11-23 08:21:57,316 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.3%) 2025-11-23 08:22:17,535 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.4%) 2025-11-23 08:22:38,230 - __main__ - INFO - 📈 Epoch progress: 1.29/6 (21.5%) 2025-11-23 08:23:24,100 - __main__ - INFO - 📊 Evaluation 10: BLEU = 0.1219 2025-11-23 08:23:24,479 - __main__ - INFO - 📊 Evaluation at step 18000 2025-11-23 08:23:28,627 - __main__ - INFO - 📤 Pushing checkpoint 18000 to HuggingFace Hub... 2025-11-23 08:23:28,628 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 08:23:37,701 - __main__ - INFO - ✅ Checkpoint 18000 pushed successfully! 2025-11-23 08:23:37,716 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-18000 2025-11-23 08:23:57,837 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.6%) 2025-11-23 08:24:18,077 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.7%) 2025-11-23 08:24:38,762 - __main__ - INFO - 📈 Epoch progress: 1.31/6 (21.9%) 2025-11-23 08:24:58,968 - __main__ - INFO - 📈 Epoch progress: 1.32/6 (22.0%) 2025-11-23 08:25:19,819 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.1%) 2025-11-23 08:25:40,288 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.2%) 2025-11-23 08:26:01,048 - __main__ - INFO - 📈 Epoch progress: 1.34/6 (22.3%) 2025-11-23 08:26:21,607 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.5%) 2025-11-23 08:26:41,975 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.6%) 2025-11-23 08:27:02,121 - __main__ - INFO - 📈 Epoch progress: 1.36/6 (22.7%) 2025-11-23 08:27:22,434 - __main__ - INFO - 📈 Epoch progress: 1.37/6 (22.8%) 2025-11-23 08:27:47,892 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:27:47,893 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (22.9%) 2025-11-23 08:27:48,203 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:28:08,098 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (23.1%) 2025-11-23 08:28:28,471 - __main__ - INFO - 📈 Epoch progress: 1.39/6 (23.2%) 2025-11-23 08:28:49,452 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.3%) 2025-11-23 08:29:09,642 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.4%) 2025-11-23 08:29:29,850 - __main__ - INFO - 📈 Epoch progress: 1.41/6 (23.5%) 2025-11-23 08:29:50,456 - __main__ - INFO - 📈 Epoch progress: 1.42/6 (23.7%) 2025-11-23 08:30:11,200 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.8%) 2025-11-23 08:30:31,277 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.9%) 2025-11-23 08:31:18,892 - __main__ - INFO - 📊 Evaluation 11: BLEU = 0.1225 2025-11-23 08:31:19,249 - __main__ - INFO - 📊 Evaluation at step 20000 2025-11-23 08:31:22,663 - __main__ - INFO - 📤 Pushing checkpoint 20000 to HuggingFace Hub... 2025-11-23 08:31:22,665 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 08:31:29,912 - __main__ - INFO - ✅ Checkpoint 20000 pushed successfully! 2025-11-23 08:31:29,916 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-20000 2025-11-23 08:31:50,195 - __main__ - INFO - 📈 Epoch progress: 1.44/6 (24.0%) 2025-11-23 08:32:10,350 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.1%) 2025-11-23 08:32:31,382 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.2%) 2025-11-23 08:32:52,241 - __main__ - INFO - 📈 Epoch progress: 1.46/6 (24.4%) 2025-11-23 08:33:12,787 - __main__ - INFO - 📈 Epoch progress: 1.47/6 (24.5%) 2025-11-23 08:33:33,381 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.6%) 2025-11-23 08:33:53,906 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.7%) 2025-11-23 08:34:14,948 - __main__ - INFO - 📈 Epoch progress: 1.49/6 (24.8%) 2025-11-23 08:34:35,256 - __main__ - INFO - 📈 Epoch progress: 1.50/6 (25.0%) 2025-11-23 08:34:56,451 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:34:56,453 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.1%) 2025-11-23 08:34:56,755 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:35:17,468 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.2%) 2025-11-23 08:35:38,056 - __main__ - INFO - 📈 Epoch progress: 1.52/6 (25.3%) 2025-11-23 08:35:58,638 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.4%) 2025-11-23 08:36:19,131 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.6%) 2025-11-23 08:36:39,411 - __main__ - INFO - 📈 Epoch progress: 1.54/6 (25.7%) 2025-11-23 08:37:00,265 - __main__ - INFO - 📈 Epoch progress: 1.55/6 (25.8%) 2025-11-23 08:37:20,391 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (25.9%) 2025-11-23 08:37:40,850 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (26.0%) 2025-11-23 08:38:00,979 - __main__ - INFO - 📈 Epoch progress: 1.57/6 (26.2%) 2025-11-23 08:38:21,565 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.3%) 2025-11-23 08:39:16,978 - __main__ - INFO - 📊 Evaluation 12: BLEU = 0.1133 2025-11-23 08:39:17,337 - __main__ - INFO - 📊 Evaluation at step 22000 2025-11-23 08:39:21,067 - __main__ - INFO - 📤 Pushing checkpoint 22000 to HuggingFace Hub... 2025-11-23 08:39:21,068 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 08:39:29,124 - __main__ - INFO - ✅ Checkpoint 22000 pushed successfully! 2025-11-23 08:39:29,126 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-22000 2025-11-23 08:39:49,626 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.4%) 2025-11-23 08:40:10,631 - __main__ - INFO - 📈 Epoch progress: 1.59/6 (26.5%) 2025-11-23 08:40:31,193 - __main__ - INFO - 📈 Epoch progress: 1.60/6 (26.6%) 2025-11-23 08:40:51,268 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.8%) 2025-11-23 08:41:12,552 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.9%) 2025-11-23 08:41:32,697 - __main__ - INFO - 📈 Epoch progress: 1.62/6 (27.0%) 2025-11-23 08:41:53,243 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.1%) 2025-11-23 08:42:13,872 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.2%) 2025-11-23 08:42:34,689 - __main__ - INFO - 📈 Epoch progress: 1.64/6 (27.4%) 2025-11-23 08:42:55,351 - __main__ - INFO - 📈 Epoch progress: 1.65/6 (27.5%) 2025-11-23 08:43:15,850 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.6%) 2025-11-23 08:43:36,025 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.7%) 2025-11-23 08:43:56,321 - __main__ - INFO - 📈 Epoch progress: 1.67/6 (27.8%) 2025-11-23 08:44:16,729 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.0%) 2025-11-23 08:44:37,239 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.1%) 2025-11-23 08:44:57,578 - __main__ - INFO - 📈 Epoch progress: 1.69/6 (28.2%) 2025-11-23 08:45:18,212 - __main__ - INFO - 📈 Epoch progress: 1.70/6 (28.3%) 2025-11-23 08:45:39,001 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.4%) 2025-11-23 08:45:59,703 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.5%) 2025-11-23 08:46:20,019 - __main__ - INFO - 📈 Epoch progress: 1.72/6 (28.7%) 2025-11-23 08:47:02,800 - __main__ - INFO - 📊 Evaluation 13: BLEU = 0.1197 2025-11-23 08:47:03,154 - __main__ - INFO - 📊 Evaluation at step 24000 2025-11-23 08:47:07,057 - __main__ - INFO - 📤 Pushing checkpoint 24000 to HuggingFace Hub... 2025-11-23 08:47:07,057 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 08:47:15,669 - __main__ - INFO - ✅ Checkpoint 24000 pushed successfully! 2025-11-23 08:47:15,671 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-24000 2025-11-23 08:47:35,776 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.8%) 2025-11-23 08:47:55,967 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.9%) 2025-11-23 08:48:16,189 - __main__ - INFO - 📈 Epoch progress: 1.74/6 (29.0%) 2025-11-23 08:48:36,698 - __main__ - INFO - 📈 Epoch progress: 1.75/6 (29.1%) 2025-11-23 08:48:56,756 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.3%) 2025-11-23 08:49:17,605 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.4%) 2025-11-23 08:49:37,883 - __main__ - INFO - 📈 Epoch progress: 1.77/6 (29.5%) 2025-11-23 08:49:58,863 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.6%) 2025-11-23 08:50:20,068 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.7%) 2025-11-23 08:50:40,801 - __main__ - INFO - 📈 Epoch progress: 1.79/6 (29.9%) 2025-11-23 08:51:01,156 - __main__ - INFO - 📈 Epoch progress: 1.80/6 (30.0%) 2025-11-23 08:51:21,244 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.1%) 2025-11-23 08:51:41,374 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.2%) 2025-11-23 08:52:01,723 - __main__ - INFO - 📈 Epoch progress: 1.82/6 (30.3%) 2025-11-23 08:52:22,182 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.5%) 2025-11-23 08:52:42,735 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.6%) 2025-11-23 08:53:02,880 - __main__ - INFO - 📈 Epoch progress: 1.84/6 (30.7%) 2025-11-23 08:53:23,224 - __main__ - INFO - 📈 Epoch progress: 1.85/6 (30.8%) 2025-11-23 08:53:43,760 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (30.9%) 2025-11-23 08:54:03,729 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (31.1%) 2025-11-23 08:54:46,422 - __main__ - INFO - 📊 Evaluation 14: BLEU = 0.1220 2025-11-23 08:54:57,420 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 08:54:57,422 - __main__ - INFO - 📊 Evaluation at step 26000 2025-11-23 08:55:01,217 - __main__ - INFO - 📤 Pushing checkpoint 26000 to HuggingFace Hub... 2025-11-23 08:55:01,218 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 08:55:09,583 - __main__ - INFO - ✅ Checkpoint 26000 pushed successfully! 2025-11-23 08:55:09,586 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-26000 2025-11-23 08:55:30,132 - __main__ - INFO - 📈 Epoch progress: 1.87/6 (31.2%) 2025-11-23 08:55:50,431 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.3%) 2025-11-23 08:56:11,046 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.4%) 2025-11-23 08:56:31,743 - __main__ - INFO - 📈 Epoch progress: 1.89/6 (31.5%) 2025-11-23 08:56:52,426 - __main__ - INFO - 📈 Epoch progress: 1.90/6 (31.7%) 2025-11-23 08:57:12,972 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.8%) 2025-11-23 08:57:33,001 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.9%) 2025-11-23 08:57:53,460 - __main__ - INFO - 📈 Epoch progress: 1.92/6 (32.0%) 2025-11-23 08:58:13,359 - __main__ - INFO - 📈 Epoch progress: 1.93/6 (32.1%) 2025-11-23 08:58:33,491 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.3%) 2025-11-23 08:58:53,787 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.4%) 2025-11-23 08:59:14,134 - __main__ - INFO - 📈 Epoch progress: 1.95/6 (32.5%) 2025-11-23 08:59:34,412 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.6%) 2025-11-23 08:59:54,653 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.7%) 2025-11-23 09:00:14,806 - __main__ - INFO - 📈 Epoch progress: 1.97/6 (32.8%) 2025-11-23 09:00:35,080 - __main__ - INFO - 📈 Epoch progress: 1.98/6 (33.0%) 2025-11-23 09:00:55,829 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.1%) 2025-11-23 09:01:16,290 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.2%) 2025-11-23 09:01:36,929 - __main__ - INFO - 📈 Epoch progress: 2.00/6 (33.3%) 2025-11-23 09:01:59,922 - __main__ - INFO - 📈 Epoch progress: 2.01/6 (33.4%) 2025-11-23 09:02:38,082 - __main__ - INFO - 📊 Evaluation 15: BLEU = 0.1212 2025-11-23 09:02:38,384 - __main__ - INFO - 📊 Evaluation at step 28000 2025-11-23 09:02:41,951 - __main__ - INFO - 📤 Pushing checkpoint 28000 to HuggingFace Hub... 2025-11-23 09:02:41,952 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 09:02:49,639 - __main__ - INFO - ✅ Checkpoint 28000 pushed successfully! 2025-11-23 09:02:49,641 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-28000 2025-11-23 09:03:09,347 - __main__ - INFO - 📈 Epoch progress: 2.01/6 (33.6%) 2025-11-23 09:03:29,574 - __main__ - INFO - 📈 Epoch progress: 2.02/6 (33.7%) 2025-11-23 09:03:50,131 - __main__ - INFO - 📈 Epoch progress: 2.03/6 (33.8%) 2025-11-23 09:04:10,657 - __main__ - INFO - 📈 Epoch progress: 2.04/6 (33.9%) 2025-11-23 09:04:30,935 - __main__ - INFO - 📈 Epoch progress: 2.04/6 (34.0%) 2025-11-23 09:04:51,362 - __main__ - INFO - 📈 Epoch progress: 2.05/6 (34.2%) 2025-11-23 09:05:12,207 - __main__ - INFO - 📈 Epoch progress: 2.06/6 (34.3%) 2025-11-23 09:05:32,445 - __main__ - INFO - 📈 Epoch progress: 2.06/6 (34.4%) 2025-11-23 09:05:52,852 - __main__ - INFO - 📈 Epoch progress: 2.07/6 (34.5%) 2025-11-23 09:06:13,060 - __main__ - INFO - 📈 Epoch progress: 2.08/6 (34.6%) 2025-11-23 09:06:33,286 - __main__ - INFO - 📈 Epoch progress: 2.09/6 (34.8%) 2025-11-23 09:06:53,703 - __main__ - INFO - 📈 Epoch progress: 2.09/6 (34.9%) 2025-11-23 09:07:14,264 - __main__ - INFO - 📈 Epoch progress: 2.10/6 (35.0%) 2025-11-23 09:07:34,690 - __main__ - INFO - 📈 Epoch progress: 2.11/6 (35.1%) 2025-11-23 09:07:55,165 - __main__ - INFO - 📈 Epoch progress: 2.11/6 (35.2%) 2025-11-23 09:08:15,733 - __main__ - INFO - 📈 Epoch progress: 2.12/6 (35.4%) 2025-11-23 09:08:36,049 - __main__ - INFO - 📈 Epoch progress: 2.13/6 (35.5%) 2025-11-23 09:08:56,649 - __main__ - INFO - 📈 Epoch progress: 2.14/6 (35.6%) 2025-11-23 09:09:17,044 - __main__ - INFO - 📈 Epoch progress: 2.14/6 (35.7%) 2025-11-23 09:09:37,555 - __main__ - INFO - 📈 Epoch progress: 2.15/6 (35.8%) 2025-11-23 09:10:16,737 - __main__ - INFO - 📊 Evaluation 16: BLEU = 0.1249 2025-11-23 09:10:17,064 - __main__ - INFO - 📊 Evaluation at step 30000 2025-11-23 09:10:22,179 - __main__ - INFO - 📤 Pushing checkpoint 30000 to HuggingFace Hub... 2025-11-23 09:10:22,180 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 09:10:32,265 - __main__ - INFO - ✅ Checkpoint 30000 pushed successfully! 2025-11-23 09:10:32,267 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-30000 2025-11-23 09:10:52,200 - __main__ - INFO - 📈 Epoch progress: 2.16/6 (36.0%) 2025-11-23 09:11:12,243 - __main__ - INFO - 📈 Epoch progress: 2.16/6 (36.1%) 2025-11-23 09:11:32,817 - __main__ - INFO - 📈 Epoch progress: 2.17/6 (36.2%) 2025-11-23 09:11:52,857 - __main__ - INFO - 📈 Epoch progress: 2.18/6 (36.3%) 2025-11-23 09:12:13,030 - __main__ - INFO - 📈 Epoch progress: 2.19/6 (36.4%) 2025-11-23 09:12:37,455 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 09:12:37,457 - __main__ - INFO - 📈 Epoch progress: 2.19/6 (36.6%) 2025-11-23 09:12:37,777 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 09:12:47,576 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 09:12:57,551 - __main__ - INFO - 📈 Epoch progress: 2.20/6 (36.7%) 2025-11-23 09:13:18,678 - __main__ - INFO - 📈 Epoch progress: 2.21/6 (36.8%) 2025-11-23 09:13:39,512 - __main__ - INFO - 📈 Epoch progress: 2.21/6 (36.9%) 2025-11-23 09:14:00,077 - __main__ - INFO - 📈 Epoch progress: 2.22/6 (37.0%) 2025-11-23 09:14:20,418 - __main__ - INFO - 📈 Epoch progress: 2.23/6 (37.1%) 2025-11-23 09:14:40,797 - __main__ - INFO - 📈 Epoch progress: 2.24/6 (37.3%) 2025-11-23 09:15:01,178 - __main__ - INFO - 📈 Epoch progress: 2.24/6 (37.4%) 2025-11-23 09:15:21,882 - __main__ - INFO - 📈 Epoch progress: 2.25/6 (37.5%) 2025-11-23 09:15:43,229 - __main__ - INFO - 📈 Epoch progress: 2.26/6 (37.6%) 2025-11-23 09:16:03,702 - __main__ - INFO - 📈 Epoch progress: 2.26/6 (37.7%) 2025-11-23 09:16:23,964 - __main__ - INFO - 📈 Epoch progress: 2.27/6 (37.9%) 2025-11-23 09:16:44,557 - __main__ - INFO - 📈 Epoch progress: 2.28/6 (38.0%) 2025-11-23 09:17:05,127 - __main__ - INFO - 📈 Epoch progress: 2.29/6 (38.1%) 2025-11-23 09:17:25,409 - __main__ - INFO - 📈 Epoch progress: 2.29/6 (38.2%) 2025-11-23 09:18:08,617 - __main__ - INFO - 📊 Evaluation 17: BLEU = 0.1247 2025-11-23 09:18:08,956 - __main__ - INFO - 📊 Evaluation at step 32000 2025-11-23 09:18:12,719 - __main__ - INFO - 📤 Pushing checkpoint 32000 to HuggingFace Hub... 2025-11-23 09:18:12,720 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 09:18:22,296 - __main__ - INFO - ✅ Checkpoint 32000 pushed successfully! 2025-11-23 09:18:22,298 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-32000 2025-11-23 09:18:42,074 - __main__ - INFO - 📈 Epoch progress: 2.30/6 (38.3%) 2025-11-23 09:19:02,439 - __main__ - INFO - 📈 Epoch progress: 2.31/6 (38.5%) 2025-11-23 09:19:31,259 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 09:19:31,262 - __main__ - INFO - 📈 Epoch progress: 2.31/6 (38.6%) 2025-11-23 09:19:31,555 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 09:19:41,425 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 09:19:51,215 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 09:19:51,218 - __main__ - INFO - 📈 Epoch progress: 2.32/6 (38.7%) 2025-11-23 09:19:52,068 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 09:20:12,055 - __main__ - INFO - 📈 Epoch progress: 2.33/6 (38.8%) 2025-11-23 09:20:32,715 - __main__ - INFO - 📈 Epoch progress: 2.34/6 (38.9%) 2025-11-23 09:20:52,820 - __main__ - INFO - 📈 Epoch progress: 2.34/6 (39.1%) 2025-11-23 09:21:13,225 - __main__ - INFO - 📈 Epoch progress: 2.35/6 (39.2%) 2025-11-23 09:21:33,172 - __main__ - INFO - 📈 Epoch progress: 2.36/6 (39.3%) 2025-11-23 09:21:53,617 - __main__ - INFO - 📈 Epoch progress: 2.37/6 (39.4%) 2025-11-23 09:22:14,037 - __main__ - INFO - 📈 Epoch progress: 2.37/6 (39.5%) 2025-11-23 09:22:34,315 - __main__ - INFO - 📈 Epoch progress: 2.38/6 (39.7%) 2025-11-23 09:22:54,536 - __main__ - INFO - 📈 Epoch progress: 2.39/6 (39.8%) 2025-11-23 09:23:15,047 - __main__ - INFO - 📈 Epoch progress: 2.39/6 (39.9%) 2025-11-23 09:23:35,587 - __main__ - INFO - 📈 Epoch progress: 2.40/6 (40.0%) 2025-11-23 09:23:55,756 - __main__ - INFO - 📈 Epoch progress: 2.41/6 (40.1%) 2025-11-23 09:24:16,224 - __main__ - INFO - 📈 Epoch progress: 2.42/6 (40.3%) 2025-11-23 09:24:36,699 - __main__ - INFO - 📈 Epoch progress: 2.42/6 (40.4%) 2025-11-23 09:25:01,406 - __main__ - INFO - 📈 Epoch progress: 2.43/6 (40.5%) 2025-11-23 09:25:21,453 - __main__ - INFO - 📈 Epoch progress: 2.44/6 (40.6%) 2025-11-23 09:26:03,414 - __main__ - INFO - 📊 Evaluation 18: BLEU = 0.1201 2025-11-23 09:26:03,728 - __main__ - INFO - 📊 Evaluation at step 34000 2025-11-23 09:26:07,747 - __main__ - INFO - 📤 Pushing checkpoint 34000 to HuggingFace Hub... 2025-11-23 09:26:07,748 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 09:26:17,915 - __main__ - INFO - ✅ Checkpoint 34000 pushed successfully! 2025-11-23 09:26:17,916 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-34000 2025-11-23 09:26:38,257 - __main__ - INFO - 📈 Epoch progress: 2.44/6 (40.7%) 2025-11-23 09:26:58,506 - __main__ - INFO - 📈 Epoch progress: 2.45/6 (40.9%) 2025-11-23 09:27:18,774 - __main__ - INFO - 📈 Epoch progress: 2.46/6 (41.0%) 2025-11-23 09:27:39,202 - __main__ - INFO - 📈 Epoch progress: 2.47/6 (41.1%) 2025-11-23 09:27:59,546 - __main__ - INFO - 📈 Epoch progress: 2.47/6 (41.2%) 2025-11-23 09:28:19,784 - __main__ - INFO - 📈 Epoch progress: 2.48/6 (41.3%) 2025-11-23 09:28:40,124 - __main__ - INFO - 📈 Epoch progress: 2.49/6 (41.4%) 2025-11-23 09:29:00,552 - __main__ - INFO - 📈 Epoch progress: 2.49/6 (41.6%) 2025-11-23 09:29:20,903 - __main__ - INFO - 📈 Epoch progress: 2.50/6 (41.7%) 2025-11-23 09:29:41,466 - __main__ - INFO - 📈 Epoch progress: 2.51/6 (41.8%) 2025-11-23 09:30:01,776 - __main__ - INFO - 📈 Epoch progress: 2.52/6 (41.9%) 2025-11-23 09:30:22,006 - __main__ - INFO - 📈 Epoch progress: 2.52/6 (42.0%) 2025-11-23 09:30:42,360 - __main__ - INFO - 📈 Epoch progress: 2.53/6 (42.2%) 2025-11-23 09:31:02,482 - __main__ - INFO - 📈 Epoch progress: 2.54/6 (42.3%) 2025-11-23 09:31:22,826 - __main__ - INFO - 📈 Epoch progress: 2.54/6 (42.4%) 2025-11-23 09:31:43,466 - __main__ - INFO - 📈 Epoch progress: 2.55/6 (42.5%) 2025-11-23 09:32:04,045 - __main__ - INFO - 📈 Epoch progress: 2.56/6 (42.6%) 2025-11-23 09:32:24,371 - __main__ - INFO - 📈 Epoch progress: 2.57/6 (42.8%) 2025-11-23 09:32:44,985 - __main__ - INFO - 📈 Epoch progress: 2.57/6 (42.9%) 2025-11-23 09:33:05,336 - __main__ - INFO - 📈 Epoch progress: 2.58/6 (43.0%) 2025-11-23 09:33:54,033 - __main__ - INFO - 📊 Evaluation 19: BLEU = 0.1199 2025-11-23 09:33:54,344 - __main__ - INFO - 📊 Evaluation at step 36000 2025-11-23 09:33:59,171 - __main__ - INFO - 📤 Pushing checkpoint 36000 to HuggingFace Hub... 2025-11-23 09:33:59,173 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 09:34:11,715 - __main__ - INFO - ✅ Checkpoint 36000 pushed successfully! 2025-11-23 09:34:11,716 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-36000 2025-11-23 09:34:31,557 - __main__ - INFO - 📈 Epoch progress: 2.59/6 (43.1%) 2025-11-23 09:34:52,089 - __main__ - INFO - 📈 Epoch progress: 2.59/6 (43.2%) 2025-11-23 09:35:12,625 - __main__ - INFO - 📈 Epoch progress: 2.60/6 (43.4%) 2025-11-23 09:35:33,297 - __main__ - INFO - 📈 Epoch progress: 2.61/6 (43.5%) 2025-11-23 09:35:53,392 - __main__ - INFO - 📈 Epoch progress: 2.62/6 (43.6%) 2025-11-23 09:36:13,883 - __main__ - INFO - 📈 Epoch progress: 2.62/6 (43.7%) 2025-11-23 09:36:34,079 - __main__ - INFO - 📈 Epoch progress: 2.63/6 (43.8%) 2025-11-23 09:36:54,116 - __main__ - INFO - 📈 Epoch progress: 2.64/6 (44.0%) 2025-11-23 09:37:14,800 - __main__ - INFO - 📈 Epoch progress: 2.64/6 (44.1%) 2025-11-23 09:37:35,136 - __main__ - INFO - 📈 Epoch progress: 2.65/6 (44.2%) 2025-11-23 09:37:55,597 - __main__ - INFO - 📈 Epoch progress: 2.66/6 (44.3%) 2025-11-23 09:38:16,316 - __main__ - INFO - 📈 Epoch progress: 2.67/6 (44.4%) 2025-11-23 09:40:15,894 - __main__ - INFO - 📈 Epoch progress: 2.67/6 (44.6%) 2025-11-23 09:40:36,039 - __main__ - INFO - 📈 Epoch progress: 2.68/6 (44.7%) 2025-11-23 09:40:55,973 - __main__ - INFO - 📈 Epoch progress: 2.69/6 (44.8%) 2025-11-23 09:41:16,576 - __main__ - INFO - 📈 Epoch progress: 2.69/6 (44.9%) 2025-11-23 09:41:37,086 - __main__ - INFO - 📈 Epoch progress: 2.70/6 (45.0%) 2025-11-23 09:41:57,577 - __main__ - INFO - 📈 Epoch progress: 2.71/6 (45.2%) 2025-11-23 09:42:17,849 - __main__ - INFO - 📈 Epoch progress: 2.72/6 (45.3%) 2025-11-23 09:42:38,187 - __main__ - INFO - 📈 Epoch progress: 2.72/6 (45.4%) 2025-11-23 09:43:19,479 - __main__ - INFO - 📊 Evaluation 20: BLEU = 0.1221 2025-11-23 09:43:19,792 - __main__ - INFO - 📊 Evaluation at step 38000 2025-11-23 09:43:24,361 - __main__ - INFO - 📤 Pushing checkpoint 38000 to HuggingFace Hub... 2025-11-23 09:43:24,362 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 09:43:34,489 - __main__ - INFO - ✅ Checkpoint 38000 pushed successfully! 2025-11-23 09:43:34,491 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-38000 2025-11-23 09:43:54,899 - __main__ - INFO - 📈 Epoch progress: 2.73/6 (45.5%) 2025-11-23 09:44:15,186 - __main__ - INFO - 📈 Epoch progress: 2.74/6 (45.6%) 2025-11-23 09:44:35,248 - __main__ - INFO - 📈 Epoch progress: 2.74/6 (45.7%) 2025-11-23 09:44:55,457 - __main__ - INFO - 📈 Epoch progress: 2.75/6 (45.9%) 2025-11-23 09:45:16,526 - __main__ - INFO - 📈 Epoch progress: 2.76/6 (46.0%) 2025-11-23 09:45:36,727 - __main__ - INFO - 📈 Epoch progress: 2.77/6 (46.1%) 2025-11-23 09:45:57,334 - __main__ - INFO - 📈 Epoch progress: 2.77/6 (46.2%) 2025-11-23 09:46:17,813 - __main__ - INFO - 📈 Epoch progress: 2.78/6 (46.3%) 2025-11-23 09:46:38,336 - __main__ - INFO - 📈 Epoch progress: 2.79/6 (46.5%) 2025-11-23 09:46:58,873 - __main__ - INFO - 📈 Epoch progress: 2.80/6 (46.6%) 2025-11-23 09:47:19,339 - __main__ - INFO - 📈 Epoch progress: 2.80/6 (46.7%) 2025-11-23 09:47:39,806 - __main__ - INFO - 📈 Epoch progress: 2.81/6 (46.8%) 2025-11-23 09:48:00,195 - __main__ - INFO - 📈 Epoch progress: 2.82/6 (46.9%) 2025-11-23 09:48:20,524 - __main__ - INFO - 📈 Epoch progress: 2.82/6 (47.1%) 2025-11-23 09:48:40,993 - __main__ - INFO - 📈 Epoch progress: 2.83/6 (47.2%) 2025-11-23 09:49:01,234 - __main__ - INFO - 📈 Epoch progress: 2.84/6 (47.3%) 2025-11-23 09:49:21,981 - __main__ - INFO - 📈 Epoch progress: 2.85/6 (47.4%) 2025-11-23 09:49:42,732 - __main__ - INFO - 📈 Epoch progress: 2.85/6 (47.5%) 2025-11-23 09:50:03,204 - __main__ - INFO - 📈 Epoch progress: 2.86/6 (47.7%) 2025-11-23 09:50:23,702 - __main__ - INFO - 📈 Epoch progress: 2.87/6 (47.8%) 2025-11-23 09:50:59,183 - __main__ - INFO - 📊 Evaluation 21: BLEU = 0.1263 2025-11-23 09:50:59,500 - __main__ - INFO - 📊 Evaluation at step 40000 2025-11-23 09:51:03,717 - __main__ - INFO - 📤 Pushing checkpoint 40000 to HuggingFace Hub... 2025-11-23 09:51:03,718 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 09:51:16,631 - __main__ - INFO - ✅ Checkpoint 40000 pushed successfully! 2025-11-23 09:51:16,632 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-40000 2025-11-23 09:51:36,962 - __main__ - INFO - 📈 Epoch progress: 2.87/6 (47.9%) 2025-11-23 09:51:57,063 - __main__ - INFO - 📈 Epoch progress: 2.88/6 (48.0%) 2025-11-23 09:52:17,257 - __main__ - INFO - 📈 Epoch progress: 2.89/6 (48.1%) 2025-11-23 09:52:37,641 - __main__ - INFO - 📈 Epoch progress: 2.90/6 (48.3%) 2025-11-23 09:52:58,165 - __main__ - INFO - 📈 Epoch progress: 2.90/6 (48.4%) 2025-11-23 09:53:18,722 - __main__ - INFO - 📈 Epoch progress: 2.91/6 (48.5%) 2025-11-23 09:53:39,389 - __main__ - INFO - 📈 Epoch progress: 2.92/6 (48.6%) 2025-11-23 09:53:59,669 - __main__ - INFO - 📈 Epoch progress: 2.92/6 (48.7%) 2025-11-23 09:54:20,128 - __main__ - INFO - 📈 Epoch progress: 2.93/6 (48.9%) 2025-11-23 09:54:40,680 - __main__ - INFO - 📈 Epoch progress: 2.94/6 (49.0%) 2025-11-23 09:55:01,174 - __main__ - INFO - 📈 Epoch progress: 2.95/6 (49.1%) 2025-11-23 09:55:21,712 - __main__ - INFO - 📈 Epoch progress: 2.95/6 (49.2%) 2025-11-23 09:55:42,453 - __main__ - INFO - 📈 Epoch progress: 2.96/6 (49.3%) 2025-11-23 09:56:02,593 - __main__ - INFO - 📈 Epoch progress: 2.97/6 (49.5%) 2025-11-23 09:56:22,738 - __main__ - INFO - 📈 Epoch progress: 2.97/6 (49.6%) 2025-11-23 09:56:43,221 - __main__ - INFO - 📈 Epoch progress: 2.98/6 (49.7%) 2025-11-23 09:57:03,321 - __main__ - INFO - 📈 Epoch progress: 2.99/6 (49.8%) 2025-11-23 09:57:23,352 - __main__ - INFO - 📈 Epoch progress: 3.00/6 (49.9%) 2025-11-23 09:57:34,917 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 09:57:46,060 - __main__ - INFO - 📈 Epoch progress: 3.00/6 (50.0%) 2025-11-23 09:58:06,345 - __main__ - INFO - 📈 Epoch progress: 3.01/6 (50.2%) 2025-11-23 09:58:47,114 - __main__ - INFO - 📊 Evaluation 22: BLEU = 0.1251 2025-11-23 09:58:47,436 - __main__ - INFO - 📊 Evaluation at step 42000 2025-11-23 09:58:51,182 - __main__ - INFO - 📤 Pushing checkpoint 42000 to HuggingFace Hub... 2025-11-23 09:58:51,183 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 09:59:02,141 - __main__ - INFO - ✅ Checkpoint 42000 pushed successfully! 2025-11-23 09:59:02,145 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-42000 2025-11-23 09:59:22,082 - __main__ - INFO - 📈 Epoch progress: 3.02/6 (50.3%) 2025-11-23 09:59:42,385 - __main__ - INFO - 📈 Epoch progress: 3.02/6 (50.4%) 2025-11-23 10:00:02,836 - __main__ - INFO - 📈 Epoch progress: 3.03/6 (50.5%) 2025-11-23 10:00:23,468 - __main__ - INFO - 📈 Epoch progress: 3.04/6 (50.6%) 2025-11-23 10:00:43,639 - __main__ - INFO - 📈 Epoch progress: 3.05/6 (50.8%) 2025-11-23 10:01:03,911 - __main__ - INFO - 📈 Epoch progress: 3.05/6 (50.9%) 2025-11-23 10:01:24,117 - __main__ - INFO - 📈 Epoch progress: 3.06/6 (51.0%) 2025-11-23 10:01:44,959 - __main__ - INFO - 📈 Epoch progress: 3.07/6 (51.1%) 2025-11-23 10:02:05,692 - __main__ - INFO - 📈 Epoch progress: 3.07/6 (51.2%) 2025-11-23 10:02:36,583 - __main__ - INFO - 📈 Epoch progress: 3.08/6 (51.4%) 2025-11-23 10:02:57,295 - __main__ - INFO - 📈 Epoch progress: 3.09/6 (51.5%) 2025-11-23 10:03:17,602 - __main__ - INFO - 📈 Epoch progress: 3.10/6 (51.6%) 2025-11-23 10:03:38,131 - __main__ - INFO - 📈 Epoch progress: 3.10/6 (51.7%) 2025-11-23 10:03:58,263 - __main__ - INFO - 📈 Epoch progress: 3.11/6 (51.8%) 2025-11-23 10:04:19,115 - __main__ - INFO - 📈 Epoch progress: 3.12/6 (52.0%) 2025-11-23 10:04:39,426 - __main__ - INFO - 📈 Epoch progress: 3.12/6 (52.1%) 2025-11-23 10:04:59,939 - __main__ - INFO - 📈 Epoch progress: 3.13/6 (52.2%) 2025-11-23 10:05:20,284 - __main__ - INFO - 📈 Epoch progress: 3.14/6 (52.3%) 2025-11-23 10:05:40,762 - __main__ - INFO - 📈 Epoch progress: 3.15/6 (52.4%) 2025-11-23 10:06:01,395 - __main__ - INFO - 📈 Epoch progress: 3.15/6 (52.6%) 2025-11-23 10:06:41,220 - __main__ - INFO - 📊 Evaluation 23: BLEU = 0.1224 2025-11-23 10:06:41,527 - __main__ - INFO - 📊 Evaluation at step 44000 2025-11-23 10:06:45,356 - __main__ - INFO - 📤 Pushing checkpoint 44000 to HuggingFace Hub... 2025-11-23 10:06:45,357 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 10:06:55,213 - __main__ - INFO - ✅ Checkpoint 44000 pushed successfully! 2025-11-23 10:06:55,215 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-44000 2025-11-23 10:07:15,185 - __main__ - INFO - 📈 Epoch progress: 3.16/6 (52.7%) 2025-11-23 10:07:35,739 - __main__ - INFO - 📈 Epoch progress: 3.17/6 (52.8%) 2025-11-23 10:07:56,198 - __main__ - INFO - 📈 Epoch progress: 3.17/6 (52.9%) 2025-11-23 10:08:16,815 - __main__ - INFO - 📈 Epoch progress: 3.18/6 (53.0%) 2025-11-23 10:08:37,084 - __main__ - INFO - 📈 Epoch progress: 3.19/6 (53.2%) 2025-11-23 10:08:57,593 - __main__ - INFO - 📈 Epoch progress: 3.20/6 (53.3%) 2025-11-23 10:09:18,413 - __main__ - INFO - 📈 Epoch progress: 3.20/6 (53.4%) 2025-11-23 10:09:38,869 - __main__ - INFO - 📈 Epoch progress: 3.21/6 (53.5%) 2025-11-23 10:09:58,906 - __main__ - INFO - 📈 Epoch progress: 3.22/6 (53.6%) 2025-11-23 10:10:19,348 - __main__ - INFO - 📈 Epoch progress: 3.23/6 (53.8%) 2025-11-23 10:10:39,588 - __main__ - INFO - 📈 Epoch progress: 3.23/6 (53.9%) 2025-11-23 10:11:00,601 - __main__ - INFO - 📈 Epoch progress: 3.24/6 (54.0%) 2025-11-23 10:11:21,144 - __main__ - INFO - 📈 Epoch progress: 3.25/6 (54.1%) 2025-11-23 10:11:41,460 - __main__ - INFO - 📈 Epoch progress: 3.25/6 (54.2%) 2025-11-23 10:12:01,734 - __main__ - INFO - 📈 Epoch progress: 3.26/6 (54.3%) 2025-11-23 10:12:22,032 - __main__ - INFO - 📈 Epoch progress: 3.27/6 (54.5%) 2025-11-23 10:12:42,864 - __main__ - INFO - 📈 Epoch progress: 3.28/6 (54.6%) 2025-11-23 10:13:03,465 - __main__ - INFO - 📈 Epoch progress: 3.28/6 (54.7%) 2025-11-23 10:13:24,120 - __main__ - INFO - 📈 Epoch progress: 3.29/6 (54.8%) 2025-11-23 10:13:43,882 - __main__ - INFO - 📈 Epoch progress: 3.30/6 (54.9%) 2025-11-23 10:14:27,364 - __main__ - INFO - 📊 Evaluation 24: BLEU = 0.1211 2025-11-23 10:14:27,682 - __main__ - INFO - 📊 Evaluation at step 46000 2025-11-23 10:14:31,358 - __main__ - INFO - 📤 Pushing checkpoint 46000 to HuggingFace Hub... 2025-11-23 10:14:31,359 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 10:14:41,433 - __main__ - INFO - ✅ Checkpoint 46000 pushed successfully! 2025-11-23 10:14:41,435 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-46000 2025-11-23 10:15:01,282 - __main__ - INFO - 📈 Epoch progress: 3.30/6 (55.1%) 2025-11-23 10:15:21,079 - __main__ - INFO - 📈 Epoch progress: 3.31/6 (55.2%) 2025-11-23 10:15:41,403 - __main__ - INFO - 📈 Epoch progress: 3.32/6 (55.3%) 2025-11-23 10:16:01,219 - __main__ - INFO - 📈 Epoch progress: 3.33/6 (55.4%) 2025-11-23 10:16:21,578 - __main__ - INFO - 📈 Epoch progress: 3.33/6 (55.5%) 2025-11-23 10:16:42,365 - __main__ - INFO - 📈 Epoch progress: 3.34/6 (55.7%) 2025-11-23 10:17:02,514 - __main__ - INFO - 📈 Epoch progress: 3.35/6 (55.8%) 2025-11-23 10:17:22,687 - __main__ - INFO - 📈 Epoch progress: 3.35/6 (55.9%) 2025-11-23 10:17:43,276 - __main__ - INFO - 📈 Epoch progress: 3.36/6 (56.0%) 2025-11-23 10:18:04,009 - __main__ - INFO - 📈 Epoch progress: 3.37/6 (56.1%) 2025-11-23 10:18:24,324 - __main__ - INFO - 📈 Epoch progress: 3.38/6 (56.3%) 2025-11-23 10:18:44,948 - __main__ - INFO - 📈 Epoch progress: 3.38/6 (56.4%) 2025-11-23 10:19:04,953 - __main__ - INFO - 📈 Epoch progress: 3.39/6 (56.5%) 2025-11-23 10:19:25,269 - __main__ - INFO - 📈 Epoch progress: 3.40/6 (56.6%) 2025-11-23 10:19:45,740 - __main__ - INFO - 📈 Epoch progress: 3.40/6 (56.7%) 2025-11-23 10:20:05,871 - __main__ - INFO - 📈 Epoch progress: 3.41/6 (56.9%) 2025-11-23 10:20:26,185 - __main__ - INFO - 📈 Epoch progress: 3.42/6 (57.0%) 2025-11-23 10:20:46,485 - __main__ - INFO - 📈 Epoch progress: 3.43/6 (57.1%) 2025-11-23 10:21:06,588 - __main__ - INFO - 📈 Epoch progress: 3.43/6 (57.2%) 2025-11-23 10:21:26,738 - __main__ - INFO - 📈 Epoch progress: 3.44/6 (57.3%) 2025-11-23 10:21:58,681 - __main__ - INFO - 📊 Evaluation 25: BLEU = 0.1286 2025-11-23 10:21:58,987 - __main__ - INFO - 📊 Evaluation at step 48000 2025-11-23 10:22:03,116 - __main__ - INFO - 📤 Pushing checkpoint 48000 to HuggingFace Hub... 2025-11-23 10:22:03,118 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 10:22:14,031 - __main__ - INFO - ✅ Checkpoint 48000 pushed successfully! 2025-11-23 10:22:14,033 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-48000 2025-11-23 10:22:34,198 - __main__ - INFO - 📈 Epoch progress: 3.45/6 (57.5%) 2025-11-23 10:22:54,429 - __main__ - INFO - 📈 Epoch progress: 3.45/6 (57.6%) 2025-11-23 10:23:15,280 - __main__ - INFO - 📈 Epoch progress: 3.46/6 (57.7%) 2025-11-23 10:23:35,391 - __main__ - INFO - 📈 Epoch progress: 3.47/6 (57.8%) 2025-11-23 10:23:55,950 - __main__ - INFO - 📈 Epoch progress: 3.48/6 (57.9%) 2025-11-23 10:24:17,499 - __main__ - INFO - 📈 Epoch progress: 3.48/6 (58.1%) 2025-11-23 10:24:38,058 - __main__ - INFO - 📈 Epoch progress: 3.49/6 (58.2%) 2025-11-23 10:24:57,657 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:25:07,623 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:25:07,623 - __main__ - INFO - 📈 Epoch progress: 3.50/6 (58.3%) 2025-11-23 10:25:07,955 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:25:17,775 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:25:28,268 - __main__ - INFO - 📈 Epoch progress: 3.50/6 (58.4%) 2025-11-23 10:25:48,893 - __main__ - INFO - 📈 Epoch progress: 3.51/6 (58.5%) 2025-11-23 10:26:09,380 - __main__ - INFO - 📈 Epoch progress: 3.52/6 (58.6%) 2025-11-23 10:26:29,817 - __main__ - INFO - 📈 Epoch progress: 3.53/6 (58.8%) 2025-11-23 10:26:50,163 - __main__ - INFO - 📈 Epoch progress: 3.53/6 (58.9%) 2025-11-23 10:27:10,420 - __main__ - INFO - 📈 Epoch progress: 3.54/6 (59.0%) 2025-11-23 10:27:30,882 - __main__ - INFO - 📈 Epoch progress: 3.55/6 (59.1%) 2025-11-23 10:27:50,942 - __main__ - INFO - 📈 Epoch progress: 3.55/6 (59.2%) 2025-11-23 10:28:11,096 - __main__ - INFO - 📈 Epoch progress: 3.56/6 (59.4%) 2025-11-23 10:28:31,403 - __main__ - INFO - 📈 Epoch progress: 3.57/6 (59.5%) 2025-11-23 10:28:51,757 - __main__ - INFO - 📈 Epoch progress: 3.58/6 (59.6%) 2025-11-23 10:29:12,648 - __main__ - INFO - 📈 Epoch progress: 3.58/6 (59.7%) 2025-11-23 10:29:49,189 - __main__ - INFO - 📊 Evaluation 26: BLEU = 0.1268 2025-11-23 10:29:49,496 - __main__ - INFO - 📊 Evaluation at step 50000 2025-11-23 10:29:53,161 - __main__ - INFO - 📤 Pushing checkpoint 50000 to HuggingFace Hub... 2025-11-23 10:29:53,165 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 10:30:05,379 - __main__ - INFO - ✅ Checkpoint 50000 pushed successfully! 2025-11-23 10:30:05,381 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-50000 2025-11-23 10:30:25,789 - __main__ - INFO - 📈 Epoch progress: 3.59/6 (59.8%) 2025-11-23 10:30:46,522 - __main__ - INFO - 📈 Epoch progress: 3.60/6 (60.0%) 2025-11-23 10:31:07,001 - __main__ - INFO - 📈 Epoch progress: 3.60/6 (60.1%) 2025-11-23 10:31:27,491 - __main__ - INFO - 📈 Epoch progress: 3.61/6 (60.2%) 2025-11-23 10:31:48,431 - __main__ - INFO - 📈 Epoch progress: 3.62/6 (60.3%) 2025-11-23 10:32:08,645 - __main__ - INFO - 📈 Epoch progress: 3.63/6 (60.4%) 2025-11-23 10:32:28,921 - __main__ - INFO - 📈 Epoch progress: 3.63/6 (60.6%) 2025-11-23 10:32:49,144 - __main__ - INFO - 📈 Epoch progress: 3.64/6 (60.7%) 2025-11-23 10:33:09,558 - __main__ - INFO - 📈 Epoch progress: 3.65/6 (60.8%) 2025-11-23 10:33:29,850 - __main__ - INFO - 📈 Epoch progress: 3.66/6 (60.9%) 2025-11-23 10:33:41,292 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:33:51,495 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:33:51,495 - __main__ - INFO - 📈 Epoch progress: 3.66/6 (61.0%) 2025-11-23 10:33:51,820 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:34:12,028 - __main__ - INFO - 📈 Epoch progress: 3.67/6 (61.2%) 2025-11-23 10:34:32,306 - __main__ - INFO - 📈 Epoch progress: 3.68/6 (61.3%) 2025-11-23 10:34:52,494 - __main__ - INFO - 📈 Epoch progress: 3.68/6 (61.4%) 2025-11-23 10:35:12,710 - __main__ - INFO - 📈 Epoch progress: 3.69/6 (61.5%) 2025-11-23 10:35:33,230 - __main__ - INFO - 📈 Epoch progress: 3.70/6 (61.6%) 2025-11-23 10:35:54,191 - __main__ - INFO - 📈 Epoch progress: 3.71/6 (61.8%) 2025-11-23 10:36:14,739 - __main__ - INFO - 📈 Epoch progress: 3.71/6 (61.9%) 2025-11-23 10:36:35,317 - __main__ - INFO - 📈 Epoch progress: 3.72/6 (62.0%) 2025-11-23 10:36:55,536 - __main__ - INFO - 📈 Epoch progress: 3.73/6 (62.1%) 2025-11-23 10:37:31,061 - __main__ - INFO - 📊 Evaluation 27: BLEU = 0.1224 2025-11-23 10:37:31,395 - __main__ - INFO - 📊 Evaluation at step 52000 2025-11-23 10:37:34,852 - __main__ - INFO - 📤 Pushing checkpoint 52000 to HuggingFace Hub... 2025-11-23 10:37:34,852 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 10:37:44,205 - __main__ - INFO - ✅ Checkpoint 52000 pushed successfully! 2025-11-23 10:37:44,216 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-52000 2025-11-23 10:38:04,587 - __main__ - INFO - 📈 Epoch progress: 3.73/6 (62.2%) 2025-11-23 10:38:24,782 - __main__ - INFO - 📈 Epoch progress: 3.74/6 (62.4%) 2025-11-23 10:38:45,142 - __main__ - INFO - 📈 Epoch progress: 3.75/6 (62.5%) 2025-11-23 10:39:06,140 - __main__ - INFO - 📈 Epoch progress: 3.76/6 (62.6%) 2025-11-23 10:39:26,435 - __main__ - INFO - 📈 Epoch progress: 3.76/6 (62.7%) 2025-11-23 10:39:46,470 - __main__ - INFO - 📈 Epoch progress: 3.77/6 (62.8%) 2025-11-23 10:40:06,885 - __main__ - INFO - 📈 Epoch progress: 3.78/6 (62.9%) 2025-11-23 10:40:28,164 - __main__ - INFO - 📈 Epoch progress: 3.78/6 (63.1%) 2025-11-23 10:40:48,905 - __main__ - INFO - 📈 Epoch progress: 3.79/6 (63.2%) 2025-11-23 10:41:09,341 - __main__ - INFO - 📈 Epoch progress: 3.80/6 (63.3%) 2025-11-23 10:41:29,882 - __main__ - INFO - 📈 Epoch progress: 3.81/6 (63.4%) 2025-11-23 10:41:50,995 - __main__ - INFO - 📈 Epoch progress: 3.81/6 (63.5%) 2025-11-23 10:42:11,398 - __main__ - INFO - 📈 Epoch progress: 3.82/6 (63.7%) 2025-11-23 10:42:31,737 - __main__ - INFO - 📈 Epoch progress: 3.83/6 (63.8%) 2025-11-23 10:42:53,318 - __main__ - INFO - 📈 Epoch progress: 3.83/6 (63.9%) 2025-11-23 10:43:13,652 - __main__ - INFO - 📈 Epoch progress: 3.84/6 (64.0%) 2025-11-23 10:43:33,771 - __main__ - INFO - 📈 Epoch progress: 3.85/6 (64.1%) 2025-11-23 10:43:54,275 - __main__ - INFO - 📈 Epoch progress: 3.86/6 (64.3%) 2025-11-23 10:44:15,406 - __main__ - INFO - 📈 Epoch progress: 3.86/6 (64.4%) 2025-11-23 10:44:35,479 - __main__ - INFO - 📈 Epoch progress: 3.87/6 (64.5%) 2025-11-23 10:45:07,040 - __main__ - INFO - 📊 Evaluation 28: BLEU = 0.1251 2025-11-23 10:45:07,376 - __main__ - INFO - 📊 Evaluation at step 54000 2025-11-23 10:45:11,318 - __main__ - INFO - 📤 Pushing checkpoint 54000 to HuggingFace Hub... 2025-11-23 10:45:11,319 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 10:45:20,609 - __main__ - INFO - ✅ Checkpoint 54000 pushed successfully! 2025-11-23 10:45:20,619 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-54000 2025-11-23 10:45:40,565 - __main__ - INFO - 📈 Epoch progress: 3.88/6 (64.6%) 2025-11-23 10:46:00,991 - __main__ - INFO - 📈 Epoch progress: 3.88/6 (64.7%) 2025-11-23 10:46:21,867 - __main__ - INFO - 📈 Epoch progress: 3.89/6 (64.9%) 2025-11-23 10:46:42,776 - __main__ - INFO - 📈 Epoch progress: 3.90/6 (65.0%) 2025-11-23 10:47:03,407 - __main__ - INFO - 📈 Epoch progress: 3.91/6 (65.1%) 2025-11-23 10:47:24,292 - __main__ - INFO - 📈 Epoch progress: 3.91/6 (65.2%) 2025-11-23 10:47:44,906 - __main__ - INFO - 📈 Epoch progress: 3.92/6 (65.3%) 2025-11-23 10:48:05,505 - __main__ - INFO - 📈 Epoch progress: 3.93/6 (65.5%) 2025-11-23 10:48:25,824 - __main__ - INFO - 📈 Epoch progress: 3.93/6 (65.6%) 2025-11-23 10:48:46,280 - __main__ - INFO - 📈 Epoch progress: 3.94/6 (65.7%) 2025-11-23 10:49:06,984 - __main__ - INFO - 📈 Epoch progress: 3.95/6 (65.8%) 2025-11-23 10:49:27,677 - __main__ - INFO - 📈 Epoch progress: 3.96/6 (65.9%) 2025-11-23 10:49:47,961 - __main__ - INFO - 📈 Epoch progress: 3.96/6 (66.1%) 2025-11-23 10:50:08,291 - __main__ - INFO - 📈 Epoch progress: 3.97/6 (66.2%) 2025-11-23 10:50:29,023 - __main__ - INFO - 📈 Epoch progress: 3.98/6 (66.3%) 2025-11-23 10:50:49,463 - __main__ - INFO - 📈 Epoch progress: 3.98/6 (66.4%) 2025-11-23 10:51:09,741 - __main__ - INFO - 📈 Epoch progress: 3.99/6 (66.5%) 2025-11-23 10:51:26,974 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:51:37,118 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:51:37,119 - __main__ - INFO - 📈 Epoch progress: 4.00/6 (66.7%) 2025-11-23 10:51:37,947 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:51:49,497 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:51:59,344 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:51:59,346 - __main__ - INFO - 📈 Epoch progress: 4.01/6 (66.8%) 2025-11-23 10:51:59,640 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:52:19,475 - __main__ - INFO - 📈 Epoch progress: 4.01/6 (66.9%) 2025-11-23 10:52:52,564 - __main__ - INFO - 📊 Evaluation 29: BLEU = 0.1264 2025-11-23 10:52:52,906 - __main__ - INFO - 📊 Evaluation at step 56000 2025-11-23 10:52:57,365 - __main__ - INFO - 📤 Pushing checkpoint 56000 to HuggingFace Hub... 2025-11-23 10:52:57,366 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 10:53:07,429 - __main__ - INFO - ✅ Checkpoint 56000 pushed successfully! 2025-11-23 10:53:07,430 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-56000 2025-11-23 10:53:27,780 - __main__ - INFO - 📈 Epoch progress: 4.02/6 (67.0%) 2025-11-23 10:53:48,060 - __main__ - INFO - 📈 Epoch progress: 4.03/6 (67.1%) 2025-11-23 10:54:08,277 - __main__ - INFO - 📈 Epoch progress: 4.03/6 (67.2%) 2025-11-23 10:54:28,768 - __main__ - INFO - 📈 Epoch progress: 4.04/6 (67.4%) 2025-11-23 10:54:49,715 - __main__ - INFO - 📈 Epoch progress: 4.05/6 (67.5%) 2025-11-23 10:55:10,410 - __main__ - INFO - 📈 Epoch progress: 4.06/6 (67.6%) 2025-11-23 10:55:30,909 - __main__ - INFO - 📈 Epoch progress: 4.06/6 (67.7%) 2025-11-23 10:55:51,172 - __main__ - INFO - 📈 Epoch progress: 4.07/6 (67.8%) 2025-11-23 10:56:11,448 - __main__ - INFO - 📈 Epoch progress: 4.08/6 (68.0%) 2025-11-23 10:56:26,100 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:56:36,167 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:56:36,170 - __main__ - INFO - 📈 Epoch progress: 4.09/6 (68.1%) 2025-11-23 10:56:36,457 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:56:46,422 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:56:56,244 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:56:56,249 - __main__ - INFO - 📈 Epoch progress: 4.09/6 (68.2%) 2025-11-23 10:56:56,549 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 10:57:16,666 - __main__ - INFO - 📈 Epoch progress: 4.10/6 (68.3%) 2025-11-23 10:57:37,502 - __main__ - INFO - 📈 Epoch progress: 4.11/6 (68.4%) 2025-11-23 10:57:58,075 - __main__ - INFO - 📈 Epoch progress: 4.11/6 (68.6%) 2025-11-23 10:58:18,500 - __main__ - INFO - 📈 Epoch progress: 4.12/6 (68.7%) 2025-11-23 10:58:38,963 - __main__ - INFO - 📈 Epoch progress: 4.13/6 (68.8%) 2025-11-23 10:58:59,536 - __main__ - INFO - 📈 Epoch progress: 4.14/6 (68.9%) 2025-11-23 10:59:20,668 - __main__ - INFO - 📈 Epoch progress: 4.14/6 (69.0%) 2025-11-23 10:59:41,543 - __main__ - INFO - 📈 Epoch progress: 4.15/6 (69.2%) 2025-11-23 11:00:02,497 - __main__ - INFO - 📈 Epoch progress: 4.16/6 (69.3%) 2025-11-23 11:00:39,692 - __main__ - INFO - 📊 Evaluation 30: BLEU = 0.1249 2025-11-23 11:00:40,019 - __main__ - INFO - 📊 Evaluation at step 58000 2025-11-23 11:00:43,865 - __main__ - INFO - 📤 Pushing checkpoint 58000 to HuggingFace Hub... 2025-11-23 11:00:43,866 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 11:00:53,580 - __main__ - INFO - ✅ Checkpoint 58000 pushed successfully! 2025-11-23 11:00:53,581 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-58000 2025-11-23 11:01:14,112 - __main__ - INFO - 📈 Epoch progress: 4.16/6 (69.4%) 2025-11-23 11:01:34,296 - __main__ - INFO - 📈 Epoch progress: 4.17/6 (69.5%) 2025-11-23 11:01:55,224 - __main__ - INFO - 📈 Epoch progress: 4.18/6 (69.6%) 2025-11-23 11:02:15,819 - __main__ - INFO - 📈 Epoch progress: 4.19/6 (69.8%) 2025-11-23 11:02:36,025 - __main__ - INFO - 📈 Epoch progress: 4.19/6 (69.9%) 2025-11-23 11:02:56,860 - __main__ - INFO - 📈 Epoch progress: 4.20/6 (70.0%) 2025-11-23 11:03:17,581 - __main__ - INFO - 📈 Epoch progress: 4.21/6 (70.1%) 2025-11-23 11:03:38,145 - __main__ - INFO - 📈 Epoch progress: 4.21/6 (70.2%) 2025-11-23 11:03:59,166 - __main__ - INFO - 📈 Epoch progress: 4.22/6 (70.4%) 2025-11-23 11:04:19,583 - __main__ - INFO - 📈 Epoch progress: 4.23/6 (70.5%) 2025-11-23 11:04:40,498 - __main__ - INFO - 📈 Epoch progress: 4.24/6 (70.6%) 2025-11-23 11:05:01,826 - __main__ - INFO - 📈 Epoch progress: 4.24/6 (70.7%) 2025-11-23 11:05:22,634 - __main__ - INFO - 📈 Epoch progress: 4.25/6 (70.8%) 2025-11-23 11:05:43,338 - __main__ - INFO - 📈 Epoch progress: 4.26/6 (71.0%) 2025-11-23 11:06:04,202 - __main__ - INFO - 📈 Epoch progress: 4.26/6 (71.1%) 2025-11-23 11:06:25,149 - __main__ - INFO - 📈 Epoch progress: 4.27/6 (71.2%) 2025-11-23 11:06:45,679 - __main__ - INFO - 📈 Epoch progress: 4.28/6 (71.3%) 2025-11-23 11:07:06,224 - __main__ - INFO - 📈 Epoch progress: 4.29/6 (71.4%) 2025-11-23 11:07:26,901 - __main__ - INFO - 📈 Epoch progress: 4.29/6 (71.5%) 2025-11-23 11:07:47,653 - __main__ - INFO - 📈 Epoch progress: 4.30/6 (71.7%) 2025-11-23 11:08:19,925 - __main__ - INFO - 📊 Evaluation 31: BLEU = 0.1275 2025-11-23 11:08:20,305 - __main__ - INFO - 📊 Evaluation at step 60000 2025-11-23 11:08:25,271 - __main__ - INFO - 📤 Pushing checkpoint 60000 to HuggingFace Hub... 2025-11-23 11:08:25,272 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 11:08:35,661 - __main__ - INFO - ✅ Checkpoint 60000 pushed successfully! 2025-11-23 11:08:35,662 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-60000 2025-11-23 11:08:55,711 - __main__ - INFO - 📈 Epoch progress: 4.31/6 (71.8%) 2025-11-23 11:09:16,419 - __main__ - INFO - 📈 Epoch progress: 4.31/6 (71.9%) 2025-11-23 11:09:36,830 - __main__ - INFO - 📈 Epoch progress: 4.32/6 (72.0%) 2025-11-23 11:09:57,754 - __main__ - INFO - 📈 Epoch progress: 4.33/6 (72.1%) 2025-11-23 11:10:18,572 - __main__ - INFO - 📈 Epoch progress: 4.34/6 (72.3%) 2025-11-23 11:10:38,994 - __main__ - INFO - 📈 Epoch progress: 4.34/6 (72.4%) 2025-11-23 11:10:59,827 - __main__ - INFO - 📈 Epoch progress: 4.35/6 (72.5%) 2025-11-23 11:11:20,341 - __main__ - INFO - 📈 Epoch progress: 4.36/6 (72.6%) 2025-11-23 11:11:41,034 - __main__ - INFO - 📈 Epoch progress: 4.36/6 (72.7%) 2025-11-23 11:12:01,805 - __main__ - INFO - 📈 Epoch progress: 4.37/6 (72.9%) 2025-11-23 11:12:22,317 - __main__ - INFO - 📈 Epoch progress: 4.38/6 (73.0%) 2025-11-23 11:12:42,707 - __main__ - INFO - 📈 Epoch progress: 4.39/6 (73.1%) 2025-11-23 11:13:03,904 - __main__ - INFO - 📈 Epoch progress: 4.39/6 (73.2%) 2025-11-23 11:13:24,153 - __main__ - INFO - 📈 Epoch progress: 4.40/6 (73.3%) 2025-11-23 11:13:44,993 - __main__ - INFO - 📈 Epoch progress: 4.41/6 (73.5%) 2025-11-23 11:14:05,562 - __main__ - INFO - 📈 Epoch progress: 4.41/6 (73.6%) 2025-11-23 11:14:25,940 - __main__ - INFO - 📈 Epoch progress: 4.42/6 (73.7%) 2025-11-23 11:14:55,966 - __main__ - INFO - 📈 Epoch progress: 4.43/6 (73.8%) 2025-11-23 11:15:16,244 - __main__ - INFO - 📈 Epoch progress: 4.44/6 (73.9%) 2025-11-23 11:15:36,765 - __main__ - INFO - 📈 Epoch progress: 4.44/6 (74.1%) 2025-11-23 11:16:13,684 - __main__ - INFO - 📊 Evaluation 32: BLEU = 0.1237 2025-11-23 11:16:14,117 - __main__ - INFO - 📊 Evaluation at step 62000 2025-11-23 11:16:18,395 - __main__ - INFO - 📤 Pushing checkpoint 62000 to HuggingFace Hub... 2025-11-23 11:16:18,396 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 11:16:28,741 - __main__ - INFO - ✅ Checkpoint 62000 pushed successfully! 2025-11-23 11:16:28,743 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-62000 2025-11-23 11:16:49,119 - __main__ - INFO - 📈 Epoch progress: 4.45/6 (74.2%) 2025-11-23 11:17:09,729 - __main__ - INFO - 📈 Epoch progress: 4.46/6 (74.3%) 2025-11-23 11:17:30,183 - __main__ - INFO - 📈 Epoch progress: 4.46/6 (74.4%) 2025-11-23 11:17:50,985 - __main__ - INFO - 📈 Epoch progress: 4.47/6 (74.5%) 2025-11-23 11:18:11,757 - __main__ - INFO - 📈 Epoch progress: 4.48/6 (74.7%) 2025-11-23 11:18:32,308 - __main__ - INFO - 📈 Epoch progress: 4.49/6 (74.8%) 2025-11-23 11:18:53,006 - __main__ - INFO - 📈 Epoch progress: 4.49/6 (74.9%) 2025-11-23 11:19:13,448 - __main__ - INFO - 📈 Epoch progress: 4.50/6 (75.0%) 2025-11-23 11:19:33,723 - __main__ - INFO - 📈 Epoch progress: 4.51/6 (75.1%) 2025-11-23 11:19:54,909 - __main__ - INFO - 📈 Epoch progress: 4.52/6 (75.3%) 2025-11-23 11:20:15,364 - __main__ - INFO - 📈 Epoch progress: 4.52/6 (75.4%) 2025-11-23 11:20:35,533 - __main__ - INFO - 📈 Epoch progress: 4.53/6 (75.5%) 2025-11-23 11:20:55,811 - __main__ - INFO - 📈 Epoch progress: 4.54/6 (75.6%) 2025-11-23 11:21:16,256 - __main__ - INFO - 📈 Epoch progress: 4.54/6 (75.7%) 2025-11-23 11:21:37,348 - __main__ - INFO - 📈 Epoch progress: 4.55/6 (75.8%) 2025-11-23 11:21:57,843 - __main__ - INFO - 📈 Epoch progress: 4.56/6 (76.0%) 2025-11-23 11:22:18,570 - __main__ - INFO - 📈 Epoch progress: 4.57/6 (76.1%) 2025-11-23 11:22:37,004 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:22:47,122 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:22:47,123 - __main__ - INFO - 📈 Epoch progress: 4.57/6 (76.2%) 2025-11-23 11:22:47,478 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:23:08,019 - __main__ - INFO - 📈 Epoch progress: 4.58/6 (76.3%) 2025-11-23 11:23:28,506 - __main__ - INFO - 📈 Epoch progress: 4.59/6 (76.4%) 2025-11-23 11:24:04,349 - __main__ - INFO - 📊 Evaluation 33: BLEU = 0.1257 2025-11-23 11:24:04,671 - __main__ - INFO - 📊 Evaluation at step 64000 2025-11-23 11:24:08,153 - __main__ - INFO - 📤 Pushing checkpoint 64000 to HuggingFace Hub... 2025-11-23 11:24:08,154 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 11:24:18,528 - __main__ - INFO - ✅ Checkpoint 64000 pushed successfully! 2025-11-23 11:24:18,530 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-64000 2025-11-23 11:24:38,954 - __main__ - INFO - 📈 Epoch progress: 4.59/6 (76.6%) 2025-11-23 11:24:59,188 - __main__ - INFO - 📈 Epoch progress: 4.60/6 (76.7%) 2025-11-23 11:25:19,820 - __main__ - INFO - 📈 Epoch progress: 4.61/6 (76.8%) 2025-11-23 11:25:40,061 - __main__ - INFO - 📈 Epoch progress: 4.62/6 (76.9%) 2025-11-23 11:26:00,309 - __main__ - INFO - 📈 Epoch progress: 4.62/6 (77.0%) 2025-11-23 11:26:20,678 - __main__ - INFO - 📈 Epoch progress: 4.63/6 (77.2%) 2025-11-23 11:26:41,241 - __main__ - INFO - 📈 Epoch progress: 4.64/6 (77.3%) 2025-11-23 11:27:01,665 - __main__ - INFO - 📈 Epoch progress: 4.64/6 (77.4%) 2025-11-23 11:27:22,445 - __main__ - INFO - 📈 Epoch progress: 4.65/6 (77.5%) 2025-11-23 11:27:42,660 - __main__ - INFO - 📈 Epoch progress: 4.66/6 (77.6%) 2025-11-23 11:28:02,981 - __main__ - INFO - 📈 Epoch progress: 4.67/6 (77.8%) 2025-11-23 11:28:23,074 - __main__ - INFO - 📈 Epoch progress: 4.67/6 (77.9%) 2025-11-23 11:28:43,752 - __main__ - INFO - 📈 Epoch progress: 4.68/6 (78.0%) 2025-11-23 11:29:04,067 - __main__ - INFO - 📈 Epoch progress: 4.69/6 (78.1%) 2025-11-23 11:29:24,386 - __main__ - INFO - 📈 Epoch progress: 4.69/6 (78.2%) 2025-11-23 11:29:45,037 - __main__ - INFO - 📈 Epoch progress: 4.70/6 (78.4%) 2025-11-23 11:30:05,369 - __main__ - INFO - 📈 Epoch progress: 4.71/6 (78.5%) 2025-11-23 11:30:25,784 - __main__ - INFO - 📈 Epoch progress: 4.72/6 (78.6%) 2025-11-23 11:30:46,081 - __main__ - INFO - 📈 Epoch progress: 4.72/6 (78.7%) 2025-11-23 11:31:06,417 - __main__ - INFO - 📈 Epoch progress: 4.73/6 (78.8%) 2025-11-23 11:31:42,452 - __main__ - INFO - 📊 Evaluation 34: BLEU = 0.1273 2025-11-23 11:31:42,774 - __main__ - INFO - 📊 Evaluation at step 66000 2025-11-23 11:31:47,184 - __main__ - INFO - 📤 Pushing checkpoint 66000 to HuggingFace Hub... 2025-11-23 11:31:47,185 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 11:31:57,845 - __main__ - INFO - ✅ Checkpoint 66000 pushed successfully! 2025-11-23 11:31:57,846 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-66000 2025-11-23 11:32:17,970 - __main__ - INFO - 📈 Epoch progress: 4.74/6 (79.0%) 2025-11-23 11:32:39,352 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:32:49,208 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:32:49,220 - __main__ - INFO - 📈 Epoch progress: 4.74/6 (79.1%) 2025-11-23 11:32:49,552 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:32:59,539 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:33:09,376 - __main__ - INFO - 📈 Epoch progress: 4.75/6 (79.2%) 2025-11-23 11:33:29,665 - __main__ - INFO - 📈 Epoch progress: 4.76/6 (79.3%) 2025-11-23 11:33:50,740 - __main__ - INFO - 📈 Epoch progress: 4.77/6 (79.4%) 2025-11-23 11:34:11,309 - __main__ - INFO - 📈 Epoch progress: 4.77/6 (79.6%) 2025-11-23 11:34:31,544 - __main__ - INFO - 📈 Epoch progress: 4.78/6 (79.7%) 2025-11-23 11:34:51,889 - __main__ - INFO - 📈 Epoch progress: 4.79/6 (79.8%) 2025-11-23 11:35:12,416 - __main__ - INFO - 📈 Epoch progress: 4.79/6 (79.9%) 2025-11-23 11:35:32,846 - __main__ - INFO - 📈 Epoch progress: 4.80/6 (80.0%) 2025-11-23 11:35:53,172 - __main__ - INFO - 📈 Epoch progress: 4.81/6 (80.2%) 2025-11-23 11:36:13,603 - __main__ - INFO - 📈 Epoch progress: 4.82/6 (80.3%) 2025-11-23 11:36:34,105 - __main__ - INFO - 📈 Epoch progress: 4.82/6 (80.4%) 2025-11-23 11:36:55,072 - __main__ - INFO - 📈 Epoch progress: 4.83/6 (80.5%) 2025-11-23 11:37:15,470 - __main__ - INFO - 📈 Epoch progress: 4.84/6 (80.6%) 2025-11-23 11:37:38,273 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:37:38,274 - __main__ - INFO - 📈 Epoch progress: 4.84/6 (80.7%) 2025-11-23 11:37:39,127 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:37:59,322 - __main__ - INFO - 📈 Epoch progress: 4.85/6 (80.9%) 2025-11-23 11:38:20,306 - __main__ - INFO - 📈 Epoch progress: 4.86/6 (81.0%) 2025-11-23 11:38:41,021 - __main__ - INFO - 📈 Epoch progress: 4.87/6 (81.1%) 2025-11-23 11:39:01,832 - __main__ - INFO - 📈 Epoch progress: 4.87/6 (81.2%) 2025-11-23 11:39:41,982 - __main__ - INFO - 📊 Evaluation 35: BLEU = 0.1258 2025-11-23 11:39:42,313 - __main__ - INFO - 📊 Evaluation at step 68000 2025-11-23 11:39:46,566 - __main__ - INFO - 📤 Pushing checkpoint 68000 to HuggingFace Hub... 2025-11-23 11:39:46,567 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 11:40:01,175 - __main__ - INFO - ✅ Checkpoint 68000 pushed successfully! 2025-11-23 11:40:01,176 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-68000 2025-11-23 11:40:20,875 - __main__ - INFO - 📈 Epoch progress: 4.88/6 (81.3%) 2025-11-23 11:40:41,158 - __main__ - INFO - 📈 Epoch progress: 4.89/6 (81.5%) 2025-11-23 11:41:01,674 - __main__ - INFO - 📈 Epoch progress: 4.90/6 (81.6%) 2025-11-23 11:41:21,984 - __main__ - INFO - 📈 Epoch progress: 4.90/6 (81.7%) 2025-11-23 11:41:42,193 - __main__ - INFO - 📈 Epoch progress: 4.91/6 (81.8%) 2025-11-23 11:42:02,598 - __main__ - INFO - 📈 Epoch progress: 4.92/6 (81.9%) 2025-11-23 11:42:22,862 - __main__ - INFO - 📈 Epoch progress: 4.92/6 (82.1%) 2025-11-23 11:42:43,564 - __main__ - INFO - 📈 Epoch progress: 4.93/6 (82.2%) 2025-11-23 11:43:04,912 - __main__ - INFO - 📈 Epoch progress: 4.94/6 (82.3%) 2025-11-23 11:43:25,907 - __main__ - INFO - 📈 Epoch progress: 4.95/6 (82.4%) 2025-11-23 11:43:46,200 - __main__ - INFO - 📈 Epoch progress: 4.95/6 (82.5%) 2025-11-23 11:44:06,657 - __main__ - INFO - 📈 Epoch progress: 4.96/6 (82.7%) 2025-11-23 11:44:27,120 - __main__ - INFO - 📈 Epoch progress: 4.97/6 (82.8%) 2025-11-23 11:44:55,607 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:44:55,616 - __main__ - INFO - 📈 Epoch progress: 4.97/6 (82.9%) 2025-11-23 11:44:55,941 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:45:16,408 - __main__ - INFO - 📈 Epoch progress: 4.98/6 (83.0%) 2025-11-23 11:45:36,998 - __main__ - INFO - 📈 Epoch progress: 4.99/6 (83.1%) 2025-11-23 11:45:57,331 - __main__ - INFO - 📈 Epoch progress: 5.00/6 (83.3%) 2025-11-23 11:46:18,794 - __main__ - INFO - 📈 Epoch progress: 5.00/6 (83.4%) 2025-11-23 11:46:38,903 - __main__ - INFO - 📈 Epoch progress: 5.01/6 (83.5%) 2025-11-23 11:46:59,014 - __main__ - INFO - 📈 Epoch progress: 5.02/6 (83.6%) 2025-11-23 11:47:38,024 - __main__ - INFO - 📊 Evaluation 36: BLEU = 0.1254 2025-11-23 11:47:38,335 - __main__ - INFO - 📊 Evaluation at step 70000 2025-11-23 11:47:42,852 - __main__ - INFO - 📤 Pushing checkpoint 70000 to HuggingFace Hub... 2025-11-23 11:47:42,852 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 11:47:53,672 - __main__ - INFO - ✅ Checkpoint 70000 pushed successfully! 2025-11-23 11:47:53,673 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-70000 2025-11-23 11:48:13,512 - __main__ - INFO - 📈 Epoch progress: 5.02/6 (83.7%) 2025-11-23 11:48:33,762 - __main__ - INFO - 📈 Epoch progress: 5.03/6 (83.9%) 2025-11-23 11:48:53,996 - __main__ - INFO - 📈 Epoch progress: 5.04/6 (84.0%) 2025-11-23 11:49:14,612 - __main__ - INFO - 📈 Epoch progress: 5.05/6 (84.1%) 2025-11-23 11:49:34,923 - __main__ - INFO - 📈 Epoch progress: 5.05/6 (84.2%) 2025-11-23 11:49:55,370 - __main__ - INFO - 📈 Epoch progress: 5.06/6 (84.3%) 2025-11-23 11:50:16,078 - __main__ - INFO - 📈 Epoch progress: 5.07/6 (84.5%) 2025-11-23 11:50:36,749 - __main__ - INFO - 📈 Epoch progress: 5.07/6 (84.6%) 2025-11-23 11:50:57,642 - __main__ - INFO - 📈 Epoch progress: 5.08/6 (84.7%) 2025-11-23 11:51:18,248 - __main__ - INFO - 📈 Epoch progress: 5.09/6 (84.8%) 2025-11-23 11:51:38,487 - __main__ - INFO - 📈 Epoch progress: 5.10/6 (84.9%) 2025-11-23 11:51:59,031 - __main__ - INFO - 📈 Epoch progress: 5.10/6 (85.0%) 2025-11-23 11:52:19,499 - __main__ - INFO - 📈 Epoch progress: 5.11/6 (85.2%) 2025-11-23 11:52:35,208 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:52:45,037 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:52:45,039 - __main__ - INFO - 📈 Epoch progress: 5.12/6 (85.3%) 2025-11-23 11:52:45,388 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 11:53:05,555 - __main__ - INFO - 📈 Epoch progress: 5.12/6 (85.4%) 2025-11-23 11:53:26,834 - __main__ - INFO - 📈 Epoch progress: 5.13/6 (85.5%) 2025-11-23 11:53:47,887 - __main__ - INFO - 📈 Epoch progress: 5.14/6 (85.6%) 2025-11-23 11:54:08,563 - __main__ - INFO - 📈 Epoch progress: 5.15/6 (85.8%) 2025-11-23 11:54:29,236 - __main__ - INFO - 📈 Epoch progress: 5.15/6 (85.9%) 2025-11-23 11:54:50,073 - __main__ - INFO - 📈 Epoch progress: 5.16/6 (86.0%) 2025-11-23 11:55:28,304 - __main__ - INFO - 📊 Evaluation 37: BLEU = 0.1242 2025-11-23 11:55:28,655 - __main__ - INFO - 📊 Evaluation at step 72000 2025-11-23 11:55:33,076 - __main__ - INFO - 📤 Pushing checkpoint 72000 to HuggingFace Hub... 2025-11-23 11:55:33,078 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 11:55:43,947 - __main__ - INFO - ✅ Checkpoint 72000 pushed successfully! 2025-11-23 11:55:43,950 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-72000 2025-11-23 11:56:03,792 - __main__ - INFO - 📈 Epoch progress: 5.17/6 (86.1%) 2025-11-23 11:56:25,145 - __main__ - INFO - 📈 Epoch progress: 5.17/6 (86.2%) 2025-11-23 11:56:45,719 - __main__ - INFO - 📈 Epoch progress: 5.18/6 (86.4%) 2025-11-23 11:57:06,529 - __main__ - INFO - 📈 Epoch progress: 5.19/6 (86.5%) 2025-11-23 11:57:26,528 - __main__ - INFO - 📈 Epoch progress: 5.20/6 (86.6%) 2025-11-23 11:57:46,986 - __main__ - INFO - 📈 Epoch progress: 5.20/6 (86.7%) 2025-11-23 11:58:07,614 - __main__ - INFO - 📈 Epoch progress: 5.21/6 (86.8%) 2025-11-23 11:58:27,841 - __main__ - INFO - 📈 Epoch progress: 5.22/6 (87.0%) 2025-11-23 11:58:48,392 - __main__ - INFO - 📈 Epoch progress: 5.22/6 (87.1%) 2025-11-23 11:59:09,079 - __main__ - INFO - 📈 Epoch progress: 5.23/6 (87.2%) 2025-11-23 11:59:30,240 - __main__ - INFO - 📈 Epoch progress: 5.24/6 (87.3%) 2025-11-23 11:59:50,776 - __main__ - INFO - 📈 Epoch progress: 5.25/6 (87.4%) 2025-11-23 12:00:11,234 - __main__ - INFO - 📈 Epoch progress: 5.25/6 (87.6%) 2025-11-23 12:00:31,761 - __main__ - INFO - 📈 Epoch progress: 5.26/6 (87.7%) 2025-11-23 12:00:52,185 - __main__ - INFO - 📈 Epoch progress: 5.27/6 (87.8%) 2025-11-23 12:01:12,397 - __main__ - INFO - 📈 Epoch progress: 5.27/6 (87.9%) 2025-11-23 12:01:32,731 - __main__ - INFO - 📈 Epoch progress: 5.28/6 (88.0%) 2025-11-23 12:01:53,514 - __main__ - INFO - 📈 Epoch progress: 5.29/6 (88.2%) 2025-11-23 12:02:13,894 - __main__ - INFO - 📈 Epoch progress: 5.30/6 (88.3%) 2025-11-23 12:02:34,676 - __main__ - INFO - 📈 Epoch progress: 5.30/6 (88.4%) 2025-11-23 12:03:14,033 - __main__ - INFO - 📊 Evaluation 38: BLEU = 0.1254 2025-11-23 12:03:14,380 - __main__ - INFO - 📊 Evaluation at step 74000 2025-11-23 12:03:18,718 - __main__ - INFO - 📤 Pushing checkpoint 74000 to HuggingFace Hub... 2025-11-23 12:03:18,720 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 12:03:30,405 - __main__ - INFO - ✅ Checkpoint 74000 pushed successfully! 2025-11-23 12:03:30,415 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-74000 2025-11-23 12:03:51,901 - __main__ - INFO - 📈 Epoch progress: 5.31/6 (88.5%) 2025-11-23 12:04:12,798 - __main__ - INFO - 📈 Epoch progress: 5.32/6 (88.6%) 2025-11-23 12:04:33,746 - __main__ - INFO - 📈 Epoch progress: 5.33/6 (88.8%) 2025-11-23 12:04:55,695 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 12:04:55,697 - __main__ - INFO - 📈 Epoch progress: 5.33/6 (88.9%) 2025-11-23 12:04:56,050 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 12:05:16,766 - __main__ - INFO - 📈 Epoch progress: 5.34/6 (89.0%) 2025-11-23 12:05:38,094 - __main__ - INFO - 📈 Epoch progress: 5.35/6 (89.1%) 2025-11-23 12:05:59,874 - __main__ - INFO - 📈 Epoch progress: 5.35/6 (89.2%) 2025-11-23 12:06:20,713 - __main__ - INFO - 📈 Epoch progress: 5.36/6 (89.3%) 2025-11-23 12:06:42,241 - __main__ - INFO - 📈 Epoch progress: 5.37/6 (89.5%) 2025-11-23 12:07:02,849 - __main__ - INFO - 📈 Epoch progress: 5.38/6 (89.6%) 2025-11-23 12:07:23,895 - __main__ - INFO - 📈 Epoch progress: 5.38/6 (89.7%) 2025-11-23 12:07:44,628 - __main__ - INFO - 📈 Epoch progress: 5.39/6 (89.8%) 2025-11-23 12:08:06,418 - __main__ - INFO - 📈 Epoch progress: 5.40/6 (89.9%) 2025-11-23 12:08:21,381 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 12:08:31,578 - __main__ - INFO - 📈 Epoch progress: 5.40/6 (90.1%) 2025-11-23 12:08:52,890 - __main__ - INFO - 📈 Epoch progress: 5.41/6 (90.2%) 2025-11-23 12:09:14,214 - __main__ - INFO - 📈 Epoch progress: 5.42/6 (90.3%) 2025-11-23 12:09:35,582 - __main__ - INFO - 📈 Epoch progress: 5.43/6 (90.4%) 2025-11-23 12:09:56,347 - __main__ - INFO - 📈 Epoch progress: 5.43/6 (90.5%) 2025-11-23 12:10:17,782 - __main__ - INFO - 📈 Epoch progress: 5.44/6 (90.7%) 2025-11-23 12:10:38,744 - __main__ - INFO - 📈 Epoch progress: 5.45/6 (90.8%) 2025-11-23 12:11:19,789 - __main__ - INFO - 📊 Evaluation 39: BLEU = 0.1258 2025-11-23 12:11:20,173 - __main__ - INFO - 📊 Evaluation at step 76000 2025-11-23 12:11:24,624 - __main__ - INFO - 📤 Pushing checkpoint 76000 to HuggingFace Hub... 2025-11-23 12:11:24,627 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 12:11:36,029 - __main__ - INFO - ✅ Checkpoint 76000 pushed successfully! 2025-11-23 12:11:36,033 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-76000 2025-11-23 12:11:57,518 - __main__ - INFO - 📈 Epoch progress: 5.45/6 (90.9%) 2025-11-23 12:12:18,678 - __main__ - INFO - 📈 Epoch progress: 5.46/6 (91.0%) 2025-11-23 12:12:39,291 - __main__ - INFO - 📈 Epoch progress: 5.47/6 (91.1%) 2025-11-23 12:13:00,780 - __main__ - INFO - 📈 Epoch progress: 5.48/6 (91.3%) 2025-11-23 12:13:21,968 - __main__ - INFO - 📈 Epoch progress: 5.48/6 (91.4%) 2025-11-23 12:13:42,669 - __main__ - INFO - 📈 Epoch progress: 5.49/6 (91.5%) 2025-11-23 12:14:03,609 - __main__ - INFO - 📈 Epoch progress: 5.50/6 (91.6%) 2025-11-23 12:14:24,429 - __main__ - INFO - 📈 Epoch progress: 5.50/6 (91.7%) 2025-11-23 12:14:45,022 - __main__ - INFO - 📈 Epoch progress: 5.51/6 (91.9%) 2025-11-23 12:15:05,930 - __main__ - INFO - 📈 Epoch progress: 5.52/6 (92.0%) 2025-11-23 12:15:27,222 - __main__ - INFO - 📈 Epoch progress: 5.53/6 (92.1%) 2025-11-23 12:15:48,168 - __main__ - INFO - 📈 Epoch progress: 5.53/6 (92.2%) 2025-11-23 12:16:09,247 - __main__ - INFO - 📈 Epoch progress: 5.54/6 (92.3%) 2025-11-23 12:16:29,893 - __main__ - INFO - 📈 Epoch progress: 5.55/6 (92.5%) 2025-11-23 12:16:51,206 - __main__ - INFO - 📈 Epoch progress: 5.55/6 (92.6%) 2025-11-23 12:17:12,182 - __main__ - INFO - 📈 Epoch progress: 5.56/6 (92.7%) 2025-11-23 12:17:38,496 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 12:17:38,498 - __main__ - INFO - 📈 Epoch progress: 5.57/6 (92.8%) 2025-11-23 12:17:38,981 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 12:17:49,457 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 The origin has been unregistered from Argo Tunnel ' 2025-11-23 12:18:00,372 - __main__ - INFO - 📈 Epoch progress: 5.58/6 (92.9%) 2025-11-23 12:18:21,453 - __main__ - INFO - 📈 Epoch progress: 5.58/6 (93.1%) 2025-11-23 12:18:42,243 - __main__ - INFO - 📈 Epoch progress: 5.59/6 (93.2%) 2025-11-23 12:19:19,558 - __main__ - INFO - 📊 Evaluation 40: BLEU = 0.1256 2025-11-23 12:19:19,903 - __main__ - INFO - 📊 Evaluation at step 78000 2025-11-23 12:19:24,396 - __main__ - INFO - 📤 Pushing checkpoint 78000 to HuggingFace Hub... 2025-11-23 12:19:24,398 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-23 12:19:35,739 - __main__ - INFO - ✅ Checkpoint 78000 pushed successfully! 2025-11-23 12:19:35,741 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-78000 2025-11-23 12:19:40,328 - __main__ - INFO - ✅ MLflow: Training finished - status_code=2.0 (FINISHED) 2025-11-23 12:19:40,878 - __main__ - INFO - 💾 Saving model to models/finetuned-sw-en-phase2 2025-11-23 12:19:43,015 - __main__ - INFO - ✅ Training complete! 2025-11-23 12:19:43,017 - __main__ - INFO - 📊 STEP 3: Final Evaluation 2025-11-23 12:19:43,367 - __main__ - INFO - ✅ MLflow: Final evaluation starting - status_code=3.0 2025-11-23 12:19:43,376 - __main__ - INFO - ====================================================================== 2025-11-23 12:19:43,378 - __main__ - INFO - 📊 Running FINAL evaluation on test set... 2025-11-23 12:19:43,381 - __main__ - INFO - ====================================================================== 2025-11-23 12:19:43,382 - __main__ - INFO - Tokenizing test dataset... 2025-11-23 12:19:45,364 - datasets.fingerprint - WARNING - Parameter 'function'=> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only shown once. Subsequent hashing failures won't be shown. 2025-11-23 13:31:14,215 - __main__ - INFO - 📊 Evaluation 41: BLEU = 0.1327 2025-11-23 13:31:14,636 - __main__ - INFO - 📊 Evaluation at step 78000 2025-11-23 13:31:14,792 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-23 13:31:14,838 - __main__ - INFO - Computing predictions for COMET (1000 samples, batch_size=32)... 2025-11-23 13:31:14,839 - __main__ - INFO - Batch 1/32... 2025-11-23 13:31:17,926 - __main__ - INFO - Batch 5/32... 2025-11-23 13:31:19,046 - __main__ - INFO - Batch 10/32... 2025-11-23 13:31:20,194 - __main__ - INFO - Batch 15/32... 2025-11-23 13:31:21,438 - __main__ - INFO - Batch 20/32... 2025-11-23 13:31:22,619 - __main__ - INFO - Batch 25/32... 2025-11-23 13:31:23,790 - __main__ - INFO - Batch 30/32... 2025-11-23 13:31:24,397 - __main__ - INFO - Running COMET-QE model... 2025-11-23 13:31:24,429 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-23 13:31:24,438 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-23 13:31:24,440 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-23 13:31:24,442 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-23 13:31:29,652 - __main__ - INFO - ✅ COMET-QE: 0.1811 ± 0.2659 2025-11-23 13:31:29,654 - __main__ - INFO - ✅ FINAL Test Results: 2025-11-23 13:31:29,655 - __main__ - INFO - BLEU: 0.1327 2025-11-23 13:31:29,655 - __main__ - INFO - chrF: 32.59 2025-11-23 13:31:29,656 - __main__ - INFO - COMET-QE: 0.1811 2025-11-23 13:31:29,656 - __main__ - INFO - ====================================================================== 2025-11-23 13:31:29,658 - __main__ - INFO - ====================================================================== 2025-11-23 13:31:29,658 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-23 13:31:29,659 - __main__ - INFO - ====================================================================== 2025-11-23 13:31:29,660 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-23 13:31:29,661 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-23 13:31:31,728 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-23 13:31:31,796 - __main__ - INFO - ✅ Domain chrF: 1.12 2025-11-23 13:31:31,797 - __main__ - INFO - Computing COMET-QE... 2025-11-23 13:31:31,825 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-23 13:31:31,833 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-23 13:31:31,833 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-23 13:31:31,835 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-23 13:31:34,250 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 2025-11-23 13:31:34,251 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-23 13:31:34,251 - __main__ - INFO - BLEU: 0.0000 2025-11-23 13:31:34,252 - __main__ - INFO - chrF: 1.12 2025-11-23 13:31:34,252 - __main__ - INFO - COMET-QE: 0.0001 2025-11-23 13:31:34,252 - __main__ - INFO - ====================================================================== 2025-11-23 13:31:34,263 - __main__ - INFO - 💾 Final results saved to models/finetuned-sw-en-phase2/test_results.json 2025-11-23 13:31:34,270 - __main__ - INFO - ✅ Verified test_results.json with 14 metrics 2025-11-23 13:31:34,272 - __main__ - INFO - 📊 Logging all metrics to TensorBoard... 2025-11-23 13:31:34,531 - __main__ - INFO - ✅ All metrics logged to TensorBoard 2025-11-23 13:31:34,533 - __main__ - INFO - Baseline metrics: 13 2025-11-23 13:31:34,535 - __main__ - INFO - Final test metrics: 10 2025-11-23 13:31:34,536 - __main__ - INFO - Final domain metrics: 4 2025-11-23 13:31:34,537 - __main__ - INFO - ====================================================================== 2025-11-23 13:31:34,537 - __main__ - INFO - 📊 BASELINE vs FINE-TUNED Comparison 2025-11-23 13:31:34,538 - __main__ - INFO - ====================================================================== 2025-11-23 13:31:34,538 - __main__ - INFO - 📈 Test Set Metrics: 2025-11-23 13:31:34,539 - __main__ - INFO - Metric Baseline Fine-tuned Improvement 2025-11-23 13:31:34,540 - __main__ - INFO - ------------------------------------------------------------ 2025-11-23 13:31:34,540 - __main__ - INFO - eval_bleu 0.0000 0.1327 N/A 2025-11-23 13:31:34,542 - __main__ - INFO - eval_chrf 0.0000 32.5863 N/A 2025-11-23 13:31:34,542 - __main__ - INFO - eval_comet 0.0000 0.1811 N/A 2025-11-23 13:31:34,543 - __main__ - INFO - 📈 Domain Metrics (Real Call Transcriptions): 2025-11-23 13:31:34,543 - __main__ - INFO - Metric Baseline Fine-tuned Improvement 2025-11-23 13:31:34,544 - __main__ - INFO - ------------------------------------------------------------ 2025-11-23 13:31:34,545 - __main__ - INFO - domain_bleu 0.0000 0.0000 -99.2% 2025-11-23 13:31:34,545 - __main__ - INFO - domain_chrf 1.2250 1.1158 -8.9% 2025-11-23 13:31:34,546 - __main__ - INFO - domain_comet 0.0001 0.0001 +1.4% 2025-11-23 13:31:34,546 - __main__ - INFO - ====================================================================== 2025-11-23 13:31:34,986 - __main__ - INFO - ✅ Logged 14 final metrics to MLflow 2025-11-23 13:31:34,988 - __main__ - INFO - Visit MLflow: https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-23 13:31:34,989 - __main__ - INFO - 📊 Captured 1601 training log entries 2025-11-23 13:31:35,055 - __main__ - INFO - 💾 Saved comprehensive metrics to models/finetuned-sw-en-phase2/training_metrics.json 2025-11-23 13:31:35,065 - __main__ - INFO - 💾 Saved training config to models/finetuned-sw-en-phase2/training_config_used.json