2025-11-26 12:52:01,389 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-26 12:52:01,419 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json 2025-11-26 12:52:01,420 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 12:52:01,421 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-26 12:52:01,424 - __main__ - INFO - 🎲 Random seed: 42 2025-11-26 12:52:01,425 - __main__ - ERROR - ====================================================================== 2025-11-26 12:52:01,426 - __main__ - ERROR - ❌ HF_TOKEN NOT FOUND! 2025-11-26 12:52:01,426 - __main__ - ERROR - ====================================================================== 2025-11-26 12:52:01,427 - __main__ - ERROR - push_to_hub is enabled but no HuggingFace token is available. 2025-11-26 12:52:01,427 - __main__ - ERROR - To fix this, either: 2025-11-26 12:52:01,428 - __main__ - ERROR - 1. Set HF_TOKEN in your .env file: 2025-11-26 12:52:01,429 - __main__ - ERROR - HF_TOKEN=hf_xxxxx 2025-11-26 12:52:01,430 - __main__ - ERROR - 2. Or export it in your shell: 2025-11-26 12:52:01,430 - __main__ - ERROR - export HF_TOKEN=hf_xxxxx 2025-11-26 12:52:01,431 - __main__ - ERROR - 3. Or disable push_to_hub in your config: 2025-11-26 12:52:01,432 - __main__ - ERROR - "push_to_hub": false 2025-11-26 12:52:01,433 - __main__ - ERROR - ====================================================================== 2025-11-26 13:01:56,894 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-26 13:01:56,929 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json 2025-11-26 13:01:56,930 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 13:01:56,931 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-26 13:01:56,935 - __main__ - INFO - 🎲 Random seed: 42 2025-11-26 13:01:56,941 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-26 13:01:56,943 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 13:01:56,944 - __main__ - INFO - Push strategy: checkpoint 2025-11-26 13:01:56,946 - __main__ - INFO - → Will push after each checkpoint save 2025-11-26 13:01:56,947 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-26 13:01:56,948 - __main__ - INFO - 🔍 Checking connectivity to https://proceedings-graphic-pressed-responded.trycloudflare.com... 2025-11-26 13:01:56,949 - __main__ - INFO - 🔍 Checking connectivity to proceedings-graphic-pressed-responded.trycloudflare.com:443... 2025-11-26 13:01:56,960 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-26 13:01:56,961 - __main__ - INFO - ✅ MLflow tracking URI set: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 13:01:57,844 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-26 13:01:59,498 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251126_130158 2025-11-26 13:01:59,514 - __main__ - INFO - Run ID: 943aef604047491e932c794709ae64e0 2025-11-26 13:01:59,515 - __main__ - INFO - Tracking URI: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 13:02:00,428 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow 2025-11-26 13:02:00,432 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-26 13:02:00,451 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-26 13:02:00,451 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-26 13:02:00,452 - __main__ - INFO - 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 13:02:09,010 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-26 13:02:09,022 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-26 13:02:09,447 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file 2025-11-26 13:02:09,721 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file 2025-11-26 13:02:09,722 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl 2025-11-26 13:02:13,017 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) 2025-11-26 13:02:16,033 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 2025-11-26 13:02:18,985 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 2025-11-26 13:02:21,608 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 2025-11-26 13:02:24,027 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 2025-11-26 13:02:26,435 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 2025-11-26 13:02:28,868 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 2025-11-26 13:02:31,759 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 2025-11-26 13:02:34,502 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 2025-11-26 13:02:36,901 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 2025-11-26 13:02:39,333 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 2025-11-26 13:02:41,749 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 2025-11-26 13:02:44,399 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 2025-11-26 13:02:46,918 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 2025-11-26 13:02:49,320 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 2025-11-26 13:02:51,725 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 2025-11-26 13:02:54,140 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 2025-11-26 13:02:56,554 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 2025-11-26 13:02:59,008 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 2025-11-26 13:03:01,447 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 2025-11-26 13:03:03,861 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 2025-11-26 13:03:06,242 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 2025-11-26 13:03:08,815 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 2025-11-26 13:03:11,442 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 2025-11-26 13:03:14,195 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 2025-11-26 13:03:16,608 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 2025-11-26 13:03:16,848 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) 2025-11-26 13:03:16,881 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl 2025-11-26 13:03:18,987 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) 2025-11-26 13:03:21,938 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 2025-11-26 13:03:24,462 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 2025-11-26 13:03:26,865 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 2025-11-26 13:03:29,237 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 2025-11-26 13:03:31,666 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 2025-11-26 13:03:34,113 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 2025-11-26 13:03:36,520 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 2025-11-26 13:03:38,967 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 2025-11-26 13:03:41,371 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 2025-11-26 13:03:43,793 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 2025-11-26 13:03:45,416 - __main__ - WARNING - ⚠️ Training interrupted by user 2025-11-26 13:03:47,340 - __main__ - INFO - ✅ MLflow run ended: 943aef604047491e932c794709ae64e0 2025-11-26 13:04:46,954 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-26 13:04:46,964 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json 2025-11-26 13:04:46,965 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 13:04:46,966 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-26 13:04:46,970 - __main__ - INFO - 🎲 Random seed: 42 2025-11-26 13:04:46,971 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-26 13:04:46,972 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 13:04:46,972 - __main__ - INFO - Push strategy: checkpoint 2025-11-26 13:04:46,973 - __main__ - INFO - → Will push after each checkpoint save 2025-11-26 13:04:46,975 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-26 13:04:46,976 - __main__ - INFO - 🔍 Checking connectivity to https://proceedings-graphic-pressed-responded.trycloudflare.com... 2025-11-26 13:04:46,977 - __main__ - INFO - 🔍 Checking connectivity to proceedings-graphic-pressed-responded.trycloudflare.com:443... 2025-11-26 13:04:46,995 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-26 13:04:46,997 - __main__ - INFO - ✅ MLflow tracking URI set: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 13:04:47,774 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-26 13:04:48,874 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251126_130448 2025-11-26 13:04:48,876 - __main__ - INFO - Run ID: e1050fe493a94824b818f08f28d430f0 2025-11-26 13:04:48,877 - __main__ - INFO - Tracking URI: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 13:04:49,354 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow 2025-11-26 13:04:49,358 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-26 13:04:49,365 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-26 13:04:49,366 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-26 13:04:49,367 - __main__ - INFO - 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 13:04:51,598 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-26 13:04:51,615 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-26 13:04:52,040 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file 2025-11-26 13:04:52,328 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file 2025-11-26 13:04:52,329 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl 2025-11-26 13:04:52,521 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) 2025-11-26 13:04:54,897 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 2025-11-26 13:04:57,451 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 2025-11-26 13:05:00,232 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 2025-11-26 13:05:02,639 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 2025-11-26 13:05:05,043 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 2025-11-26 13:05:07,478 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 2025-11-26 13:05:09,861 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 2025-11-26 13:05:12,452 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 2025-11-26 13:05:14,945 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 2025-11-26 13:05:17,372 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 2025-11-26 13:05:19,794 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 2025-11-26 13:05:22,553 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 2025-11-26 13:05:24,981 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 2025-11-26 13:05:27,398 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 2025-11-26 13:05:29,842 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 2025-11-26 13:05:32,249 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 2025-11-26 13:05:34,653 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 2025-11-26 13:05:37,067 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 2025-11-26 13:05:39,861 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 2025-11-26 13:05:42,277 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 2025-11-26 13:05:44,672 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 2025-11-26 13:05:47,628 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 2025-11-26 13:05:50,193 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 2025-11-26 13:05:52,618 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 2025-11-26 13:05:55,055 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 2025-11-26 13:05:55,296 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) 2025-11-26 13:05:55,358 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl 2025-11-26 13:05:55,568 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) 2025-11-26 13:05:58,047 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 2025-11-26 13:06:00,483 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 2025-11-26 13:06:02,890 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 2025-11-26 13:06:05,304 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 2025-11-26 13:06:08,205 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 2025-11-26 13:06:10,666 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 2025-11-26 13:06:13,087 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 2025-11-26 13:06:15,493 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 2025-11-26 13:06:17,903 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 2025-11-26 13:06:20,320 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 2025-11-26 13:06:22,778 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 2025-11-26 13:06:25,558 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 2025-11-26 13:06:28,067 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 2025-11-26 13:06:30,508 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 2025-11-26 13:06:32,949 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 2025-11-26 13:06:35,467 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 2025-11-26 13:06:37,702 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) 2025-11-26 13:06:37,760 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 2025-11-26 13:06:37,762 - __main__ - INFO - 📝 Sample data check: 2025-11-26 13:06:37,763 - __main__ - INFO - Sample 1: 2025-11-26 13:06:37,766 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... 2025-11-26 13:06:37,767 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... 2025-11-26 13:06:37,767 - __main__ - INFO - Sample 2: 2025-11-26 13:06:37,768 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... 2025-11-26 13:06:37,771 - __main__ - INFO - Target (6 words): Job had said no such thing.... 2025-11-26 13:06:37,772 - __main__ - INFO - Sample 3: 2025-11-26 13:06:37,773 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... 2025-11-26 13:06:37,775 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... 2025-11-26 13:06:43,119 - __main__ - INFO - Filtering statistics: {'kept': 4202243} 2025-11-26 13:06:43,125 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 2025-11-26 13:06:47,486 - __main__ - INFO - Removed 518 duplicates 2025-11-26 13:06:47,701 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 2025-11-26 13:06:54,433 - __main__ - INFO - ====================================================================== 2025-11-26 13:06:54,445 - __main__ - INFO - ✅ Dataset splits: 2025-11-26 13:06:54,448 - __main__ - INFO - Train: 3,571,465 samples 2025-11-26 13:06:54,451 - __main__ - INFO - Validation: 420,173 samples 2025-11-26 13:06:54,452 - __main__ - INFO - Test: 210,087 samples 2025-11-26 13:06:54,453 - __main__ - INFO - ====================================================================== 2025-11-26 13:06:54,894 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-26 13:10:05,211 - __main__ - INFO - ✅ Dataset validation passed 2025-11-26 13:10:05,223 - __main__ - INFO - Train: 3,571,465 samples 2025-11-26 13:10:05,225 - __main__ - INFO - Validation: 420,173 samples 2025-11-26 13:10:05,227 - __main__ - INFO - Test: 210,087 samples 2025-11-26 13:10:07,926 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: 59e2a5c4) 2025-11-26 13:10:07,934 - __main__ - INFO - Next run will skip tokenization and load from cache 2025-11-26 13:10:07,936 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... 2025-11-26 13:10:34,363 - __main__ - ERROR - ❌ Pipeline failed: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. Traceback (most recent call last): File "/workspace/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1992, in main trainer_manager = TranslationTrainer(config, model, tokenizer, dataset_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1331, in __init__ self.tokenized_datasets = self.dataset_dict.map( ^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/datasets/dataset_dict.py", line 953, in map dataset_dict[split] = dataset.map( ^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 562, in wrapper out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3332, in map for rank, done, content in iflatmap_unordered( File "/venv/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 619, in iflatmap_unordered raise RuntimeError( RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. 2025-11-26 13:10:36,392 - __main__ - INFO - ✅ MLflow run ended: e1050fe493a94824b818f08f28d430f0 2025-11-26 14:17:28,234 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-26 14:17:28,246 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json 2025-11-26 14:17:28,247 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 14:17:28,248 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-26 14:17:28,251 - __main__ - INFO - 🎲 Random seed: 42 2025-11-26 14:17:28,252 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-26 14:17:28,253 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 14:17:28,255 - __main__ - INFO - Push strategy: checkpoint 2025-11-26 14:17:28,256 - __main__ - INFO - → Will push after each checkpoint save 2025-11-26 14:17:28,257 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-26 14:17:28,258 - __main__ - INFO - 🔍 Checking connectivity to https://proceedings-graphic-pressed-responded.trycloudflare.com... 2025-11-26 14:17:28,260 - __main__ - INFO - 🔍 Checking connectivity to proceedings-graphic-pressed-responded.trycloudflare.com:443... 2025-11-26 14:17:28,277 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-26 14:17:28,278 - __main__ - INFO - ✅ MLflow tracking URI set: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 14:17:29,065 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-26 14:17:30,255 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251126_141729 2025-11-26 14:17:30,256 - __main__ - INFO - Run ID: 7185822b56ad49388b2566fd03e2a229 2025-11-26 14:17:30,261 - __main__ - INFO - Tracking URI: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 14:17:30,730 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow 2025-11-26 14:17:30,734 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-26 14:17:30,753 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-26 14:17:30,754 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-26 14:17:30,755 - __main__ - INFO - 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 14:17:35,248 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-26 14:17:35,250 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-26 14:17:35,722 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file 2025-11-26 14:17:36,035 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file 2025-11-26 14:17:36,037 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl 2025-11-26 14:17:39,086 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) 2025-11-26 14:17:41,999 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 2025-11-26 14:17:44,556 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 2025-11-26 14:17:47,056 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 2025-11-26 14:17:49,518 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 2025-11-26 14:17:51,962 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 2025-11-26 14:17:54,480 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 2025-11-26 14:17:56,916 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 2025-11-26 14:17:59,364 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 2025-11-26 14:18:01,847 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 2025-11-26 14:18:04,352 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 2025-11-26 14:18:06,881 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 2025-11-26 14:18:09,358 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 2025-11-26 14:18:11,829 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 2025-11-26 14:18:14,330 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 2025-11-26 14:18:16,847 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 2025-11-26 14:18:19,295 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 2025-11-26 14:18:21,734 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 2025-11-26 14:18:24,181 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 2025-11-26 14:18:26,628 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 2025-11-26 14:18:29,053 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 2025-11-26 14:18:31,486 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 2025-11-26 14:18:33,936 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 2025-11-26 14:18:36,408 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 2025-11-26 14:18:38,861 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 2025-11-26 14:18:41,313 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 2025-11-26 14:18:41,558 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) 2025-11-26 14:18:41,590 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl 2025-11-26 14:18:42,885 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) 2025-11-26 14:18:45,399 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 2025-11-26 14:18:47,870 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 2025-11-26 14:18:50,286 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 2025-11-26 14:18:52,733 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 2025-11-26 14:18:55,188 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 2025-11-26 14:18:57,622 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 2025-11-26 14:19:00,054 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 2025-11-26 14:19:02,462 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 2025-11-26 14:19:04,871 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 2025-11-26 14:19:07,288 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 2025-11-26 14:19:09,700 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 2025-11-26 14:19:12,140 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 2025-11-26 14:19:14,574 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 2025-11-26 14:19:16,993 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 2025-11-26 14:19:19,449 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 2025-11-26 14:19:21,849 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 2025-11-26 14:19:24,108 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) 2025-11-26 14:19:24,165 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 2025-11-26 14:19:24,166 - __main__ - INFO - 📝 Sample data check: 2025-11-26 14:19:24,167 - __main__ - INFO - Sample 1: 2025-11-26 14:19:24,168 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... 2025-11-26 14:19:24,169 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... 2025-11-26 14:19:24,170 - __main__ - INFO - Sample 2: 2025-11-26 14:19:24,171 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... 2025-11-26 14:19:24,172 - __main__ - INFO - Target (6 words): Job had said no such thing.... 2025-11-26 14:19:24,173 - __main__ - INFO - Sample 3: 2025-11-26 14:19:24,174 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... 2025-11-26 14:19:24,175 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... 2025-11-26 14:19:29,479 - __main__ - INFO - Filtering statistics: {'kept': 4202243} 2025-11-26 14:19:29,481 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 2025-11-26 14:19:34,042 - __main__ - INFO - Removed 518 duplicates 2025-11-26 14:19:34,313 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 2025-11-26 14:19:41,590 - __main__ - INFO - ====================================================================== 2025-11-26 14:19:41,592 - __main__ - INFO - ✅ Dataset splits: 2025-11-26 14:19:41,593 - __main__ - INFO - Train: 3,571,465 samples 2025-11-26 14:19:41,594 - __main__ - INFO - Validation: 420,173 samples 2025-11-26 14:19:41,595 - __main__ - INFO - Test: 210,087 samples 2025-11-26 14:19:41,596 - __main__ - INFO - ====================================================================== 2025-11-26 14:19:42,055 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-26 14:22:54,655 - __main__ - INFO - ✅ Dataset validation passed 2025-11-26 14:22:54,658 - __main__ - INFO - Train: 3,571,465 samples 2025-11-26 14:22:54,660 - __main__ - INFO - Validation: 420,173 samples 2025-11-26 14:22:54,661 - __main__ - INFO - Test: 210,087 samples 2025-11-26 14:22:57,099 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: 59e2a5c4) 2025-11-26 14:22:57,117 - __main__ - INFO - Next run will skip tokenization and load from cache 2025-11-26 14:22:57,120 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... 2025-11-26 14:23:22,483 - __main__ - ERROR - ❌ Pipeline failed: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. Traceback (most recent call last): File "/workspace/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1992, in main trainer_manager = TranslationTrainer(config, model, tokenizer, dataset_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1331, in __init__ self.tokenized_datasets = self.dataset_dict.map( ^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/datasets/dataset_dict.py", line 953, in map dataset_dict[split] = dataset.map( ^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 562, in wrapper out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3332, in map for rank, done, content in iflatmap_unordered( File "/venv/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 619, in iflatmap_unordered raise RuntimeError( RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. 2025-11-26 14:23:24,870 - __main__ - INFO - ✅ MLflow run ended: 7185822b56ad49388b2566fd03e2a229 2025-11-26 14:52:14,188 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-26 14:52:14,224 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json 2025-11-26 14:52:14,226 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 14:52:14,227 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-26 14:52:14,232 - __main__ - INFO - 🎲 Random seed: 42 2025-11-26 14:52:14,233 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-26 14:52:14,234 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 14:52:14,235 - __main__ - INFO - Push strategy: checkpoint 2025-11-26 14:52:14,236 - __main__ - INFO - → Will push after each checkpoint save 2025-11-26 14:52:14,237 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-26 14:52:14,238 - __main__ - INFO - 🔍 Checking connectivity to https://proceedings-graphic-pressed-responded.trycloudflare.com... 2025-11-26 14:52:14,239 - __main__ - INFO - 🔍 Checking connectivity to proceedings-graphic-pressed-responded.trycloudflare.com:443... 2025-11-26 14:52:14,261 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-26 14:52:14,263 - __main__ - INFO - ✅ MLflow tracking URI set: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 14:52:15,107 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-26 14:52:16,508 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251126_145215 2025-11-26 14:52:16,516 - __main__ - INFO - Run ID: 56240a85d2814079a1e568351bd90cef 2025-11-26 14:52:16,519 - __main__ - INFO - Tracking URI: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 14:52:16,969 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow 2025-11-26 14:52:16,972 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-26 14:52:17,115 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-26 14:52:17,117 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-26 14:52:17,118 - __main__ - INFO - 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 14:52:25,971 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-26 14:52:25,973 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-26 14:52:26,457 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file 2025-11-26 14:52:26,739 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file 2025-11-26 14:52:26,740 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl 2025-11-26 14:52:37,140 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) 2025-11-26 14:52:40,247 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 2025-11-26 14:52:43,241 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 2025-11-26 14:52:45,871 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 2025-11-26 14:52:48,447 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 2025-11-26 14:52:51,105 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 2025-11-26 14:52:53,747 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 2025-11-26 14:52:56,348 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 2025-11-26 14:52:58,964 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 2025-11-26 14:53:01,618 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 2025-11-26 14:53:04,186 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 2025-11-26 14:53:06,761 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 2025-11-26 14:53:09,408 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 2025-11-26 14:53:12,072 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 2025-11-26 14:53:14,680 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 2025-11-26 14:53:17,357 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 2025-11-26 14:53:20,027 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 2025-11-26 14:53:22,667 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 2025-11-26 14:53:25,362 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 2025-11-26 14:53:28,018 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 2025-11-26 14:53:30,678 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 2025-11-26 14:53:33,290 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 2025-11-26 14:53:36,230 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 2025-11-26 14:53:39,427 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 2025-11-26 14:53:41,946 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 2025-11-26 14:53:44,471 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 2025-11-26 14:53:44,716 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) 2025-11-26 14:53:44,752 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl 2025-11-26 14:53:53,135 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) 2025-11-26 14:53:56,107 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 2025-11-26 14:53:59,022 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 2025-11-26 14:54:01,537 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 2025-11-26 14:54:03,970 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 2025-11-26 14:54:06,435 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 2025-11-26 14:54:08,903 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 2025-11-26 14:54:11,364 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 2025-11-26 14:54:13,833 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 2025-11-26 14:54:16,294 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 2025-11-26 14:54:18,765 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 2025-11-26 14:54:21,232 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 2025-11-26 14:54:23,706 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 2025-11-26 14:54:26,181 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 2025-11-26 14:54:28,627 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 2025-11-26 14:54:31,077 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 2025-11-26 14:54:33,538 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 2025-11-26 14:54:35,808 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) 2025-11-26 14:54:35,867 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 2025-11-26 14:54:35,868 - __main__ - INFO - 📝 Sample data check: 2025-11-26 14:54:35,871 - __main__ - INFO - Sample 1: 2025-11-26 14:54:35,873 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... 2025-11-26 14:54:35,874 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... 2025-11-26 14:54:35,875 - __main__ - INFO - Sample 2: 2025-11-26 14:54:35,875 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... 2025-11-26 14:54:35,876 - __main__ - INFO - Target (6 words): Job had said no such thing.... 2025-11-26 14:54:35,877 - __main__ - INFO - Sample 3: 2025-11-26 14:54:35,878 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... 2025-11-26 14:54:35,879 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... 2025-11-26 14:54:41,232 - __main__ - INFO - Filtering statistics: {'kept': 4202243} 2025-11-26 14:54:41,233 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 2025-11-26 14:54:45,682 - __main__ - INFO - Removed 518 duplicates 2025-11-26 14:54:45,894 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 2025-11-26 14:54:53,322 - __main__ - INFO - ====================================================================== 2025-11-26 14:54:53,325 - __main__ - INFO - ✅ Dataset splits: 2025-11-26 14:54:53,326 - __main__ - INFO - Train: 3,571,465 samples 2025-11-26 14:54:53,326 - __main__ - INFO - Validation: 420,173 samples 2025-11-26 14:54:53,327 - __main__ - INFO - Test: 210,087 samples 2025-11-26 14:54:53,328 - __main__ - INFO - ====================================================================== 2025-11-26 14:54:53,835 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-26 14:58:05,583 - __main__ - INFO - ✅ Dataset validation passed 2025-11-26 14:58:05,586 - __main__ - INFO - Train: 3,571,465 samples 2025-11-26 14:58:05,588 - __main__ - INFO - Validation: 420,173 samples 2025-11-26 14:58:05,589 - __main__ - INFO - Test: 210,087 samples 2025-11-26 14:58:07,067 - __main__ - INFO - ✅ Found cached tokenized datasets (hash: 59e2a5c4) 2025-11-26 14:58:07,068 - __main__ - INFO - Skipping tokenization - this saves significant time! 2025-11-26 14:58:07,069 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... 2025-11-26 14:58:42,055 - __main__ - INFO - ✅ Tokenization complete! 2025-11-26 14:58:42,057 - __main__ - INFO - 🔧 Initializing COMET-QE model... 2025-11-26 14:58:42,159 - comet.models.download_utils - INFO - Downloading wmt20-comet-qe-da.tar.gz 2025-11-26 14:59:21,457 - comet.models.download_utils - INFO - Extracting /root/.cache/torch/unbabel_comet/wmt20-comet-qe-da.tar.gz 2025-11-26 14:59:39,749 - comet.models.download_utils - INFO - Extracted /root/.cache/torch/unbabel_comet/wmt20-comet-qe-da.tar.gz 2025-11-26 14:59:41,205 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` 2025-11-26 14:59:49,994 - comet.models.base - INFO - Encoder model frozen. 2025-11-26 14:59:50,547 - __main__ - INFO - ✅ COMET-QE model loaded successfully 2025-11-26 14:59:50,549 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ 2025-11-26 14:59:50,597 - __main__ - INFO - ✅ Trainer initialized 2025-11-26 14:59:50,598 - __main__ - INFO - 🔬 STEP 1: Baseline Evaluation 2025-11-26 14:59:51,327 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 2025-11-26 14:59:51,328 - __main__ - INFO - ====================================================================== 2025-11-26 14:59:51,330 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... 2025-11-26 14:59:51,331 - __main__ - INFO - ====================================================================== 2025-11-26 15:00:31,842 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.1216 2025-11-26 15:00:31,844 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-26 15:00:31,875 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... 2025-11-26 15:00:31,877 - __main__ - INFO - Batch 1/16... 2025-11-26 15:00:33,243 - __main__ - INFO - Batch 5/16... 2025-11-26 15:00:34,320 - __main__ - INFO - Batch 10/16... 2025-11-26 15:00:35,558 - __main__ - INFO - Batch 15/16... 2025-11-26 15:00:35,946 - __main__ - INFO - Running COMET-QE model... 2025-11-26 15:00:35,958 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-26 15:00:35,967 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-26 15:00:35,967 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-26 15:00:35,968 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision 2025-11-26 15:00:35,970 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-26 15:00:39,691 - __main__ - INFO - ✅ COMET-QE: 0.1977 ± 0.2784 2025-11-26 15:00:39,692 - __main__ - INFO - ✅ Baseline Results (Test Set): 2025-11-26 15:00:39,692 - __main__ - INFO - BLEU: 0.1216 2025-11-26 15:00:39,693 - __main__ - INFO - chrF: 32.36 2025-11-26 15:00:39,693 - __main__ - INFO - COMET: 0.1977 2025-11-26 15:00:39,695 - __main__ - INFO - ====================================================================== 2025-11-26 15:00:39,695 - __main__ - INFO - ====================================================================== 2025-11-26 15:00:39,696 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-26 15:00:39,696 - __main__ - INFO - ====================================================================== 2025-11-26 15:00:39,697 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-26 15:00:39,698 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-26 15:00:40,792 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-26 15:00:40,878 - __main__ - INFO - ✅ Domain chrF: 1.22 2025-11-26 15:00:40,879 - __main__ - INFO - Computing COMET-QE... 2025-11-26 15:00:40,886 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-26 15:00:40,894 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-26 15:00:40,894 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-26 15:00:40,896 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-26 15:00:43,200 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 2025-11-26 15:00:43,213 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-26 15:00:43,214 - __main__ - INFO - BLEU: 0.0000 2025-11-26 15:00:43,215 - __main__ - INFO - chrF: 1.22 2025-11-26 15:00:43,216 - __main__ - INFO - COMET-QE: 0.0001 2025-11-26 15:00:43,217 - __main__ - INFO - ====================================================================== 2025-11-26 15:00:43,831 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 2025-11-26 15:00:43,847 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase2/baseline_results.json 2025-11-26 15:00:43,858 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics 2025-11-26 15:00:43,860 - __main__ - INFO - 🚀 STEP 2: Fine-tuning 2025-11-26 15:00:43,861 - __main__ - INFO - Metrics are being logged to MLflow in real-time 2025-11-26 15:00:43,863 - __main__ - INFO - MLflow URL: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 15:00:43,865 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs 2025-11-26 15:00:43,866 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') 2025-11-26 15:00:43,868 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 15:00:43,869 - __main__ - INFO - ====================================================================== 2025-11-26 15:00:43,870 - __main__ - INFO - 🚀 Starting training... 2025-11-26 15:00:43,871 - __main__ - INFO - ====================================================================== 2025-11-26 15:00:45,126 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) 2025-11-26 15:01:06,238 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.1%) 2025-11-26 15:01:26,944 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.2%) 2025-11-26 15:01:48,288 - __main__ - INFO - 📈 Epoch progress: 0.02/6 (0.4%) 2025-11-26 15:02:09,155 - __main__ - INFO - 📈 Epoch progress: 0.03/6 (0.5%) 2025-11-26 15:02:29,947 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.6%) 2025-11-26 15:02:50,639 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.7%) 2025-11-26 15:03:11,496 - __main__ - INFO - 📈 Epoch progress: 0.05/6 (0.8%) 2025-11-26 15:03:32,002 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.0%) 2025-11-26 15:03:52,078 - urllib3.connectionpool - WARNING - Retrying (Retry(total=6, connect=7, read=6, redirect=7, status=7)) after connection broken by 'RemoteDisconnected('Remote end closed connection without response')': /api/2.0/mlflow/runs/log-batch 2025-11-26 15:03:52,531 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.1%) 2025-11-26 15:04:13,047 - __main__ - INFO - 📈 Epoch progress: 0.07/6 (1.2%) 2025-11-26 15:19:54,631 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-26 15:19:54,643 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json 2025-11-26 15:19:54,643 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 15:19:54,645 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-26 15:19:54,647 - __main__ - INFO - 🎲 Random seed: 42 2025-11-26 15:19:54,648 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-26 15:19:54,649 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 15:19:54,651 - __main__ - INFO - Push strategy: checkpoint 2025-11-26 15:19:54,651 - __main__ - INFO - → Will push after each checkpoint save 2025-11-26 15:19:54,652 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-26 15:19:54,652 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... 2025-11-26 15:19:54,653 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... 2025-11-26 15:19:54,675 - __main__ - WARNING - ⚠️ DNS resolution failed: [Errno -2] Name or service not known 2025-11-26 15:19:54,676 - __main__ - ERROR - ❌ Cannot reach MLflow server at https://food-accepts-blocked-refinance.trycloudflare.com 2025-11-26 15:19:54,677 - __main__ - ERROR - Will proceed with file logging only (MLflow disabled) 2025-11-26 15:19:54,679 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-26 15:19:54,716 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-26 15:19:54,717 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-26 15:19:54,718 - __main__ - INFO - 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 15:19:59,423 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-26 15:19:59,425 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-26 15:19:59,839 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file 2025-11-26 15:20:00,106 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file 2025-11-26 15:20:00,116 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl 2025-11-26 15:20:08,129 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) 2025-11-26 15:20:10,899 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 2025-11-26 15:20:13,823 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 2025-11-26 15:20:16,634 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 2025-11-26 15:20:19,094 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 2025-11-26 15:20:21,446 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 2025-11-26 15:20:23,821 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 2025-11-26 15:20:26,130 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 2025-11-26 15:20:28,448 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 2025-11-26 15:20:30,819 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 2025-11-26 15:20:33,225 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 2025-11-26 15:20:35,647 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 2025-11-26 15:20:38,038 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 2025-11-26 15:20:40,449 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 2025-11-26 15:20:42,846 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 2025-11-26 15:20:45,232 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 2025-11-26 15:20:47,601 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 2025-11-26 15:20:49,975 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 2025-11-26 15:20:52,373 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 2025-11-26 15:20:55,083 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 2025-11-26 15:20:57,461 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 2025-11-26 15:20:59,815 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 2025-11-26 15:21:02,179 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 2025-11-26 15:21:04,556 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 2025-11-26 15:21:06,954 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 2025-11-26 15:21:09,332 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 2025-11-26 15:21:09,569 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) 2025-11-26 15:21:09,608 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl 2025-11-26 15:21:13,615 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) 2025-11-26 15:21:16,561 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 2025-11-26 15:21:19,432 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 2025-11-26 15:21:21,830 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 2025-11-26 15:21:24,197 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 2025-11-26 15:21:26,610 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 2025-11-26 15:21:29,005 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 2025-11-26 15:21:31,439 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 2025-11-26 15:21:33,860 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 2025-11-26 15:21:36,301 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 2025-11-26 15:21:38,899 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 2025-11-26 15:21:41,759 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 2025-11-26 15:21:44,223 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 2025-11-26 15:21:46,651 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 2025-11-26 15:21:49,074 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 2025-11-26 15:21:51,521 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 2025-11-26 15:21:53,990 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 2025-11-26 15:21:56,241 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) 2025-11-26 15:21:56,290 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 2025-11-26 15:21:56,291 - __main__ - INFO - 📝 Sample data check: 2025-11-26 15:21:56,292 - __main__ - INFO - Sample 1: 2025-11-26 15:21:56,292 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... 2025-11-26 15:21:56,293 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... 2025-11-26 15:21:56,293 - __main__ - INFO - Sample 2: 2025-11-26 15:21:56,294 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... 2025-11-26 15:21:56,295 - __main__ - INFO - Target (6 words): Job had said no such thing.... 2025-11-26 15:21:56,295 - __main__ - INFO - Sample 3: 2025-11-26 15:21:56,296 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... 2025-11-26 15:21:56,296 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... 2025-11-26 15:22:01,627 - __main__ - INFO - Filtering statistics: {'kept': 4202243} 2025-11-26 15:22:01,628 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 2025-11-26 15:22:06,238 - __main__ - INFO - Removed 518 duplicates 2025-11-26 15:22:06,460 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 2025-11-26 15:22:13,232 - __main__ - INFO - ====================================================================== 2025-11-26 15:22:13,233 - __main__ - INFO - ✅ Dataset splits: 2025-11-26 15:22:13,233 - __main__ - INFO - Train: 3,571,465 samples 2025-11-26 15:22:13,234 - __main__ - INFO - Validation: 420,173 samples 2025-11-26 15:22:13,235 - __main__ - INFO - Test: 210,087 samples 2025-11-26 15:22:13,235 - __main__ - INFO - ====================================================================== 2025-11-26 15:22:13,701 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-26 15:25:20,487 - __main__ - INFO - ✅ Dataset validation passed 2025-11-26 15:25:20,490 - __main__ - INFO - Train: 3,571,465 samples 2025-11-26 15:25:20,490 - __main__ - INFO - Validation: 420,173 samples 2025-11-26 15:25:20,491 - __main__ - INFO - Test: 210,087 samples 2025-11-26 15:25:21,869 - __main__ - INFO - ✅ Found cached tokenized datasets (hash: 59e2a5c4) 2025-11-26 15:25:21,870 - __main__ - INFO - Skipping tokenization - this saves significant time! 2025-11-26 15:25:21,871 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... 2025-11-26 15:26:03,304 - __main__ - INFO - ✅ Tokenization complete! 2025-11-26 15:26:03,318 - __main__ - INFO - 🔧 Initializing COMET-QE model... 2025-11-26 15:26:03,452 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. 2025-11-26 15:26:04,582 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` 2025-11-26 15:26:10,765 - comet.models.base - INFO - Encoder model frozen. 2025-11-26 15:26:11,304 - __main__ - INFO - ✅ COMET-QE model loaded successfully 2025-11-26 15:26:11,320 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ 2025-11-26 15:26:11,352 - __main__ - INFO - 📊 Limiting evaluation dataset from 420225 to 2000 samples 2025-11-26 15:26:11,366 - __main__ - INFO - ✅ Trainer initialized 2025-11-26 15:26:11,367 - __main__ - INFO - 🔬 STEP 1: Baseline Evaluation 2025-11-26 15:26:11,367 - __main__ - INFO - ====================================================================== 2025-11-26 15:26:11,368 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... 2025-11-26 15:26:11,368 - __main__ - INFO - ====================================================================== 2025-11-26 15:26:43,143 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.1215 2025-11-26 15:26:43,145 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-26 15:26:43,182 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... 2025-11-26 15:26:43,183 - __main__ - INFO - Batch 1/16... 2025-11-26 15:26:44,523 - __main__ - INFO - Batch 5/16... 2025-11-26 15:26:45,642 - __main__ - INFO - Batch 10/16... 2025-11-26 15:26:46,850 - __main__ - INFO - Batch 15/16... 2025-11-26 15:26:47,234 - __main__ - INFO - Running COMET-QE model... 2025-11-26 15:26:47,246 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-26 15:26:47,255 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-26 15:26:47,255 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-26 15:26:47,256 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision 2025-11-26 15:26:47,258 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-26 15:26:50,282 - __main__ - INFO - ✅ COMET-QE: 0.1977 ± 0.2784 2025-11-26 15:26:50,283 - __main__ - INFO - ✅ Baseline Results (Test Set): 2025-11-26 15:26:50,285 - __main__ - INFO - BLEU: 0.1215 2025-11-26 15:26:50,286 - __main__ - INFO - chrF: 32.36 2025-11-26 15:26:50,286 - __main__ - INFO - COMET: 0.1977 2025-11-26 15:26:50,287 - __main__ - INFO - ====================================================================== 2025-11-26 15:26:50,288 - __main__ - INFO - ====================================================================== 2025-11-26 15:26:50,288 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-26 15:26:50,289 - __main__ - INFO - ====================================================================== 2025-11-26 15:26:50,290 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-26 15:26:50,291 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-26 15:26:51,371 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-26 15:26:51,431 - __main__ - INFO - ✅ Domain chrF: 1.22 2025-11-26 15:26:51,432 - __main__ - INFO - Computing COMET-QE... 2025-11-26 15:26:51,438 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-26 15:26:51,447 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-26 15:26:51,449 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-26 15:26:51,452 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-26 15:26:53,552 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 2025-11-26 15:26:53,553 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-26 15:26:53,554 - __main__ - INFO - BLEU: 0.0000 2025-11-26 15:26:53,555 - __main__ - INFO - chrF: 1.22 2025-11-26 15:26:53,555 - __main__ - INFO - COMET-QE: 0.0001 2025-11-26 15:26:53,556 - __main__ - INFO - ====================================================================== 2025-11-26 15:26:53,565 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase2/baseline_results.json 2025-11-26 15:26:53,570 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics 2025-11-26 15:26:53,570 - __main__ - INFO - 🚀 STEP 2: Fine-tuning 2025-11-26 15:26:53,571 - __main__ - INFO - Metrics are being logged to MLflow in real-time 2025-11-26 15:26:53,571 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs 2025-11-26 15:26:53,572 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') 2025-11-26 15:26:53,572 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 15:26:53,573 - __main__ - INFO - ====================================================================== 2025-11-26 15:26:53,573 - __main__ - INFO - 🚀 Starting training... 2025-11-26 15:26:53,574 - __main__ - INFO - ====================================================================== 2025-11-26 15:34:05,454 - __main__ - INFO - 📊 Evaluation 2: BLEU = 0.1169 2025-11-26 15:34:13,047 - __main__ - INFO - ✅ Repository brendaogutu/mul-sw-en-translation-phase2 ready 2025-11-26 15:34:13,049 - __main__ - INFO - 📤 Pushing checkpoint 2000 to HuggingFace Hub... 2025-11-26 15:34:13,050 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 15:34:33,303 - __main__ - INFO - ✅ Checkpoint 2000 pushed successfully! 2025-11-26 15:34:33,316 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-2000 2025-11-26 15:38:14,833 - __main__ - INFO - ✅ COMET available for evaluation 2025-11-26 15:38:14,848 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json 2025-11-26 15:38:14,850 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 15:38:14,850 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 2025-11-26 15:38:14,855 - __main__ - INFO - 🎲 Random seed: 42 2025-11-26 15:38:14,856 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK 2025-11-26 15:38:14,857 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 15:38:14,857 - __main__ - INFO - Push strategy: checkpoint 2025-11-26 15:38:14,864 - __main__ - INFO - → Will push after each checkpoint save 2025-11-26 15:38:14,865 - __main__ - INFO - ✅ MLflowManager initialized 2025-11-26 15:38:14,867 - __main__ - INFO - 🔍 Checking connectivity to https://proceedings-graphic-pressed-responded.trycloudflare.com... 2025-11-26 15:38:14,867 - __main__ - INFO - 🔍 Checking connectivity to proceedings-graphic-pressed-responded.trycloudflare.com:443... 2025-11-26 15:38:14,885 - __main__ - INFO - ✅ MLflow server is reachable 2025-11-26 15:38:14,886 - __main__ - INFO - ✅ MLflow tracking URI set: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 15:38:16,109 - __main__ - INFO - ✅ Using experiment: swahili-english-translation 2025-11-26 15:38:17,370 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251126_153816 2025-11-26 15:38:17,371 - __main__ - INFO - Run ID: d9a43848ca0a46218f55909e1f84eaee 2025-11-26 15:38:17,373 - __main__ - INFO - Tracking URI: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 15:38:17,868 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow 2025-11-26 15:38:17,873 - __main__ - INFO - 📥 Loading domain evaluation data... 2025-11-26 15:38:17,898 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs 2025-11-26 15:38:17,929 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... 2025-11-26 15:38:17,939 - __main__ - INFO - 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 2025-11-26 15:38:22,708 - __main__ - INFO - ✅ Model loaded on cuda:0 2025-11-26 15:38:22,713 - __main__ - INFO - 🔍 Verifying dataset files... 2025-11-26 15:38:23,112 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file 2025-11-26 15:38:23,381 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file 2025-11-26 15:38:23,382 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl 2025-11-26 15:38:36,717 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) 2025-11-26 15:38:39,716 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 2025-11-26 15:38:42,275 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 2025-11-26 15:38:44,735 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 2025-11-26 15:38:47,169 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 2025-11-26 15:38:49,607 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 2025-11-26 15:38:52,120 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 2025-11-26 15:38:54,561 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 2025-11-26 15:38:56,986 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 2025-11-26 15:38:59,421 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 2025-11-26 15:39:01,888 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 2025-11-26 15:39:04,346 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 2025-11-26 15:39:06,753 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 2025-11-26 15:39:09,161 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 2025-11-26 15:39:11,570 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 2025-11-26 15:39:13,995 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 2025-11-26 15:39:16,378 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 2025-11-26 15:39:18,770 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 2025-11-26 15:39:21,205 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 2025-11-26 15:39:23,649 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 2025-11-26 15:39:26,111 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 2025-11-26 15:39:28,503 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 2025-11-26 15:39:30,905 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 2025-11-26 15:39:33,336 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 2025-11-26 15:39:35,755 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 2025-11-26 15:39:38,159 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 2025-11-26 15:39:38,393 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) 2025-11-26 15:39:38,430 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl 2025-11-26 15:39:41,057 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) 2025-11-26 15:39:43,501 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 2025-11-26 15:39:45,923 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 2025-11-26 15:39:48,326 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 2025-11-26 15:39:50,702 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 2025-11-26 15:39:53,128 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 2025-11-26 15:39:55,545 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 2025-11-26 15:39:57,960 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 2025-11-26 15:40:00,396 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 2025-11-26 15:40:02,809 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 2025-11-26 15:40:05,238 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 2025-11-26 15:40:07,634 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 2025-11-26 15:40:10,037 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 2025-11-26 15:40:12,439 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 2025-11-26 15:40:14,830 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 2025-11-26 15:40:17,241 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 2025-11-26 15:40:19,628 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 2025-11-26 15:40:21,855 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) 2025-11-26 15:40:21,899 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 2025-11-26 15:40:21,914 - __main__ - INFO - 📝 Sample data check: 2025-11-26 15:40:21,916 - __main__ - INFO - Sample 1: 2025-11-26 15:40:21,917 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... 2025-11-26 15:40:21,917 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... 2025-11-26 15:40:21,918 - __main__ - INFO - Sample 2: 2025-11-26 15:40:21,919 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... 2025-11-26 15:40:21,920 - __main__ - INFO - Target (6 words): Job had said no such thing.... 2025-11-26 15:40:21,920 - __main__ - INFO - Sample 3: 2025-11-26 15:40:21,921 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... 2025-11-26 15:40:21,921 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... 2025-11-26 15:40:27,246 - __main__ - INFO - Filtering statistics: {'kept': 4202243} 2025-11-26 15:40:27,250 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 2025-11-26 15:40:31,728 - __main__ - INFO - Removed 518 duplicates 2025-11-26 15:40:31,942 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 2025-11-26 15:40:38,598 - __main__ - INFO - ====================================================================== 2025-11-26 15:40:38,598 - __main__ - INFO - ✅ Dataset splits: 2025-11-26 15:40:38,615 - __main__ - INFO - Train: 3,571,465 samples 2025-11-26 15:40:38,616 - __main__ - INFO - Validation: 420,173 samples 2025-11-26 15:40:38,617 - __main__ - INFO - Test: 210,087 samples 2025-11-26 15:40:38,618 - __main__ - INFO - ====================================================================== 2025-11-26 15:40:39,055 - __main__ - INFO - 🔍 Validating dataset before training... 2025-11-26 15:43:48,131 - __main__ - INFO - ✅ Dataset validation passed 2025-11-26 15:43:48,134 - __main__ - INFO - Train: 3,571,465 samples 2025-11-26 15:43:48,135 - __main__ - INFO - Validation: 420,173 samples 2025-11-26 15:43:48,135 - __main__ - INFO - Test: 210,087 samples 2025-11-26 15:43:49,920 - __main__ - INFO - ✅ Found cached tokenized datasets (hash: 59e2a5c4) 2025-11-26 15:43:49,921 - __main__ - INFO - Skipping tokenization - this saves significant time! 2025-11-26 15:43:49,922 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... 2025-11-26 15:44:27,150 - __main__ - INFO - ✅ Tokenization complete! 2025-11-26 15:44:27,152 - __main__ - INFO - 🔧 Initializing COMET-QE model... 2025-11-26 15:44:27,269 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. 2025-11-26 15:44:28,434 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` 2025-11-26 15:44:34,787 - comet.models.base - INFO - Encoder model frozen. 2025-11-26 15:44:35,315 - __main__ - INFO - ✅ COMET-QE model loaded successfully 2025-11-26 15:44:35,316 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ 2025-11-26 15:44:35,351 - __main__ - INFO - 📊 Limiting evaluation dataset from 420225 to 2000 samples 2025-11-26 15:44:35,365 - __main__ - INFO - ✅ Trainer initialized 2025-11-26 15:44:35,366 - __main__ - INFO - 🔬 STEP 1: Baseline Evaluation 2025-11-26 15:44:35,931 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 2025-11-26 15:44:35,932 - __main__ - INFO - ====================================================================== 2025-11-26 15:44:35,933 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... 2025-11-26 15:44:35,933 - __main__ - INFO - ====================================================================== 2025-11-26 15:45:07,851 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.1215 2025-11-26 15:45:07,854 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-26 15:45:07,891 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... 2025-11-26 15:45:07,892 - __main__ - INFO - Batch 1/16... 2025-11-26 15:45:09,406 - __main__ - INFO - Batch 5/16... 2025-11-26 15:45:10,628 - __main__ - INFO - Batch 10/16... 2025-11-26 15:45:12,010 - __main__ - INFO - Batch 15/16... 2025-11-26 15:45:12,404 - __main__ - INFO - Running COMET-QE model... 2025-11-26 15:45:12,443 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-26 15:45:12,451 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-26 15:45:12,452 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-26 15:45:12,453 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision 2025-11-26 15:45:12,456 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-26 15:45:16,167 - __main__ - INFO - ✅ COMET-QE: 0.1977 ± 0.2784 2025-11-26 15:45:16,168 - __main__ - INFO - ✅ Baseline Results (Test Set): 2025-11-26 15:45:16,169 - __main__ - INFO - BLEU: 0.1215 2025-11-26 15:45:16,170 - __main__ - INFO - chrF: 32.36 2025-11-26 15:45:16,170 - __main__ - INFO - COMET: 0.1977 2025-11-26 15:45:16,170 - __main__ - INFO - ====================================================================== 2025-11-26 15:45:16,171 - __main__ - INFO - ====================================================================== 2025-11-26 15:45:16,171 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-26 15:45:16,172 - __main__ - INFO - ====================================================================== 2025-11-26 15:45:16,172 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-26 15:45:16,173 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-26 15:45:17,286 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-26 15:45:17,373 - __main__ - INFO - ✅ Domain chrF: 1.22 2025-11-26 15:45:17,374 - __main__ - INFO - Computing COMET-QE... 2025-11-26 15:45:17,380 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-26 15:45:17,389 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-26 15:45:17,390 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-26 15:45:17,392 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-26 15:45:19,457 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 2025-11-26 15:45:19,458 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-26 15:45:19,460 - __main__ - INFO - BLEU: 0.0000 2025-11-26 15:45:19,461 - __main__ - INFO - chrF: 1.22 2025-11-26 15:45:19,462 - __main__ - INFO - COMET-QE: 0.0001 2025-11-26 15:45:19,463 - __main__ - INFO - ====================================================================== 2025-11-26 15:45:19,966 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 2025-11-26 15:45:19,978 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase2/baseline_results.json 2025-11-26 15:45:19,988 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics 2025-11-26 15:45:19,989 - __main__ - INFO - 🚀 STEP 2: Fine-tuning 2025-11-26 15:45:19,990 - __main__ - INFO - Metrics are being logged to MLflow in real-time 2025-11-26 15:45:19,991 - __main__ - INFO - MLflow URL: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 15:45:19,992 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs 2025-11-26 15:45:19,994 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') 2025-11-26 15:45:19,995 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 15:45:19,996 - __main__ - INFO - ====================================================================== 2025-11-26 15:45:19,997 - __main__ - INFO - 🚀 Starting training... 2025-11-26 15:45:19,998 - __main__ - INFO - ====================================================================== 2025-11-26 15:45:21,185 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) 2025-11-26 15:45:42,465 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.1%) 2025-11-26 15:46:03,034 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.2%) 2025-11-26 15:46:24,110 - __main__ - INFO - 📈 Epoch progress: 0.02/6 (0.4%) 2025-11-26 15:46:45,084 - __main__ - INFO - 📈 Epoch progress: 0.03/6 (0.5%) 2025-11-26 15:47:05,886 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.6%) 2025-11-26 15:47:26,381 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.7%) 2025-11-26 15:47:47,439 - __main__ - INFO - 📈 Epoch progress: 0.05/6 (0.8%) 2025-11-26 15:48:08,096 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.0%) 2025-11-26 15:48:28,758 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.1%) 2025-11-26 15:48:50,137 - __main__ - INFO - 📈 Epoch progress: 0.07/6 (1.2%) 2025-11-26 15:49:11,147 - __main__ - INFO - 📈 Epoch progress: 0.08/6 (1.3%) 2025-11-26 15:49:31,904 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.4%) 2025-11-26 15:49:52,850 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.6%) 2025-11-26 15:50:13,654 - __main__ - INFO - 📈 Epoch progress: 0.10/6 (1.7%) 2025-11-26 15:50:34,024 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.8%) 2025-11-26 15:50:54,355 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.9%) 2025-11-26 15:51:15,161 - __main__ - INFO - 📈 Epoch progress: 0.12/6 (2.0%) 2025-11-26 15:51:36,131 - __main__ - INFO - 📈 Epoch progress: 0.13/6 (2.2%) 2025-11-26 15:51:56,801 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.3%) 2025-11-26 15:52:17,573 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.4%) 2025-11-26 15:53:04,601 - __main__ - INFO - 📊 Evaluation 2: BLEU = 0.1169 2025-11-26 15:53:05,031 - __main__ - INFO - 📊 Evaluation at step 2000 2025-11-26 15:53:09,974 - __main__ - INFO - ✅ Repository brendaogutu/mul-sw-en-translation-phase2 ready 2025-11-26 15:53:09,975 - __main__ - INFO - 📤 Pushing checkpoint 2000 to HuggingFace Hub... 2025-11-26 15:53:09,976 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 15:53:17,857 - __main__ - INFO - ✅ Checkpoint 2000 pushed successfully! 2025-11-26 15:53:17,859 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-2000 2025-11-26 15:53:38,219 - __main__ - INFO - 📈 Epoch progress: 0.15/6 (2.5%) 2025-11-26 15:53:59,290 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.6%) 2025-11-26 15:54:20,049 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.7%) 2025-11-26 15:54:40,579 - __main__ - INFO - 📈 Epoch progress: 0.17/6 (2.9%) 2025-11-26 15:55:01,123 - __main__ - INFO - 📈 Epoch progress: 0.18/6 (3.0%) 2025-11-26 15:55:21,745 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.1%) 2025-11-26 15:55:41,969 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.2%) 2025-11-26 15:56:03,309 - __main__ - INFO - 📈 Epoch progress: 0.20/6 (3.3%) 2025-11-26 15:56:23,794 - __main__ - INFO - 📈 Epoch progress: 0.21/6 (3.5%) 2025-11-26 15:56:44,598 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.6%) 2025-11-26 15:57:05,030 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.7%) 2025-11-26 15:57:25,619 - __main__ - INFO - 📈 Epoch progress: 0.23/6 (3.8%) 2025-11-26 15:57:46,533 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (3.9%) 2025-11-26 15:58:07,360 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (4.1%) 2025-11-26 15:58:28,104 - __main__ - INFO - 📈 Epoch progress: 0.25/6 (4.2%) 2025-11-26 15:58:49,607 - __main__ - INFO - 📈 Epoch progress: 0.26/6 (4.3%) 2025-11-26 15:59:10,293 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.4%) 2025-11-26 15:59:30,763 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.5%) 2025-11-26 15:59:51,217 - __main__ - INFO - 📈 Epoch progress: 0.28/6 (4.7%) 2025-11-26 16:00:12,027 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.8%) 2025-11-26 16:00:58,821 - __main__ - INFO - 📊 Evaluation 3: BLEU = 0.1186 2025-11-26 16:00:59,196 - __main__ - INFO - 📊 Evaluation at step 4000 2025-11-26 16:01:05,960 - __main__ - INFO - 📤 Pushing checkpoint 4000 to HuggingFace Hub... 2025-11-26 16:01:05,961 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 16:01:18,111 - __main__ - INFO - ✅ Checkpoint 4000 pushed successfully! 2025-11-26 16:01:18,115 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-4000 2025-11-26 16:01:39,069 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.9%) 2025-11-26 16:01:59,618 - __main__ - INFO - 📈 Epoch progress: 0.30/6 (5.0%) 2025-11-26 16:02:19,926 - __main__ - INFO - 📈 Epoch progress: 0.31/6 (5.1%) 2025-11-26 16:02:40,522 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.3%) 2025-11-26 16:03:01,310 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.4%) 2025-11-26 16:03:22,321 - __main__ - INFO - 📈 Epoch progress: 0.33/6 (5.5%) 2025-11-26 16:03:43,805 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.6%) 2025-11-26 16:04:04,893 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.7%) 2025-11-26 16:04:25,380 - __main__ - INFO - 📈 Epoch progress: 0.35/6 (5.9%) 2025-11-26 16:04:46,071 - __main__ - INFO - 📈 Epoch progress: 0.36/6 (6.0%) 2025-11-26 16:05:06,903 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.1%) 2025-11-26 16:05:28,054 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.2%) 2025-11-26 16:05:48,374 - __main__ - INFO - 📈 Epoch progress: 0.38/6 (6.3%) 2025-11-26 16:06:09,404 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.5%) 2025-11-26 16:06:30,087 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.6%) 2025-11-26 16:06:50,702 - __main__ - INFO - 📈 Epoch progress: 0.40/6 (6.7%) 2025-11-26 16:07:11,672 - __main__ - INFO - 📈 Epoch progress: 0.41/6 (6.8%) 2025-11-26 16:07:32,187 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (6.9%) 2025-11-26 16:07:52,929 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (7.0%) 2025-11-26 16:08:13,223 - __main__ - INFO - 📈 Epoch progress: 0.43/6 (7.2%) 2025-11-26 16:09:01,150 - __main__ - INFO - 📊 Evaluation 4: BLEU = 0.1098 2025-11-26 16:09:01,557 - __main__ - INFO - 📊 Evaluation at step 6000 2025-11-26 16:09:06,281 - __main__ - INFO - 📤 Pushing checkpoint 6000 to HuggingFace Hub... 2025-11-26 16:09:06,282 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 16:09:17,586 - __main__ - INFO - ✅ Checkpoint 6000 pushed successfully! 2025-11-26 16:09:17,604 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-6000 2025-11-26 16:09:37,393 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.3%) 2025-11-26 16:09:58,395 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.4%) 2025-11-26 16:10:19,357 - __main__ - INFO - 📈 Epoch progress: 0.45/6 (7.5%) 2025-11-26 16:10:40,495 - __main__ - INFO - 📈 Epoch progress: 0.46/6 (7.6%) 2025-11-26 16:11:01,094 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.8%) 2025-11-26 16:11:22,057 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.9%) 2025-11-26 16:11:42,721 - __main__ - INFO - 📈 Epoch progress: 0.48/6 (8.0%) 2025-11-26 16:12:02,880 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.1%) 2025-11-26 16:12:23,276 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.2%) 2025-11-26 16:12:43,535 - __main__ - INFO - 📈 Epoch progress: 0.50/6 (8.4%) 2025-11-26 16:13:04,351 - __main__ - INFO - 📈 Epoch progress: 0.51/6 (8.5%) 2025-11-26 16:13:25,533 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.6%) 2025-11-26 16:13:46,002 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.7%) 2025-11-26 16:14:06,815 - __main__ - INFO - 📈 Epoch progress: 0.53/6 (8.8%) 2025-11-26 16:14:27,612 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.0%) 2025-11-26 16:14:48,687 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.1%) 2025-11-26 16:15:09,813 - __main__ - INFO - 📈 Epoch progress: 0.55/6 (9.2%) 2025-11-26 16:15:30,687 - __main__ - INFO - 📈 Epoch progress: 0.56/6 (9.3%) 2025-11-26 16:15:50,864 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.4%) 2025-11-26 16:16:11,185 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.6%) 2025-11-26 16:16:57,438 - __main__ - INFO - 📊 Evaluation 5: BLEU = 0.1175 2025-11-26 16:16:57,820 - __main__ - INFO - 📊 Evaluation at step 8000 2025-11-26 16:17:04,619 - __main__ - INFO - 📤 Pushing checkpoint 8000 to HuggingFace Hub... 2025-11-26 16:17:04,620 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 16:17:15,572 - __main__ - INFO - ✅ Checkpoint 8000 pushed successfully! 2025-11-26 16:17:15,574 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-8000 2025-11-26 16:17:35,586 - __main__ - INFO - 📈 Epoch progress: 0.58/6 (9.7%) 2025-11-26 16:17:56,297 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.8%) 2025-11-26 16:18:17,087 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.9%) 2025-11-26 16:18:37,822 - __main__ - INFO - 📈 Epoch progress: 0.60/6 (10.0%) 2025-11-26 16:18:58,361 - __main__ - INFO - 📈 Epoch progress: 0.61/6 (10.2%) 2025-11-26 16:19:18,885 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.3%) 2025-11-26 16:19:39,385 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.4%) 2025-11-26 16:19:59,998 - __main__ - INFO - 📈 Epoch progress: 0.63/6 (10.5%) 2025-11-26 16:20:20,640 - __main__ - INFO - 📈 Epoch progress: 0.64/6 (10.6%) 2025-11-26 16:20:41,239 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.8%) 2025-11-26 16:21:01,604 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.9%) 2025-11-26 16:21:21,953 - __main__ - INFO - 📈 Epoch progress: 0.66/6 (11.0%) 2025-11-26 16:21:42,831 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.1%) 2025-11-26 16:22:03,702 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.2%) 2025-11-26 16:22:23,768 - __main__ - INFO - 📈 Epoch progress: 0.68/6 (11.3%) 2025-11-26 16:22:44,385 - __main__ - INFO - 📈 Epoch progress: 0.69/6 (11.5%) 2025-11-26 16:23:05,353 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.6%) 2025-11-26 16:23:26,172 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.7%) 2025-11-26 16:23:46,306 - __main__ - INFO - 📈 Epoch progress: 0.71/6 (11.8%) 2025-11-26 16:24:06,386 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (11.9%) 2025-11-26 16:24:52,276 - __main__ - INFO - 📊 Evaluation 6: BLEU = 0.1201 2025-11-26 16:24:52,658 - __main__ - INFO - 📊 Evaluation at step 10000 2025-11-26 16:24:56,162 - __main__ - INFO - 📤 Pushing checkpoint 10000 to HuggingFace Hub... 2025-11-26 16:24:56,163 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 16:25:06,006 - __main__ - INFO - ✅ Checkpoint 10000 pushed successfully! 2025-11-26 16:25:06,015 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-10000 2025-11-26 16:25:26,160 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (12.1%) 2025-11-26 16:25:47,389 - __main__ - INFO - 📈 Epoch progress: 0.73/6 (12.2%) 2025-11-26 16:26:07,910 - __main__ - INFO - 📈 Epoch progress: 0.74/6 (12.3%) 2025-11-26 16:26:28,628 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.4%) 2025-11-26 16:26:48,827 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.5%) 2025-11-26 16:27:09,519 - __main__ - INFO - 📈 Epoch progress: 0.76/6 (12.7%) 2025-11-26 16:27:29,858 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.8%) 2025-11-26 16:27:50,683 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.9%) 2025-11-26 16:28:16,023 - __main__ - INFO - 📈 Epoch progress: 0.78/6 (13.0%) 2025-11-26 16:28:37,030 - __main__ - INFO - 📈 Epoch progress: 0.79/6 (13.1%) 2025-11-26 16:28:57,372 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.3%) 2025-11-26 16:29:17,511 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.4%) 2025-11-26 16:29:38,325 - __main__ - INFO - 📈 Epoch progress: 0.81/6 (13.5%) 2025-11-26 16:29:59,091 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.6%) 2025-11-26 16:30:19,444 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.7%) 2025-11-26 16:30:40,279 - __main__ - INFO - 📈 Epoch progress: 0.83/6 (13.9%) 2025-11-26 16:31:00,631 - __main__ - INFO - 📈 Epoch progress: 0.84/6 (14.0%) 2025-11-26 16:31:21,133 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.1%) 2025-11-26 16:31:41,554 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.2%) 2025-11-26 16:32:01,981 - __main__ - INFO - 📈 Epoch progress: 0.86/6 (14.3%) 2025-11-26 16:32:55,177 - __main__ - INFO - 📊 Evaluation 7: BLEU = 0.1172 2025-11-26 16:32:56,101 - __main__ - INFO - 📊 Evaluation at step 12000 2025-11-26 16:33:00,167 - __main__ - INFO - 📤 Pushing checkpoint 12000 to HuggingFace Hub... 2025-11-26 16:33:00,167 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 16:33:10,375 - __main__ - INFO - ✅ Checkpoint 12000 pushed successfully! 2025-11-26 16:33:10,376 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-12000 2025-11-26 16:33:31,660 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.5%) 2025-11-26 16:33:51,937 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.6%) 2025-11-26 16:34:12,412 - __main__ - INFO - 📈 Epoch progress: 0.88/6 (14.7%) 2025-11-26 16:34:33,310 - __main__ - INFO - 📈 Epoch progress: 0.89/6 (14.8%) 2025-11-26 16:34:53,571 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (14.9%) 2025-11-26 16:35:14,347 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (15.1%) 2025-11-26 16:35:34,720 - __main__ - INFO - 📈 Epoch progress: 0.91/6 (15.2%) 2025-11-26 16:35:55,386 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.3%) 2025-11-26 16:36:15,968 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.4%) 2025-11-26 16:36:36,705 - __main__ - INFO - 📈 Epoch progress: 0.93/6 (15.5%) 2025-11-26 16:36:57,025 - __main__ - INFO - 📈 Epoch progress: 0.94/6 (15.6%) 2025-11-26 16:37:17,355 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.8%) 2025-11-26 16:37:37,781 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.9%) 2025-11-26 16:37:58,781 - __main__ - INFO - 📈 Epoch progress: 0.96/6 (16.0%) 2025-11-26 16:38:19,375 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.1%) 2025-11-26 16:38:40,368 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.2%) 2025-11-26 16:39:00,906 - __main__ - INFO - 📈 Epoch progress: 0.98/6 (16.4%) 2025-11-26 16:39:22,058 - __main__ - INFO - 📈 Epoch progress: 0.99/6 (16.5%) 2025-11-26 16:39:42,516 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.6%) 2025-11-26 16:40:04,615 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.7%) 2025-11-26 16:40:52,808 - __main__ - INFO - 📊 Evaluation 8: BLEU = 0.1178 2025-11-26 16:40:53,273 - __main__ - INFO - 📊 Evaluation at step 14000 2025-11-26 16:40:57,556 - __main__ - INFO - 📤 Pushing checkpoint 14000 to HuggingFace Hub... 2025-11-26 16:40:57,557 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 16:41:12,071 - __main__ - INFO - ✅ Checkpoint 14000 pushed successfully! 2025-11-26 16:41:12,077 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-14000 2025-11-26 16:41:32,297 - __main__ - INFO - 📈 Epoch progress: 1.01/6 (16.8%) 2025-11-26 16:41:53,076 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.0%) 2025-11-26 16:42:13,244 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.1%) 2025-11-26 16:42:33,859 - __main__ - INFO - 📈 Epoch progress: 1.03/6 (17.2%) 2025-11-26 16:42:54,826 - __main__ - INFO - 📈 Epoch progress: 1.04/6 (17.3%) 2025-11-26 16:43:15,509 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.4%) 2025-11-26 16:43:36,088 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.6%) 2025-11-26 16:43:57,218 - __main__ - INFO - 📈 Epoch progress: 1.06/6 (17.7%) 2025-11-26 16:44:17,864 - __main__ - INFO - 📈 Epoch progress: 1.07/6 (17.8%) 2025-11-26 16:44:38,282 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (17.9%) 2025-11-26 16:44:58,756 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (18.0%) 2025-11-26 16:45:19,109 - __main__ - INFO - 📈 Epoch progress: 1.09/6 (18.2%) 2025-11-26 16:45:39,663 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.3%) 2025-11-26 16:46:00,369 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.4%) 2025-11-26 16:46:21,166 - __main__ - INFO - 📈 Epoch progress: 1.11/6 (18.5%) 2025-11-26 16:46:41,702 - __main__ - INFO - 📈 Epoch progress: 1.12/6 (18.6%) 2025-11-26 16:47:02,646 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.8%) 2025-11-26 16:47:23,510 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.9%) 2025-11-26 16:47:44,422 - __main__ - INFO - 📈 Epoch progress: 1.14/6 (19.0%) 2025-11-26 16:48:04,903 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.1%) 2025-11-26 16:48:53,083 - __main__ - INFO - 📊 Evaluation 9: BLEU = 0.1183 2025-11-26 16:48:53,626 - __main__ - INFO - 📊 Evaluation at step 16000 2025-11-26 16:48:57,078 - __main__ - INFO - 📤 Pushing checkpoint 16000 to HuggingFace Hub... 2025-11-26 16:48:57,078 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 16:49:06,782 - __main__ - INFO - ✅ Checkpoint 16000 pushed successfully! 2025-11-26 16:49:06,784 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-16000 2025-11-26 16:49:26,693 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.2%) 2025-11-26 16:49:47,398 - __main__ - INFO - 📈 Epoch progress: 1.16/6 (19.4%) 2025-11-26 16:50:07,998 - __main__ - INFO - 📈 Epoch progress: 1.17/6 (19.5%) 2025-11-26 16:50:28,528 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.6%) 2025-11-26 16:50:49,162 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.7%) 2025-11-26 16:51:10,379 - __main__ - INFO - 📈 Epoch progress: 1.19/6 (19.8%) 2025-11-26 16:51:31,188 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (19.9%) 2025-11-26 16:51:51,598 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (20.1%) 2025-11-26 16:52:12,375 - __main__ - INFO - 📈 Epoch progress: 1.21/6 (20.2%) 2025-11-26 16:52:32,087 - __main__ - INFO - 📈 Epoch progress: 1.22/6 (20.3%) 2025-11-26 16:52:52,896 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.4%) 2025-11-26 16:53:13,262 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.5%) 2025-11-26 16:53:33,806 - __main__ - INFO - 📈 Epoch progress: 1.24/6 (20.7%) 2025-11-26 16:53:54,820 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.8%) 2025-11-26 16:54:15,617 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.9%) 2025-11-26 16:54:36,064 - __main__ - INFO - 📈 Epoch progress: 1.26/6 (21.0%) 2025-11-26 16:54:57,165 - __main__ - INFO - 📈 Epoch progress: 1.27/6 (21.1%) 2025-11-26 16:55:17,640 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.3%) 2025-11-26 16:55:38,662 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.4%) 2025-11-26 16:55:59,432 - __main__ - INFO - 📈 Epoch progress: 1.29/6 (21.5%) 2025-11-26 16:56:44,724 - __main__ - INFO - 📊 Evaluation 10: BLEU = 0.1219 2025-11-26 16:56:45,222 - __main__ - INFO - 📊 Evaluation at step 18000 2025-11-26 16:56:49,564 - __main__ - INFO - 📤 Pushing checkpoint 18000 to HuggingFace Hub... 2025-11-26 16:56:49,565 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 16:57:00,156 - __main__ - INFO - ✅ Checkpoint 18000 pushed successfully! 2025-11-26 16:57:00,157 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-18000 2025-11-26 16:57:20,382 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.6%) 2025-11-26 16:57:41,369 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.7%) 2025-11-26 16:58:01,591 - __main__ - INFO - 📈 Epoch progress: 1.31/6 (21.9%) 2025-11-26 16:58:22,170 - __main__ - INFO - 📈 Epoch progress: 1.32/6 (22.0%) 2025-11-26 16:58:43,488 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.1%) 2025-11-26 16:59:04,061 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.2%) 2025-11-26 16:59:24,749 - __main__ - INFO - 📈 Epoch progress: 1.34/6 (22.3%) 2025-11-26 16:59:45,462 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.5%) 2025-11-26 17:00:06,941 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.6%) 2025-11-26 17:00:27,513 - __main__ - INFO - 📈 Epoch progress: 1.36/6 (22.7%) 2025-11-26 17:00:47,989 - __main__ - INFO - 📈 Epoch progress: 1.37/6 (22.8%) 2025-11-26 17:01:08,467 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (22.9%) 2025-11-26 17:01:29,023 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (23.1%) 2025-11-26 17:01:49,659 - __main__ - INFO - 📈 Epoch progress: 1.39/6 (23.2%) 2025-11-26 17:02:10,319 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.3%) 2025-11-26 17:02:31,722 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.4%) 2025-11-26 17:02:54,391 - __main__ - INFO - 📈 Epoch progress: 1.41/6 (23.5%) 2025-11-26 17:03:15,204 - __main__ - INFO - 📈 Epoch progress: 1.42/6 (23.7%) 2025-11-26 17:03:35,749 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.8%) 2025-11-26 17:03:56,077 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.9%) 2025-11-26 17:04:43,169 - __main__ - INFO - 📊 Evaluation 11: BLEU = 0.1225 2025-11-26 17:04:43,613 - __main__ - INFO - 📊 Evaluation at step 20000 2025-11-26 17:04:52,366 - __main__ - INFO - 📤 Pushing checkpoint 20000 to HuggingFace Hub... 2025-11-26 17:04:52,367 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 17:05:03,979 - __main__ - INFO - ✅ Checkpoint 20000 pushed successfully! 2025-11-26 17:05:03,980 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-20000 2025-11-26 17:05:24,366 - __main__ - INFO - 📈 Epoch progress: 1.44/6 (24.0%) 2025-11-26 17:05:44,722 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.1%) 2025-11-26 17:06:05,254 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.2%) 2025-11-26 17:06:26,544 - __main__ - INFO - 📈 Epoch progress: 1.46/6 (24.4%) 2025-11-26 17:06:47,089 - __main__ - INFO - 📈 Epoch progress: 1.47/6 (24.5%) 2025-11-26 17:07:07,849 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.6%) 2025-11-26 17:07:28,326 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.7%) 2025-11-26 17:07:49,542 - __main__ - INFO - 📈 Epoch progress: 1.49/6 (24.8%) 2025-11-26 17:08:09,977 - __main__ - INFO - 📈 Epoch progress: 1.50/6 (25.0%) 2025-11-26 17:08:30,453 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.1%) 2025-11-26 17:08:51,842 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.2%) 2025-11-26 17:09:12,306 - __main__ - INFO - 📈 Epoch progress: 1.52/6 (25.3%) 2025-11-26 17:09:32,993 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.4%) 2025-11-26 17:09:53,319 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.6%) 2025-11-26 17:10:13,578 - __main__ - INFO - 📈 Epoch progress: 1.54/6 (25.7%) 2025-11-26 17:10:34,380 - __main__ - INFO - 📈 Epoch progress: 1.55/6 (25.8%) 2025-11-26 17:10:54,895 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (25.9%) 2025-11-26 17:11:15,327 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (26.0%) 2025-11-26 17:11:35,577 - __main__ - INFO - 📈 Epoch progress: 1.57/6 (26.2%) 2025-11-26 17:11:57,376 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.3%) 2025-11-26 17:12:52,830 - __main__ - INFO - 📊 Evaluation 12: BLEU = 0.1133 2025-11-26 17:12:53,225 - __main__ - INFO - 📊 Evaluation at step 22000 2025-11-26 17:12:57,360 - __main__ - INFO - 📤 Pushing checkpoint 22000 to HuggingFace Hub... 2025-11-26 17:12:57,361 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 17:13:06,308 - __main__ - INFO - ✅ Checkpoint 22000 pushed successfully! 2025-11-26 17:13:06,314 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-22000 2025-11-26 17:13:26,399 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.4%) 2025-11-26 17:13:47,465 - __main__ - INFO - 📈 Epoch progress: 1.59/6 (26.5%) 2025-11-26 17:14:08,586 - __main__ - INFO - 📈 Epoch progress: 1.60/6 (26.6%) 2025-11-26 17:14:28,839 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.8%) 2025-11-26 17:14:50,346 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.9%) 2025-11-26 17:15:10,980 - __main__ - INFO - 📈 Epoch progress: 1.62/6 (27.0%) 2025-11-26 17:15:31,737 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.1%) 2025-11-26 17:15:52,488 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.2%) 2025-11-26 17:16:13,006 - __main__ - INFO - 📈 Epoch progress: 1.64/6 (27.4%) 2025-11-26 17:16:34,452 - __main__ - INFO - 📈 Epoch progress: 1.65/6 (27.5%) 2025-11-26 17:16:55,065 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.6%) 2025-11-26 17:17:15,489 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.7%) 2025-11-26 17:17:35,891 - __main__ - INFO - 📈 Epoch progress: 1.67/6 (27.8%) 2025-11-26 17:17:56,335 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.0%) 2025-11-26 17:18:17,309 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.1%) 2025-11-26 17:18:37,633 - __main__ - INFO - 📈 Epoch progress: 1.69/6 (28.2%) 2025-11-26 17:18:58,357 - __main__ - INFO - 📈 Epoch progress: 1.70/6 (28.3%) 2025-11-26 17:19:20,368 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.4%) 2025-11-26 17:19:41,773 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.5%) 2025-11-26 17:20:02,469 - __main__ - INFO - 📈 Epoch progress: 1.72/6 (28.7%) 2025-11-26 17:20:45,791 - __main__ - INFO - 📊 Evaluation 13: BLEU = 0.1197 2025-11-26 17:20:46,181 - __main__ - INFO - 📊 Evaluation at step 24000 2025-11-26 17:20:49,966 - __main__ - INFO - 📤 Pushing checkpoint 24000 to HuggingFace Hub... 2025-11-26 17:20:49,966 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 17:21:00,484 - __main__ - INFO - ✅ Checkpoint 24000 pushed successfully! 2025-11-26 17:21:00,486 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-24000 2025-11-26 17:21:21,002 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.8%) 2025-11-26 17:21:41,788 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.9%) 2025-11-26 17:22:02,130 - __main__ - INFO - 📈 Epoch progress: 1.74/6 (29.0%) 2025-11-26 17:22:23,172 - __main__ - INFO - 📈 Epoch progress: 1.75/6 (29.1%) 2025-11-26 17:22:43,451 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.3%) 2025-11-26 17:23:04,553 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.4%) 2025-11-26 17:23:25,507 - __main__ - INFO - 📈 Epoch progress: 1.77/6 (29.5%) 2025-11-26 17:23:46,026 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.6%) 2025-11-26 17:24:07,282 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.7%) 2025-11-26 17:24:28,108 - __main__ - INFO - 📈 Epoch progress: 1.79/6 (29.9%) 2025-11-26 17:24:48,818 - __main__ - INFO - 📈 Epoch progress: 1.80/6 (30.0%) 2025-11-26 17:25:09,138 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.1%) 2025-11-26 17:25:29,625 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.2%) 2025-11-26 17:25:50,234 - __main__ - INFO - 📈 Epoch progress: 1.82/6 (30.3%) 2025-11-26 17:26:10,569 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.5%) 2025-11-26 17:26:31,382 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.6%) 2025-11-26 17:26:51,848 - __main__ - INFO - 📈 Epoch progress: 1.84/6 (30.7%) 2025-11-26 17:27:12,730 - __main__ - INFO - 📈 Epoch progress: 1.85/6 (30.8%) 2025-11-26 17:27:33,559 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (30.9%) 2025-11-26 17:27:54,145 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (31.1%) 2025-11-26 17:28:37,769 - __main__ - INFO - 📊 Evaluation 14: BLEU = 0.1220 2025-11-26 17:28:38,281 - __main__ - INFO - 📊 Evaluation at step 26000 2025-11-26 17:28:42,091 - __main__ - INFO - 📤 Pushing checkpoint 26000 to HuggingFace Hub... 2025-11-26 17:28:42,092 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 17:28:51,807 - __main__ - INFO - ✅ Checkpoint 26000 pushed successfully! 2025-11-26 17:28:51,815 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-26000 2025-11-26 17:29:12,107 - __main__ - INFO - 📈 Epoch progress: 1.87/6 (31.2%) 2025-11-26 17:29:33,590 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.3%) 2025-11-26 17:29:54,623 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.4%) 2025-11-26 17:30:15,939 - __main__ - INFO - 📈 Epoch progress: 1.89/6 (31.5%) 2025-11-26 17:30:37,067 - __main__ - INFO - 📈 Epoch progress: 1.90/6 (31.7%) 2025-11-26 17:30:58,061 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.8%) 2025-11-26 17:31:18,541 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.9%) 2025-11-26 17:31:39,182 - __main__ - INFO - 📈 Epoch progress: 1.92/6 (32.0%) 2025-11-26 17:31:59,476 - __main__ - INFO - 📈 Epoch progress: 1.93/6 (32.1%) 2025-11-26 17:32:20,115 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.3%) 2025-11-26 17:32:40,678 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.4%) 2025-11-26 17:33:01,358 - __main__ - INFO - 📈 Epoch progress: 1.95/6 (32.5%) 2025-11-26 17:33:22,398 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.6%) 2025-11-26 17:33:43,021 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.7%) 2025-11-26 17:34:03,906 - __main__ - INFO - 📈 Epoch progress: 1.97/6 (32.8%) 2025-11-26 17:34:25,041 - __main__ - INFO - 📈 Epoch progress: 1.98/6 (33.0%) 2025-11-26 17:34:45,997 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.1%) 2025-11-26 17:35:07,381 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.2%) 2025-11-26 17:35:28,654 - __main__ - INFO - 📈 Epoch progress: 2.00/6 (33.3%) 2025-11-26 17:35:50,983 - __main__ - INFO - 📈 Epoch progress: 2.01/6 (33.4%) 2025-11-26 17:36:28,715 - __main__ - INFO - 📊 Evaluation 15: BLEU = 0.1212 2025-11-26 17:36:29,111 - __main__ - INFO - 📊 Evaluation at step 28000 2025-11-26 17:36:36,019 - __main__ - INFO - 📤 Pushing checkpoint 28000 to HuggingFace Hub... 2025-11-26 17:36:36,020 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 2025-11-26 17:36:49,430 - __main__ - INFO - ✅ Checkpoint 28000 pushed successfully! 2025-11-26 17:36:49,434 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-28000 2025-11-26 17:36:56,016 - __main__ - INFO - ✅ MLflow: Training finished - status_code=2.0 (FINISHED) 2025-11-26 17:36:56,297 - __main__ - INFO - 💾 Saving model to models/finetuned-sw-en-phase2 2025-11-26 17:36:57,686 - __main__ - INFO - ✅ Training complete! 2025-11-26 17:36:57,687 - __main__ - INFO - 📊 STEP 3: Final Evaluation 2025-11-26 17:36:58,087 - __main__ - INFO - ✅ MLflow: Final evaluation starting - status_code=3.0 2025-11-26 17:36:58,089 - __main__ - INFO - ====================================================================== 2025-11-26 17:36:58,090 - __main__ - INFO - 📊 Running FINAL evaluation on test set... 2025-11-26 17:36:58,091 - __main__ - INFO - ====================================================================== 2025-11-26 17:36:58,091 - __main__ - INFO - Tokenizing test dataset... 2025-11-26 17:37:00,087 - datasets.fingerprint - WARNING - Parameter 'function'=> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only shown once. Subsequent hashing failures won't be shown. 2025-11-26 18:55:14,071 - __main__ - INFO - 📊 Evaluation 16: BLEU = 0.1289 2025-11-26 18:55:14,655 - __main__ - INFO - 📊 Evaluation at step 28000 2025-11-26 18:55:14,800 - __main__ - INFO - 📊 Computing COMET on test set... 2025-11-26 18:55:14,860 - __main__ - INFO - Computing predictions for COMET (1000 samples, batch_size=32)... 2025-11-26 18:55:14,861 - __main__ - INFO - Batch 1/32... 2025-11-26 18:55:18,346 - __main__ - INFO - Batch 5/32... 2025-11-26 18:55:19,409 - __main__ - INFO - Batch 10/32... 2025-11-26 18:55:20,560 - __main__ - INFO - Batch 15/32... 2025-11-26 18:55:21,821 - __main__ - INFO - Batch 20/32... 2025-11-26 18:55:22,971 - __main__ - INFO - Batch 25/32... 2025-11-26 18:55:24,126 - __main__ - INFO - Batch 30/32... 2025-11-26 18:55:24,716 - __main__ - INFO - Running COMET-QE model... 2025-11-26 18:55:24,731 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-26 18:55:24,739 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-26 18:55:24,740 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-26 18:55:24,741 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-26 18:55:29,735 - __main__ - INFO - ✅ COMET-QE: 0.1763 ± 0.2628 2025-11-26 18:55:29,736 - __main__ - INFO - ✅ FINAL Test Results: 2025-11-26 18:55:29,737 - __main__ - INFO - BLEU: 0.1289 2025-11-26 18:55:29,738 - __main__ - INFO - chrF: 32.25 2025-11-26 18:55:29,738 - __main__ - INFO - COMET-QE: 0.1763 2025-11-26 18:55:29,738 - __main__ - INFO - ====================================================================== 2025-11-26 18:55:29,739 - __main__ - INFO - ====================================================================== 2025-11-26 18:55:29,740 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... 2025-11-26 18:55:29,741 - __main__ - INFO - ====================================================================== 2025-11-26 18:55:29,742 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... 2025-11-26 18:55:29,743 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... 2025-11-26 18:55:30,630 - __main__ - INFO - ✅ Domain BLEU: 0.0000 2025-11-26 18:55:30,741 - __main__ - INFO - ✅ Domain chrF: 1.18 2025-11-26 18:55:30,742 - __main__ - INFO - Computing COMET-QE... 2025-11-26 18:55:30,748 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. 2025-11-26 18:55:30,757 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True 2025-11-26 18:55:30,757 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores 2025-11-26 18:55:30,759 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2025-11-26 18:55:33,071 - __main__ - INFO - ✅ Domain COMET-QE: 0.0000 2025-11-26 18:55:33,072 - __main__ - INFO - ✅ Domain Evaluation Results: 2025-11-26 18:55:33,072 - __main__ - INFO - BLEU: 0.0000 2025-11-26 18:55:33,073 - __main__ - INFO - chrF: 1.18 2025-11-26 18:55:33,073 - __main__ - INFO - COMET-QE: 0.0000 2025-11-26 18:55:33,074 - __main__ - INFO - ====================================================================== 2025-11-26 18:55:33,091 - __main__ - INFO - 💾 Final results saved to models/finetuned-sw-en-phase2/test_results.json 2025-11-26 18:55:33,115 - __main__ - INFO - ✅ Verified test_results.json with 14 metrics 2025-11-26 18:55:33,117 - __main__ - INFO - 📊 Logging all metrics to TensorBoard... 2025-11-26 18:55:33,454 - __main__ - INFO - ✅ All metrics logged to TensorBoard 2025-11-26 18:55:33,455 - __main__ - INFO - Baseline metrics: 13 2025-11-26 18:55:33,456 - __main__ - INFO - Final test metrics: 10 2025-11-26 18:55:33,457 - __main__ - INFO - Final domain metrics: 4 2025-11-26 18:55:33,457 - __main__ - INFO - ====================================================================== 2025-11-26 18:55:33,458 - __main__ - INFO - 📊 BASELINE vs FINE-TUNED Comparison 2025-11-26 18:55:33,458 - __main__ - INFO - ====================================================================== 2025-11-26 18:55:33,459 - __main__ - INFO - 📈 Test Set Metrics: 2025-11-26 18:55:33,459 - __main__ - INFO - Metric Baseline Fine-tuned Improvement 2025-11-26 18:55:33,460 - __main__ - INFO - ------------------------------------------------------------ 2025-11-26 18:55:33,460 - __main__ - INFO - eval_bleu 0.0000 0.1289 N/A 2025-11-26 18:55:33,461 - __main__ - INFO - eval_chrf 0.0000 32.2491 N/A 2025-11-26 18:55:33,462 - __main__ - INFO - eval_comet 0.0000 0.1763 N/A 2025-11-26 18:55:33,463 - __main__ - INFO - 📈 Domain Metrics (Real Call Transcriptions): 2025-11-26 18:55:33,463 - __main__ - INFO - Metric Baseline Fine-tuned Improvement 2025-11-26 18:55:33,463 - __main__ - INFO - ------------------------------------------------------------ 2025-11-26 18:55:33,464 - __main__ - INFO - domain_bleu 0.0000 0.0000 +1417.7% 2025-11-26 18:55:33,465 - __main__ - INFO - domain_chrf 1.2250 1.1786 -3.8% 2025-11-26 18:55:33,466 - __main__ - INFO - domain_comet 0.0001 0.0000 -26.2% 2025-11-26 18:55:33,466 - __main__ - INFO - ====================================================================== 2025-11-26 18:55:33,938 - __main__ - INFO - ✅ Logged 14 final metrics to MLflow 2025-11-26 18:55:33,939 - __main__ - INFO - Visit MLflow: https://proceedings-graphic-pressed-responded.trycloudflare.com 2025-11-26 18:55:33,940 - __main__ - INFO - 📊 Captured 576 training log entries 2025-11-26 18:55:33,967 - __main__ - INFO - 💾 Saved comprehensive metrics to models/finetuned-sw-en-phase2/training_metrics.json 2025-11-26 18:55:33,983 - __main__ - INFO - 💾 Saved training config to models/finetuned-sw-en-phase2/training_config_used.json